One year ago, I wrote about how creating applications for the cloud presented new opportunities in dev-ops management. OpenLogic's just-released product was built from the ground up to run in the cloud. We designed the application to allow for largely zero-downtime deployments, and constructed our deployment strategy to build everything we needed from scratch, if necessary, and swap the running servers on the fly. This process worked extremely well for us. We could spin up new app servers in minutes, and the process gave us the flexibility to start from scratch or from intermediate saved baselines to improve the deployment times when possible. Coupled with a Kanban-style continuous release process, we deployed new features quickly, confidently, and safely. This was a long way from where we started over ten years ago when we delivered CDs and DVDs every 6 months.
Over the past year, my team set some long-term goals to take the best of the approach we built from scratch on CloudSwing, and apply them to our OpenLogic Exchange (OLEX) product suite. Here are three items from my original blog and the improvements achieved this past year in streamlining our development processes.
1. Agile processes turn features into bite size stories that can deliver value rapidly.
We have adjusted from a mostly Scrum-based engineering process to a hybrid, combining features of Scrum with a continuous release process similar to Kanban. With a large system, there can be a fair amount of regression to perform before putting new code into production, so our goal was to move features through the QA (Quality Assurance) process faster. If features do not move through QA faster than they come into QA, continuous releases will not work. To do this, we leveraged every engineer, as well as other internal teams, as part of the QA process. We emphasized creating an automated testing platform where everyone is responsible for the QA regression suite instead of just our smaller QA team.
2. Repeatable builds, functional tests, and continuous integration, turn quality control into a predictable, automatable procedure.
In order to accomplish the process changes above, we involved the entire team in the regression suite maintenance. Our development tests and QA tests were in two different systems so we scrutinized the existing landscape of testing tools, chose the best, given the criteria that it must be automatable and run in continuous integration.
With open source, taking advantage of the latest trends requires keeping dependencies up-to-date as much as possible. In our case, we required a major upgrade to Rails to take full advantage of new testing tools such as Capybara. After completing the upgrades necessary, we implemented a new continuous, repeatable build process that everyone on the team maintains. This spreads out the testing load so even acceptance and regression test issues are identified earlier and resolved before the code is done.
As this new system has matured, our release regression times have dropped significantly, and we maintain a high release cadence. We deliver new features to our customers quickly.
3. Dev-ops engages developers in how to build for a production environment and test that environment early and often.
OLEX is a much larger and more complex product than CloudSwing, with many disparate systems involved during deployment. Since we are big on automation, many parts of the deployment process were already automated, using tools like Capistrano, Webistrano, and internal scripts. However, the process required downtime, as all the services switched to the new code. By utilizing inherent features of projects like Unicorn™ and Resque®, organizing our feature delivery schedule, and updating our deployment model using tools like Whiskey Disk and Sprinkle, we created a process that enables us to release code at will, usually with zero downtime, and in as little as a few minutes. This is especially helpful as more and more of our customers include international users.
All of this is a continuation of mindset change from large releases with longer cycles, to many smaller releases, quickly. By keeping our open source dependencies current, we take advantage of new projects and new ideas without a large investment. It is the essence of being agile and continually capitalizing on our agility.