DevOps for modern software development
Software is everywhere. It runs almost everything including our economy. In today’s digital age, enterprises must be customer-obsessed and systematically reinvent themselves to serve informed and empowered customers. Everyone understands how to download software from the “App Store” and wants your app to be there too. The demand for high quality software that is continually updated and improved will continue into the foreseeable future as the installed base of mobile devices expands around the globe.
Because strong IT performance is a competitive advantage, the pressure on IT Leaders to create quality software in short time frames has never been higher. Expectations for solutions that work on all devices is now a mandate. And, with many “apps” available for free, IT faces a lot of questions from business leaders about the increasing costs and time to build and maintain their portfolio of software applications.
Understanding the opportunities to improve software development involve four core areas:
- Hardware—System Administrators configure and support hardware with its operating system
- Software Development—Software Engineers writing code
- Software Quality Assurance—People testing the software
- Software Deployment—People that install the finished system on the hardware— or release management.
It can be even more complex in large corporate environments with a big infrastructure foot print, application integration that requires architectural standards, databases for managing huge amounts of transactions and customer information and information security to protect it.
But for sake of simplicity—I will focus on how software development at the core of those 4 roles—since that is the area IT Leaders have the most control and impact.
I have heard it said, more than once, that software development is more “art” than “science”—because getting all of the people in those 4 key areas working together using the same processes at the same pace can be challenging. If all parts of the process don’t mesh together the finished product will be late, be missing features or have defects—which are the big 3 things you want to get right when building software.
The working friction between the 4 core areas is usually the largest barrier to success for IT managers to meet business needs, cost effectively with the desired quality as quickly as possible.
Typical areas of friction are (or reasons why software development is error prone and time consuming):
- The desire of the system administrator to know the complete solution hardware configuration, capacity and performance requirements from the software engineer before the solution is designed; in order to perform their time consuming tasks to provision the hardware and create system environments—along with all their other competing priorities. Weeks or months are needed to acquire, install and configure hardware in the data center—especially when there are other competing projects already on the schedule.
- Lack of an automated method to automate build, deploy and test with every change—means performing this manually. This can mean the difference between performing a build and test of the entire system 5 times a day versus once a week or longer.
- Hardware and software not configured consistently in all environments throughout the development process causing delays for the team while issues are fixed. You know this is the case when you hear the software engineer say: “It works on my machine”, or “we are waiting for the environment to be ready”.
- When Dev and testing/QA are separate functions, the cycle time between coding and testing takes up all the scheduled slacked time to iron out issues and make sure code coverage is complete.
- Code merges from multiple software engineers or code branches collide and require re-work to fix.
- When the system is complete and handed back to the Operations to deploy. If deployment is performed manually it is subject to an errors when touching dozens of servers and thousands of settings—which could crash the system on go-live.
- We can only deploy off-hours or once a month and need the full team to deploy the software
If you have been involved in software development projects, no doubt you have experienced these issues and could easily add a dozen more examples. Not to mention people have feelings and when combined with other life situations stress is a factor. You know this is the case when you see finger pointing during crunch time. Sometimes an IT Manager is part technical manager and part Dr. Phil.
Can DevOps Help?
DevOps takes the core four roles and brings them together in a single unit with a mandate of using automation tools in order to create an iterative delivery process. In other words using automated software build, test and release tools a team has more control over the entire end-to-end process and eliminates a lot of the friction between the functional silos.
A big target of DevOps is product delivery. Whether on a mobile platform or web-based solution—the team can focus on set of product features with each iteration. This is very effective for getting new features into the hands of customers as quickly as possible.
For best results, DevOps is paired with Scrum and agile software development practices to achieve a continuous delivery cycle, (Development and QA activities are collapsed together with the support of tools), as shown below.
For purposes of illustration of how the tools support the DevOps process, the software testing is shown in step 5. In DevOps, Development and QA are combined into a single process step. In effect steps 2 through 6 are collapsed into a fluid continuous integration process and are performed together at least once daily. Ideally, the team would be able to perform steps 2 – 6 several times a day.
Using a combination of agile + DevOps and cloud infrastructure with a dedicated team is the best way to get started with a pilot project.
The Agile concepts are important—start with customer collaboration, keep things small so work can be accomplished in sprints or iterations, agree on a clear definition of done, transparency on progress, inspect artifacts, adapt to continuous activities to meet the sprint goal which is expressed in 30 day intervals. There are social media use cases where sprints are in one week timeframes.
In Scrum and agile practice, testing and QA are part of the development activity, performed continuously as part of the sprint toward meeting the Definition of Done. For those IT Shops using the waterfall process, Dev and QA are separate–with a hand-off–a noticeable fiction point. A key difference are those functions are combined in continuous process within the Scrum and Agile practice.
With continuous deployment (think daily builds deployed to test environments) you can engage your stakeholders actively in user acceptance testing and ops with load and external integration as development progresses. Everyone is working everyday towards a common definition of done.
A key benefit of the Scrum and agile practice is it provides valuable feedback early and often.
When Scrum and agile is used with DevOps, the team is empowered to accomplish their goals with minimal-to-no wait states on processes inside of their control. The team is 100% focused on converting the Product Backlog into a working product Increment. Business people are happy because they control the product backlog, have complete transparency on the team priorities and progress. Quality is improved because testing is performed early and often. The team is happy because they can focus on shipping working software.
In 2014, a survey with over 9,000 responses was conducted with technical professionals around the world found that:
“IT organizations (with DevOps adoption) were more agile and reliable, deploying code 30 times more frequently with 50 percent fewer failures.”
Once the DevOps team is running properly the ability to release software will dramatically improve well past monthly releases to weekly, daily, even multiple times per day if the business requires it.
To get the full benefit of DevOps you need to be using automation tools for build, test and deploy. This automation provides a rinse and repeat action to building software that cannot be achieved through legacy process or manual steps.
DevOps requires using tools that automate your software build and deploys the software—then will run an automated set of test scripts on that build and immediately show what passes or fails.
Automation tools require skills investment and time to get in place with your application. Depending if you run in the Cloud or on-premise, use Linux or Windows will influence your tool selection.
The process steps you need to consider for tools support are:
- Request capture and ticket workflow
- Source Control
- Agile Planning
- Test Case Management
- Build automation
- Continuous deployment
- Release Management
- Automated test scripts and Load Testing
- Feedback Management
- Team Collaboration
- Application Telemetry
- Lab Management
- Cloud provider integration
If you are a Microsoft shop this is a great place to start: Visual Studio Team Services
To enable DevOps, it is necessary to align your infrastructure with the right supporting process and environments.
To overcome initial infrastructure challenges, using Cloud based infrastructure allows a fast start for the team with virtual servers that support the most commonly used software development stack and automation tools for both Linux and Windows. Getting infrastructure provisioning semi-automated is critical to successful software development. The magic can happen only when the hardware is available and properly configured to run the application.
Sellers of Cloud infrastructure like to tell you they can provision a server in less than hour. That’s great, but without all the software your software engineering team needs to build, run and test the application—all you have is a server with a base OS still requires more work to be usable.
Documenting the “Desired State Configuration” (DSC) for your application is the first step to automating infrastructure provisioning. The DSC is everything the application needs to run. That includes OS configuration settings, 3rd party software and supporting files with their installation file directory locations. I have been successful with this approach on all types of applications and hardware situations.
Once you have the DSC, PowerShell, and a number of other tools and products enable automated infrastructure configuration with code, can be used to create your gold copy template for the server that a developer can create without the assistance of a System Administrator.
The days were you can go dark for 6 to 12 months to build software are gone.
Today many organizations have software releases every 30 days or less.
So how do you are get there?
This is a lot of change and too often existing teams cannot make the transition without a sophisticated change management process where new roles are defined, processes are piloted and tool training is provided. It is easier to start by picking a pilot team and letting them work completely independently using new processes and tools and Cloud based or pre-provisioned (on-site Data Center) infrastructure.
Once a pilot team has achieved success—then lessons learned can be distilled and shared to determine where the next step makes sense within your organization.
DevOps is gaining traction in corporate IT Shops, but is still a relatively new practice. Often times, using a consultant or Service provider already skilled in their area is the best way to get started.
Firms like IBM, Amazon, Microsoft, Accenture, Pivotal Labs+Cloud Foundry, EMC have solid tools and/or thoughtful service offerings. Accentient provides great training on how to use Microsoft ALM tools for Scrum and testing automation.
DevOps does not eliminate roles or jobs. Nor does DevOps combine roles. It uses automation to enable roles to be more effective with less friction because there are less hand-offs during the software development process. In DevOps, the System Administer will create the template that the Software Engineer to user over and over again to configure and create environments in a self-service style workflow.
Managing and operating your production systems and environments does not change. For example, you still need operational DBA’s to monitor and perform database maintenance to keep them running correctly. These roles actively participate in the agile DevOps process.
DevOps practices improve IT performance. Because of the use of tools to perform work and automate processes you get faster turn times, better quality at a reduced cost. As a bonus, you can also get metrics on the development process.
Finally, in addition to speeding up your software development process to delivery business value quickly while improving quality—DevOps with the right tools set also provides traceability. Traceability is essential for meeting compliance requirements to show how work is initiated, approved, tested and deployed as approved. With traceability it is much easier for your software development process to pass an internal or external audit—because you can show evidence of your work products along with the check points.