How do you know if a company is making progress on their DevOps journey?

A road leading back through a green valley into the bright sun.

Google’s “Accelerate the State of DevOps 2021” report shows that 26% of survey respondents are in the “Elite” category of DevOps adoption and Puppet’s “State of DevOps Report 2021” shows that 18% of respondents are “highly evolved” in DevOps. Both reports show more and more companies reaching this status every year. Engineers have higher job satisfaction with more automation and companies have better performance. You may be lucky enough to be in one of those elite companies, but others may be looking to work for one of them. What can ask during your interviews to know if a company is making progress on their DevOps journey?

How do you define an “Elite” DevOps company?

Many companies say they value automation, but they show it by tracking and improving DevOps metrics. Adoption of DevOps is incremental and happens at stages. We can’t expect everything to be automated immediately. However they should be able to show progress on the following key DevOps metrics. See a summary below and more details in the table below. (See a similar table on p. 10 of the Puppet’s State of DevOps Report 2021.)

Deployment frequency. Elite companies: multiple per day. Seriously. Elite performers deploy code 973 times more frequently than low performers. Wow, and I thought monthly releases were so much faster than bi-annual releases.
Lead time from change to production. Elite companies: less than an hour. Elite performers heave lead times 6,570 times faster than low performers.
Time to restore service. Elite companies: less than an hour.
Change failure rate. Elite companies: less than 15%.

Accelerate State of DevOps 2021, pg. 9

More and more companies are achieving Elite status every year. See the chart below.

Accelerate State of DevOps 2021, pg. 11

Similar trends according to the Puppet report. But it says that far too many companies are stuck in the middle of their DevOps journey.

67% of mid-evolution companies say their team has automated most repetitive tasks, but 58% say there are multiple handoffs among teams to deploy products and services.

Does it matter if a company uses DevOps?

Here is what DevOps can deliver (from Accelerate the State of DevOps 2021):

Beat competitors with faster value to customers with 973x more frequent code deployments (p. 13).
Less pain to customers making them more likely to stay with 3x lower change failure rate (p. 13). They are 3.7 times more likely to meet their reliability targets because of continuous integration (CI) that incorporates automated testing (p. 26- 27).
Impact the bottom line by being 1.6x more likely to meet or exceed organizational goals when integrating security best practices (p. 24).

Engineers have higher job satisfaction with more automation.

89% are more satisfied with their job and 84% are more satisfied with their company as a result of using automation in the workplace. Employee satisfaction increases as a result of automation in the workplace

It takes a commitment to focus resources on automation for improved performance later. The Puppet State of DevOps Report 2021 found that companies with high DevOps adoption have a lot of support from their leadership.

Puppet State of DevOps Report, p. 11

How do you find a company destined for “elite” status?

Ok, so now you may be getting excited: there are really companies like this out there because DevOps makes good business sense. But how can you find a company like this to work for? Ask questions during the interview. What is their software development process? What Continuous Integration (CI) best practices are implemented? Do they use the pull request in Git to automatically run a static analysis tool? Do they use a code coverage tool to determine how many lines of code are tested in their automated tests? Do software engineers (SWE) write the unit test first and then write the code? Do SWE add unit tests and component tests to the automated process before they ask SDET (Software Development Engineering in Test) teams to implement End-to-End (E2E) tests or manual tests?

Find out more about the culture. Is there a shared commitment to excellence, not passing the buck, so DevOps engineers don’t have to hound SWE to fix a problem? Do engineers trust in the tests enough so they can trust the automated deployments? Is there sufficient test coverage so engineers can trust the code? (from DevOps: The Big Picture, by Richard Seroter on Pluralsight.)

Determine if the automations create a process or pattern that will not be dropped under pressure. Many manual checks or steps are dropped when schedule pressure hits. But with a robust and well-built automation, the team will still use version control, and the build will still use unit tests to prevent pushing defects to production. Ask if the team is oriented toward overall outcome (less bugs found in testing, more test coverage by SWE), not individual team outcome that are often at odds (Ex: SWE done with code faster, SDET small issue queue). (DevOps Foundations: Core Concepts and Fundamentals, by Chris Behrens on Pluralsight).

What did you find?

Now you know there are companies out there who can push new code within an hour instead of accumulate enhancements for 6 months before they release them. And now you know how to find them. What kind of answers do you hear from employers? Do they even track any of the key DevOps KPIs or can they quantify recent improvements? Do they struggle to make progress on the journey or do DevOps Engineers spend more than 50% of their time creating/improving automations?