Agile in Government Summit was two days of great speakers and roundtables from a broad range of government agencies provided a view of the state of agile in some pockets of our intelligence and military organizations.
I had the opportunity to speak on the topic of Agile HW. My talk entitled Breadboard to Battlefield considered a large scale aerospace project that exhibited the principles of Lean & Agile.
Below is my summary of the conference and some takeaways.
Opportunities and Barriers
DevOps holds more promise for the IT leaders in many of the intelligence organizations, the since it requires less moving parts and a more pragmatic approach to solving the problems of security and delivery.
SAFe® had great billing at the conference and appears the front-runner among the larger organizations who want to join the Agile bandwagon.
Nevertheless, the biggest barrier for many was getting an Authority To Operate (ATO) for security sensitive software applications. The DevOps technologies and test automation are shrinking this long delay (18 months in some cases), but the lowest figure quoted at the conference was around two weeks. In comparison to Amazon, which in 2011 did 1079 deployment per HOUR to production, the ATO for the best is class security sensitive deployment is 2 billion times slower. That was five years ago and let’s speculate that Amazon does one or two orders of magnitude more deploys per hour in 2017.
Obvious the risks are much higher with some software deployments vs. others. Sensitive software needs the right level of review as the size of the consequences for faulty security can balloon from single person to a large group and then to entire country. Amazon proved how faulty controls can take down an entire region when a developer fat fingered a command and shutdown S3 services for all the US-EAST-1 region for several hours.
Balancing the speed versus consequences will never be an easy task. However, when it comes to rolling out solutions that are “secure,” the DevOps model has much to offer when coupled with infrastructure as code (IaC) and automated testing (AT). With IaC and AT an environment can be either destructively tested (e.g. intrusion tested with “gray hat” attacks) and when completed, you destroy the environment with all its possible infections and stand up a production environment from scratch.
DevOps, Lean and Agile are taking root and making large and small public areas faster to adapt to changing circumstances.
DevOps Moves from The Pheonix Project to Government Agencies
One of the best books I’ve read on DevOps and Lean is The Pheonix Project. Many of you have read it and if you haven’t I highly recommend you pick up a copy and dash through it. The book is an exhilarating ride about a company that absolutely cannot get anything right with technology, but they are dependent on technology to handle the complexity of their business. Through many twists and turns they use Lean thinking to focus and execute, but not without some harrowing adventures.
Many organizations are organically adopting the tools and processes that enable DevOps behaviors in small areas. For example, DevOps might only equip a single product among hundreds because the skills and vision of a single team pushed through the learning curve and found value in the paradigm. In some cases, the pressure to deliver frequently, perhaps daily or hourly, demands a different style and many teams adopt DevOps tools and processes purely for survival.
In larger organizations, such as the National Geospatial Administration (NGA), DevOps is institutionalized with a management structure that includes DevOps Manager, DevOps Engineer, etc. Organizations who have adopted DevOps practices and tools with an official mandate and an organizational structure are fun to talk with because they tell stories of successful and speedy enterprise level deployments. And they also have many frustrations and stories of failure, but they can speak the DevOps lingo and have a vision for continued improvement.
It’s SAFe® to be Agile
The second biggest topic of the summit was the Scale Agile Framework (SAFe®). The notion of putting large systems together in an agile pipeline is the nut that so many companies want to crack. Strange that Google and Amazon have not chosen the top down process path. Interestingly when I interviewed for a Technical Project Manager position at Google, they said, “No we don’t enforce a process. Each team picks their own.” At Amazon, I heard the same message. The release chain in these organizations is so fast that there isn’t time for a Release Train (RT), the releases or deployment in Google and Amazon come like bullets from an automatic rifle.
The SAFe acronym and it’s associated process are a good model for executives to digest. Leaders see a deterministic mechanism for turning their lethargic organizations into lean operating machines.
Lean and Kaizen and the Toyota Production System (TPS) are somewhat contrary to SAFe’s more top-down approach in that Lean processes are typical driven by bottoms up refinement.
Nevertheless, when an organization needs to start a change, it can be very helpful to adopt a “system” to change behavior and then observe the new behaviors. Observing the new behaviors help reinforce a new way of thinking and the new way of thinking can then help you break from old thinking and adapt. The important thing to keep in mind is that necessity to adapt. Customizing process and removing barriers is a moving target and requires analysis and thinking, not just large scale processes.
Requests for Proposal are out for solicitation that requires the contractor to perform using SAFe®. The goal of the agencies and the military in adopting agile methodologies are two fold 1) rapid response, 2) cost savings. The adoptions of SAFe® probably has an effectiveness curve that is months to years in the future once adoption is mandated. No complaints allowed if the adoption proves successful and execution improves along with the frugal use of tax payers dollars.
There are numerous situations and different levels of security and criticality for each agency, but I observe that organizations seem to operate better when they have leadership that understands technology, and both mindset and practices of the agile manifesto.
If you would like to connect with me, please click on the Contact Agile Leadership Edge tab. I would love to hear more about your specific experience with DevOps, Lean and Agile and the pain points that you currently face.