Focus on Configuration Management Debt First

It has been a few years since Managing Software Debt was published and I think about how it could have been better on a regular basis. I think the content was useful to put in a book and aligns with current movements such as DevOps, Microservices, Continuous Delivery, Lean and Agile. On the other hand, there may have been a tendency by me to focus on Technical Debt at the beginning of the book that was not warranted in terms of importance for managing software debt. It has been my experience that managing technical debt, the activities that a team or team members choose not to do well now and will impede future development if left undone, is less important than the other four types of software debt. In fact, here are the priorities that I typically used with companies to help them manage software debt more effectively:

  1. Configuration Management Debt: Integration and release management become more risky, complex, and error-prone.
  2. Platform Experience Debt: The availability of people to work on software changes is becoming limited or cost-prohibitive.
  3. Design Debt: The cost of adding features is increasing toward the point where it is more than the cost of writing from scratch.
  4. Quality Debt: There is a diminishing ability to verify the functional and technical quality of software.

Along the way, I might work with teams on processes and techniques that can help with reducing technical debt but that was to get them ready to take advantage of the efficiencies that result from the other four. I wonder if the popularity of the topic “technical debt” and my background as a developer lead me to focus chapters 2 through 4 on managing technical debt. No matter what the reasoning, I think it is good to discuss the reasons for demoting it in priority with regards to managing software debt on your team and in your organizations.

Why Configuration Management Debt First?

The value of software is only potential value until it is put into a user’s hands. There can be many roadblocks to software getting into user’s hands in an organization’s processes:

  • Proliferation of long-lived branches
  • Over burdened release engineering and operations teams
  • Poor integration processes across architecture components and scaled team delivery
  • High coupling with centrally managed architecture element/component
  • Too many variations/versions of the software supported in production
  • Code changes feel too risky and takes too long to validate before releasing into production
  • Poor documentation practices
  • Too many hand-offs between teams in order to release software to users

Just as the tag line of the chapter 6 “Configuration Management Debt” says:

If releases are like giving birth, then you must be doing something wrong.
— Robert Benefield

In organizations that have effective configuration management practices it is common to see deployment pipelines that have a smaller number of hand-offs between teams, architectures that tend to be more malleable, and efficient validation processes. What is interesting about the list that I just wrote in the previous sentence is that they align with managing three other types of software debt more effectively:

  • Smaller number of hand-offs: Platform Experience Debt
  • Malleable architectures: Design Debt
  • Efficient validation processes: Quality Debt

What I have found is that by focusing on Configuration Management Debt it is simpler to identify aspects of the integration and release management process that need to be tackled in order to get working software in the hands of users sooner while reducing the bottlenecks in the organizational processes and practices therefore leading to further optimizations in the future.

Alignment with Today’s Industry Conversations

It is interesting to note that this focus aligns well with the tenets of the DevOps movement luminaries. Going towards Feature Teams organizational configurations that have people with all of the capabilities to design, implement, validate, integrate, configure, deploy and maintain reduces hand-offs and increases quality because the team that builds the software maintains it, as well. Its amazing how quality goes up when a team member on rotation has to respond to production issues in the middle of the night.

Not only does this align with the DevOps but malleable architectures tends to align with the Microservices movement. Given Feature Teams we can take advantage of Conway’s Law, rather than be a victim of it, to align team delivery with providing a specific capability in the architecture. Since these capabilities are more specific the implementation tends to be smaller and therefore easier to change later. Now there are still operational issues to overcome in order to make this an efficient approach. Platforms such as Cloud Foundry that provide application runtimes and access to services with low operational overhead will make Microservices based architectures even more approachable and efficient to attain.

The Agile movement has already encouraged significant changes in how we validate software. Continuous Delivery has continued to push the validation efficiencies forward. As more teams become aware and more effective with test frameworks, tools, platforms and SaaS offerings we will continue to see more efficient validation processes in our delivery pipelines.

There is a great book by Jez Humble, Lean Enterprise: How High Performance Organizations Innovate at Scale, that I recommend if you want to explore the topics above further. Let me know in the comments section if you have any additional or different thoughts around focusing on configuration management debt first. I’m always interested in learning from others tackling difficult problems and the approaches that they see working.

Managing Software Debt 2015 Predictions

2015 is fast approaching and for the first time I felt the urge to make public predictions on what the new year will bring through the lens of Managing Software Debt. Interestingly enough, many of these predictions revolve around topics this blog was constructed to discuss. This blog has been a long time coming for me since my last official blog post prior to November was in 2012 so let me take a slight diversion to describe my reflections on how this blog came into being.

My last blog gettingagile.com had run its course after 178 posts from 2005 to 2012 and it seemed to me that we were Beyond Agile and I needed a different focus. After publishing Managing Software Debt: Building for Inevitable Change in 2010, a reference on 5 types of software debt and how they can be managed and monitored, it seemed that focusing on leading change through Continuous Delivery and reducing Configuration Management Debt had the best results in my consulting experience. Since that time, the DevOps movement has hit a full stride and embodies this approach for leading change in organizations. I think we are on the precipice of our next industry revolution to reduce the cost of change for software as cloud has become a common enterprise platform of choice. With that, here are areas of the software development, deployment and operations ecosystem that I think are going to see significant interest in 2015.

  • PaaS (Platform as a Service)
  • Twelve-Factor Apps and Microservices
  • Feature Teams around Business Capabilities
  • Deployment Orchestration

PaaS

OK, OK. I know that my new role is Product Owner for PaaS at CenturyLink Cloud but there is a good reason I took this role in November (time to note my disclaimer that the views expressed on this blog are mine alone). In my last few roles the impact of infrastructure development enabling deployment foo applications and services to cloud platforms was significant enough to cause me pause. It seemed that the problems being solved were similar across development efforts: load balancing apps & services, event publishing & processing, service discovery, continuous delivery pipelining, blue/green deployments and infrastructure provisioning just to name a few. We had looked at multiple vendor offerings but what we saw prior to 2014 had been, in our opinion, immature. As 2014 progressed, the popularity of containers kicked off a valuable conversation about separation of concerns for deployment and infrastructure. Containers provided a piece of the solution that allowed infrastructure and application/service development to execute within their own life cycles beyond what Puppet and Chef had done for configuration management.

After I heard about the Product Owner role here at CenturyLink Cloud, where I’d be working with a team to deliver PaaS based on Cloud Foundry, I updated my knowledge of the PaaS space. I had already been playing with Docker and worked with others who implemented a build pipeline for our services based on Docker containers. Through this process it was clear that there were still significant problems to solve beyond the ease of development story that Docker was just an introductory chapter to. While researching Cloud Foundry again, after trying it out back in 2012 and deciding not to use it, I was pleasantly surprised how far the platform had come. Immediately I took notice of an aspect of Cloud Foundry called Warden which manages isolated, ephemeral, and resource-controlled environments (aka containers). It been around since November 2011 and had the full Cloud Foundry ecosystem surrounding it which looked to help solve more of the problems in the infrastructure and application/service deployment space than other alternatives available today.

As more enterprise developers see the benefits of PaaS, such as ease of development and deployment with low overhead for configuration management and operations involvement, there will be a large upswing in its adoption. Also, folks in operations will further benefit and enable the DevOps culture as PaaS continues to mature allowing for more self-service provisioning and deployment while still getting the visibility needed to support service level agreements and control operating costs. Look for more on PaaS in this blog as we learn more about how customers innovate and deliver on our upcoming PaaS offering.

Twelve-Factor Apps and Microservices

My last post on The Imminent Acceleration of the Twelve-Factor Apps Approach already discussed why I think The Twelve-Factor App and Microservices will be big in 2015. Some folks in our industry are already realizing the benefits of these approaches. It is probably not surprising that this realization was found mostly outside of the monolithic ESB and SOA tool vendor offerings. Instead, the rise of SPAs (single page apps), RESTful APIs, OpenID/OAuth, cloud computing, open source and many other emergent approaches from the community have been the potion for increased adoption of service-oriented architectures. Look for significant changes in enterprise application development to support The Twelve-Factor Apps approach and implementation of Microservices (or at least less monolithic) as 2015 progresses.

Feature Teams around Business Capabilities

As the chapter “Platform Experience Debt” from the Managing Software Debt book explained, organizations are more flexible when there is clearer alignment of teams to business capabilities and ultimately to their users. The rise of DevOps has brought the cultural changes needed to be more adaptive (and dare I say “agile”) to light and ignited a follow on movement from the software development centric agile movement to incorporating production operations as an aspect of team responsibility. This has made it even more apparent that the Feature Teams collaborative team configuration helps drive alignment of business capabilities with user needs along. Not only that, this organizational alignment creates less brittle boundaries between teams than component (or functional) team configurations do. DevOps has definitely made its mark with Gene Kim’s book The Phoenix Project and the outbreak of DevOps oriented conferences around the world. Look for the DevOps movement to accelerate as more real world change stories are shared in the new year.

Deployment Orchestration

With the rise of PaaS, The Twelve-Factor App and Microservices, the need for more effective deployment orchestration tools and processes will grow. Enterprise Operations groups will need strategies for dealing with the increased frequency of deployment, proliferation of environments and running processes, and hybrid models with internal data centers and cloud architectures used in conjunction with public cloud provider offerings. Continuous Integration servers and access to server instances are not enough. The number of deployment models, platforms, and network topologies will make governance a mess. In 2015 we will need to start finding solutions to orchestrating deployments from build to validation to deployment and to governance. There is a lot of room to innovate and make a significant impact in Deployment Orchestration. I’m excited to see what is coming to solve Deployment Orchestration challenges in the new year.

This is my first attempt at a predictions blog. Let me know how I did on Twitter (@csterwa). I’m always looking for feedback.

Have a Happy New Year 2015!

Managing Configuration Management Debt

Introduction

In my book “Managing Software Debt: Building for Inevitable Change”, chapter 6 discusses Configuration Management Debt and approaches to help keep this type of debt to a minimum. Since I originally wrote this chapter a lot has gone on in the software development and operations communities. There has been a couple of huge uprisings: Continuous Delivery and DevOps. Of course, the ideas expressed as part of these movements are not necessarily new but rather standing on proverbial shoulders of giants.

Managing Configuration Management Debt

When I wrote in this chapter is only a small portion of ideas that are being feverishly adopted across our industry. It is exciting to see just how fast truly Lean approaches such as DevOps are motivating teams, enhancing customer experiences, and driving business results. Application of DevOps and Continuous Delivery are quickly maturing and seem to be a stronger force than even the Manifesto for Agile Software Development.

I would like to review the 3 areas that organizations can work on to improve their Configuration Management performance to put some perspective on how these hold up 5 or so years later:

  • Transfer some Configuration Management responsibilities to the teams
  • Increase automated feedback
  • Track issues in a collaborative manner

Transfer Responsibilities to Teams

Release Engineering teams tend to get overburdened as time goes by. They tend to be centralized and support more teams than they have people. The DevOps movement has put a spotlight onto this problem and provided cultural, process, and technical approaches for reducing the risk of overburdening Release Engineering teams. Taking a DevOps approach implies moving some responsibilities for configuration management to the team.

Increase Automated Feedback

To reduce the impact of increasing the team’s platform complexity by adding these responsibilities, organizations must invest in increasing automation and developer user experience of automation tools. The other aspect of transferring responsibilities to teams from the book was automating Install and Rollback procedures. Going beyond Continuous Integration towards the fully automated build, test deployment and monitoring approaches contained in Continuous Delivery encompasses this idea. There has been a tremendous amount of innovation that makes Continuous Delivery more approachable including the growth of Cloud Computing adoption to Chef & Puppet to automate infrastructure to the promise of Platform as a Service (PaaS).

Track Issues Collaboratively

Automated build pipelines and teams that have more responsibility for software operations leads us to ask about who tracks issues. Again, the DevOps movement has pushed our organizations to think about the responsibility for production deployments. The more teams can take responsibility, given appropriate automation and developer user experience with the tools, the more responsive they will be to customer issues and operational changes needed around their software. It is my contention that the more responsibility teams have for production operations the better production will operate.

Conclusion

In review, I probably write enough on Configuration Management Debt. In my consulting and product development activities since the book I have realized that focusing on Configuration Management Debt first makes steps to address the other four types of software debt (technical, quality, design, and platform experience) easier to see. The build pipeline(s) and the spread of responsibilities can tell teams a lot about where they can find optimizations.

From Test-Driven to DevOps and PaaS

Over a decade ago one of the main practices that I looked for in teams were automated testing and continuous integration. If teams were able to do these well then it seemed like the “golden bits” would be ready at all times for Operations to deploy. Many teams were thrilled to have delivery from planning to local development to continuously integrated and then let Ops handle it from there.

Earlier in my career I had worked in a couple of companies where there was no separate Ops team. We had to deploy, manage, and maintain our own operational environments. I recognize now that not having the ability to take delivery from plan to operational made me feel uncomfortable.

Around a decade ago I started asking the teams I was working on to focus on “Push Button Release”. Even if we couldn’t access the deployment environment we would ensure that the number of steps to deploy our software was 1 or as close to 1 as we could possibly get. Our team should automate the deployment process as though we would have to manage the actual deployment ourselves. Not only that, we should also tend to the rollback deployment process in the same way. In fact, we created CI jobs that deployed and rolled back just to ensure that these automated scripts acted as expected after all the builds, tests, and packaging jobs completed.

A decade later our industry is tending towards new possibilities due to progressive and somewhat obsessive approaches revealed publicly by companies such as IMVU, Netflix, Etsy, and others. It is no longer enough for teams to hand off a validated build with a “push button” deployment script. Teams now run their own configuration management via Puppet, Chef, Ansible and Docker and are finding many different ways to manage the deployment environment configuration dynamically.

The next few years will bring many tools and platforms that will attempt to tackle the deployment environment configuration challenges such as service discovery, log aggregation, monitoring, alerting and notification, attached storage, security, policy management, data transport and more. It is an exciting time to see how we are progressing past automated testing and continuous integration and as an industry looking to automate all the way into operations. With the momentum of the DevOps movement and PaaS (Platform as a Service) innovation, this is an exciting time and I intend to be part of it as much as I can. Lets make the promise of DevOps and PaaS a reality.