The Monolith: Penny Wise, Pound Foolish

I just arrived in Santa Clara for the 2016 Cloud Foundry Summit. I’ve been meaning to write this article for some time but for some reason haven’t found the time to do so. Although not typical nor preferable, I’m sitting in a hotel lobby with a beer and a burger with a bit of time on my hands before the Unconference begins later this afternoon. This must be the right time to get this off my mind and out into public to spark feedback and conversation.

The Monolith

So lets jump right in head first. I make it no secret that I’m a fan of the Microservices and Serverless architectural approaches, Continuous Delivery, Containers and Feature Teams which I deem as aspects of the Cloud Native development approach. This is not just theoretical admiration but rather hands on and experiential from years of applying and consulting with similar or direct use of these approaches and technologies. Over and over again I’ve seen that these have lead to more scalable, understandable and, most importantly, changeable systems. The alternative, developing a Monolith architecture approach, has, in my experience, tended in the other direction on all of these dimensions.

The Monolith seems easier to develop and understand for quite some time. Although it is well understood that “project size is easily the most significant determinant of effort, cost and schedule [for a software project]” as Steve McConnell pointed out in his book “Software Estimation: Demystifying the Black Art”. Lets break down some of the reasons why a Monolith approach usually turns into a Big Ball of Mud.

The Scalability Problem

When most people think of scaling they think of “web scale”, cluster size or problems only the largest companies have to deal with. Although these are areas where the Monolith architecture approach breaks down, there are situations that are much more common around scalability to deal with.

Have you ever tried to get two or more teams working on the same codebase? What problems start to emerge? How do we handle them? These are the questions that all software development organizations should ask themselves. Here are some of the problems that I’ve seen and how organizations attempted to solve them:

  • Trampling each others changes tends to lead towards complicated source control management, build and deployment approaches including:
    • Slow code review processes
    • Delayed merging of code
    • Reverse code trampling into developer environment
    • Not allowing multiple teams to work on sections of codebase
  • Breaking changes to dependent code leads to:
    • Processes for more up front design reviews
    • More comprehensive documentation of proposed changes

These problems seem to most as just the way software development works. As time passes on a system the unapologetic grip of legacy code takes hold. Unintended coupling between different parts of the system creep in and lead to additional cyclomatic complexity as McCabe defined in 1976. Later studies showed a positive correlation between complexity and lower cohesion and defect count in systems with higher cycolmatic complexity. Rather than responding with apathy and synacism about the onslaught of legacy code, this should scream at us there must be a better way.

The Understandability Problem

How long does it take for a brand, spanking new team member to get up to speed with your system or even an aspect of the system? Can a new team member be trusted to check in at least somewhat significant code on day 1 and not cause great amounts of harm? Do all of your existing team members all understand the system enough to make changes?

We’ve all been there. Downloading an existing codebase with tens or hundreds of thousands of lines of code. Taking a look at it to figure out how you should get started understanding it. Even the best intentioned and capable teams can find themselves in this dilemma of the Monolith architecture. Where did it all go wrong?

Requirements change. Business changes. Code changes. When developing in a Monolith architecture these new perspectives tend to result in leaky abstractions, muddled models, and broken boundary contexts. I believe this is due to the convenience allowed, as a type of moral hazard, to make changes that cross boundary contexts inside a single system. These changes are not done with ill intentions or even without forethought. As the great Gerald Weinberg said:

Things are the way they are because they got that way.

The Changeability Problem

Have you ever been in a situation where you’ve had to tell a product manager, stakeholder or customer that a seemingly simple change may take weeks or months because of technical debt? Were there ever situations where an architectural redesign of a component within the system was deemed as too risky with a significant and unknown duration to resolve? Are there components of the system that team members aren’t supposed to touch and instead there are wrappers around the component to interract with it from the outside?

I’ve seen all of these and have also been part of a team that has brought such codebases back to life. How did we do it? Break it apart into understandable, isolated, cohesive pieces through careful refactoring of the code and infrastructure over long periods of time. Why did we do it? Because we knew it would lead to accelerated delivery of improvements to those components. We could then scale up the number of teams developing on the system by creating feature teams around those components.

Changeability is one of the most important aspects of any software system. Smaller, cohesive components that are decoupled from other components allows the system to be malleable as the need for changes emerge. An application should not equal a single codebase. Instead, an application is the interactions of multiple components that provide the desired behavior and value to users. How can we get to this desired state?

Which Way Next?

It is my opinion that the Cloud Native development approach is going to change how we deliver software even more than most people think. The days of developing using monolithic software architectures are numbered. The platforms, organizational team structures and architectural patterns & practices that will support this change are maturing quickly. My upcoming articles will describe how I think some of this will play out in the industry. Make way for the Cloud Native way!

Focus on Configuration Management Debt First

It has been a few years since Managing Software Debt was published and I think about how it could have been better on a regular basis. I think the content was useful to put in a book and aligns with current movements such as DevOps, Microservices, Continuous Delivery, Lean and Agile. On the other hand, there may have been a tendency by me to focus on Technical Debt at the beginning of the book that was not warranted in terms of importance for managing software debt. It has been my experience that managing technical debt, the activities that a team or team members choose not to do well now and will impede future development if left undone, is less important than the other four types of software debt. In fact, here are the priorities that I typically used with companies to help them manage software debt more effectively:

  1. Configuration Management Debt: Integration and release management become more risky, complex, and error-prone.
  2. Platform Experience Debt: The availability of people to work on software changes is becoming limited or cost-prohibitive.
  3. Design Debt: The cost of adding features is increasing toward the point where it is more than the cost of writing from scratch.
  4. Quality Debt: There is a diminishing ability to verify the functional and technical quality of software.

Along the way, I might work with teams on processes and techniques that can help with reducing technical debt but that was to get them ready to take advantage of the efficiencies that result from the other four. I wonder if the popularity of the topic “technical debt” and my background as a developer lead me to focus chapters 2 through 4 on managing technical debt. No matter what the reasoning, I think it is good to discuss the reasons for demoting it in priority with regards to managing software debt on your team and in your organizations.

Why Configuration Management Debt First?

The value of software is only potential value until it is put into a user’s hands. There can be many roadblocks to software getting into user’s hands in an organization’s processes:

  • Proliferation of long-lived branches
  • Over burdened release engineering and operations teams
  • Poor integration processes across architecture components and scaled team delivery
  • High coupling with centrally managed architecture element/component
  • Too many variations/versions of the software supported in production
  • Code changes feel too risky and takes too long to validate before releasing into production
  • Poor documentation practices
  • Too many hand-offs between teams in order to release software to users

Just as the tag line of the chapter 6 “Configuration Management Debt” says:

If releases are like giving birth, then you must be doing something wrong.
— Robert Benefield

In organizations that have effective configuration management practices it is common to see deployment pipelines that have a smaller number of hand-offs between teams, architectures that tend to be more malleable, and efficient validation processes. What is interesting about the list that I just wrote in the previous sentence is that they align with managing three other types of software debt more effectively:

  • Smaller number of hand-offs: Platform Experience Debt
  • Malleable architectures: Design Debt
  • Efficient validation processes: Quality Debt

What I have found is that by focusing on Configuration Management Debt it is simpler to identify aspects of the integration and release management process that need to be tackled in order to get working software in the hands of users sooner while reducing the bottlenecks in the organizational processes and practices therefore leading to further optimizations in the future.

Alignment with Today’s Industry Conversations

It is interesting to note that this focus aligns well with the tenets of the DevOps movement luminaries. Going towards Feature Teams organizational configurations that have people with all of the capabilities to design, implement, validate, integrate, configure, deploy and maintain reduces hand-offs and increases quality because the team that builds the software maintains it, as well. Its amazing how quality goes up when a team member on rotation has to respond to production issues in the middle of the night.

Not only does this align with the DevOps but malleable architectures tends to align with the Microservices movement. Given Feature Teams we can take advantage of Conway’s Law, rather than be a victim of it, to align team delivery with providing a specific capability in the architecture. Since these capabilities are more specific the implementation tends to be smaller and therefore easier to change later. Now there are still operational issues to overcome in order to make this an efficient approach. Platforms such as Cloud Foundry that provide application runtimes and access to services with low operational overhead will make Microservices based architectures even more approachable and efficient to attain.

The Agile movement has already encouraged significant changes in how we validate software. Continuous Delivery has continued to push the validation efficiencies forward. As more teams become aware and more effective with test frameworks, tools, platforms and SaaS offerings we will continue to see more efficient validation processes in our delivery pipelines.

There is a great book by Jez Humble, Lean Enterprise: How High Performance Organizations Innovate at Scale, that I recommend if you want to explore the topics above further. Let me know in the comments section if you have any additional or different thoughts around focusing on configuration management debt first. I’m always interested in learning from others tackling difficult problems and the approaches that they see working.