Summary: 

• Efficiency and Effectiveness are the two vectors we want to control when trying to meet market demand. When we fail to meet market fit we generate debt that we need to pay off by refining our product to align with the market. This was the original analogy used for technical debt.
• Technical debt has grown to encompass anything in software or software delivery that doesn’t add value. By decomposing those categories it becomes easier to manage them. 
• Managing technical debt by visualizing it and budgeting its removal enables ongoing product growth. By ignoring the cost of the debt, it will grow until it consumes the budget and more. 
• There are methods for delivering and designing products to help minimize the overall amount of debt that is taken on. Unfortunately, you cannot deliver a technical product without some technical debt. 

There’s an iceberg up ahead

In a recent article, I highlighted the need to increase adaptability to match market changes. Our systems have liabilities that don’t show up on a balance sheet that prevent our company from moving forward. One example was Southwest Airlines’ 2022 scheduling system issues, which cost the company billions of dollars when it failed. But this is the tip of the iceberg. 

• We aren’t seeing the cost of delay for optimizing flights daily. (Effective)
• We aren’t seeing the opportunity cost of having to spend extra time maintaining old systems. (Effective)
• We aren’t seeing how long it takes simple changes to be made inside the company. (Efficiency) 
• We aren’t seeing the extra maintenance cost for any change to the system. (Efficiency)
• We aren’t seeing the extra time it takes each employee to use a system not optimized for their current workflow (Effective/Efficiency)

What is technical debt and where does it come from

Where does technical debt come from? How does it get into the system? Let’s break down the problem into 3 main categories, Market Fit Debt, Architectural Debt, and Waste. 

Market Fit Debt

Ward Cunningham is credited with coming up with the debt metaphor as it relates to software. He used this metaphor to try and highlight that when writing software we inherently don’t know what to correct until we deploy it and the software gets used. This excess or unnecessary work is a loan we are taking out to learn the right outcome. We have to do this in software especially because all product work is discovery. According to a recent Pendo study, 80% of product features are never used. This number varies widely across studies but it is never 0%. Also just because a feature isn’t used much doesn’t mean it isn’t needed, after all, I’ve never used my car’s airbags but I’m glad they are there. 

Software development takes on debt to pay for learning what is used when it is deployed. One product discovery goal is to minimize the cost of learning what the customer will use. When doing product development we don’t know the recipe yet. We are making the dish while we are trying to figure out the recipe. An unknown outcome can’t be optimized. 

Until we fix the discrepancy between what is coded and actual business use, we will have a clunky non-optimized recipe. Whenever we release a new system or software we must accept we are inherently taking on debt because of our lack of knowledge. Some debt is unavoidable but we can make ourselves prone to excess debt. The bigger the release without feedback the larger the risk of the debt we are taking on.

This is one of many reasons why small releases with frequent user feedback loops are important. Ideas always come up during product development. They come in the form of, “I bet the user would like…” or “What would be really cool…” Ideas are great but a conversation with users to validate them is much cheaper than a team of developers making them. You should only build to the level of your understanding of the problem. Or avoid building if you don’t know the real problem. Taiichi Ohno would remind us maximizing the utilization of a team is maximizing waste. As we test our assumptions we hope to minimize the debt we take on.

Debt is a good metaphor, as Cunningham points out because this problem doesn’t go away. You are paying interest on the lack of understanding. There are clear signs in your system that debt is in your system.

1. Users are not fully utilizing what you made. If a user asks for the option to export/import to excel, then you might be dealing with some technical debt. 
2. The system is difficult to change because you are working around something that shouldn’t be there. If you are storing extra data with dummy values for a data model, then you might be dealing with some technical debt. 
3. Support things that don’t have value. If users aren’t using some sections of your website, then you might be dealing with some technical debt. 

Architectural Debt and Waste

There are two other factors that have been added to the original debt metaphor because without fixing them your overall cost will go up and adaptability will go down. Martin Fowler created a visualization that includes these categories:

The bottom row highlights technical debt as it relates to the product itself, an architecture debt. In the “Prudent” category, this architecture comes from hindsight, which is always part of an evolving system. What worked in version one is an impediment in version two.  Under “Reckless” we see the source of architecture debt comes from incompetence like not separating functionality into logical layers, non-cohesive design, or failing to compartmentalize logic. What you may have saved in developer costs, you will end up spending more on increased support, lack of adaptability, and because of those bad market fit over time. 

Why? We are always incurring debt when we release. If you only learn after going live how to change things but the architecture is messy and doesn’t allow change. Things that can’t change have to be rewritten, which means the whole first product is waste.

A more insidious waste is the ever-evolving software architecture landscape. A good practice of today may be the security hole of tomorrow. This is like clay sewer pipes, which were a revolution when they were installed but trees with aggressive roots made the idea foolish over time. This is to say there is debt building in our systems even if they match the market perfectly, the market hasn’t changed, and we built them perfectly for the time. 

The “Reckless” category is cost without any value, aka waste. In the “Prudent” category we do get some value or we simply didn’t know something until we did it. This is still within the scope of the original metaphor, we gain value but rework needs to happen or we will incur ongoing costs. 

To summarize there are 3 things that need to be cleaned up on a regular basis: market fit debt, antiquated architecture, and waste. Since the market and the software landscape are continually changing, this debt grows without us changing anything. 

Technical Debt Management

Since there is no way to permanently remove technical debt, we need to develop strategies to minimize it. Here are a few ways I have worked to keep it under control. 

1. Visualize the cost. 
2. Budget to fix. 

Visualize the Cost

When the debt is hidden it doesn’t get addressed. It can build a long time before the costs are visible. As a result, we need to make the costs visible as early as possible. How do we do that? We measure the impact of the costs continuously. As mentioned before there are signs that technical debt is in our system and we should be able to measure if it is increasing or decreasing. The key to any of these metrics is not just to take a snapshot but to watch the trend over time to see if the debt is going up or down. Here are some I have used:

• Bottom 10 lowest used features – Market fit debt
• Number of “how-to” requests – Market fit debt
• The time it takes a new contractor to have code checked in – Market fit debt
• The time it takes from a user expressing a need to production deployment of a solution – Architecture debt
• Lines of code that aren’t used – Architecture Debt / Waste
• Defect rate – Waste

The web is full of key performance indicators that “every team should have”. I do not buy into metrics that one size fits all. Figure out what is an indicator for your software of bad Market Fit, bad architecture, and waste and track it.

Budget to fix 

When budgeting for a new project, it is important to include the “fix” cost after the release. If we know there will always be debt, then the budget needs to extend beyond the release. Additionally, the support cost for a project somehow goes away over time in many organizations, perhaps this is an old function of the depreciation of a machine in a factory. Assuming that after 5 years we don’t need to account for it anymore. Releasing software isn’t like putting a machine in a factory it is more like planting a tree in the jungle. Because of the interdependence of software, there are a large number of external factors affecting its efficacy. 

There is a tendency to want to try and budget capacity into only two categories: new features and support. But if we do this without managing our technical debt we quickly squeeze out the ability to deliver new features:

Instead, if we are more intentional with improving our learning, delivering, and reducing debt our ability to deliver new features to our users remains consistent. 

This can be a hard sell initially because it seems like we are spending less time delivering new features than we could be. But our overall goal isn’t maximizing production it is maximizing value. Maximizing production as always leads to maximizing waste. 

Design for Minimizing Debt

1. Build small, fail small
2. Learning Architecture 

Build small, fail small

There is a tendency in many organizations for large (aka big bang) releases, and this may be for a variety of reasons: marketing, budget cycles, or the need to align a large number of groups. The problem is we don’t learn anything until users start using our product. If we have to build big we need to minimize the market fit debt by getting continual feedback. One way to do this is to deliver to a small group of customers and get feedback before the big bang release. 

Picking the right group of customers for feedback is important. I once released a code-first interface that our expert developers had given glowing feedback on, but our new contractors had no idea what to do with it. Unfortunately for me, we had a surge of new contractors. To satisfy them we scrambled to create a simple plug-play interface, but we kept the expert interface as an alternative. Some of the experts switched to using the simple interface and gave feedback on how much they liked it. Key learning for me was: Just because your customers like what they have doesn’t mean you can’t serve them and more customers better. After this, I use a user experience method called crazy 8’s to mockup 8 completely different designs (sometimes multiple times) before picking a single way forward. 

Learning Architecture 

If bad architecture can box us in, then good architecture can set us free. What is good architecture? Architecture that enables you to run experiments to validate assumptions. In the software industry, one of the most common is the ability to do “A/B testing”. This means one customer base gets version A and another gets version B. Then you test to see which one gets better user adoption/feedback/etc. It is a way to test the waters with your toe before jumping in the freezing river. As a software platform owner, we made sure to enable A/B deployments on our platform. One of my surprises was the customers used it in a variety of ways. They used it for production deployment but they also used it for a variety of user experience testing and security impact testing. 

Final thoughts

Whether you pay attention to it or not technical debt can and will catch up with you. Left hidden it will continue to impede efficiency to the point you run out of time and resources to be effective. One of the added bonuses for me is as I deal with and remove waste, I end up learning more. I learn why things weren’t used or what made something brittle. That learning can be rolled back into a better project. 

If ignored these underlying costs can cripple a company’s ability to adapt or deliver, as in the case of Southwest. Some organizations may try and mask these costs but then the iceberg only gets bigger. A recent study indicated that the average system has a million dollars worth of technical debt. In 2010 Gartner estimated the cost of technical debt worldwide was $1 trillion. These are not small numbers and are a drag on our company’s ability to adapt and keep up with the market.