Part 3: Technical Debt

Technical debt is a broad term that doesn't get enough attention. Engineers and stakeholders understand the concept, and typically do allocate a reasonable amount of time to 'paying it down'. But what exactly is technical debt, and why should it be more important? How does it compare to software testing and architecture?

What is Technical Debt?

When you are writing new code you are starting with nothing and working towards something. That 'something', in its perfect, completed form is typically quite complex and would take a lot of time to do 100% correctly. Engineers are in a perpetual state of "I'll do that later" due to the significant time delay that occurs between the mind and the physical realm. In order to deliver anything of any use to anyone, engineers are forced to constantly do 'good enough' solutions to meet project requirements.

There is no end to the amount of tests, error cases and architectural options in a project. Nothing would ever get done if engineers had to account for all of these in order to ship useful software. Luckily, engineers have learned how to live with insurmountable imperfection and still deliver working results. The problem is, however, that this behavior ultimately leads to bankruptcy when the debt becomes too much, and projects have no choice but to start over.

I have absolutely zero facts to base this on, but I'd guess that most software projects have a shelf life of 3-5 years before they either completely implode or the business just decides rebuilding the system would be easier than maintaining it. Add a poor language choice or bad architecture and the only thing missing from the party is the tremendous business success you've been working towards, pushing the system to its limits.

Realistically if you get 5 good years out of your software before needing to rebuild it you've done pretty well. But also think about 5 years of all those engineer salaries, and you have a multi-million dollar asset you've basically decided to throw in the trash. If that's the case, you should at least hope it's a strategic, planned move that will give your company a competitive edge rather than a need due to limitations of a poorly built project. In other words, hopefully it's because you choose to not because you need to.

Technical debt is the main reason codebases die. They become unwieldy, buggy, bloated, unreliable and difficult to maintain or improve. Hundreds, thousands or even millions of small compromises from years and years of work, piled on top of each other; hacks on hacks on hacks.

As the saying goes "one lie begets another" so does the engineer equivalent "one hack begets another".

Types of Technical Debt

A few examples of technical debt include:

- Hard-coding secrets & configuration instead of using environment variables- Lack of proper abstractions or patterns- Suboptimal architecture or technology choices- Outdated dependencies or tools- Failure to use best practices- Incomplete or inconsistent error handling- Temporary hacks used in place of more mature or correct implementations- Insecure or unsafe but working solutions- Code optimizations and linting

An even broader definition I like to use includes testing and documentation, which are typically considered separate activities. I like to group them together because a lot of teams spend significant time testing and documenting software as a dogma when they should instead be prioritized along with other technical debt. Testing may be lower priority than architecture and abstractions up front, but more relevant afterwards. After all, couldn't a lack of testing be considered technical debt? Or a lack of documentation?

Most of the time it's not a question of 'if' but a question of 'when'. If a codebase is mature, well-architected and well-maintained but light on testing it would make a ton of sense to invest in testing to further solidify the already winning combination. If the product is still evolving, architecture hasn't been seriously discussed and the code is a mess why would you want to bother with tests? As a preventative measure in a TDD environment, sure, but even still it is questionable if that's appropriate in the given context, especially before the company finds product/market fit.

The Dangers of Technical Debt

Aside from needing to completely rebuild an application every 3-5 years, technical debt creates additional problems. Early on in the project it doesn't matter, but as the codebase grows it becomes more complex to maintain. The edge cases and complicated business logic start to bleed over into areas they don't belong, separation of concerns starts to become blurred, and tight coupling begins to occur. Microservices help minimize this simply by minimizing the size and scope of an application, but otherwise it just takes a tremendous amount of talent, discipline and communication to keep a software project sane over the years, especially in a startup that's likely to go through many product iterations and experiments.

Technical debt limits your ability to adapt quickly and iterate, and establishing testing as a requirement means you're likely doubling your development efforts, cutting into time that could be used for technical debt. Technical debt also frustrates engineers and makes their job harder than it needs to be. It can put them in a 'reactive' state where they are simply on standby waiting for an issue to pop up with a legacy system. Instead of improving products they are stuck on damage control, just hoping the whole thing doesn't burn to the ground. Engineers want to build things, not fight in vain against a monster they didn't even create.