What is technical debt?
Technical debt is a collection of technical activities (architecture updates, code refactoring, or cleanup) that need to take place in order to properly complete a particular job, such as scaling-up the platform or meeting regulatory constraints.
Consider the analogy of building brick wall. If some of the lower bricks are not placed properly as you continue to build, the tower will become unstable and will inevitably collapse. That is, if the technical debt is not repaid, the platform will keep on accumulating interest, making it harder and harder to implement changes later on. Unaddressed technical debt increases software entropy.
Why does technical debt occur?
The main reason technical debt accumulates is because addressing the technical debt is often perceived as something that is important but not urgent. The effort required to clean-up the technical debt (through refactoring, upgrading, architecture revamp, or code-cleanup) is perceived as less important than the financial impact of adding features to the product (and thus increasing the value of your product) or engaging in custom development work to secure an important client (and thus achieving both professional services revenue and increasing monthly recurring revenue).
Examples of technical debt and how we addressed them
Here are some examples of technical debt and how we helped our customers overcome them.
Dealing with legacy codebase
Our client had a legacy codebase that was written in C++ about 10 years ago. A number of APIs were created to expose this functionality to the newer Java layer built on top of the C++ codebase. Unfortunately, there was little knowledge or understanding of the business logic written in the C++ codebase, so developers had difficulty creating the required APIs. The result was that customers were reporting bugs in the API , because the new features delivered were incorrectly addressing the new user scenarios.
Revamped the overall system architecture and progressively migrated the C++ codebase to Java. This ensure the new APIs were not only more robust, but the team had also a much more solid understand of the solution built using the refactored codebase.
By working closely with the client on implementing the solution, they experienced a reduction in the defects reported in production, an increase in customer satisfaction and customer retention rate, which lead to a significant increase in revenue year-to-year.
Rapid growth hindered by non-scalable architecture
Our client had an aging delivered ecommerce platform was now required to support a 100% increase in traffic year-to-year. The degradation in performance lead to longer response times, which resulted in unhappy customers and revenue loss.
We audited the architecture to identify and resolve several of the performance bottlenecks. For example, we moved from a hosted to a cloud solution, so that the platform could scale quickly to meet the increased traffic (for example, to deal with spikes during the Black Friday).
We also reviewed the performance of the SQL queries and optimized those that were not performing as required.
Our client grew over 100% year -over-year as the platform was now able to scale and handle the increased traffic. A 600% reduction in the response time for the real-time quote for shipping lead to an estimated $20M increase in revenue over 3 years through a higher conversion rate.
Suboptimal deployment of new technologies during the MVP
Another transactional platform we worked on used CMS and the search platform in a suboptimal way, as the development team was new to these technologies and there was tremendous pressure to deliver the MVP to production under aggressive timelines. Consequently, publishing new content to production was quirky and took a lot of time, and the indexing of the product catalog was lengthy, too. This forced the web team to have everything locked down at least one week before each event or marketing campaign.
We engaged in a refactoring of the codebase to ensure the search platform behaved in an optimal way, with support from SMEs. We also migrated to newer versions of the CMS and the search platform, which were both faster and more scalable.
With ongoing support and preventive maintenance, the technology solution we architected initially has enabled the company to grow its online revenue from $500M to $1.5B in that last 8 years.
Cutting costs by not implementing regression testing
In order to deliver the project under tight time and budget constraints, the client chose not to include regression tests as part of the MVP. Consequently, each new deployment to production was delayed by a full week of manual testing at a cost of $10k for QA and User Acceptance Testing.
We spent one iteration and $25k to implement a set of regression tests which only took 15 min to run and we integrated them as part of the regular build process.
Since deployment to production was scheduled at the end of each month-long sprint, by implementing the automated regression testing we were able to deliver value to production faster (12 releases per year instead of 10 releases per year) while saving over $100k on the annual cost for testing.
What is the impact of technical debt?
The accumulating technical debt may lead to the following types of negative outcomes to the business:
Slower time to market: at one point, the complexity of the system increases so much that it becomes increasingly difficult to maintain or to add new features. This leads to a slower time to market and an increased cost to add features / business value
Loss of revenue and market share: the code / architecture is too old and rigid to adapt to a competitive and fast-moving market. More defects are reported by clients in production, due to features that are inaccurate or which incompletely meet the intended user scenarios. This will lead to increased customer churn, bad reputation, and loss of revenue.
Escalating costs and difficulty maintaining the legacy application: the application is too complex and fragile and breaks-down often, and only a few of the original developers dare to touch some of the core areas
Growth stops as your product/platform is unable to scale to meet the increased demand: as an analogy, imagine building a 2 story house, and then incrementally adding 1 story every year. Soon you realize the foundation of the building was never meant to support such a weight / scale, so you have to tear the building down and rebuild it from scratch.
Non-compliance: the architecture and technical design of the system no longer reflect the needs of the business (evolving functional requirements, non-functional requirements changed: scalability, load, security etc.) or lead to the system failing to meet the SLAs agreed-upon with your customers
Code fragmentation and silo work: developers working in parallel made technical decisions focused on GTD on an aggressive timeline, and focused on local optimization rather than understanding the implications / tradeoffs of the system
What is the solution
The solution is to have a champion who can make a business case to the executive team (typically, to the CEO or the board, and potentially to the head of Product or Sales) about the long term benefit of tackling the technical debt.
Alignment and visibility: Ensure the executive team understands that when technical debt accumulates past a certain point, it can jeopardize your business and even take down your platform. For example, in 1999 eBay almost collapsed due to their technical debt and the liability caused by the 25 million lines of inflexible code
Speak business not technical: Make a business case that ongoing maintenance / prevention is more cost effective than rebuilding a failing platform: the reason you take your car to a $50 regular oil change every 3-4 months is to prevent a major failure of the powertrain which can cost $10k to fix and will stop you from driving you car for weeks.
Plan ahead: Work with the Product Owner to allocate ongoing capacity from your team’s velocity (typically 10-20% during each release) to handle technical debt. Ensure that addressing technical debt is not continuously deferred under the pressure to deliver more functionality, faster.
Is technical debt preventing you from scaling up and delivering business value? What have you done to address technical debt, what worked and what did not work for you?