Month: February 2022

Overview of Technical Debt

“Technical debt is a metaphor commonly used by software professionals in reference to short-term compromises made during the design, development, testing and deployment processes”.

In order to stay competitive, many organizations opt for software development methodologies like Agile, to accelerate the overall development processes. Cramped up release schedules often force teams to skip the standard practices, resulting in the accumulation of technical debt. Some technical debt is given less priority/importance during the rapid release cycles and is addressed post the production release. 

Organizations often push large, complex changes to speed up the release process. Short-term compromises are acceptable to a certain extent, however, long-term debt can damage an organization’s IT infrastructure and reputation. Sometimes, it comes with a heavy penalty of re-engineering and post-release fixes. These damages could be in the form of high costs for:

• Remediating pending technical debt
• Customer dissatisfaction due to scalability and performance issues 
• Increased hiring and training 
• Increased modernization time 

The cost of refactoring, re-engineering, re-base, and re-platform could be much higher than the original cost during initial development. These compromises should be thoroughly analyzed and approved by IT stakeholders and CXOs. This involves looking at future tradeoffs, risk appetite (risk capacity), and cost. Organizations also need to evaluate the pros and cons of taking technical debt decisions.

Taking on technical debt can be both tricky and risky for organizations. Hence organizations must factor in associated risks and operational costs. One of the consequences of Technical debt is the implied cost of reworking on applications and their architecture. Hence, organizations should choose easy development paths and limited solutions to shorten the production time.

If these technical debts are not addressed over time, the accrued interest makes it more challenging to implement changes, resulting in business and technical challenges.

A Scandinavian study reveals that developers waste, on average, 23% of their time due to technical debt.

As if that wasn’t alarming enough, Stripe published data that shows that software developers on an average spend 42% of their workweek dealing with technical debt and bad code

Major Drivers of Technical Debt 

• Faster solution design process
• Faster development of source code
• Quick releases
• Cut throat business competition to release new and unique features early in market

Impact of accumulating Technical Debt 

• It results in daily operational costs to accommodate remediation.
• A longer development cycle leads to slower application releases.
• It incurs long term financial loss due to technical debt accumulation.
• It may result in compliance issues and lack of proper standards.
• Code quality and design gets compromised.
• More time is spent on debugging rather than development.
• Failures that can put an organization’s reputation at risk.
• It can be a cause of security breaches and hefty fines.
• It can potentially lead to Loss of agility, lower productivity due to outages.

Types of Technical Debt

• Design/Architecture Debt: 

It represents a design work with backlogs which may include a lack of design thinking processes, UI bugs, and other design flaws that were neglected. Most organizations do follow standard design practices like The Open Group Architecture Framework(TOGAF) due to the agile way of designing. Tools and techniques like the ADM and TOGAF implementation governance, provide the required format and standard of solution design.

• Code Debt: 

It is the most common debt, which is skipped due to speedy agile delivery, complexity, or lack of subject knowledge. In some cases, new features are added in the latest version, which the dev team may not be aware of. This might result in the dev team working on the same feature again, resulting in unnecessary cost and time investment. Sometimes, the development team doesn’t follow standard best practices for coding or use quick workarounds or they do not refactor code because of time-bound release cycles.

• NFR/Infrastructure Debt

Introduced during designing and implementing Non Functional Requirements (NFR) such as:

• Inaccurate scalability configuration may crash applications on high load.
• Improper availability planning leads to outage issues when any data center is down.
• Inaccurate caching, logging leads to slower application performance.
• Repetitive code of error/exception handling may create refactoring and performance issues.
• Additional auditing and tracing may lead to performance issues and occupy unnecessary database storage.
• Ignorance of security may lead to serious data breaches and financial loss.
• Improper observability and monitoring may not give an alert on time for any major issues in application and infrastructure. 
• Testing Debt

The pressure of quick agile releases may force organizations to miss out on most of the manual and automated testing scenarios. Frequent unit and detailed end-to-end integration testing can detect major production issues. Sometimes, these detailed testings are skipped during the development phase which leads to major production bugs. 

• Process Debt

It is introduced when a few less important business and technical process flow steps are skipped. In agile development, there are many processes that are followed like sprint planning, Kanban, Scrum, retro meetings, and some other project management processes such as Capability Maturity Model(CMM) and Project Management Institute(PMI), etc. Sometimes these processes are not followed religiously due to time constraint pressure, which may have a severe impact later.

• Defect Debt

It is introduced when minor technical bugs are skipped during the testing phase like frontend UI cosmetic bugs, etc. These low severity bugs are deferred to the following releases, which may later have an impact in the form of production bugs. These production bugs spoil an organization’s reputation and profit margin.

• Documentation Debt

It is introduced when some of the less important technical contents in the document are skipped. Improper documentation always creates an issue for customers and developers to understand and operate after the release. The engineering team may not properly document the release and feature/fix details due to quick release schedules. As a result, users find it difficult to test and use new features. 

• Known or Deliberate  Debt

Known or deliberate debt is injected on purpose to accelerate releases. This acceleration is achieved by workarounds or alternate methods or technologies that use simple algorithms. For example, sometimes the dev team does not evaluate and consider better algorithms to avoid cyclomatic code complexity in the source code. As a result, it reduces the performance of the code.

• Unknown Outdated/Accidental Debt

It is introduced unknowingly by developers/designers and other stakeholders. It is sometimes introduced by regression of other related code changes, independent applications and libraries. For example, if all applications use the same error handling library code and if there is a regression issue in that error handling library, it may impact all dependent applications.

• Bit Rot Technical Debt

According to Wired, it involves, “a component or system slowly devolving into unnecessary complexity through lots of incremental changes, often exacerbated when worked upon by several people who might not fully understand the original design.” In practice, many old and new engineers work on the same module code without knowing the background details of the code. New engineers may rewrite or redesign code without understanding the initial design and background. It may create issues like regression issues, etc. This happens over time, and it should be avoided.

Causes of Technical Debt

• Business competition

Competitive business markets may force organizations to roll out frequent feature releases to better/surpass/overtake their competitors and keep the customers interested.

• Time constraints due to agile releases

With tighter deadlines, the development team doesn’t have enough time to follow all coding/design standards such as language-specific coding standards, TOGAF enterprise design, suitable design patterns, review, testing/validation, and other best development practices.   

• Save short-term cost

Some organizations want to develop and release features faster to save additional development costs on coding and design effort. They may prefer employing a small development team for faster releases with minimal short-term cost. These organizations may also hire junior or unskilled developers for more profit margin.  

• Lack of knowledge and  training

The development team may change very frequently during exit, internal movement, and new hiring. Faster release cycles may result in undertrained resources due to lack of functional or technical training and little to no knowledge transfers about product and design.

• Improper project planning

Tighter release schedules may result in improper project planning which plays a major role in introducing technical debt. For example, skipping important meetings with all business stakeholders or project planning phases such as agile retro, scrum, and sprint plan meetings, etc. 

• Complex technical design and technical solution

The development teams prefer simple technical design and solution over a complex one, because they don’t want to spend more time and effort understanding complex algorithms and technical solutions. Complex solutions take more time to understand and implement. They also need more POC evaluation and effort.

• Poor development practices

Most development teams prefer shortcuts by following poor development practices. Due to aggressive release timelines and lack of knowledge, dev teams don’t follow standard coding and design practices. 

• Insufficient testing

It is a major contributor to technical debt. Regular unit and integration testing for even a small code change are very important. Testing and validation are the only mechanisms to identify bugs and shortfalls in software applications. They also find technical and functional bugs. Insufficient testing can lead to the introduction of technical debt. 

• Delayed refactoring

Tight deadlines may force developments in giving less priority to refactoring code in the early stages. Hence they defer and delay code refactoring to prioritize quick releases

Due to high pressure, development teams defer and delay code refactoring to release software products early, or they don’t give priority to refactor code during initial development. Sometimes they backtrack, review and refactor with a delayed path with low priority. 

• Constant change

‘Change is the only constant.’ Software applications evolve and adopt new designs and technologies over time. It’s hard to cope with these constant changes in parallel. It takes time to revisit the source code, design and then implements the latest design and technologies.

• Outdated technology 

Most traditional organizations use outdated technologies. They make late decisions to upgrade or defer modernization with modern technologies. They miss a lot of new modern features which are considered to be technical debt. This debt can be mitigated only by shifting to modern technologies.  

• No involvement and mentoring by senior developer and architect

It’s very common to have less or no involvement of senior developers and architects during design and development. It’s very important to have senior mentors who can review and guide the development team to avoid technical debt. Those senior developers/architects might have a better understanding and experience of working on the same project or software applications.

Identifying and Analyzing Technical Debt

• User feedback

User feedback/reviews are very important in identifying technical debt and mitigating it. Organizations should listen and act on users’ feedback for improvement and handling bugs. These feedbacks and bugs are considered to be technical debt.

• Analyze bad code smell

Use manual and automated code review to understand bad code smell like memory leakage of JVM Java applications. There are many code analyzers or tools like SonarQube, PMD, FindBug, Checkstyle, etc, that can help. It could be integrated with automated build and deployment of CI/CD pipelines for every release. 

• Monitoring and observability tools

Application Performance Monitoring (APM) tools are the best tools to monitor software applications continuously, for example, VMware Wavefront/Tanzu observability, Dynatrace, DataDog, etc. They have special algorithms to check the performance of applications and underlying infrastructure. They also analyze application logs and generate failure reasons reports. These reports are a great source of identifying technical debt.

• Manual and automated code review 

Continuous, manual, and automated code review processes definitely help to identify technical debt using static and automated code analyzers.  

• Operational profit and loss analysis

It’s done through business and senior CxO people by analyzing operational cost (Opex) and loss analysis reports. These reports give a fair idea of improvement and address important technical debt quickly. Addressing this technical debt is very important for any organization because it impacts their business revenue. 

• Performance metrics

Application Performance Monitoring (APM) and load testing tools also generate performance reports of the software application on high load. This is the best way to identify and mitigate technical debt of Non-functional requirements (NFR) configurations like the performance of application and infrastructure, read caching availability, scalability, etc.

• Understand long-term or short-term requirement

Organizations identify technical debt by understanding long-term and short-term technical requirements. Accordingly, they prioritize, plan and remediate. These requirements are prioritized based on business criticality and urgency. 

• Review with latest industry-standard best practices

Some technical debt can be analyzed by comparing with the latest industry-standard software application design and development such as Agile, TDD, BDD, Scrum, Kanban, Cloud-native, microservices, micro frontends, TOGAF, and latest technology trends like Cloud, etc.

• Code refactoring tools and techniques

There are modern tools available that are capable of analyzing legacy monolithic apps and providing suggestions or refactoring partially to modern cloud-native microservices design. They also provide tools to migrate on-prem VM (Virtual Machine) to cloud VM with easy lift and shift rebase.

• Security analysis

Some security-related technical debt is identified during the security analysis phase. There are some security analysis tools available like CheckMarx, SonarQube which generate security reports for applications. There are many other infrastructure security tools like Vmware Carbon black endpoint in security, RSA, Aquasec, Claire aqua security etc.

Best Practices to Avoid Technical Debt

To reduce technical debt, it’s essential to analyze and measure it. You can calculate technical debt by using remediation and development costs as parameters. These are a few techniques to avoid technical debt:

• Listen to user feedback comments and remediate application technical debt.
• Religiously follow consistent code review practices. Have multiple rounds of manual code and design reviews by senior peers and architects.
• Do automated testing after every build and release.
• Monitor and analyze reports based on observability and monitoring tools.  
• Analyze and evaluate the performance and business impact of any new code and design change before implementing.
• Follow standard coding best practices.
• Follow the manual and automated static code review for any release.
• Use incident management and issue tracker to report and track bugs.
• Always review and validate solution architecture before implementation. 
• Follow static and dynamic code analysis using code analyzer tools like Somarqube, PMD, FindBug, etc.
• Follow agile iterative development approach and regularly do retrospective meetings. Also, measure technical debt/TDR in each iteration.
• Use project management tools like Jira, Trello, etc.
• Do code refactoring of legacy code. Always revisit code and modularize common code components.  
• Strictly follow test-driven development (TDD) and Behavioral Driven Development (BDD) approach for every module of code.
• Follow continuous build, integration, test, and validate the approach on all releases.
• Last but not the least, technical debt should be documented, measured, and prioritized. 

Estimating Technical Debt Cost

It’s very important to measure technical debt cost as it helps stakeholders and senior management to analyze and prioritize remediation costs. This should be a measurable number to make business decisions. It also helps to track the technical debt remediation status. There are so many measurable variables to measure code and constraints to calculate technical debt.

There are various tools available like SonarQube to check code quality, code complexities, lines of code, etc.

We can calculate technical debt as a ratio of the cost to fix a software system [Remediation Cost] to the cost of developing it [Development Cost]. This ratio is called the Technical Debt Ratio [TDR]:

Technical Debt Ratio (TDR) = (Remediation Cost / Development Cost) x 100%

Good TDR should be <=5%. High TDR shows bad code quality, which involves more remediation costs. 

Optionally, remediation cost (RC) and Development cost (DC) could be also replaced by hours, which will help to calculate remediation time in terms of total efforts in hours. 

Key Takeaways

These are some key points about technical debt cost:

• The average organization wastes 25% to 45% of its development cost.
• Hiring and training new engineers involve additional costs and an increase in coordination costs.
• Operational overhead cost by spending 15 to 20% on unplanned work.
• Impacts organizations’ revenue for additional and unplanned work.
• Waste of time in analyzing improvement of source code and design.
• Lower productivity rate around 60 to 70%.
• Cost of  project management and modern tooling.

Conclusion

Technical Debt can impact different factors like overall operations cost, velocity, quality of the product and can easily end up impacting teams’ productivity and morale. Hence avoiding technical debt or addressing it at the right intervals during the development process is the best way forward. We hope this blog helps you have a better understanding of Technical debt and best practices of their remediation.