How to recover your architecture after drift and erosion

In 2002, the internal motto at Danger, Inc. was “Ship or die.” This intense focus on delivery encapsulates a common sentiment in the software industry: get the product out the door.

But this approach can also be misleading, suggesting that the job is done once the product ships. In reality, software is a living entity, continuously evolving to incorporate new technologies, comply with regulations, and satisfy emerging customer demands.

Yet, if these ongoing changes are not meticulously managed, you start accumulating what is known as "technical debt": sub-optimal design or implementation decisions or nonperforming or broken code. What many teams overlook, is that the underlying architecture also begins to degrade, accumulating what is known as architectural debt.

Over time, systems burdened with significant technical debt become increasingly challenging to maintain and adapt.

This blog post will explore the nature of architectural technical debt, providing insights into how to assess, manage, and mitigate this debt along with strategies to handle architectural drift and erosion. By gaining a comprehensive and real-time understanding of your system's current state, you can make informed decisions that enhance your team's effectiveness and your software’s resilience.

What is architectural technical debt?

“We rarely reward, recognize, or teach code stewardship the way that we do feature development skills. But code stewardship skills—documenting systems, recovering context from code, and designing for future changes—make the difference between a team that hums along for a decade or more and a team that repeatedly mires itself in declarations of code bankruptcy, rewrites, and despair.”

- Chelsea Troy, Senior Staff Software Engineer, Data Platform and Machine Learning Operations @ Mozilla

Technical debt encapsulates the compromises and expedient choices made during software development that offer short-term benefits at the expense of long-term health. In other words, it’s a metaphor that reflects the accumulated consequences of past decisions and shortcuts in software development.

Technical debt is an inevitable component of software development and it can be manageable and strategically be addressed. But an overwhelming accumulation can severely impact software maintainability and longevity.

Architectural Technical Debt (ATD), a specific category of technical debt, includes both intentional and unintentional decisions that compromise a system’s architecture. This form of debt can lead to decreased performance and scalability, ultimately threatening the system’s adaptability.

Over the last few years snowballing ATD has been cropping up across a wide variety of applications. Among the contributing factors we see:

• The increasing complexity of software systems driven by the rise of SaaS, APIs, composable platforms, and legacy systems, making them challenging to manage and understand. Not to mention that the adoption of new technologies like CI/CD pipelines, DevOps tools, etc. increased the velocity with which changes manifest in an architecture.
• Neglected architectural processes in agile environments, can lead to the inconsistent implementation of best practices like continuous system design reviews.
• The lack of real-time, full-stack visibility when evolving, debugging, refactoring. Immediately understanding how a system actually behaves in production, is often an effort-intensive job that requires times and hopping between tools to correlate frontend user actions, backend traces, logs, request/response payloads, etc.

ATD is sometimes inevitable and can even be necessary when the goal is rapid delivery followed by iterative improvements. However, it’s crucial for teams to recognize ATD early and implement robust management strategies to prevent the architecture from becoming outdated, unreliable, and rigid against the demands of evolving business needs.

Differences between architectural drift and erosion

Software architecture serves as a vital framework for the development and maintenance of software systems, offering teams a shared abstraction to understand and communicate about complex systems. However, this architecture can degrade in two distinct ways: drift and erosion.

Applications that were initially well-architected can deteriorate due to various factors such as neglect, shifting priorities, developer turnover, release pressures, a lack of awareness of these changes and many other factors.

When architectural decisions drift or erode, the symptoms often appear as user-reported bugs, performance issues, or unexplained failures.

• Architectural drift is defined by the discrepancies between the planned architecture and the actual implementation. This form of degradation introduces design elements that, while not part of the initial architectural plan, do not necessarily contravene it. The architecture remains fundamentally intact, but accumulates unaccounted-for decisions like inconsistent coding practices, redundant components, or tangled dependencies. These elements often go undocumented, rendering the original architecture misleading and potentially undermining trust in both the system architecture and its associated documentation.
• Architectural erosion, in contrast, occurs when new design elements directly conflict with or undermine the system's foundational architecture, thus violating its guiding principles. Examples include tightly coupled modules, bypassing security protocols, and ignoring performance constraints. Erosion not only compromises the system's integrity but leads to a fragile architecture that is likely to encounter significant issues in the future.

While architectural drift and erosion are distinct concepts, both manifest as real production issues that affect end-users. A database call that bypasses caching (erosion) looks identical to a poorly optimized query (drift) when users report slow page loads, but the solutions are fundamentally different.

Effective ATD management requires more than static architecture diagrams. Teams need visibility into how their systems actually execute in production: which services communicate, what dependencies are called, how data flows through middleware, and where architectural violations occur under real user loads. Session-level observability that connects frontend behavior to backend execution provides the ground truth needed to distinguish intentional trade-offs from unintended degradation.

Transforming your strategy for managing architectural debt

Teams facing architectural technical debt typically adopt one of two strategies:

• continuously patching the system to circumvent ATD limitations while putting out fires as they crop up,
• OR engaging in extensive refactoring efforts.

Unfortunately, both approaches often fall short and might even intensify the existing technical debt.

👎 Patchwork and reactive measures: Simply tweaking the code tends to be a superficial fix. Without a comprehensive understanding of the system’s architecture or the root causes of issues, teams operate on a reactive basis, which seldom resolves the underlying problems. When developers debug issues without full-stack context (seeing only error logs or frontend symptoms) they often apply fixes that address immediate pain points while missing the architectural violations underneath. This reactive debugging perpetuates drift by treating symptoms rather than causes.

👎 Challenges of major refactoring: On the other hand, refactoring, especially of deeply intertwined subsystems, requires immense effort, substantial budget, and extensive engineering resources. During this period, other parts of the team must maintain the existing, often outdated, platform while simultaneously integrating new features. Even thorough refactoring efforts may not be sustainable if they fail to address the fundamental processes that initially led to the accumulation of debt.

The most effective strategy for managing all types of technical debt involves shifting from these REACTIVE measures a holistic, PROACTIVE approach to software development.

Addressing these architectural mismatches promptly reduces the "interest" accrued from technical debt, which manifests as bugs, increased time for understanding the system, and extended development time for new features.

Delivering smaller batches of work more frequently, can be more beneficial than delivering the same value in fewer, larger batches - as it’s perfectly highlighted in Kent Beck’s “Principle of Flow”.

This supports the idea of continuous improvement and adjustment, which can be crucial in managing technical debt effectively.

An effective approach to architectural recovery

All engineering teams, whether they follow Agile or Waterfall methodologies (or something in between), are susceptible to the accumulation of architectural technical debt (ADT). Even teams committed to the Agile principle of continuous improvement may find themselves focusing solely on product enhancements while inadvertently neglecting the long-term architectural integrity of their systems.

For example, the focus on rapid delivery often results in inadequate documentation and a lack of clear design, which complicates the understanding of the system’s architecture and how its components interconnect.

Additionally, recent shifts in the tech industry (such as layoffs and a heightened focus on operational efficiency and profitability) tend to prioritize short-term gains over the essential modernization of development processes. This often exacerbates the challenges of ADT.

Change can be difficult, and inertia within organizations often sustains the status quo until it becomes a significant impediment. Leadership that can effectively navigate and instigate change is crucial in these scenarios.

To combat the accumulation of ADT, two proactive and manageable strategies can be particularly effective:

(1) Implement real-time architecture visibility: Begin by conducting a thorough inventory of your existing architecture. This helps assess the extent of architectural drift from the original design and identify any instances of architectural erosion. Tools that automatically map system components can significantly streamline this process.

(2) Implement collaborative, debugging workflows: When architectural drift introduces undocumented dependencies or erosion creates hidden coupling, even simple bugs require extensive investigation.

Traditional debugging workflows compound this problem. Support teams collect incomplete bug reports, developers hunt through data across multiple systems, and everyone wastes hours reconstructing context that should be immediately available. The back-and-forth between "what did the user do?" and "what happened in the backend?" burns engineering capacity that could be spent on strategic refactoring.

The lack of visibility and collaboration also doesn't allow teams to foresee and align on system design changes, minimizing the risk of unsuitable architectural decisions.

Final thoughts

Building applications has become more accessible than ever. But, this ease of creation often leads to increased complexity in software systems, making them difficult to manage and understand. This complexity can slow down teams as they navigate, with little visibility, through mazes of technical debt, dependencies, and bugs, causing them to miss opportunities to adapt to changes in the market.

The maintenance of software systems often boils down to their economic value. Decision-makers frequently weigh whether it's more cost-effective to rewrite an entire system or to continue maintaining and updating the existing one, despite the challenges involved.

However, investing in real-time architecture visibility and collaborative debugging can drastically reduce architectural debt, making it more manageable and strategically addressable. These measures can shift the economic balance, making it more viable to maintain and evolve existing systems as business needs change.