Designing Systems: Anti-Patterns and What Experience Teaches

This is the final part of the Designing Systems series.

• Part 1: How to Approach a System Design: Framing, requirements vs constraints vs tradeoffs, the hidden assumptions, and the design as a communication artifact.
• Part 2: Core Elements: Data, state, control flow, failure, and observability.
• Part 3: Structuring for Change: Boundaries, contracts, evolution, and ownership.
• Part 4: Anti-Patterns and What Experience Teaches: The failure modes that show up repeatedly, what a good design actually feels like, and a few habits that compound.

The first three parts (Part 1, Part 2, Part 3) described how to approach a design well. Part 4 is about how designs go wrong, how to catch them before they ship, and what the positive case looks like once you have seen enough designs to recognise it.

Table of Contents

• The Recurring Anti-Patterns
  • Premature Generalisation
  • Hidden Coupling
  • Over-Engineering for Scale That Is Not There
  • Under-Engineering for Failure That Will Be
  • The Blank-Page Trap
  • The Org-Chart Imprint
  • Documentation as Afterthought
• What a Good Design Actually Feels Like
• Habits That Compound
• Conclusion

The Recurring Anti-Patterns

A small number of failure modes account for most of the bad system designs an experienced engineer sees over a career. The patterns are not a complete list, and any specific design might fail in its own original way. But these are the ones that show up often enough to be worth recognising on sight.

Premature Generalisation

The designer anticipates future requirements that have not arrived and builds the system to handle them. The code paths multiply. The configuration surface expands. The abstractions creep upward to make room for cases nobody has asked for. The system is now ready for several futures, none of which materialises, and the work of changing it has become harder than it would have been to write the simple version three times.

How to catch it:

• Ask, for each abstraction, what concrete case it serves today. If the answer is "none, but we might need it later," it is probably premature.
• Apply the rule of three: three concrete instances of a pattern before generalising it. Fewer than three, and the abstraction is guessing.
• Notice when the design conversation drifts from "how do we solve this problem" to "how do we build a framework for problems like this." The drift is usually a sign of premature generalisation.

The cost is paid by everyone who works on the system after the original designer leaves.

Hidden Coupling

Two components look independent on the diagram and behave as one in practice. The hidden coupling can be a shared queue that serialises their work, a shared database that locks them together under load, a shared library that forces them to deploy in lockstep, or a shared assumption about timing that is never written down.

How to catch it:

• Ask, for each pair of components that look independent: what would break if one of them stopped existing tomorrow? If the answer is more than expected, the coupling is hidden.
• Look at the change history. If two components are routinely modified together, they are coupled, even if the diagram says otherwise.
• Watch the on-call pattern. Pages on one service that turn out to be caused by another service are a reliable indicator of hidden coupling.

Hidden coupling is more dangerous than visible coupling because the design conversations do not address it. The system behaves as if the coupling does not exist, until it does.

Over-Engineering for Scale That Is Not There

The system is partitioned, sharded, queued, cached, and load-balanced for a workload an order of magnitude beyond what it will ever see. Every operation has to traverse three components that exist only because someone imagined a future bottleneck. The simple thing has been buried under the architecture for the imagined thing.

How to catch it:

• Match capacity decisions to the actual or projected load, not to a category label. "We need to scale" is not a design input. "We expect 5x growth to 20k requests/sec in 18 months" is.
• Treat scaling as a problem to solve when it arrives, not a problem to over-solve in advance. The simple version of the system, run on hardware sized for the next eighteen months, is usually a better starting point than the partitioned version built for the next five years.
• Be honest about what you actually know about the load profile. A lot of over-engineering is justified by load assumptions that nobody has measured.

Under-Engineering for Failure That Will Be

The mirror image. The happy path is well thought through, and the failure paths are sketched in vague language ("we will retry," "we will alert"). The system goes to production and the on-call team discovers, one incident at a time, what the design did not say.

How to catch it:

• Run the failure mode table from Part 2. For each component, what fails, how often, and what recovery looks like. If the answers are vague, the design is incomplete.
• Specifically ask about slow failures, not just down failures. Most production incidents are partial: latency rises, error rates climb, but the system technically still responds. Designs that have not thought about partial failure tend to handle it badly.
• Pressure-test the assumptions about recovery. A retry policy nobody has measured is a guess. A failover mechanism that has never been exercised is a hope.

The Blank-Page Trap

The designer treats the new system as if no constraints exist outside it. Existing data shapes, existing dependencies, existing operational habits, existing team capabilities - all of them are ignored in favour of a clean design that exists only on the whiteboard. When the design meets the world, the world is the thing that has to change.

How to catch it:

• Before designing, read the systems that the new design has to live alongside. The existing systems set the context. A design that pretends the context does not exist is a design that will be expensive to integrate.
• Talk to the people who run the adjacent systems. Their tacit knowledge is part of the design constraint, whether or not it gets written down.
• Be suspicious of designs that are pure on the whiteboard. Real systems live in the middle of constraints they did not choose.

The Org-Chart Imprint

Conway's law shows up as a bug when the system is drawn around the current team structure for short-term convenience, and the team structure changes a year later. The components that mapped cleanly to teams are now stranded across multiple teams, and the boundaries that made sense for the original org actively get in the way of the new one.

How to catch it:

• Distinguish between boundaries that reflect the domain and boundaries that reflect the org chart. The first survive reorganisation; the second do not.
• Pay attention to which decisions in the design are easy to revisit and which are not. The expensive decisions should be domain-driven; the cheap ones can be org-driven.
• Ask, before drawing service boundaries: would this still be the right shape if the org changed in twelve months?

Documentation as Afterthought

The system ships, the original team moves on, and the next team inherits a working system with no explanation of why it works that way. The on-call playbook is a wiki page that was last updated two reorganisations ago. The design decisions are in someone's head, and that someone is no longer available.

How to catch it:

• Treat the design document as a deliverable, not a side artefact. It is read more often than the code.
• Capture decisions with reasoning attached, in a form that survives the original team. Architecture decision records exist for this purpose; they are cheap to write and disproportionately valuable later.
• The on-call experience is part of the design. A system that cannot be operated by a new on-call engineer reading the runbooks is incomplete, even if the code is correct.

A Quick Reference

The two most common, in my experience, are hidden coupling and documentation as afterthought. The first is the most expensive to fix later. The second is the most easily prevented and the most often skipped.

What a Good Design Actually Feels Like

Anti-patterns are easier to write about than positive cases. It is worth being explicit about what good feels like, because most engineers see fewer good designs than bad ones in any given decade.

A few signals that a system is well-designed:

• Operations are quiet. The system runs without surprising the team that owns it. Pages are rare, and when they happen, they have a known cause and a known response. Quiet operations are usually a result of the failure mode work in Part 2 having been done early.
• Changes are cheap. Most pull requests touch one component. Schema changes follow a known migration pattern. New requirements map to existing boundaries more often than they require new ones. The cost of the next change is roughly proportional to the size of the change, not to the size of the system.
• The design is explainable. A new engineer can be brought up to speed in days, not months. A senior engineer can describe the system to a peer in fifteen minutes. The design document and the code agree with each other.
• The contracts are honest. What each component promises is what it actually delivers. There are no informal side-channel agreements that hold the system together.
• Failure is boring. When something does fail, the recovery is rehearsed, the impact is contained, and the post-incident review is short. Most incidents are smaller than the system, not larger.

None of these is a property of one decision. Each one is the result of many small decisions made well over the life of the system. The point of a good design is to make it possible for the next thousand small decisions to be made well.

Habits That Compound

A few engineering habits compound over a career and are worth naming, because they are not visible in any single design and are present in everyone who designs well consistently.

• Read before drawing. Before designing anything that has to live alongside an existing system, read the existing system. The time saved on the read is paid back many times over in the time not wasted on the design.
• Write the framing down. The NFRs, constraints, decisions, and hidden assumptions all benefit from being written down. The writing is almost always the cheap part of the design.
• Sketch alternatives, even when you already have a preferred design. Two or three sketches sharpen the preferred one. A single sketch is hard to evaluate, including by the person who drew it.
• Pay attention to the boring machinery. Migrations, rollouts, feature flags, observability, runbooks. The long-term health of the system is mostly determined by how well the boring machinery works.
• Bring the team in at the framing stage, not the result stage. The team is more useful as a participant in the decisions than as an audience for them.
• Write decisions down with their reasoning. The reasoning is what makes the decision revisable. A decision without reasoning is a fact that has to be relitigated.
• Accept that you will be wrong about some of it. The point of designing for change is that the original designer does not need to be right about everything. The system is well designed if the next round of decisions can be made well, regardless of who is making them.

Conclusion

The four parts of this series have walked through the same idea from four angles. Part 1 argued for slowing down at the front of the design to frame the problem before drawing it. Part 2 named the five elements that every system design has to make decisions about: data, state, control flow, failure, and observability. Part 3 covered the four levers that determine whether a system can survive change: boundaries, contracts, evolution, and ownership. Part 4 closed with the failure modes that show up most often, the positive case for what a good design feels like, and the habits that compound.

The deeper claim across all four parts is that system design is mostly the work of being honest about what you know and what you do not know, and of structuring decisions so that they can be revisited cheaply as the system and the world around it change. The best designs are not the ones that anticipate every future. They are the ones that make the next decision cheaper than the last one.

That is the whole series.