A second producer on a shared counter silently changes what every existing consumer is reading

The lesson

Adding a producer to an existing counter is not an additive change. It is a silent redefinition of every consumer that already reads it.

The reason it goes unnoticed is that the second producer is almost always reasonable. The event genuinely is a dismissal, the recording API genuinely does take dismissals, and reusing it is genuinely cheaper than building a parallel store. Every argument for the reuse is about the producer's side, where it is correct. The cost is entirely on the consumer's side, where nobody is looking, because those consumers were written before the second producer existed and cannot mention it.

What makes this different from ordinary metric drift is the direction of the damage. A shared counter has consumers that care about different provenance, and the second producer's events are maximally irrelevant to at least one of them. So the error is not spread evenly — it concentrates on the consumer that had the least to do with the change, and it grows in proportion to how successful the new subsystem is. The better the second producer works, the more wrong the old consumer gets. That is the opposite of the failure mode people watch for, which is a new feature being underused.

The counter's name is what seals it. It was named for the original event, so it keeps reading as correct forever. A reviewer looking at the increment sees the right verb; a reviewer looking at the consumer sees the right field. Only someone holding both at once sees that they no longer mean the same thing, and the two are usually in different modules owned by different concerns.

The detection is an invariant, not an inspection. If the counter was originally incremented only as a consequence of some parent event, then it can never exceed that parent's count. When it does, that is not a rounding problem or a bad week of data — it is arithmetic proof that something else is writing to it. This is the cheapest available test and it needs no instrumentation: hold the sub-count against the count it should be a subset of, and look for rows where the subset is larger. The rows that violate it are the contaminated ones, and their excess is a floor on how much contamination exists, never the total.

The failure that taught it

A retrieval system recorded per-document outcomes so that documents which kept surfacing without ever being useful would sink in search. One of those outcomes was a dismissal: this was returned, read, and correctly ignored. A ranking function read that dismissal count and applied a demotion, capped at a maximum.

A second subsystem was built later, on the same corpus but a different channel. Instead of returning documents in answer to a query, it matched documents against code as it was written and surfaced them unprompted. It needed a way for its output to be dismissed too, and the obvious one already existed, so its output text instructed the reader to call the same recording verb with the same dismissal value.

Both halves worked. The unprompted channel worked well — it was independently the more productive of the two. But its firings were frequent by design and mostly not applicable in the moment, so it generated dismissals at many times the rate the query channel ever had. Every one of those landed in the counter the ranking function read, and the ranking function had no idea a second channel existed.

The result was exactly inverted. The single most productive document in the unprompted channel — the one that had just caught a real defect in freshly written code — was pinned at the maximum demotion in search, because it fired often and most firings were correctly waved away. Being good at the second job was what made it invisible to the first.

It surfaced through the invariant. A document showed more dismissals than it had ever been retrieved — eleven against nine, and another at twenty-one against one. A dismissal that only ever follows a retrieval cannot outnumber retrievals, so the excess had to come from somewhere else, and there was only one somewhere else. Across the corpus, roughly half of all dismissals sat on documents that also belonged to the second channel.

The reports from users described the symptom one layer up and misattributed the cause: they asked for the unprompted channel to stop repeating itself, which was a real irritation but a different mechanism, and one whose own deduplication was already working as designed. Nobody reported the ranking damage, because it is not observable from either side — the dismissing user never sees the search results, and the searcher never learns which results were held back.

A second, smaller instance of the same shape was visible in the same data: a third consumer, a report that suggested retiring unproductive patterns, was reading that same counter and wanted the second producer's events. It had been quietly getting a mixture too, just one that happened to be closer to what it meant.

How to apply it

• Before wiring a new subsystem into an existing counter, list the counter's current consumers and ask, for each one, whether the new event means the same thing to it. If the answer differs between two consumers, the counter is already two counters and reusing it will merge them.
• Say what the counter measures in a sentence that names the producer. If the sentence needs an "or" once the new call site lands, split the field rather than widening the sentence. The split is cheap at that moment and expensive afterwards, because historical rows lose their provenance permanently the moment they are mixed.
• Treat one component instructing the user to write into another component's counter as the smell. It is easy to spot in review — a literal call to another module's recording verb, embedded in the first module's user-facing text — and it is the exact moment provenance is lost, because the string carries no indication of who emitted it.
• When you do split it, check that every consumer moves to the half it actually meant, not just the one that broke. A consumer that was accidentally getting the right mixture is still reading an unowned field, and it will drift the next time either producer changes.
• Add the invariant as a check, not as a habit: assert that a subset counter cannot exceed its parent. It is one comparison, it needs no new data, and it is the only thing here that fires without someone already suspecting the problem.
• For data already mixed, do not silently rewrite it. The excess over the invariant is a lower bound on the contamination, not a correction — rows within the bound are indistinguishable from clean ones. Decide explicitly between resetting the affected rows, carrying them with a marker, or keeping them and starting the split from today, and record which you chose and why. Quietly "fixing" the numbers to what they should have been invents a history nobody observed.

Why this carries no signature

The mechanism is cross-module by construction: the producer, the counter and the injured consumer live in three different files, and each one reads as correct in isolation. A grep over any single file cannot see it, and a pattern loose enough to flag every string that mentions a recording verb would fire constantly on legitimate instruction text — manufacturing exactly the kind of dismissal-generating noise this lesson is about. The invariant check in How to apply it is the real check; it belongs in whatever reports on the counter, not in a scanner.