Metric Cardinality Is an Architecture Budget

A dashboard can look perfectly healthy while the number on it is wrong. Not wrong because data was lost, and not wrong because someone mistyped a query. Wrong because a safety mechanism inside the metrics SDK did exactly what it was designed to do, and nobody told the people reading the chart.

That is the part of the OpenTelemetry project’s practical guide to metric cardinality limits, published on August 6, 2026, that deserves more attention than the limit itself. The guide is blunt about the consequence: when a metric stream overflows, the total value remains correct while queries that filter or group by attributes can undercount. A service level objective built on a filtered breakdown can drift away from reality without a single alert firing about it.

The reflex response is to treat this as a configuration detail owned by whoever runs the collector. It is not. Cardinality is a budget your application design spends, your runtime consumes, and your operations teams pay for. A platform team can cap the damage. It cannot decide whether tenant_id, order_id, or an exception message belongs on a metric, because that decision depends on the question the measurement is supposed to answer.

What the limit actually does

The mechanics are worth stating precisely, because the intuitive version is wrong in a way that matters.

The SDK aggregates measurements by the complete set of attribute values, and every unique combination becomes a data point held in process memory. The metrics SDK specification sets a default of 2000 combinations per metric stream when neither a View nor the MetricReader supplies its own value.

When that ceiling is reached, measurements are not discarded. They are folded into a synthetic data point carrying a single attribute, otel.metric.overflow=true. The specification is explicit that every measurement must be reflected in exactly one aggregator, and that measurements must not be double counted or dropped during an overflow.

The lost information is the breakdown. In an illustrative counter example, an overflowing failed request contributes to the overflow total without retaining its route or outcome attributes. The total request count stays right. A failure rate calculated from the remaining per-route counts can be biased; its direction depends on which observations overflowed.

This is the detail that makes the problem hard to spot in review. Overflow replaces the whole attribute combination, not just the dimension that caused the growth. A bounded, well-designed boolean like success becomes unreliable for filtering the moment it travels alongside a high-cardinality attribute on the same measurement. The blast radius is the metric stream, not the offending label. Resource attributes and instrumentation scope attributes survive overflow because they sit outside the measurement attribute set, which is a useful modeling fact and a poor substitute for getting the attributes right.

Temporality changes the shape of the risk for synchronous instruments. Cumulative aggregation retains state across collection cycles and continues exporting the attribute sets admitted before overflow. Delta aggregation budgets the combinations within a collection cycle. The same tenant vocabulary can therefore need different capacity under the two configurations. Check the SDK’s behavior for the instrument and version you actually deploy, rather than applying a synchronous-counter assumption to every instrument.

One phrase, two systems, two owners

“Cardinality limit” names a process memory protection. It is not a backend limit, and conflating the two produces confident wrong conclusions in capacity planning.

Each process has its own aggregation state, while the backend receives data from the whole fleet. Different instances, resource identities, and collection cycles can contribute different series. A local limit can reduce exported detail, but it does not establish a fleetwide ceiling on ingestion, storage, or query cost. Capacity planning still needs to account for how many producers exist and how their series change over time.

Those are two budgets with two owners. Application teams own which attributes exist and what the process holds in memory. Platform and observability teams own storage, retention, and query cost across the fleet. Each budget needs its own estimate and its own alert. A team that has only the first one will be surprised by the bill. A team that has only the second one will be surprised by an out of memory kill.

The protection is not evenly distributed

There is a quieter assumption underneath most cardinality conversations: that the SDK is protecting you. Sometimes it is not.

The specification says SDKs should support being configured with a cardinality limit. The spec compliance matrix is a starting point for checking support, not a substitute for the release documentation of your installed SDK. Global configuration, per-metric configuration through Views, and overflow reporting are separate capabilities to verify.

In a polyglot estate the same instrumentation mistake may therefore encounter different protections. The default of 2000 is not a fleetwide guarantee, and an overflow query that returns nothing may simply mean the SDK in question has nothing to report. Where a limit is absent, monitor process memory alongside series counts and instrumentation growth. A controlled load test should establish which warning arrives before the service runs out of headroom.

A budget needs an owner and a trigger

The useful response is not a universal number. It is a small set of observable signals mapped to decisions someone is accountable for.

Signal you can observe	The decision it should trigger
Overflow appears on a metric backing paging, SLOs, or autoscaling	Treat filtered breakdowns as untrustworthy until fixed. This is an incident in the measurement, not a tuning ticket.
Attribute values are accidental or unbounded, such as raw URLs or exception text	Remove or normalize them with a View or fix the instrumentation upstream. Do not raise the limit to accommodate accidental growth.
Attributes are bounded, but the expected combinations exceed the limit	Size the limit from the estimate with headroom, and record the estimate where the next reviewer will find it.
The dimension is high cardinality and operationally necessary	Evaluate delta temporality where supported, size for the active set per cycle, and monitor overflow. Check backend costs separately.
Nobody can name the decision the metric supports	Do not ship the dimension. Collection without a decision is cost with no return.
No one owns schema changes for the metric	Treat it as an unapproved interface. Dashboards, alerts, and capacity models are its consumers.

The value of this shape is that it is local. A small vocabulary can still be wrong if nobody uses the metric. A larger one can be justified when the question matters, the values are controlled, and the failure behavior is understood. The point is not to make every metric tiny. The point is to make every expansion deliberate and attributable.

What a healthy schema looks like, independent of the limit

It helps to separate the backstop from the design target, and the Prometheus project has been unambiguous about the design target for years. Its instrumentation practices suggest keeping the cardinality of a metric below 10 as a rule of thumb, and investigating alternatives such as reducing dimensions or moving the analysis out of monitoring entirely once a metric passes 100 or looks likely to. Its naming conventions put it plainly: do not use labels for high-cardinality dimensions such as user IDs, email addresses, or other unbounded sets of values.

These figures serve different purposes. Prometheus offers conservative instrumentation guidance; the SDK default provides a runtime backstop. Neither establishes a universal safe series count for your system. A stream below the SDK limit still deserves a cost review, but its count alone does not prove it will time out. Explain the decision it supports, benchmark the relevant queries, and justify the dimensions against the available capacity.

Relocating the detail is not free

The standard advice is to move high-entropy context into logs and traces. That advice is right about signal shape and incomplete about consequences.

Aggregate metrics can answer recurring questions without retaining one record per request. Logs and traces can retain individual events, so their volume depends on traffic, sampling, and record size as well as indexing and retention choices. Backend pricing varies. Before relocating a field, estimate the destination’s ingestion and storage costs using your own configuration. Moving a dimension changes the cost model; it does not make the information free.

Review access and retention at the destination too. Identifiers can expose sensitive information in any telemetry signal, including metric labels. Moving them into a record with additional context can make that exposure broader. Keep only the detail needed for investigation, apply the relevant handling policy, and avoid collecting request bodies or personal identifiers by default. Sampling also means a particular trace may not survive, so an investigative workflow should not promise a complete record of every request.

The placement rule therefore has two halves. Does the shape of the signal match the shape of the question, and does the receiving system have the retention, access, and deletion behavior this data requires? When the answer to the second half is uncomfortable, the honest option is often to collect nothing. Deleting telemetry is a real observability improvement when no decision was ever attached to it.

The four failures that keep producing this

Limit as policy. The team can quote the configured limit but cannot say which metrics are expected to approach it, what degrades near the boundary, or who investigates a sudden increase. Raising the limit becomes the default response to a warning, which converts a circuit breaker into a ratchet.

Label laundering. A high-entropy value is renamed until it reads like a normal dimension. request_id becomes request, exception_message becomes error, repository_branch becomes branch. The author knows the difference. The next reviewer sees a tidy name.

Combinatorial blindness. Labels get reviewed one at a time because that is how the diff presents them. Four independently bounded dimensions still multiply, and a review that examined four inputs and never estimated the product has not reviewed the metric.

Dashboard-first observability. Dimensions are added until the chart can be sliced every imaginable way, and the series cost is discovered afterwards. In production the metric is what consumes memory and backend capacity. The dashboard is only one consumer of it.

A rollout that does not require a rewrite

Start by finding out whether this is already happening to you. Prometheus-compatible exporters typically expose the overflow marker as a label, so the guide’s detection query is short:

count by (__name__, job) (
  last_over_time({otel_metric_overflow="true"}[1d])
)

Adapt the label names to your exporter and backend, and narrow the selector to the services you are investigating before running it across a large fleet. Silence is a baseline, not a proof. The SDK may not implement limits, the overflow may predate your retention window, or your backend may represent the attribute differently.

Then make the signal continuous rather than archaeological. Alert when any metric emits an overflow data point, and split severity by consequence: page when it lands on metrics used for paging, SLOs, autoscaling, or critical dashboards, and raise a ticket when it lands on exploratory ones. Put an overflow indicator next to any chart that groups by a measurement attribute, so the reader can tell whether the breakdown is complete.

Classify each dimension by where its values come from, not by what it is called. A label named type can hold arbitrary input. A label named path is safe when it is drawn from a documented route vocabulary and dangerous when it is a raw URL. Then assign an owner and a change path, because a metric schema is an interface whose consumers break silently. Finally, exercise the failure path in a controlled environment and write the response into the runbook. A limit that has never been reached on purpose is an assumption.

The constraints are real

Legacy instrumentation is spread across services owned by teams with different priorities, and some dashboards depend on dimensions nobody remembers adding. Removing a label can reduce cost and still create a political problem when a team loses a slice it used once during an incident. There is also a lag between a schema change and operational learning: a vocabulary bounded in code can become open-ended after a product decision, months later, with no observability change to blame.

Small teams without a dedicated observability engineer should not skip the budget, they should shrink the default. Require a named decision, reject free-text dimensions, and route rich context elsewhere unless someone argues the metric shape. Large teams have the opposite problem: standards abstract enough to be unarguable and useless in a pull request. A standard earns its keep when it lists accepted vocabularies, rejected dimensions, expected failure behavior, and the escalation path for the unusual case.

The operating rule

Metric cardinality is an architecture budget because application design creates it, runtime systems consume it, and the teams operating both pay for it. The SDK limit is a boundary around that practice, not a substitute for it.

Decide which questions deserve aggregate metrics. Give every dimension a vocabulary and an owner. Estimate combinations rather than labels. Check whether your language’s SDK enforces anything at all. Alert on the overflow marker before an SLO quietly stops describing reality. And when you move detail to logs or traces, price the move and check the privacy consequences before you call it a cleanup.

If your telemetry is growing faster than your confidence in it, another dashboard or a higher limit will not help. What you need is a schema decision someone can defend.

Further reading: the OpenTelemetry concept documentation on cardinality limits, which walks through the overflow properties and how temporality affects the limit.

If your metrics have outgrown their original design, we can help you assess high-cardinality metric families, plan changes to instrumentation, and establish ownership and overflow monitoring. Our observability consultancy turns that assessment into a practical improvement plan for your engineering team. Talk with us about your observability needs.