OpenTelemetry can give engineering teams a common way to generate, collect, process and export metrics, logs and traces. But gathering that telemetry consumes resources, and the pipeline can itself become a source of CPU load, memory pressure, network traffic and storage expense if its design does not match the workload.
In a November 17, 2026, discussion of her Cloud Native Days Austria 2026 presentation, “How to Defeat OTel’s Silent Killer: The Performance Overhead,” Diana Todea outlined ways to investigate and manage those costs. Todea is a CNCF Ambassador, an OpenTelemetry documentation approver and a Romanian localization maintainer. Her central point: there is no single overhead percentage that applies to every deployment. The impact depends on request volume, runtime behavior, span counts, attributes and the work performed throughout the pipeline.
That work begins in application instrumentation but continues through collection, processing, export and storage. A useful performance review should therefore track CPU and memory use, application latency, telemetry volume and Collector behavior under load. Resource reduction also has to be balanced against observability needs: overload controls can protect applications and Collectors, but may reject or drop telemetry.
Five configuration areas to examine
Todea used Telemetrygen, a utility for generating synthetic OpenTelemetry traces, metrics and logs at controlled rates, to isolate Collector behavior from the additional resource demands of a full microservices environment. The investigation highlighted five configuration areas where unnecessary work can arise.
-
Batching and export behavior: Sending small payloads frequently can increase serialization, request and network overhead. Batching in an exporter’s sending queue can combine telemetry before export, but larger batches can require more buffering and affect latency. Todea’s example uses an item-based queue and batching thresholds; its values are documented defaults, not universal tuning recommendations. Teams should test settings against their own traffic and queue behavior.
-
Resource attributes: System metadata adds context, but attributes that are not used can enlarge payloads and add downstream processing. A team may need host names and operating-system types for troubleshooting without needing detailed CPU model or cache information. The Collector’s resource detection processor can be configured to control which system attributes are collected. Effects on storage and cardinality depend on the backend.
-
Receiver polling intervals: Host, container, database and network receivers may continue collecting on a schedule even when application traffic is low. Where operational needs permit, increasing an interval—for example, from 10 seconds to 30 seconds—can reduce collection frequency. The trade-off is less temporal detail, so intervals should reflect how metrics are used for alerting and diagnosis.
-
Trace sampling: Exporting every trace can create substantial pipeline and backend work. Head-based sampling makes an early decision and can limit downstream processing; tail-based sampling can make decisions using more of a trace, such as retaining errors, but requires buffering and additional memory and processing. Sampling policies should balance diagnostic value, resource costs and the chance of retaining relevant requests.
-
Memory-limiter placement: The Collector’s memory_limiter processor helps manage pressure, but it is a safeguard, not a general performance optimization or replacement for capacity planning. Placing it early can prevent later, more expensive processors from working on data that may be refused. Teams also need to understand how receivers handle refusal to avoid unnecessary telemetry loss.
Design for production conditions
Local tests are useful for isolating components, but synthetic workloads cannot represent every production pattern. In Kubernetes, traffic and telemetry can shift as applications scale and deployments change. Todea recommends treating Collector architecture as part of performance planning, rather than relying only on individual processor settings.
A two-tier design can separate lightweight edge collection from more intensive processing at gateway Collectors, allowing the tiers to scale for different workloads. Filtering unnecessary attributes and data early can reduce processing and transfer volume, provided teams preserve the context needed for debugging and correlation. Bounded queues, Kubernetes memory limits, Collector safeguards and retry mechanisms address different failure modes; persistent queues can help during temporary backend outages but still need capacity planning and cannot guarantee delivery through every failure.
Scaling also requires attention to signal-specific behavior. Tail sampling generally depends on spans from the same trace reaching the same processing instance, which can complicate horizontal scaling. A design that increases throughput must account for such stateful processing requirements.
Optimization can start before data reaches the Collector. Excessive spans, unnecessary instrumentation and high-cardinality attributes increase work across the pipeline. For metrics, unique request IDs can produce unbounded numbers of time series; for traces and logs, excess attributes and events enlarge payloads and can increase backend processing or indexing costs.
Telemetrygen helped Todea focus on Collector configuration without the background activity of a complete application environment. A full environment such as OpenTelemetry Astronomy Shop remains useful for studying instrumentation in context. The broader operational lesson is to benchmark under controlled conditions, change settings individually, and evaluate CPU, memory, latency, throughput and telemetry loss. Configuration examples are starting points, not production-ready prescriptions; observability pipelines should be measured and tuned like other production systems.