# Migration for `batchprocessor` and exporterhelper batching ## Overview The Collector now offers two batching mechanisms: 1. The `batchprocessor` component, used in pipelines for many years and widely deployed, notably as a default in the OpenTelemetry Helm chart and other distributions. 2. The `exporterhelper` combined queue/batch sender, a replacement for the batch processor, is ready to replace it. We find there are a small number of users and known use-cases that require a batch processor, for example: - When multiple exporters are in use, the processor form uses less memory because the exporters share batched requests. - Aggregation processors (e.g., `groupbyattrs`) benefit from larger batches and recommend applying the batch processor first. At the same time, the existing batch processor has several known defects (e.g., concurrency limit, error propagation, trace context), and fixing the defects require breaking changes. Our goal is to migrate most users away from the batch processor and enable batching by default for all exporterhelper users. This will require a careful sequence of steps for a successful migration. ## Background This supersedes an unmerged draft in [#11947](https://github.com/open-telemetry/opentelemetry-collector/pull/11947) and answers [#15047](https://github.com/open-telemetry/opentelemetry-collector/issues/15047), [#13766](https://github.com/open-telemetry/opentelemetry-collector/issues/13766), [#13582](https://github.com/open-telemetry/opentelemetry-collector/issues/13582), [#13583](https://github.com/open-telemetry/opentelemetry-collector/pull/13583), [#14038](https://github.com/open-telemetry/opentelemetry-collector/issues/14038), [#12022](https://github.com/open-telemetry/opentelemetry-collector/issues/12022), [#8122](https://github.com/open-telemetry/opentelemetry-collector/issues/8122). ### `batchprocessor` defects The processor is widely used. It has well-documented defects: - Suppresses errors (always returns `nil`) - Returns to the caller before the export has completed - Single thread concurrency - Sizes batches only by item count - Interrupts trace context. The single thread concurrency problem is particularly a problem for error propagation. Performance can suffer when the downstream components block waiting for export. If the exporter is synchronous, as by exporterhelper's `wait_for_result: true`, the batch processor creates a single-export bottleneck. Even while suppressing errors, batchprocessor creates backpressure. This operational detail must be preserved as we deprecate this component. Below, we propose that the new `queuebatchprocessor` adopt `block_on_overflow: true` and `wait_for_result: true` as its defaults so that users replacing `batchprocessor` retain the same blocking behavior. The exporterhelper queue defaults are not changing on this dimension. ### Exporterhelper replacement The `exporterhelper` queue/batch sender resolves the known batchprocessor defects and integrates new features: - Choice of sizing logic by items, bytes, and requests - Optional synchronous (`wait_for_result`) or asynchronous behavior - Optional persistent storage extension - Optional blocking (`block_on_overflow`). The `exporterhelper` queue/batch sender has feature parity with `batchprocessor`. However, exporterhelper default settings remain aligned and compatible with `batchprocessor`: - `wait_for_result: false` asynchronous export - `block_on_overflow: false` drop when full - `batch::enabled: false` batching disabled. The exporterhelper supports opt-in persistent storage, and when no storage extension is configured it uses an in-memory queue with `wait_for_result: false`, meaning the default behavior is to accept data and immediately return success. This is what `batchprocessor` expects, and this is also how `batchprocessor` behaves, returning success, not waiting for potential errors. The exporterhelper's built-in batching support is critical for vendor-specific components, because it gives exporters a way to control request size in terms of bytes, even considering custom export protocols. The ideal final state is that we enable exporterhelper batching by default. Many exporters already do this, but not all, the choice is made per-component. ### What success looks like The `batchprocessor` is removed from the standard distribution and, eventually, from the core repository. A re-implemented `queuebatchprocessor` will be created (in the core) using exporterhelper as the implementation. This will use the combined QueueBatchConfig struct and support all batching modes. Unlike the exporterhelper defaults, the `queuebatchprocessor` defaults are chosen to preserve `batchprocessor`'s blocking behavior: it sets `wait_for_result: true` and `block_on_overflow: true` so that pipelines previously relying on the batch processor for backpressure continue to behave the same way after migration. OpenTelemetry documentation will not refer to `batchprocessor`. The OpenTelemetry Helm charts will not refer to `batchprocessor`. #### Recommendation: no error propagation by default in exporterhelper The `wait_for_result` feature will remain off by default in exporterhelper. The Collector's default position for exporters is to accept data in-memory and return success. Note that configuring `wait_for_result: true` together with a persistent storage extension is not supported, and should become checked as an invalid configuration (tracked under "Additional work items" below). #### Recommendation: do not block on overflow by default in exporterhelper The `block_on_overflow` feature will remain off by default in exporterhelper. Exporters that need to apply backpressure can opt in explicitly, and pipelines that want batch-processor-style blocking should use the new `queuebatchprocessor` (which sets it true by default) instead. #### Recommendation: queuebatchprocessor defaults preserve `batchprocessor` behavior The new `queuebatchprocessor` is the direct replacement for `batchprocessor` in complex pipelines (e.g., when multiple exporters share a single batching step, or when aggregation processors require larger batches upstream). To minimize migration surprises, its default configuration uses: - `wait_for_result: true` — errors propagate back to the receiver, matching the synchronous handoff users expect from a processor. - `block_on_overflow: true` — when the queue is full the processor blocks rather than returning an error, preserving the batch processor's existing backpressure semantics. - `batch::enabled: true` — batching is on by default; this is the whole point of the component. These defaults apply only to `queuebatchprocessor` and do not change the exporterhelper defaults used by individual exporters. #### Recommendation: exporterhelper batching enabled by default Currently, components have a mixture of `configoptional.Some`, `configoptional.Default`, and `configoptional.None`, usually passing `exporterhelper.DefaultQueueBatchConfig()`. However, many of these configurations are historical, set to `Default` or `None` because the exporterhelper was evolving. Not all exporters want queue/batching enabled by default. Concrete examples raised during review include: - **Pull-based exporters** (e.g., `prometheus`) have no concept of an outbound queue or batch at all. - **Order-sensitive exporters** (e.g., `prometheusremotewrite`) cannot use the standard queue because it does not guarantee in-order delivery; they typically run with `num_consumers: 1` and no batching. - **Exporters whose client library already implements retry/queueing** (e.g., `googlecloud`) disable retries and may not want a second layer of queue/batching on top. Even so, we believe many exporters will choose to enable batching by default now that the feature is stable. This RFC calls for an audit of the components, evaluating for every exporter whether a `Default` or `None` should become `Some` in each exporter's default queue config. A feature flag for this change will be defined, `pkg.exporterhelper.queueBatchEnabled`. #### Recommended final default queue configuration The `NewDefaultQueueConfig` function will begin being influenced by the `pkg.exporterhelper.queueBatchEnabled` feature flag. Only the `Batch` field's default changes; `BlockOnOverflow` and `WaitForResult` remain `false`. ```go func NewDefaultQueueConfig() queuebatch.Config { batchCfg := queuebatch.BatchConfig{ FlushTimeout: 200 * time.Millisecond, Sizer: request.SizerTypeItems, MinSize: 8192, } var batch configoptional.Optional[queuebatch.BatchConfig] if metadata.PkgExporterhelperQueueBatchEnabledFeatureGate.IsEnabled() { // CHANGED: Batching enabled by default under the feature gate. batch = configoptional.Some(batchCfg) } else { batch = configoptional.Default(batchCfg) } return queuebatch.Config{ Sizer: request.SizerTypeRequests, NumConsumers: 10, QueueSize: 1_000, BlockOnOverflow: false, WaitForResult: false, Batch: batch, } } ``` ### Double-batching problem One concern preventing the migration is that we may unintentionally apply multiple batching processes in a single pipeline. This is a coordination problem. If a pipeline with a default- or user-configured batch processor has settings that are out of line with the new exporterhelper defaults, users will pay the cost of batching and then re-batching. Users have 6 releases (~3 months) during which the batch processor will print a deprecation warning. During this window the new exporterhelper defaults are still off, so unmodified pipelines do not double-batch. Adopters who manually enable the Beta-on values to provide feedback, and custom builds that continue to ship `batchprocessor` after it is dropped from the default distribution in Phase 3, are protected by the startup warning introduced in Phase 1, step 5. ## Risks The phased timeline below is designed around the following risks: - **Custom exporters that do not use exporterhelper lose batching entirely** once `batchprocessor` is removed. The Phase 2 deprecation window gives these users time to adopt `queuebatchprocessor` or move to exporterhelper. - **`queuebatchprocessor` defaults preserve blocking semantics.** Users migrating from `batchprocessor` to `queuebatchprocessor` will see the same backpressure behavior (`block_on_overflow: true`, `wait_for_result: true`). Users who want non-blocking behavior must set these fields explicitly. The exporterhelper defaults are unchanged on this dimension. - **Batching-on-by-default adds latency.** Exporters adopting the new defaults introduce up to `FlushTimeout` of latency where there was none before. - **Silent double batching during Phase 2** when users keep `batchprocessor` alongside an exporter with batching enabled. Mitigated by the startup warning under "Additional work items". - **Distribution drift.** Non-OpenTelemetry distros and vendor docs referencing `batchprocessor` become invalid on removal. Mitigated by the Phase 1 advisory and the Phase 2 Helm chart switch. - **Insufficient real-world feedback.** Mitigated by the Phase 2 exit criteria requiring confirmed early adopters. ## Migration timeline We treat the batchprocessor deprecation as separate from the change of exporterhelper default with an interleaved timeline. That is: 1. Batch processor will become deprecated with printed instructions and documentation for users to explicitly set exporterhelper queue/batch configuration instead, or to use the new `queuebatchprocessor` (which preserves blocking semantics). 2. Exporterhelper will introduce a feature flag and proceed to enable batching (`batch::enabled: true`) by default. The `block_on_overflow` and `wait_for_result` defaults are not changing for exporterhelper. The sequence of events is ordered in phases. The following table summarizes what an operator running a typical pipeline observes at each phase: | Phase | `batchprocessor` in config | exporterhelper defaults | Net behavior | Release version | |-------|-------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------| | <1 | OK | `batch::enabled: false` | Documented behavior | | | 1 | OK | Unchanged; feature gate is Alpha (default off) | No change for users; new `queuebatchprocessor` and docs available for opt-in migration. | v0.158.0+ | | 2 | Works, prints deprecation warning pointing to migration docs | Unchanged; gate still Alpha/off | Users see warnings and have at least six releases to migrate; Helm chart switches to the new path while `batchprocessor` still works. Exit gated on early-adopter success. | v0.160.0 | | 3 | Removed from default distribution; still available to custom builds. Pipelines using the removed component in the default binary fail to start. | Gate Beta (default on): `batch::enabled: true` | Pipelines that didn't migrate fail fast; exporters batch by default; users can still opt out via gate for six releases. | v0.166.0 | | 4 | n/a | Gate Stable and removed; new default permanent | Users choose `exporterhelper` or `queuebatchprocessor` for batching | v0.172.0 | ### Phase 1 These are loosely dependent, 1. Ensure all exporters in OpenTelemetry repos define `queue_sender` having type `configoptional.Optional[exporterhelper.QueueBatchConfig]`. Add an mdatagen-generated check that fails when an exporter's default config omits `queue_sender` or sets it to a non-standard value. Exporters that legitimately need different defaults (e.g., pull-based exporters) must declare the opt-out in their `metadata.yaml`. [#15964](https://github.com/open-telemetry/opentelemetry-collector/pull/15964) 2. Introduce the `pkg.exporterhelper.queueBatchEnabled` feature gate at Alpha stability. [#15690](https://github.com/open-telemetry/opentelemetry-collector/pull/15690) 3. Support a configurable metrics prefix to distinguish processor batching from exporter batching ([#14038](https://github.com/open-telemetry/opentelemetry-collector/issues/14038)). [#15962](https://github.com/open-telemetry/opentelemetry-collector/pull/15962) 4. Implement `queuebatchprocessor`. [#15500](https://github.com/open-telemetry/opentelemetry-collector/pull/15500) 5. Detect pipelines that combine `batchprocessor` with an exporterhelper that has batching enabled, and emit a startup warning identifying the affected pipeline and exporter. This protects users from silently paying for double batching during Phase 2 and continues to protect custom-build users who keep `batchprocessor` through Phase 3 and beyond. The check catches most cases; notify affected Collector Contrib code owners that wrapped exporters can opt in by exposing exporterhelper's batching capability. [#15966](https://github.com/open-telemetry/opentelemetry-collector/pull/15966) 6. Update documentation explaining `batchprocessor` deprecation with instructions to adopt explicit exporterhelper `queue_sender` settings or the new `queuebatchprocessor`. Issue release notes and advisories to distros to remove `batchprocessor` from documentation. Create a blog post about what's happening to `batchprocessor`, get it approved. [open-telemetry/opentelemetry.io#12073](https://github.com/open-telemetry/opentelemetry.io/pull/12073) #### Exit criteria for Phase 1 All of the above must be completed. Additionally, in [#15966](https://github.com/open-telemetry/opentelemetry-collector/pull/15966): - Integration tests verify immediate pass-through of right-sized requests across queuebatchprocessor and exporterhelper stages, without flush delays or unnecessary merging, splitting, or copying. - Benchmarks measure multistage batching overhead for `min_size` alone and `min_size` with `max_size`, compared with queue-only operation. ### Phase 2 7. Deprecate `batchprocessor`. Release the blog post. The first release where this lands is tentatively **v0.160.0** (September 2026). 8. Full audit of exporters and default settings. Exporters can opt-in to the migration here by electing `Some` instead of `Default` or `None`. Exporters that do not want default blocking/batching will override and/or validate that their exporterhelper queue/batch configuration are valid; they must stop using the `NewDefaultQueueConfig()`. 9. Update various documentation pointing to `queuebatchprocessor`, which will advise users to configure exporter batching in most cases. ([#13766](https://github.com/open-telemetry/opentelemetry-collector/issues/13766)) After the batch processor is deprecated, we will wait at least +6 releases (~3 months) with `batchprocessor` printing warnings and referring to the documentation, which will guide them to fill explicit exporterhelper settings. During this window we actively solicit early-adopter feedback on the feature gate set to its Beta-on value. #### Exit criteria for Phase 2 Phase 2 does not end purely on a release-count schedule. Before the Helm chart switch (step 10) and the transition to Phase 3, we must have: - At least three production adopters of the feature gate (set to the Beta-on value manually) who have reported back, of which at least one operates the Collector at non-trivial scale. Adopters will be tracked in a running list maintained alongside this RFC (e.g., in the deprecation tracking issue). - No open regressions reported by those adopters against the new defaults. If these criteria are not met after +6 releases, Phase 2 is extended in 3-release increments until they are met. Once the exit criteria are satisfied: 10. Update the standard OpenTelemetry Helm chart to no longer reference `batchprocessor`. The chart instead configures exporterhelper `queue_sender` with batching enabled (or uses `queuebatchprocessor`). `batchprocessor` is still available as a fallback for users who have not yet migrated. Shipping the updated chart is the signal that we are ready to enter Phase 3. ### Phase 3 In a single release cycle: 11. Remove `batchprocessor` from the standard OpenTelemetry Collector distribution manifest. The component's Go module remains in the core repository so that custom builds (e.g., via `ocb`) can continue to include it as an explicit dependency. The startup warning from Phase 1, step 5 keeps flagging misconfigurations in such custom builds. 12. Promote the feature gate to **Beta** (default on). Exporterhelper enables batching by default; users can still opt out via the gate for one cycle. ### Phase 4 We will wait another +6 releases (~3 months). 13. Promote the feature gate to **Stable** and remove it. The new exporterhelper batching default becomes permanent and can no longer be toggled via the gate. 14. Remove the `batchprocessor` Go module from the core repository. Custom builds that still reference it must move to `queuebatchprocessor` or vendor the component themselves. ## Additional work items Items noted in this document that should be tracked separately, accomplished during Phase 1: - Reject `wait_for_result: true` combined with a configured storage extension at configuration validation time (see "Recommendation: no error propagation by default" above). [#12660](https://github.com/open-telemetry/opentelemetry-collector/pull/12660) ## Conclusion At the end, we can close a whole bunch of issues going back to [#8122](https://github.com/open-telemetry/opentelemetry-collector/issues/8122).