# Queue Batch Processor | Status | | | ------------- |-----------| | Stability | [alpha]: profiles | | | [beta]: traces, metrics, logs | | Distributions | [] | | Issues | [](https://github.com/open-telemetry/opentelemetry-collector/issues?q=is%3Aopen+is%3Aissue+label%3Aprocessor%2Fqueuebatch) [](https://github.com/open-telemetry/opentelemetry-collector/issues?q=is%3Aclosed+is%3Aissue+label%3Aprocessor%2Fqueuebatch) | | [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@jmacd](https://www.github.com/jmacd), [@iblancasa](https://www.github.com/iblancasa) | [alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha [beta]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#beta The queue/batch processor accepts spans, metrics, logs, or profiles and places them into batches, using the [exporterhelper sending queue and batch](../../exporter/exporterhelper/README.md#sending-queue) logic and configuration. This component replaces the [batchprocessor](../batchprocessor); see [docs/rfcs/batching-migration.md](../../docs/rfcs/batching-migration.md). This component improves on the former batch processor while maintaining its general behavior. This component uses default settings taken from the `batchprocessor` mapped into `exporterhelper` settings: ```yaml # The processor will produce and export one batch at a time. num_consumers: 1 # The queue is measured in request units. sizer: requests # The queue admits 10 requests. queue_size: 10 # Backpressure: The queue blocks the caller when full until # canceled. block_on_overflow: true # Error propagation: The caller receives success immediately # after the request enters the queue. Enable to wait for success # or failure. wait_for_result: false # These defaults match both batchprocessor and exporterhelper. batch: enabled: true sizer: items min_size: 8192 flush_timeout: 200ms ``` Refer to the [exporterhelper documentation](../../exporter/exporterhelper/README.md) for more details on this configuration structure. Other than changing the default configuration, the entire `queuebatch` functionality is available through this processor, including partitioning and storage. ## Examples The following examples show two capabilities that Collector users frequently rely on: batching by client metadata and buffering the queue on disk. ### Partitioning with `metadata_keys` The `metadata_keys` setting enables forming distinct batch for each combination of `client.Metadata` values, so that a multi-tenant pipeline could batch each tenant's data separately. The queue/batch processor offers the same behavior through the `batch::partition::metadata_keys` setting. ```yaml receivers: otlp: protocols: grpc: # Capture client metadata so it is available for partitioning. include_metadata: true processors: queue_batch: batch: partition: cache_size: 10000 # Produce one batch per distinct tenant_id value. metadata_keys: - tenant_id service: pipelines: traces: receivers: [otlp] processors: [queue_batch] exporters: [otlp] ``` Receivers must be configured with `include_metadata: true` so the metadata keys are available to the processor. An empty value and unset metadata are treated as distinct cases, and entries are case-insensitive. `batch::partition::cache_size` (default = 10000) limits how many distinct partition batchers are kept in memory at once. The value must be positive. When the limit is reached, the least recently used partition is flushed and removed. The current cache size and configured capacity are exported as `otelcol_processor_queuebatch_partition_cache_size` and `otelcol_processor_queuebatch_partition_cache_capacity`. `batch::partition::idle_timeout` (default = 90s) controls how long a partition may stay empty before it is removed. Keep it above the data arrival interval so partitions are not churned on every scrape. The value must be positive. ### Persisting the queue with a storage extension By default the queue is held in memory, so any buffered data is lost if the Collector stops. Set the top-level `storage` field to the ID of a [storage extension][filestorage] to persist the queue on disk, so that buffered data is recovered and exporting continues after a restart. ```yaml extensions: file_storage: directory: /var/lib/otelcol/queue_batch processors: queue_batch: # Persist the queue using the file_storage extension. storage: file_storage service: extensions: [file_storage] pipelines: logs: receivers: [otlp] processors: [queue_batch] exporters: [otlp] ``` When `storage` is set there is no in-memory queue. Each request is serialized as it enters the queue and read back from disk before export. The `wait_for_result` setting is not supported together with `storage`. [filestorage]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/storage/filestorage