pub(crate) fn warn_if_small_batch_coverage_missing(shapes: &[(usize, usize)])Expand description
Emits a startup WARN when the resolved warmup shape grid does not cover
the small-batch shapes that real router traffic routinely bin-packs to.
Concretely, real /v1/embeddings, /v1/sparse-embeddings, and
/v1/embeddings:both calls produce chunk batches of 1–2 (single-text or
two-text requests are the dominant traffic pattern for both ad-hoc queries
and bulk indexers). When the warmup grid omits both batch=1 and batch=2,
the first such request triggers in-band TRT JIT compilation, which on the
:both route has been observed to trigger pathological autotuner
allocation requests (multiple terabytes) and a fatal failed to create engine from network error.
Greppable tag: trt_warmup_shape_coverage_gap. Operators are not blocked
from deploying a batch-1-only grid (e.g. local dev workstations) — the
surface is informational so legitimate edge configurations still start.