const DEFAULT_TRT_WARMUP_SHAPES: &[(usize, usize)];Expand description
Default TRT warmup shapes: a 2D {1, 2, 4, 8, 16, 32} × {128, 512, 2048, 8192}
grid composed in batch-major order.
Batch is the outer dimension so the smallest batches (which dominate
real router traffic — single-text and two-text requests are the most
common pattern for both ad-hoc queries and bulk indexers) are fully
compiled first; larger batches typical of bulk re-indexing fill in
afterwards. Within each batch the sequence dimension grows monotonically
so the cheap _ × 128 shape comes before the expensive _ × 8192 shape —
operators watching /health see progress quickly.
Previously-unseen shapes trigger in-band engine compilation in the middle
of a real request, producing tens-to-hundreds-of-seconds inference_ms
values. In the worst case, TRT JIT for the dual-output
/v1/embeddings:both graph at unseen small-batch shapes can request
pathological autotuner allocations (multiple terabytes on a fused
LayerNorm + MatMul foreign-node) that the CUDA allocator cannot
satisfy, producing a fatal failed to create engine from network error.
Including (1, _), (2, _), (4, _), and (8, _) rows closes the JIT
window for the common router pack-sizes.
This 24-shape grid covers the full realistic shape space so every router request hits a pre-compiled engine.