Skip to main content

is_trt_jit_oom

bge_m3_embedding_server::embedder::worker::trt_retry

Function is_trt_jit_oom 

Source 
pub(super) fn is_trt_jit_oom(e: &Error) -> bool
Expand description

Returns true when an ORT error string indicates a TRT JIT workspace overflow — a condition that may resolve with a smaller batch or halved workspace budget.

Patterns verified against ORT 2.0.0-rc.12 TensorRT EP (ort/src/ep/tensorrt.rs). Re-verify on every ORT version bump.

§Patterns matched

  1. user allocator error — direct CUDA allocation failure surfaced by ORT’s user-allocator shim during TRT kernel autotuning.
  2. could not find any implementation + (workspace | alloc) — TRT kernel-autotuner declared no tactic fits, and the qualifier confirms the cause is allocation-driven (otherwise this string also matches genuine unsupported-op cases where retry is pointless).
  3. failed to create engine + (workspace | alloc | memory | oom | tactic) — TRT EP build-time failure observed in production on large fused-route requests (e.g. /v1/embeddings:both with high batch and token counts). The qualifier is mandatory: without it, this same family also covers unsupported-op and corrupted-cache cases where retrying with a halved workspace is pointless and doubles caller-visible latency. alloc subsumes cuMemAlloc; memory subsumes out of memory.

§Known gap

The verbatim production error string often does NOT include any qualifier — the TRT logger appears to emit workspace/alloc detail to a separate tracing target rather than propagating it into the outer Status Message. We chose Option A here (require a qualifier) over Option B (retry every failed to create engine unconditionally) so we don’t regress does_not_match_unsupported-style cases. If a follow-up CloudWatch investigation confirms the TRT root-cause is reliably surfaced only in a sibling target=ort event and never in the embed error string, we may need to relax this to Option B (or pipe the TRT logger output into the embed error chain). Until then, this function will continue to return false for the verbatim production message and callers will see HTTP 500 on first build failure.

Tracking: https://github.com/Fulton-Engineering-Services/bge-m3-embedding-server/issues/78