Skip to main content

embed_with_trt_retry

bge_m3_embedding_server::embedder::worker::trt_retry

Function embed_with_trt_retry 

Source 
pub(super) fn embed_with_trt_retry<T, F>(
    embed_fn: F,
    base_cm: &CostModel,
    worker_id: usize,
    route: &'static str,
) -> Result<T>
where
    F: FnMut(&CostModel) -> Result<T>,
Expand description

Wraps an embed call with the standard TRT JIT-OOM retry-once-with-halved-budget pattern.

If embed_fn fails and is_trt_jit_oom matches the error, retries once with max_workspace_bytes / 2. Logs trt_jit_retry on the first attempt and trt_jit_retry_exhausted when the retry also fails. Returns the final result.