Skip to main content

log_inference_complete

Function log_inference_complete 

Source
pub(super) fn log_inference_complete(
    stats: &EmbedStats,
    worker_id: usize,
    _route: &'static str,
    jit_suspect_tx: Option<&Sender<(usize, usize)>>,
    engine_propagation_tx: Option<&Sender<(usize, usize)>>,
    batch_len: usize,
) -> Option<(usize, usize)>
Expand description

Emits the chunk_run INFO event and, on a cache miss, notifies both the JIT-suspect channel (adaptive warmup scheduling) and the engine propagation broadcast channel (peer worker fast disk-load).

Returns Some((batch_len, max_chunk_seq)) when a shape was broadcast on the engine propagation channel. The call site MUST insert this shape into warmed_local so the originating worker self-skips its own broadcast on the next drain_engine_propagation iteration (COR-1).

§5000 ms threshold heuristic (COR-10)

CHUNK_CACHE_HIT_THRESHOLD_MS (5 s) is a heuristic proxy for “TRT engine JIT compile occurred”, not a semantic guarantee. False negatives are possible for fast-JIT small shapes; false positives are impossible because a cache-hit path never exceeds ~100 ms. The trade-off is acceptable: the worst outcome of a false negative is that the adaptive warmup task eventually resubmits the shape on the next real cache miss.