Module adaptive_warmup 

Adaptive in-process background warmup loop.

Detects TRT engine cache misses during live inference (shapes whose total inference_ms >= CACHE_HIT_THRESHOLD_MS) and compiles their engines during idle windows (queue empty for quiet_secs consecutive seconds) so subsequent requests hit the cache.

Integration

1. Before spawning the worker pool, create a bounded channel with [mpsc::channel::<(usize, usize)>] (capacity 64 is sufficient).
2. Store the sender half in crate::embedder::WorkerConfig::jit_suspect_tx. Workers call try_send((batch, seq)) after any inference whose inference_ms exceeds the cache-hit threshold.
3. After spawning the pool, call spawn_adaptive_warmup with the receiver half and a clone of the pool.

The background task accumulates suspected miss shapes, waits for an idle window, and compiles one shape at a time via EmbedPool::send_adaptive_warmup. Successfully compiled shapes are not re-submitted in the current session.

Accepted tradeoffs

Non-atomic idle detection (L-4): The queue_depth == 0 check and the subsequent send_adaptive_warmup call are not atomic. A traffic burst arriving between these two operations will queue behind the adaptive warmup compile. This is intentional — the compile runs inside a normal worker slot and the burst simply waits, same as any other request. Introducing atomic coordination across the pool boundary would add complexity without meaningful latency improvement.

Homogeneous-SM assumption (L-5): The adaptive warmup dispatches to whichever worker accepts the message first. For this to benefit all workers the instance must have a homogeneous GPU SM version (e.g. all L40S sm_89) so the compiled engine plan on EFS is valid for every worker on restart. Mixed-SM deployments (e.g. mixing g6e and g5 instances in the same ASG) will compile plans only for the SM of whichever worker runs first. See CLAUDE.md “TRT plans are compute-capability-specific” for the full constraint.

Structs

AdaptiveWarmupConfig 🔒

Configuration for the adaptive background warmup task.

Functions

drain_rx 🔒

Drains any available items from rx into pending, skipping already-warmed and already-pending shapes. Non-blocking: returns as soon as try_recv fails.

run_adaptive_warmup_loop 🔒

spawn_adaptive_warmup 🔒

Spawns the adaptive background warmup task.

wait_for_quiet_window 🔒

Waits until the pool queue has been idle for quiet_secs consecutive seconds.