Module sm_detect 

Per-device GPU compute-capability detection for SM-aware TRT cache filtering.

ORT names every TRT engine plan with a _smXX suffix tied to the GPU’s compute capability (sm75 = T4, sm86 = A10G, sm89 = L40S/L4, sm120 = Blackwell). Plans built for one SM cannot be loaded by another — the TRT runtime silently refuses them and JIT-compiles instead. Filtering the engine cache by the worker’s own SM is the only way to produce a truthful cache_hit signal on heterogeneous-SM fleets (or on fresh hosts where a previous-SM cache survives on EFS).

Detection mechanism: shell out to nvidia-smi --query-gpu=compute_cap --format=csv,noheader -i <device_id>. The subprocess is cheap (single-digit ms) and avoids pulling in a CUDA driver crate that the project does not otherwise need. The parser is a pure function so the bulk of the surface area can be unit-tested without a GPU.

Failure modes (missing nvidia-smi, non-zero exit, parse error) return None and the caller falls back to the legacy unfiltered behaviour so operators rolling forward mid-deploy never see a hard regression.

Functions

detect_sm_for_device 🔒

Returns the GPU compute capability as a smXY string for the given CUDA device, or None if detection fails for any reason.

parse_compute_capability 🔒

Parses nvidia-smi --query-gpu=compute_cap stdout ("X.Y\n" or "X.Y\nX.Y\n…" for multi-device queries without -i) into an ORT-compatible smXY string.