Struct Config 

pub struct Config {

    pub cache_dir: String,
    pub bind_addr: String,
    pub tls_cert_path: Option<PathBuf>,
    pub tls_key_path: Option<PathBuf>,
    pub workers: usize,
    pub intra_threads: usize,
    pub max_batch: usize,
    pub max_seq_length: usize,
    pub idle_timeout: Option<Duration>,
    pub model_variant: ModelVariant,
    pub memory_safety_factor: f64,
    pub cost_model_override: Option<CostModel>,
    pub heartbeat_secs: u64,
    pub ep: EpSelection,
    pub gpu_count: usize,
    pub gpu_vram_budget_bytes: Option<usize>,
    pub trt_max_workspace_bytes: Option<usize>,
    pub gpu_mem_limit_bytes: Option<usize>,
    pub adaptive_warmup_enabled: bool,
    pub engine_propagation_enabled: bool,
    pub adaptive_warmup_quiet_secs: u64,
    pub adaptive_warmup_max_shapes_per_hour: u32,
    pub trt_warmup_shapes: Vec<(usize, usize)>,
    pub warmup_only: bool,
    pub prewarm_strict: bool,
    pub max_body_bytes: usize,
    pub circuit_breaker_threshold: usize,
    pub trt_inband_jit_guard_enabled: bool,
    pub trt_inband_jit_guard_seq: usize,
}

Runtime configuration loaded from environment variables.

All fields are read once at startup via Config::from_env. Changes to environment variables after startup have no effect.

Fields

cache_dir: String

Path to the directory where ONNX model files are cached.

Set with BGE_M3_CACHE_DIR. Defaults to /cache.

bind_addr: String

TCP bind address for the HTTP server.

Set with BGE_M3_BIND. Defaults to 0.0.0.0:8081. The 0.0.0.0 default is intentional for Docker container deployments.

tls_cert_path: Option<PathBuf>

Path to the TLS certificate PEM file.

Set with BGE_M3_TLS_CERT_PATH. When set together with BGE_M3_TLS_KEY_PATH and the tls Cargo feature is compiled in, the server binds HTTPS instead of HTTP.

tls_key_path: Option<PathBuf>

Path to the TLS private key PEM file.

Set with BGE_M3_TLS_KEY_PATH.

workers: usize

Number of embedding worker threads to spawn.

Set with BGE_M3_WORKERS. Defaults to 2. Minimum effective value is 1. Each worker loads its own model instance.

intra_threads: usize

Number of intra-op threads each ORT session may use for a single session.run() call (matmul / attention kernels).

Set with BGE_M3_INTRA_THREADS. Defaults to 1. Minimum effective value is 1.

The default of 1 preserves predictable per-worker RSS (the workspace probe and quadratic cost model are calibrated against single-threaded MLAS runs). Raise this on under-utilized hosts where BGE_M3_WORKERS * intra_threads <= num_cpus: e.g. on an 8 vCPU task with workers=2, setting intra_threads=4 lets each worker fan out to four cores during inference, taking CPU utilization from ~25% to ~100% under load. Going above floor(num_cpus / workers) causes thread oversubscription and hurts throughput.

Re-run the startup probe (do not pin coefficients) after changing this value so the cost model captures any new scratch-buffer overhead.

max_batch: usize

Maximum number of input texts accepted in a single request.

Set with BGE_M3_MAX_BATCH. Defaults to 256. Minimum effective value is 1.

max_seq_length: usize

Maximum sequence length (tokens) for a single text.

Set with BGE_M3_MAX_SEQ_LENGTH. Defaults to 8192 (BGE-M3’s published max). Range: [1, 8192]. Set lower to reduce memory footprint on constrained hardware.

The tokenizer will silently truncate any input exceeding this length. The probe and bin-packer use this as the upper bound when computing workspace costs.

idle_timeout: Option<Duration>

Duration of inactivity after which workers unload their model instances from memory.

Set with BGE_M3_IDLE_TIMEOUT_SECS. Defaults to 300 (5 minutes). Set to 0 to disable idle unloading entirely.

When unloaded, models are automatically reloaded on the next incoming request. The reload blocks the request until complete (~5–10 s from CoreML compiled cache; ~15–30 s cold).

model_variant: ModelVariant

ONNX model variant to load.

Set with BGE_M3_MODEL. Accepts "fp32", "fp16", or "int8". Defaults to "fp16" for fleet-wide embedding consistency and reduced RAM on Linux/Intel deployments. Set BGE_M3_MODEL=fp32 on Apple Silicon to recover CoreML GPU acceleration. See ModelVariant for per-variant performance and memory trade-offs.

memory_safety_factor: f64

Fraction of estimated available workspace to actually use per worker.

Set with BGE_M3_MEMORY_SAFETY_FACTOR. Defaults to 0.7 (30% headroom for ORT arena fragmentation and spike overhead not captured by the probe). Range: 0.1..=1.0.

cost_model_override: Option<CostModel>

If Some, skip the startup probe and use this cost model directly.

Populated when:

• BGE_M3_DISABLE_AUTO_BUDGET=1 is set (uses conservative defaults), or
• BGE_M3_TOKEN_BUDGET is set (translates the legacy token count to a max_workspace_bytes using conservative a/b coefficients), or
• BGE_M3_COST_MODEL_A and BGE_M3_COST_MODEL_B are both set with BGE_M3_AVAILABLE_MEMORY_BYTES (full explicit override).

heartbeat_secs: u64

Interval (seconds) between periodic heartbeat log events.

Set with BGE_M3_HEARTBEAT_SECS. Defaults to 60. Set to 0 to disable heartbeat logging entirely.

Heartbeat events log RSS, live/loaded worker counts, queue depth, available request permits, and current probe status — useful for detecting slow memory leaks or queue saturation between requests.

ep: EpSelection

ONNX Runtime execution provider to use.

Set with BGE_M3_EP. Accepts "cpu", "cuda", or "tensorrt". Defaults to "cpu". On macOS, CoreML is always used regardless of this setting. Requires the corresponding Cargo feature (cuda or tensorrt) to be enabled at build time for GPU EPs.

When set to "cuda" or "tensorrt", the host-RAM probe is bypassed in favour of the VRAM budget, and BGE_M3_WORKERS is clamped to BGE_M3_GPU_COUNT in EmbedPool::spawn.

gpu_count: usize

Number of GPU devices available on this instance.

Set with BGE_M3_GPU_COUNT. When a GPU execution provider (cuda or tensorrt) is active, BGE_M3_WORKERS is clamped to this value in EmbedPool::spawn so each worker is pinned to a distinct CUDA device (device_id = worker_index % gpu_count).

Auto-detected on Linux from /proc/driver/nvidia/gpus/ entry count. Defaults to 1 on macOS (CoreML is always single-device) and on Linux when the driver proc path is absent. Override explicitly on multi-GPU ECS instances: BGE_M3_GPU_COUNT=8.

gpu_vram_budget_bytes: Option<usize>

VRAM workspace ceiling (bytes) when a GPU execution provider is active.

Set with BGE_M3_GPU_VRAM_BUDGET_BYTES. Ignored when ep == Cpu. Defaults to None, which causes the server to use 10 GiB as the ceiling (suitable for GPUs with ≥ 16 GiB VRAM such as A10G / L4). Lower this on GPUs with less VRAM (e.g. 8589934592 for 8 GiB).

trt_max_workspace_bytes: Option<usize>

TRT EP workspace size cap in bytes.

Set with BGE_M3_TRT_MAX_WORKSPACE_BYTES. When None, TRT uses its default “as large as possible” workspace — which can OOM on saturated VRAM. Set to a value that leaves room for resident model weights and cached engine plans (e.g. 4 GiB = 4294967296 on an L40S with 4 workers).

gpu_mem_limit_bytes: Option<usize>

CUDA EP device-level memory limit in bytes.

Set with BGE_M3_GPU_MEM_LIMIT_BYTES. When None, CUDA EP uses all available device memory. Symmetric to trt_max_workspace_bytes.

adaptive_warmup_enabled: bool

Enable the in-process adaptive background warmup loop.

Set with BGE_M3_ADAPTIVE_WARMUP_ENABLED=1. When enabled, the server detects unseen (batch, seq) shapes during live inference and compiles TRT engines for them during idle windows.

engine_propagation_enabled: bool

Enable cross-worker engine cache propagation via broadcast channel.

When true, after any worker writes a new TRT engine plan to EFS (via the adaptive_warmup loop or a real-inference JIT compile), a (batch, seq) shape notification is broadcast to every peer worker so they eagerly run trt_prewarm against their own session (~1-3s fast disk-load).

Defaults to adaptive_warmup_enabled. Set BGE_M3_ENGINE_PROPAGATION_ENABLED=0 to disable propagation while keeping adaptive warmup active (debugging).

adaptive_warmup_quiet_secs: u64

Seconds the server must be idle (queue_depth == 0, all workers free) before the adaptive warmup loop fires a shape.

Set with BGE_M3_ADAPTIVE_WARMUP_QUIET_SECS. Default: 3.

adaptive_warmup_max_shapes_per_hour: u32

Maximum number of shapes the adaptive warmup loop may compile per hour.

Set with BGE_M3_ADAPTIVE_WARMUP_MAX_SHAPES_PER_HOUR. Default: 12. Prevents pathological traffic patterns from compiling indefinitely.

trt_warmup_shapes: Vec<(usize, usize)>

List of (batch_size, seq_len) shapes to pre-compile as TensorRT engine files during worker startup.

Set with BGE_M3_TRT_WARMUP_SHAPES as a comma-separated list of BxL tokens (e.g. "1x128,4x512,16x2048,32x8192"). Only used when ep == EpSelection::TensorRt. Invalid tokens are skipped with a WARN. An empty or all-invalid value falls back to the default set. Operators can shrink the grid for local development (e.g. "1x128") — the env var override path is the canonical way to keep cold-start tractable on workstations.

Default: a 2D {1, 2, 4, 8, 16, 32} × {128, 512, 2048, 8192} grid (24 shapes) composed in batch-major order so the smallest batches finish first and the most common router shapes (single-text and two-text requests) hit a pre-compiled engine on the very first real request. The expensive _ × 8192 shapes compile last (~30–170 s each) — total cold-cache compile budget is roughly 9–18 minutes on first deploy. Subsequent starts on the same EC2 instance reuse cached engine files (seconds).

Each shape may take 30–170 s to compile on the first run; the worker signals ready only after all shapes finish, so /health returns 503 during this window.

warmup_only: bool

Exit cleanly after TensorRT engine compilation and cache flush.

Set with BGE_M3_WARMUP_ONLY. Default false.

When true the server initialises the model and ORT session exactly as normal — loading ONNX weights, configuring the TRT EP, and running the pre-warm shape compilation via the existing warmup path. After all engines have been compiled and fsynced to the EFS cache the process logs a single INFO line and calls process::exit(0). No TCP listener is bound; the HTTP server never starts.

Primary use-case: ECS init container that pre-populates the shared EFS engine cache before the main container starts, so the main container always sees a warm cache and skips the 6–12 minute cold-compile window.

A WARN is logged if this flag is set with BGE_M3_EP other than tensorrt — warmup-only on CPU is a no-op (there is nothing to compile) but the server still exits 0 cleanly rather than erroring.

prewarm_strict: bool

When true, prewarm postcondition failures cause workers to refuse to signal ready (the pool init handle errors out and the readiness probe triggers a hard process exit). When false, postcondition failures only log a WARN and workers still signal ready — preserving pre-fix behaviour for debugging and operators who explicitly opt out of fail-loud startup.

Set with BGE_M3_PREWARM_STRICT. Defaults to true.

A production incident motivated this default: every worker on a multi-GPU Blackwell task hit a TRT autotuner workspace OOM mid-build (IBuilder::buildSerializedNetwork: Error Code 10), the postcondition logged a WARN, /health returned 200 ok, and the task served HTTP 500 traffic on the same shape that failed prewarm. Strict-mode would have forced an immediate task exit, which ECS retries — far better than routing traffic to a known-broken pool.

max_body_bytes: usize

Maximum HTTP request body size in bytes.

Set with BGE_M3_MAX_BODY_BYTES. Defaults to 33_554_432 (32 MiB). Raise this value when embedding large batches with long function bodies that exceed the default limit (HTTP 413 Content Too Large).

circuit_breaker_threshold: usize

Number of consecutive inference failures that trips the per-worker circuit breaker.

Set with BGE_M3_CIRCUIT_BREAKER_THRESHOLD. Defaults to 5.

When a worker returns N consecutive errors from embed_dense, embed_sparse, or embed_both, it unloads its ORT session (dropping the CUDA arena) and decrements loaded_workers. /health transitions to idle (200) when loaded_workers == 0 and fail (503) when live_workers == 0. On the next incoming request the worker reloads from the on-disk model cache, resetting the counter. This limits blast radius from a broken GPU state to ~5 requests before self-healing.

trt_inband_jit_guard_enabled: bool

Enables the in-band TensorRT JIT admission guard.

Set with BGE_M3_TRT_INBAND_JIT_GUARD. Defaults to true.

When enabled (and ep == tensorrt), the worker refuses any chunk whose padded sequence length is at/above Self::trt_inband_jit_guard_seq and exceeds the pool’s warmed engine coverage, returning HTTP 503 instead of issuing the session.run() that would trigger an in-band TRT JIT. On the fused /v1/embeddings:both graph at seq=8192 that JIT can request a pathological autotuner allocation (tens of GiB to multiple TiB) that crashes the worker via SIGSEGV / OOM-kill — a hard process death no Result-based safety net can catch. Refusing the rare uncovered request is strictly safer. Set to 0 to disable (restoring the pre-guard crash-on-uncovered-large-shape behaviour).

trt_inband_jit_guard_seq: usize

Sequence-length threshold for the in-band JIT guard.

Set with BGE_M3_TRT_INBAND_JIT_GUARD_SEQ. Defaults to 4096.

Chunks with seq < guard_seq are always admitted (a cold JIT at small or medium sequence lengths is bounded and lets the engine profile grow naturally); only seq >= guard_seq chunks that are also uncovered by the warmed profile are refused. The default sits between the second- highest (2048) and highest (8192) default warmup tiers, so it targets the genuinely-pathological large-sequence region and is a no-op for deployments whose max_seq_length is below it.

Implementations

impl Config

pub fn from_env() -> Result<Self>

Creates a Config by reading environment variables.

Unrecognized or missing variables fall back to their defaults.

Errors

Returns Err when exactly one of BGE_M3_TLS_CERT_PATH / BGE_M3_TLS_KEY_PATH is set: a half-configured TLS pair would cause the server to silently fall back to plain HTTP rather than fail loudly.

pub(crate) fn validate(&self) -> Result<()>

Validates configuration invariants that cannot be enforced by the type system alone.

Errors

Returns Err when exactly one of tls_cert_path / tls_key_path is Some. Both must be present or both must be absent.

pub(crate) fn from_lookup<F: Fn(&str) -> Option<String>>(lookup: F) -> Self

Creates a Config by resolving each setting through lookup.

lookup receives an env-var name and returns its value if set, or None to fall back to the default for that setting. Used by Config::from_env with the real environment and in tests with a closure over a HashMap.

Side effect: when BGE_M3_EP=tensorrt, emits a WARN via tracing if the resolved trt_warmup_shapes grid does not cover batch=1 or batch=2. Tests that construct a Config with TRT EP and a partial grid will see this log output.