Enum ModelVariant 

pub enum ModelVariant {
    Fp32,
    Fp16,
    Int8,
}

ONNX model variant to load.

Controlled by BGE_M3_MODEL. Defaults to ModelVariant::Fp16.

Variants

Fp32

BAAI/bge-m3 FP32 model (~2.16 GB per session).

Set BGE_M3_MODEL=fp32 to enable. Recommended for Apple Silicon CoreML deployments where latency is the primary constraint: the FP32 ONNX graph contains no Cast nodes, so ORT can dispatch the entire multi-head attention + FFN block as one contiguous CoreML subgraph to the GPU — delivering 20–61% lower latency than the MLAS CPU baseline.

Not the default. Linux/Intel (MLAS-only) deployments should prefer ModelVariant::Fp16 for lower RAM and fleet-wide embedding consistency.

Fp16

Xenova/bge-m3 FP16 model (~1.08 GB per session). Default. Halves per-session memory vs FP32 (~50% reduction; ~1.08 GB vs ~2.16 GB).

This is the fleet default: all Apple Silicon LaunchAgent deployments set BGE_M3_MODEL=fp16 explicitly, and the server default matches so that Linux/Docker deployments produce consistent embeddings without any additional configuration.

Latency caveat (CoreML only). The Xenova FP16 ONNX model contains FP16↔FP32 Cast nodes at every transformer-layer boundary. ORT’s CoreML EP cannot fuse these into the attention/FFN subgraphs; each Cast executes on CPU and the transformer block never forms a single contiguous GPU subgraph. Result: FP16 + CoreML EP runs 6–10× slower than FP32 + CoreML. On MLAS/CPU EP (Linux, Intel), this Cast overhead is similarly present but the MLAS FP16 penalty (~6–9×) is the accepted trade-off for lower RAM and fleet consistency. Use BGE_M3_MODEL=fp32 on Apple Silicon to recover CoreML GPU acceleration.

Int8

Xenova/bge-m3 INT8 quantized model (~568 MB per session). Weights-only quantization; ORT dequantizes to f32 internally. Reduces peak memory by ~74% per worker vs FP32.

Embedding quality validated: dense cosine similarity ≥ 0.963 vs FP32 reference across a 184-text corpus — suitable for ANN search and semantic ranking. Avoid for applications requiring ranking precision within very small similarity margins (< 0.05 apart).

Use with MLAS (CPU EP) only. DequantizeLinear nodes fragment the CoreML execution plan identically to FP16 Cast nodes; INT8 + CoreML EP runs 42–79% slower than INT8 + MLAS with no GPU benefit.

Trait Implementations

impl Clone for ModelVariant

fn clone(&self) -> ModelVariant

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ModelVariant

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for ModelVariant

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for ModelVariant

fn eq(&self, other: &ModelVariant) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for ModelVariant

impl Eq for ModelVariant

impl StructuralPartialEq for ModelVariant