Skip to main content

load_models

Function load_models 

Source
pub(super) fn load_models(
    cfg: &GpuSessionConfig<'_>,
    show_download_progress: bool,
) -> Result<(Session, Tokenizer)>
Expand description

Downloads (if not already cached) and loads both the ORT session and the tokenizer for the given model variant, returning them as a pair.

cfg.device_id selects the CUDA/TRT GPU device for this session. Computed by EmbedPool::spawn as worker_index % gpu_count. Ignored on CPU EP and macOS.

cfg.trt_max_workspace_bytes and cfg.gpu_mem_limit_bytes are forwarded verbatim to execution_providers; see that function’s documentation for semantics.