Function load_tokenizer 

pub(super) fn load_tokenizer(
    tokenizer_path: &Path,
    max_seq_length: usize,
) -> Result<Tokenizer>

Loads and configures the BGE-M3 tokenizer with truncation at max_seq_length but no padding. Padding is applied per-chunk in build_chunk_arrays.