Skip to main content

tokenize_no_pad

Function tokenize_no_pad 

Source
pub(super) fn tokenize_no_pad(
    tokenizer: &Tokenizer,
    texts: &[String],
) -> Result<Vec<Encoding>>
Expand description

Tokenizes texts without applying any padding. Returns one Encoding per text, each truncated to the tokenizer’s configured max_length.