Skip to main content

sparse_project

Function sparse_project 

Source
pub(super) fn sparse_project(
    hidden: &[f32],
    weight: &ArrayView1<'_, f32>,
    bias: f32,
) -> f32
Expand description

Projects a single token’s hidden state through the sparse-linear layer.

Returns max(0, dot(hidden, weight) + bias) (ReLU-gated score).