Bumblebee.Text.NomicBert (Bumblebee v0.7.0)

View Source

Nomic BERT model family.

This is a variant of BERT that uses:

  • Rotary position embeddings (RoPE) instead of absolute position embeddings
  • SwiGLU activation in the feed-forward network
  • Post-normalization (like original BERT)
  • No biases in attention and feed-forward layers

Architectures

  • :base - plain Nomic BERT without any head on top

Inputs

  • "input_ids" - {batch_size, sequence_length}

    Indices of input sequence tokens in the vocabulary.

  • "attention_mask" - {batch_size, sequence_length}

    Mask indicating which tokens to attend to. This is used to ignore padding tokens, which are added when processing a batch of sequences with different length.

  • "position_ids" - {batch_size, sequence_length}

    Indices of positions of each input sequence token used when applying rotary position embeddings (RoPE).

Global layer options

  • :output_hidden_states - when true, the model output includes all hidden states

  • :output_attentions - when true, the model output includes all attention weights

Configuration

  • :vocab_size - the vocabulary size of the token embedding. This corresponds to the number of distinct tokens that can be represented in model input and output . Defaults to 30528

  • :max_positions - the maximum sequence length that this model can process. Typically this is set to a large value just in case, such as 512, 1024 or 2048 . Defaults to 8192

  • :max_token_types - the vocabulary size of the token type embedding (also referred to as segment embedding). This corresponds to how many different token groups can be distinguished in the input . Defaults to 2

  • :hidden_size - the dimensionality of hidden layers. Defaults to 768

  • :num_blocks - the number of Transformer blocks in the encoder. Defaults to 12

  • :num_attention_heads - the number of attention heads for each attention layer in the encoder. Defaults to 12

  • :intermediate_size - the dimensionality of the intermediate layer in the transformer feed-forward network (FFN) in the encoder. Defaults to 4 * hidden_size

  • :activation - the activation function. Defaults to :silu

  • :rotary_embedding_base - base for computing rotary embedding frequency. Defaults to 1000

  • :rotary_embedding_percentage - percentage of hidden size to use for rotary embeddings. Defaults to 1.0

  • :layer_norm_epsilon - the epsilon used by the layer normalization layers. Defaults to 1.0e-5

  • :initializer_scale - the standard deviation of the normal initializer used for initializing kernel parameters. Defaults to 0.02

  • :ffn_gate_bias - whether to use bias in the up and gate projections of the FFN. Defaults to true

  • :ffn_output_bias - whether to use bias in the output projection of the FFN. Defaults to true

  • :num_labels - the number of labels to use in the last layer for the classification task. Defaults to 2

  • :id_to_label - a map from class index to label. Defaults to %{}