Why does this model have no biases?

#48
by Inoob - opened

I used this code to extract the weight and biases of the model:

from transformers import AutoModelForCausalLM

model_name = "./Llama-3.2-3b-Instruct"
model = AutoModelForCausalLM.from_pretrained(model_name)
model_weights = model.state_dict()
weights = {}
biases = {}

for key, value in model_weights.items():
    if 'weight' in key:
        weights[key] = value
    else:
        biases[key] = value

print("Weights:", weights)
print("Biases:", biases)

However, the biases dict is empty.

Is this intentional to not have biases?

Sign up or log in to comment