answerdotai

/

ModernBERT-base

like 435

Follow

Answer.AI 231

Hey!

As of right now, ModernBERT requires transformers to be installed from the (stable) main branch of the transformers repository. After the next transformers release (4.48.x), it will be supported in the python package available everywhere.

Sadly it can't be used on older versions of transformers, as it's a whole new model type internally so it needs the new code, sorry about that.

For anyone wondering run the following command:

pip install git+https://github.com/huggingface/transformers

running o

For anyone wondering run the following command:

pip install git+https://github.com/huggingface/transformers

tried to run with pip install git+https://github.com/huggingface/transformers on python 3.9
lib/python3.9/site-packages/transformers/utils/import_utils.py", line 1805, in _get_module
raise RuntimeError(
RuntimeError: Failed to import transformers.models.modernbert.modeling_modernbert because of the following error (look up to see its traceback):
unsupported operand type(s) for |: '_GenericAlias' and '_GenericAlias'

simple script

from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("answerdotai/ModernBERT-base")
model = AutoModelForMaskedLM.from_pretrained("answerdotai/ModernBERT-base")
input_text = "He walked to the [MASK]."
print(model(input_text))

I believe this is due to Python 3.9. Using a newer version should fix it.
As a workaround, you can add from __future__ import annotations to the top of transformers/models/modernbert/modeling_modernbert.py but I wouldn't go that route.

• Tom Aarsen

I believe this is due to Python 3.9. Using a newer version should fix it.
As a workaround, you can add from __future__ import annotations to the top of transformers/models/modernbert/modeling_modernbert.py but I wouldn't go that route.

• Tom Aarsen

    # Check only the first and last input IDs to reduce overhead.
    if self.config.pad_token_id in input_ids[:, [-1, 0]]:
        warn_string = (
return forward_call(*args, **kwargs)

File "/opt/homebrew/anaconda3/envs/market_prediction_cpu/lib/python3.10/site-packages/transformers/models/modernbert/modeling_modernbert.py", line 871, in forward
self.warn_if_padding_and_no_attention_mask(input_ids, attention_mask)
File "/opt/homebrew/anaconda3/envs/market_prediction_cpu/lib/python3.10/site-packages/transformers/modeling_utils.py", line 5033, in warn_if_padding_and_no_attention_mask
if self.config.pad_token_id in input_ids[:, [-1, 0]]:
TypeError: unhashable type: 'slice'
python-BaseException

python3.10
running on mac os
inputs = tokenizer(
'He walked to the',
return_tensors="pt"
)
model(inputs['input_ids'], inputs['attention_mask'])

nvm, worked after some fixes

I believe this is due to Python 3.9. Using a newer version should fix it.
As a workaround, you can add from __future__ import annotations to the top of transformers/models/modernbert/modeling_modernbert.py but I wouldn't go that route.

• Tom Aarsen

    # Check only the first and last input IDs to reduce overhead.
    if self.config.pad_token_id in input_ids[:, [-1, 0]]:
        warn_string = (
return forward_call(*args, **kwargs)

File "/opt/homebrew/anaconda3/envs/market_prediction_cpu/lib/python3.10/site-packages/transformers/models/modernbert/modeling_modernbert.py", line 871, in forward
self.warn_if_padding_and_no_attention_mask(input_ids, attention_mask)
File "/opt/homebrew/anaconda3/envs/market_prediction_cpu/lib/python3.10/site-packages/transformers/modeling_utils.py", line 5033, in warn_if_padding_and_no_attention_mask
if self.config.pad_token_id in input_ids[:, [-1, 0]]:
TypeError: unhashable type: 'slice'
python-BaseException

python3.10
running on mac os
inputs = tokenizer(
'He walked to the',
return_tensors="pt"
)
model(inputs)

Works with my current python version : Python 3.10.14
OS : Windows 11 and Ubuntu 24.04 LTS

I wonder whether using mean pooling is enough for ModernBERT.

No sentence-transformers model found with name answerdotai/ModernBERT-base. Creating a new one with mean pooling.

running o

For anyone wondering run the following command:

pip install git+https://github.com/huggingface/transformers

tried to run with pip install git+https://github.com/huggingface/transformers on python 3.9
lib/python3.9/site-packages/transformers/utils/import_utils.py", line 1805, in _get_module
raise RuntimeError(
RuntimeError: Failed to import transformers.models.modernbert.modeling_modernbert because of the following error (look up to see its traceback):
unsupported operand type(s) for |: '_GenericAlias' and '_GenericAlias'

simple script

from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("answerdotai/ModernBERT-base")
model = AutoModelForMaskedLM.from_pretrained("answerdotai/ModernBERT-base")
input_text = "He walked to the [MASK]."
print(model(input_text))

It works when I change my python version from 3.9 to 3.10