LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders
LLM2Vec is a simple recipe to convert decoder-only LLMs into text encoders. It consists of 3 simple steps: 1) enabling bidirectional attention, 2) masked next token prediction, and 3) unsupervised contrastive learning. The model can be further fine-tuned to achieve state-of-the-art performance.
- Repository: https://github.com/McGill-NLP/llm2vec
- Paper: https://arxiv.org/abs/2404.05961
Overview:
This is a bi-directional version of Sheared-LLaMA-1.3B trained with masked token prediction on the Wikipedia dataset. Modern decoder models offer several advantages over classical encoders like BERT:
They are pre-trained on more recent textual corpora They are trained on larger and more diverse datasets Modern decoders have better support for long-context windows Flash-attention support is available for these models
Considering these benefits, we are excited to release a series of decoder models tuned to work in a bi-directional setting. This approach combines the strengths of modern decoder architectures with the versatility of bi-directional context understanding, potentially opening up new possibilities for various natural language processing tasks, such as NER.
In comparison to original LLM2Vec we trained all weights of LLama model, it potentially improve bi-directional abilities of the model.
Installation
pip install llm2vec
Usage
from llm2vec.models import LlamaBiModel
import torch
from transformers import AutoTokenizer
# Loading base Mistral model, along with custom code that enables bidirectional connections in decoder-only LLMs. MNTP LoRA weights are merged into the base model.
tokenizer = AutoTokenizer.from_pretrained(
"knowledgator/Sheared-LLaMA-encoder-1.3B"
)
model = LLamaBiModel.from_pretrained("knowledgator/Sheared-LLaMA-encoder-1.3B")
text = "The quick brown fox jumps over the lazy dog."
inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True, max_length=512)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)
inputs = {k: v.to(device) for k, v in inputs.items()}
with torch.no_grad():
outputs = model(**inputs)
last_hidden_states = outputs.last_hidden_state
Here's an improved and expanded version of the README snippet:
Adapting for Different Discriminative Tasks
Our bi-directional LLaMA model can be easily adapted for various discriminative tasks such as text classification, question answering, and token classification. To use these specialized versions, we provide a fork of LLM2Vec with additional functionality.
Installation
To get started, clone our fork of LLM2Vec and install it:
git clone https://github.com/Knowledgator/llm2vec.git
cd llm2vec
pip install -e .
Using -e
flag installs the package in editable mode, which is useful for development.
Usage
Here's how to import and use the models for different tasks:
from llm2vec import (
AutoLLMEncoderForSequenceClassification,
AutoLLMEncoderForQuestionAnswering,
AutoLLMEncoderForTokenClassification
)
# Load models for different tasks
classification_model = AutoLLMEncoderForSequenceClassification.from_pretrained('knowledgator/Sheared-LLaMA-encoder-1.3B')
question_answering_model = AutoLLMEncoderForQuestionAnswering.from_pretrained('knowledgator/Sheared-LLaMA-encoder-1.3B')
token_classification_model = AutoLLMEncoderForTokenClassification.from_pretrained('knowledgator/Sheared-LLaMA-encoder-1.3B')
Example: Text Classification
Here's a basic example of how to use the model for text classification:
from transformers import AutoTokenizer
# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained('knowledgator/Sheared-LLaMA-encoder-1.3B')
# Prepare input
text = "This movie is great!"
inputs = tokenizer(text, return_tensors="pt")
# Get classification logits
outputs = classification_model(**inputs)
logits = outputs.logits
# The logits can be used with a softmax function to get probabilities
# or you can use torch.argmax(logits, dim=1) to get the predicted class
Fine-tuning
To fine-tune these models on your specific task:
- Prepare your dataset in a format compatible with HuggingFace's
datasets
library. - Use the
Trainer
class from HuggingFace'stransformers
library to fine-tune the model.
Here's a basic example:
from transformers import Trainer, TrainingArguments
from datasets import load_dataset
# Load your dataset
dataset = load_dataset("your_dataset")
# Define training arguments
training_args = TrainingArguments(
output_dir="./results",
num_train_epochs=3,
per_device_train_batch_size=8,
per_device_eval_batch_size=8,
warmup_steps=500,
weight_decay=0.01,
logging_dir="./logs",
)
# Initialize Trainer
trainer = Trainer(
model=classification_model,
args=training_args,
train_dataset=dataset["train"],
eval_dataset=dataset["test"],
)
# Fine-tune the model
trainer.train()
Contributing
We welcome contributions! If you have suggestions for improvements or encounter any issues, please open an issue or submit a pull request on our GitHub repository.
- Downloads last month
- 13