riccardo45/prova

Model features

Grounded Generation with Jamba:

A common use-case for LLMs is grounded generation and RAG, where the model is required to answer a question or follow an instruction based on a given set of documents or document snippets. To standardize this process, Jamba was trained with a specific "documents" section in its chat template. The model was trained to attend to this section, and grounded generation tasks show improved performance when the task is formatted in this way.

Similar to tools, which are given as an external argument to the model in addition to the conversation, documents are provided in a similar way. To support document-level metadata, a document is defined as a dictionary with key-values of your choosing. These are formatted within the chat template. Two keys that get special treatment are "title", which is formatted at the top of the document if present, and "text" which is a required field and defines the actual text of the document.

Ataching documents to Jamba 1.5 prompt

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("ai21labs/AI21-Jamba-1.5-Large")
messages = [
        {
            "role": "user",
            "content": "Who wrote Harry Potter?"
        }
]
documents = [
        {
            "text": "Harry Potter is a series of seven fantasy novels written by British author J. K. Rowling.",
            "title": "Harry Potter"
        },
        {
            "text": "The Great Gatsby is a novel by American writer F. Scott Fitzgerald.",
            "title": "The Great Gatsby",
            "country": "United States",
            "genre": "Novel"
        }
]
prompt = tokenizer.apply_chat_template(
    messages,
    documents=documents,
    tokenize=False,
)
# Output: J. K. Rowling