Hugging Face's logo

PleIAs
/
Pleias-Nano

Safetensors
llama
Model card Files Community
4
Pleias-Nano / README.md
Pclanglais's picture
Pclanglais
Update README.md
83a13d9 verified
preview code
|
raw
history blame contribute delete
11.3 kB
metadata
license: apache-2.0
datasets:
  - PleIAs/common_corpus
language:
  - en
  - fr
  - de
  - es
  - it
  - nl
  - la
  - pt

Pleias-nano-1.2b-RAG 0.1 is a specialized language model designed by Pleias and trained with Tracto AI for Retrieval-Augmented Generation.

Similarly to its base model, Pleias-nano-1.2b-Preview, Pleias-nano-1.2b-RAG 0.1 aims to be a fully open model (weights, code, data), only trained on content with a permissible license and fully compliant with the upcoming European AI Act.

Description

Pleias-nano-1.2b-RAG is continuous pretraining of Pleias-nano-1.2b-Preview on a new dataset of 45,088,768,000 tokens modeling common retrieval tasks. All the content of the dataset is ultimately coming from Common Corpus.

Pleias-nano-1.2b-RAG includes the main features of the original base model:

  • Only trained on open data under a permissible license and in compliance with the European AI Act. By design, all Pleias model are unable to output copyrighted content.
  • Extensive multilingual support for main European languages: English, French, German, Spanish, Italian, Dutch, Latin, Portuguese and Polish.
  • Extremely low level of toxicity and problematic content.

Pleias-nano-1.2b-RAG supports retrieval-augmented generation with enhanced verifiability, source analysis and grounding on submitted sources. This includes:

  • Standardized structure and special tokens to include queries, sources, .
  • Anticipation of various query forms in multiple languages, from actual drafted questions to unstructured list of keyword search.
  • Source analysis/criticism which also acts as an integrated reranker step.
  • Generation of ground answers with references and excerpts linked to the original sources.

While the base model Pleias-nano-1.2b-RAG has been made available as an experimental preview, we release Pleias-nano-1.2b-RAG 0.1 as an early version. Pleias-nano-1.2b-RAG 0.1 has been already tested and integrated into multiple applied RAG projects, including Pleias flagship application Scholasticai.

Training

Pleias-nano-1.2b-RAG was trained pretrained on TractoAI on ISEG GPU cluster by Nebius AI on the fork Nanotron developed by TractoAI. We provide the complete settings as a yaml file as part of our release.

Pleias-nano-1.2b-RAG derives from the last checkpoint of Pleias-nano-1.2b-Preview (369,000). The training schedule reused the last learning rate value (5e-6) without decay for 43,000 steps. Each step is about 10 time smaller than the original steps from the base model training (roughly 1M tokens per step vs. 12M tokens)

Training covers the entire RAG dataset we have been designing based on Common Corpus for 3 epochs.

Further experiments were made with different learning rate values: none of theses tests have provided a better convergence than the one obtained with the final learning rate from the base model.

Inference

Pleias-nano-1.2b-RAG relies on special tokens to encode the core RAG functionalities:

A typical example, with excerpts drawn from a Wikipedia article on Wikipedia

<|query_start|>Is Wikipedia reliable?<|query_end|>
<|source_start|><|source_id_start|>ebea70a3502acfbd<|source_id_end|>Articles for traditional encyclopedias such as Encyclopædia Britannica are written by experts, lending such encyclopedias a reputation for accuracy.[144] However, a peer review in 2005 of forty-two scientific entries on both Wikipedia and Encyclopædia Britannica by the science journal Nature found few differences in accuracy, and concluded that "the average science entry in Wikipedia contained around four inaccuracies; Britannica, about three."[145] Joseph Reagle suggested that while the study reflects "a topical strength of Wikipedia contributors" in science articles, "Wikipedia may not have fared so well using a random sampling of articles or on humanities subjects."<|source_end|>
<|source_start|><|source_id_start|>5f862e733d38288e<|source_id_end|>As a consequence of the open structure, Wikipedia "makes no guarantee of validity" of its content, since no one is ultimately responsible for any claims appearing in it.[W 54] Concerns have been raised by PC World in 2009 regarding the lack of accountability that results from users' anonymity, the insertion of false information,[152] vandalism, and similar problems. Legal Research in a Nutshell (2011), cites Wikipedia as a "general source" that "can be a real boon" in "coming up to speed in the law governing a situation" and, "while not authoritative, can provide basic facts as well as leads to more in-depth resources".<|source_end|>
<|source_start|><|source_id_start|>354fa4908152b336<|source_id_end|>Wikipedia's open structure inherently makes it an easy target for Internet trolls, spammers, and various forms of paid advocacy seen as counterproductive to the maintenance of a neutral and verifiable online encyclopedia.[70][W 55] In response to paid advocacy editing and undisclosed editing issues, Wikipedia was reported in an article in The Wall Street Journal to have strengthened its rules and laws against undisclosed editing.[162] The article stated that: "Beginning Monday [from the date of the article, June 16, 2014], changes in Wikipedia's terms of use will require anyone paid to edit articles to disclose that arrangement. Katherine Maher, the nonprofit Wikimedia Foundation's chief communications officer, said the changes address a sentiment among volunteer editors that 'we're not an advertising service; we're an encyclopedia.'"<|source_end|>
<|source_analysis_start|>

As a specialized language model, PleIAs-1.2b-RAG will be unable to work properly with prompts that detracts from that design.

RAG Evaluation

We evaluate Pico and Nano models on a RAG task. As existing benchmarks are largely limited to English, we develop a custom multilingual RAG benchmark. We synthetically generate queries and small sets of documents. To evaluate, we prompted models with the query and documents. We then ran a head-to-head ELO-based tournament with GPT-4o as judge. We release the prompts and generations for all models we compared. Our nano (1.2B) model outperforms Llama 3.2 1.1B and EuroLLM 1.7B. Our pico (350M) model outperforms other models in its weight class, such as SmolLM 360M and Qwen2.5 500M, in addition to much larger models, such as Llama 3.2 1.1B and EuroLLM 1.7B.

Rank Model ELO
1 Qwen2.5-Instruct-7B 1294.6
2 Llama-3.2-Instruct-8B 1269.8
3 Pleias-nano-1.2B-RAG 1137.5
4 Llama-3.2-Instruct-3B 1118.1
5 Qwen2.5-Instruct-3B 1078.1
6 Pleias-pico-350M-RAG 1051.2
7 Llama-3.2-1B-Instruct 872.3
8 EuroLLM-1.7B-Instruct 860.0
9 SmolLM-360M-Instruct 728.6
10 Qwen2.5-0.5B-Instruct 722.2
11 SmolLM-1.7B-Instruct 706.3

Acceptable use

Pleias-nano-1.2b-RAG includes a much wider range of support for verifiability and grounding than most generalist models.

The model is not a substitute for an integrated RAG application. Retrieval errors as well as challenging texts and questions can still create a range of issues. We especially encourage end users to take advantage of the citations and the references to provide better indicators of accuracy.

For best results we recommend the following setting:

  • Deterministic generation (temp = 0) and no repetition penalty (which is unsurprisingly detrimental to the accuracy of citations).
  • Standardized hashes of 16 characters. While the model has been trained on many other patterns (including full bibliographic entries), this has proven the most convenient for systematic citation parsing.

Training Greenhouse Gas Emissions: Estimated total location-based greenhouse gas emissions were 4 tons CO2eq for training.

Ethical Considerations

pleias-nano-1.2b-RAG model, like all large language models, carries inherent ethical risks that require careful consideration. Our approach to mitigating these risks begins at the data level, where we exclusively use vetted sources, deliberately excluding CommonCrawl. The primary challenge comes from our public domain dataset component, which contains historical texts that may reflect outdated social norms and potentially harmful language, particularly regarding minoritized groups.

To address this, we implemented a systematic ethical filtering process using toxicity classifiers to identify extremely harmful content. We also employed synthetic rewriting techniques to transform mildly problematic passages while preserving the underlying informational value. This process significantly reduced potential societal harm without compromising the dataset's size or textual quality, resulting in notably low toxicity scores in benchmarks compared to other models.

Despite these preventive measures, users should be aware that the model has not undergone additional safety alignment procedures and may still produce problematic outputs. The model's capabilities in generative AI tasks must be balanced against the risks of bias, misinformation propagation, and autonomous decision-making challenges. We explicitly prohibit any malicious utilization and emphasize the responsibility of users to implement appropriate safeguards.

At Pleias, we continue to research and develop improved methods for creating safer and more equitable models and datasets. This includes ongoing work in toxicity reduction, bias mitigation, and the development of more sophisticated ethical filtering techniques.

Acknowledgements

This work would not have been possible without the substantial support and technical expertise from TractoAI, a serverless AI platform for running data and compute-intensive workloads at scale.

We are deeply grateful to the Mozilla Foundation Local AI Program for their generous support.

Finally, we acknowledge the significant contributions from the open science LLM community, particularly HuggingFace, Eleuther AI and Allen AI whose insights and cooperation have been invaluable to our work.

Future updates

Pleias-nano-1.2b-RAG will be continuously improved through iterative retraining/adaptation.

The current roadmap includes the following features:

  • Context length expansion.
  • Better handling of multilingual sources. In its current form, Pleias-nano-1.2b-RAG will generally switch language if a query is made to sources in a different language.
  • New sampling methods inspired by Entropix for a better combined support of text creativity and accuracy.
  • Interactive/conversational RAG.

End users are encouraged to update to the latest version whenever possible.