Lemone-Embed: A Series of Fine-Tuned Embedding Models for French Taxation
This series is made up of 7 models, 3 basic models of different sizes trained on 1 epoch, 3 models trained on 2 epochs making up the Boost series and a Pro model with a non-Roberta architecture.
This sentence transformers model, specifically designed for French taxation, has been fine-tuned on a dataset comprising 43 million tokens, integrating a blend of semi-synthetic and fully synthetic data generated by GPT-4 Turbo and Llama 3.1 70B, which have been further refined through evol-instruction tuning and manual curation.
The model is tailored to meet the specific demands of information retrieval across large-scale tax-related corpora, supporting the implementation of production-ready Retrieval-Augmented Generation (RAG) applications. Its primary purpose is to enhance the efficiency and accuracy of legal processes in the taxation domain, with an emphasis on delivering consistent performance in real-world settings, while also contributing to advancements in legal natural language processing research.
This is a sentence-transformers model finetuned from intfloat/multilingual-e5-large. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
Model Details
Model Description
- Model Type: Sentence Transformer
- Base model: intfloat/multilingual-e5-large
- Maximum Sequence Length: 512 tokens
- Output Dimensionality: 1024 tokens
- Similarity Function: Cosine Similarity
- Developed by: Louis Brulé Naudet
- Funded by: Microsoft for Startups
- Shared by: Louis Brulé Naudet
- Model type: Sentence Transformers
- Language(s) (NLP): FR
- License: Apache 2
- Finetuned from model: intfloat/multilingual-e5-large
Model Sources
- Documentation: Sentence Transformers Documentation
- Repository: Sentence Transformers on GitHub
- Hugging Face: Sentence Transformers on Hugging Face
Full Model Architecture
SentenceTransformer(
(0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: XLMRobertaModel
(1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
(2): Normalize()
)
Usage
Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("louisbrulenaudet/lemone-embed-l")
# Run inference
sentences = [
"query: Décrivez avec précision les étapes détaillées requises pour traiter les réclamations collectives résultant de désastres agricoles comme définies dans l'article R*198-2 du Livre des procédures fiscales.",
"passage: Selon l'article R*198-2 du Livre des procédures fiscales, le traitement des réclamations collectives en cas de catastrophes naturelles impactant les cultures agricoles, incluant des phénomènes tels que la grêle ou les inondations, exige la collaboration de plusieurs entités administratives. Initialement, deux commissaires sont nommés par l'administration fiscale pour superviser le processus. Ils sont assistés par un délégué de l'administration des impôts. Avant toute action, le maire de la commune affectée est notifié au moins dix jours avant l'inspection prévue, et il est chargé de communiquer cette date aux résidents via des affichages publics. Les agriculteurs affectés doivent alors rapporter leurs pertes à la mairie avant la réalisation d'un constat officiel par l'inspecteur des impôts, qui sera consigné dans un procès-verbal. Une fois ce document clôturé, aucune réclamation supplémentaire n'est acceptée.",
"passage: Selon les dispositions de l'article R*196-6 du Livre des procédures fiscales, il est attribué aux sujets fiscaux un intervalle précisément défini pour élever des réclamations à l'égard des taxes, cotisations et autres prélèvements relatifs aux céréales et leurs transformés. Ce délai se prolonge jusqu'à la fin de la campagne agricole suivante celle au cours de laquelle l'avis de mise en recouvrement de la taxe a été notifié ou le règlement de l'imposition contestée effectué, permettant ainsi aux parties prenantes de se prévaloir de leurs prérogatives contestataires avec une certitude temporelle.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
Evaluation
Metrics
Information Retrieval
- Dataset:
Lemone-information-retrieval
- Evaluated with
InformationRetrievalEvaluator
Metric | Value |
---|---|
cosine_accuracy@1 | 0.9756 |
cosine_accuracy@3 | 0.9936 |
cosine_accuracy@5 | 0.9955 |
cosine_accuracy@10 | 0.9981 |
cosine_precision@1 | 0.9756 |
cosine_precision@3 | 0.3312 |
cosine_precision@5 | 0.1991 |
cosine_precision@10 | 0.0998 |
cosine_recall@1 | 0.9756 |
cosine_recall@3 | 0.9936 |
cosine_recall@5 | 0.9955 |
cosine_recall@10 | 0.9981 |
cosine_ndcg@10 | 0.988 |
cosine_mrr@10 | 0.9847 |
cosine_map@100 | 0.9847 |
dot_accuracy@1 | 0.9756 |
dot_accuracy@3 | 0.9936 |
dot_accuracy@5 | 0.9955 |
dot_accuracy@10 | 0.9981 |
dot_precision@1 | 0.9756 |
dot_precision@3 | 0.3312 |
dot_precision@5 | 0.1991 |
dot_precision@10 | 0.0998 |
dot_recall@1 | 0.9756 |
dot_recall@3 | 0.9936 |
dot_recall@5 | 0.9955 |
dot_recall@10 | 0.9981 |
dot_ndcg@10 | 0.988 |
dot_mrr@10 | 0.9847 |
dot_map@100 | 0.9847 |
Training Details
Training Dataset
- Size: 296,234 training samples
- Columns:
query
,positive
, andnegative
- Approximate statistics based on the first 1000 samples:
query positive negative type string string string details - min: 24 tokens
- mean: 54.2 tokens
- max: 179 tokens
- min: 75 tokens
- mean: 182.28 tokens
- max: 332 tokens
- min: 53 tokens
- mean: 190.2 tokens
- max: 456 tokens
- Loss:
CachedGISTEmbedLoss
with these parameters:{'guide': SentenceTransformer( (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: NewModel (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) (2): Normalize() ), 'temperature': 0.01}
Training Hyperparameters
Non-Default Hyperparameters
eval_strategy
: stepsper_device_train_batch_size
: 128learning_rate
: 1e-05num_train_epochs
: 1warmup_ratio
: 0.1fp16
: Truebatch_sampler
: no_duplicates
All Hyperparameters
Click to expand
overwrite_output_dir
: Falsedo_predict
: Falseeval_strategy
: stepsprediction_loss_only
: Trueper_device_train_batch_size
: 128per_device_eval_batch_size
: 8per_gpu_train_batch_size
: Noneper_gpu_eval_batch_size
: Nonegradient_accumulation_steps
: 1eval_accumulation_steps
: Nonetorch_empty_cache_steps
: Nonelearning_rate
: 1e-05weight_decay
: 0.0adam_beta1
: 0.9adam_beta2
: 0.999adam_epsilon
: 1e-08max_grad_norm
: 1.0num_train_epochs
: 1max_steps
: -1lr_scheduler_type
: linearlr_scheduler_kwargs
: {}warmup_ratio
: 0.1warmup_steps
: 0log_level
: passivelog_level_replica
: warninglog_on_each_node
: Truelogging_nan_inf_filter
: Truesave_safetensors
: Truesave_on_each_node
: Falsesave_only_model
: Falserestore_callback_states_from_checkpoint
: Falseno_cuda
: Falseuse_cpu
: Falseuse_mps_device
: Falseseed
: 42data_seed
: Nonejit_mode_eval
: Falseuse_ipex
: Falsebf16
: Falsefp16
: Truefp16_opt_level
: O1half_precision_backend
: autobf16_full_eval
: Falsefp16_full_eval
: Falsetf32
: Nonelocal_rank
: 0ddp_backend
: Nonetpu_num_cores
: Nonetpu_metrics_debug
: Falsedebug
: []dataloader_drop_last
: Falsedataloader_num_workers
: 0dataloader_prefetch_factor
: Nonepast_index
: -1disable_tqdm
: Falseremove_unused_columns
: Truelabel_names
: Noneload_best_model_at_end
: Falseignore_data_skip
: Falsefsdp
: []fsdp_min_num_params
: 0fsdp_config
: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}fsdp_transformer_layer_cls_to_wrap
: Noneaccelerator_config
: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}deepspeed
: Nonelabel_smoothing_factor
: 0.0optim
: adamw_torchoptim_args
: Noneadafactor
: Falsegroup_by_length
: Falselength_column_name
: lengthddp_find_unused_parameters
: Noneddp_bucket_cap_mb
: Noneddp_broadcast_buffers
: Falsedataloader_pin_memory
: Truedataloader_persistent_workers
: Falseskip_memory_metrics
: Trueuse_legacy_prediction_loop
: Falsepush_to_hub
: Falseresume_from_checkpoint
: Nonehub_model_id
: Nonehub_strategy
: every_savehub_private_repo
: Falsehub_always_push
: Falsegradient_checkpointing
: Falsegradient_checkpointing_kwargs
: Noneinclude_inputs_for_metrics
: Falseeval_do_concat_batches
: Truefp16_backend
: autopush_to_hub_model_id
: Nonepush_to_hub_organization
: Nonemp_parameters
:auto_find_batch_size
: Falsefull_determinism
: Falsetorchdynamo
: Noneray_scope
: lastddp_timeout
: 1800torch_compile
: Falsetorch_compile_backend
: Nonetorch_compile_mode
: Nonedispatch_batches
: Nonesplit_batches
: Noneinclude_tokens_per_second
: Falseinclude_num_input_tokens_seen
: Falseneftune_noise_alpha
: Noneoptim_target_modules
: Nonebatch_eval_metrics
: Falseeval_on_start
: Falseeval_use_gather_object
: Falsebatch_sampler
: no_duplicatesmulti_dataset_batch_sampler
: proportional
Environmental Impact
Carbon emissions were measured using CodeCarbon.
- Energy Consumed: 1.732 kWh
- Carbon Emitted: 0.639 kg of CO2
- Hours Used: 3.122 hours
Training Hardware
- On Cloud: No
- GPU Model: 1 x NVIDIA H100 NVL
- CPU Model: AMD EPYC 9V84 96-Core Processor
- RAM Size: 314.69 GB
Framework Versions
- Python: 3.10.12
- Sentence Transformers: 3.1.1
- Transformers: 4.44.2
- PyTorch: 2.3.0+cu121
- Accelerate: 0.33.0
- Datasets: 2.21.0
- Tokenizers: 0.19.1
Citation
BibTeX
Sentence Transformers
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
If you use this code in your research, please use the following BibTeX entry.
@misc{louisbrulenaudet2024,
author = {Louis Brulé Naudet},
title = {Lemone-Embed: A Series of Fine-Tuned Embedding Models for French Taxation},
year = {2024}
howpublished = {\url{https://huggingface.co./datasets/louisbrulenaudet/lemone-embed-l}},
}
Feedback
If you have any feedback, please reach out at [email protected].
- Downloads last month
- 20
Model tree for louisbrulenaudet/lemone-embed-l
Base model
intfloat/multilingual-e5-largeDatasets used to train louisbrulenaudet/lemone-embed-l
Collection including louisbrulenaudet/lemone-embed-l
Evaluation results
- Cosine Accuracy@1 on Lemone information retrievalself-reported0.976
- Cosine Accuracy@3 on Lemone information retrievalself-reported0.994
- Cosine Accuracy@5 on Lemone information retrievalself-reported0.996
- Cosine Accuracy@10 on Lemone information retrievalself-reported0.998
- Cosine Precision@1 on Lemone information retrievalself-reported0.976
- Cosine Precision@3 on Lemone information retrievalself-reported0.331
- Cosine Precision@5 on Lemone information retrievalself-reported0.199
- Cosine Precision@10 on Lemone information retrievalself-reported0.100
- Cosine Recall@1 on Lemone information retrievalself-reported0.976
- Cosine Recall@3 on Lemone information retrievalself-reported0.994