xezpeleta/whisper-medium-eu

Whisper Medium Basque

This model is a fine-tuned version of openai/whisper-medium specifically for Basque (eu) language Automatic Speech Recognition (ASR). It was trained on the asierhv/composite_corpus_eu_v2.1 dataset, which is a composite corpus designed to improve Basque ASR performance.

Key improvements and results compared to the base model:

• Significant WER reduction: The fine-tuned model achieves a Word Error Rate (WER) of 9.1045 on the validation set of the asierhv/composite_corpus_eu_v2.1 dataset, demonstrating improved accuracy compared to the base whisper-medium model for Basque.
• Performance on Common Voice: When evaluated on the Mozilla Common Voice 18.0 dataset, the model achieved a WER of 7.14. This indicates strong generalization capabilities and highlights the benefits of the medium-sized model for enhanced accuracy.

Model description

This model utilizes the whisper-medium architecture, which offers a substantial increase in capacity compared to smaller variants, leading to improved accuracy. By fine-tuning this model on a dedicated Basque speech corpus, it specializes in accurately transcribing Basque speech. The whisper-medium model strikes a balance between high accuracy and manageable computational requirements.

Intended uses & limitations

Intended uses:

• High-precision automatic transcription of Basque speech for professional and research applications.
• Development of advanced Basque speech-based applications requiring very high accuracy.
• Research in Basque speech processing where the highest possible accuracy is crucial.
• Professional transcription services and applications where accuracy is paramount.
• Use in scenarios where the computational cost is acceptable for the significant improvement in accuracy.

Limitations:

• Performance remains influenced by audio quality, with challenges arising from background noise and poor recording conditions.
• Accuracy may be affected by highly dialectal or informal Basque speech.
• Despite improved performance, the model may still produce errors, particularly with complex linguistic structures or rare words.
• The medium model is larger than the small, base, and tiny models, so inference will be slower and require more resources.

Training and evaluation data

• Training dataset: asierhv/composite_corpus_eu_v2.1. This dataset is a meticulously curated collection of Basque speech data, designed to maximize the performance of Basque ASR systems.
• Evaluation Dataset: The test split of asierhv/composite_corpus_eu_v2.1.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 6.25e-06
• train_batch_size: 16
• eval_batch_size: 8
• seed: 42
• optimizer: AdamW with betas=(0.9, 0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• training_steps: 10000
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	WER
0.3412	0.05	500	0.5112	28.8685
0.1464	0.1	1000	0.4178	20.5570
0.2504	0.15	1500	0.3625	18.1279
0.2615	0.2	2000	0.3236	15.5364
0.1648	0.25	2500	0.3209	13.8129
0.0933	0.3	3000	0.2991	12.8887
0.1016	0.35	3500	0.2823	12.4329
0.1449	0.4	4000	0.2741	11.7460
0.151	0.45	4500	0.2791	11.5774
0.0917	0.5	5000	0.2744	11.2402
0.0913	0.55	5500	0.2901	11.1340
0.1085	0.6	6000	0.2663	10.3285
0.0928	0.65	6500	0.2705	10.2910
0.0725	0.7	7000	0.2506	10.3035
0.1216	0.75	7500	0.2758	9.7103
0.131	0.8	8000	0.2519	9.4292
0.0525	0.85	8500	0.2602	9.3106
0.0729	0.9	9000	0.2549	9.3606
0.0939	0.95	9500	0.2470	9.1920
0.0639	1.0	10000	0.2488	9.1045

Framework versions

• Transformers 4.49.0.dev0
• Pytorch 2.6.0+cu124
• Datasets 3.3.1.dev0
• Tokenizers 0.21.0