metadata

license: cc-by-4.0
datasets:
  - mozilla-foundation/common_voice_17_0
  - google/fleurs
  - MASC
language:
  - ar
pipeline_tag: automatic-speech-recognition
library_name: NeMo
metrics:
  - WER
  - CER
tags:
  - speech-recognition
  - ASR
  - Arabic
  - Conformer
  - Transducer
  - CTC
  - NeMo
  - hf-asr-leaderboard
  - speech
  - audio
model-index:
  - name: stt_ar_fastconformer_hybrid_large_pc_v1.0
    results:
      - task:
          name: Automatic Speech Recognition
          type: automatic-speech-recognition
        dataset:
          name: MASC
          split: test
          type: masc
          args:
            language: ar
        metrics:
          - name: Test WER
            type: wer
            value: 11.46
      - task:
          name: Automatic Speech Recognition
          type: automatic-speech-recognition
        dataset:
          name: MCV17
          type: mozilla-foundation/common_voice_17_0
          split: test
          args:
            language: ar
        metrics:
          - name: Test WER
            type: wer
            value: 10.2
      - task:
          type: Automatic Speech Recognition
          name: automatic-speech-recognition
        dataset:
          name: FLEURS
          type: google/fleurs
          split: test
          args:
            language: ar
        metrics:
          - name: Test WER
            type: wer
            value: 8.18

NVIDIA FastConformer-Hybrid Large (ar)

| | | |

This model transcribes speech in Arabic language with punctuation marks support. It is a "large" version of FastConformer Transducer-CTC (around 115M parameters) model and is trained on two losses: Transducer (default) and CTC. See the section Model Architecture and NeMo documentation for complete architecture details. The model transcribes text in Arabic without diacritical marks and supports periods, Arabic commas and Arabic question marks.

This model is ready for commercial and non-commercial use.

License

License to use this model is covered by the CC-BY-4.0. By downloading the public and release version of the model, you accept the terms and conditions of the CC-BY-4.0 license.

References

[1] Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognition

[2] Google Sentencepiece Tokenizer

[3] NVIDIA NeMo Toolkit

[4] HuggingFace ASR Leaderboard

Model Architecture

FastConformer [1] is an optimized version of the Conformer model with 8x depthwise-separable convolutional downsampling. The model is trained in a multitask setup with hybrid Transducer decoder (RNNT) and Connectionist Temporal Classification (CTC) loss. You may find more information on the details of FastConformer here: Fast-Conformer Model.

Model utilizes a Google Sentencepiece Tokenizer [2] tokenizer with a vocabulary size of 1024.

Input

Input Type: Audio
Input Format(s): .wav files
Other Properties Related to Input: 16000 Hz Mono-channel Audio, Pre-Processing Not Needed

Output

This model provides transcribed speech as a string for a given audio sample.

Output Type: Text
Output Format: String
Output Parameters: One Dimensional (1D)
Other Properties Related to Output: May Need Inverse Text Normalization; Does Not Handle Special Characters; Outputs text in Arabic without diacritical marks

Limitations

The model is non-streaming and outputs the speech as a string without diacritical marks. Not recommended for word-for-word transcription and punctuation as accuracy varies based on the characteristics of input audio (unrecognized word, accent, noise, speech type, and context of speech). Since this model was trained on publicly available speech datasets, the performance of this model might degrade for speech which includes technical terms, or vernacular that the model has not been trained on.

How to Use this Model

The model is available for use in the NeMo toolkit [3], and can be used as a pre-trained checkpoint for inference or for fine-tuning on another dataset.

Automatically instantiate the model

import nemo.collections.asr as nemo_asr
asr_model = nemo_asr.models.EncDecHybridRNNTCTCBPEModel.from_pretrained(model_name="nvidia/stt_ar_fastconformer_hybrid_large_pc_v1.0")

Transcribing using Python

First, let's get a sample

wget https://dldata-public.s3.us-east-2.amazonaws.com/2086-149220-0033.wav

Then simply do:

asr_model.transcribe(['2086-149220-0033.wav'])

Transcribing many audio files

Using Transducer mode inference:

python [NEMO_GIT_FOLDER]/examples/asr/transcribe_speech.py 
 pretrained_name="nvidia/stt_ar_fastconformer_hybrid_large_pc_v1.0" 
 audio_dir="<DIRECTORY CONTAINING AUDIO FILES>"

Using CTC mode inference:

python [NEMO_GIT_FOLDER]/examples/asr/transcribe_speech.py 
 pretrained_name="nvidia/stt_ar_fastconformer_hybrid_large_pc_v1.0" 
 audio_dir="<DIRECTORY CONTAINING AUDIO FILES>"
 decoder_type="ctc"

Training

The [NVIDIA NeMo Toolkit] [3] was used for training the model for two hundred epochs. Model is trained with this example script.

The tokenizer for these model was built using the text transcripts of the train set with this script.

Training, Testing, and Evaluation Datasets

Training Datasets

The model is trained on composite dataset comprising of around 760 hours of Arabic speech:

Massive Arabic Speech Corpus (MASC) [690h]
- Data Collection Method: Automated
- Labeling Method: Automated
Mozilla Common Voice 17.0 Arabic [65h]
- Data Collection Method: by Human
- Labeling Method: by Human
Google Fleurs Arabic [5h]
- Data Collection Method: by Human
- Labeling Method: by Human

Evaluation Datasets

Massive Arabic Speech Corpus (MASC)
- Data Collection Method: Automated
- Labeling Method: Automated
Mozilla Common Voice 17.0 Arabic
- Data Collection Method: by Human
- Labeling Method: by Human
Google Fleurs Arabic
- Data Collection Method: by Human
- Labeling Method: by Human

Test Datasets

Massive Arabic Speech Corpus (MASC)
- Data Collection Method: Automated
- Labeling Method: Automated
Mozilla Common Voice 17.0 Arabic
- Data Collection Method: by Human
- Labeling Method: by Human
Google Fleurs Arabic
- Data Collection Method: by Human
- Labeling Method: by Human

Software Integration

Supported Hardware Microarchitecture Compatibility:

NVIDIA Ampere
NVIDIA Blackwell
NVIDIA Jetson
NVIDIA Hopper
NVIDIA Lovelace
NVIDIA Pascal
NVIDIA Turing
NVIDIA Volta

Runtime Engine

Nemo 2.0.0

Preferred Operating System

Linux

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Please report security vulnerabilities or NVIDIA AI Concerns here.

Explainability

High-Level Application and Domain: Automatic Speech Recognition
- Describe how this model works: The model transcribes audio input into text for the Arabic language
Verified to have met prescribed quality standards: Yes
Performance Metrics: Word Error Rate (WER), Character Error Rate (CER), Real-Time Factor
Potential Known Risks: Transcripts may not be 100% accurate. Accuracy varies based on the characteristics of input audio (Domain, Use Case, Accent, Noise, Speech Type, Context of speech, etcetera).

Performance

Test Hardware: A5000 GPU

The performance of Automatic Speech Recognition models is measured using Word Error Rate (WER) and Char Error Rate (CER). Since this dataset is trained on multiple domains, it will generally perform well at transcribing audio in general.

The following tables summarize the performance of the available models in this collection with the Transducer decoder. Performances of the ASR models are reported in terms of Word Error Rate (WER%) and Inverse Real-Time Factor (RTFx) with greedy decoding on test sets.

Transducer

Version	Tokenizer	Vocabulary Size	MASC Test WER	MASC Test RTFx	MCV test WER	MCV test RTFx	FLEURS test WER	FLEURS test RTFx
2.0.0	SentencePiece Unigram	1024	11.46	1654.80	10.20	1535.45	8.18	1144.34

CTC

Version	Tokenizer	Vocabulary Size	MASC Test WER	MASC Test RTFx	MCV test WER	MCV test RTFx	FLEURS test WER	FLEURS test RTFx
2.0.0	SentencePiece Unigram	1024	12.11	2060.66	11.38	1891.04	9.23	1565.59

These are greedy WER numbers without external LM. More details on evaluation can be found at HuggingFace ASR Leaderboard [4].

Bias

Was the model trained with a specific accent? No
Have any special measures been taken to mitigate unwanted bias? No
Participation considerations from adversely impacted groups [protected classes] (https://www.senate.ca.gov/content/protected-classes) in model design and testing: No

Privacy

Generatable or reverse engineerable personal data? No
If applicable, was a notice provided to the individuals prior to the collection of any personal data used? Not applicable
If personal data was collected for the development of the model, was it collected directly by NVIDIA? Not applicable
Is there dataset provenance? Yes
If data is labeled, was it reviewed to comply with privacy laws? Yes
Is data compliant with data subject requests for data correction or removal, if such a request was made? No, not possible with externally-sourced data
Is a mechanism in place to honor data subject rights of access or deletion of personal data? No
How often is the training dataset reviewed?: Before Release

Safety & Security

Use Case Restrictions:

Non-streaming ASR model
Model outputs text in Arabic without diacritical marks
Output text requires Inverse Text Normalization
Model is noise-sensitive
Model can have poor performance in dialectal Arabic speech

Model is not applicable for life-critical applications.

Access Reactions:

The Principle of Least Privilege (PoLP) is applied limiting access for dataset generation and model development. Restrictions enforce dataset access during training and dataset license constraints adhered to.

NVIDIA Riva: Deployment

NVIDIA Riva is an accelerated speech AI SDK deployable on-prem, in all clouds, multi-cloud, hybrid, on edge, and embedded. Additionally, Riva provides:

World-class out-of-the-box accuracy for the most common languages with model checkpoints trained on proprietary data with hundreds of thousands of GPU-compute hours
Best in class accuracy with run-time word boosting (e.g., brand and product names) and customization of acoustic model, language model, and inverse text normalization
Streaming speech recognition, Kubernetes compatible scaling, and enterprise-grade support

Although this model isn’t supported yet by Riva, the list of supported models is here.
Check out Riva live demo.