Create an audio dataset

You can share a dataset with your team or with anyone in the community by creating a dataset repository on the Hugging Face Hub:

from datasets import load_dataset

dataset = load_dataset("<username>/my_dataset")

There are several methods for creating and sharing an audio dataset:

• Create an audio dataset from local files in python with Dataset.push_to_hub(). This is an easy way that requires only a few steps in python.

• Create an audio dataset repository with the AudioFolder builder. This is a no-code solution for quickly creating an audio dataset with several thousand audio files.

Local files

You can load your own dataset using the paths to your audio files. Use the cast_column() function to take a column of audio file paths, and cast it to the Audio feature:

>>> audio_dataset = Dataset.from_dict({"audio": ["path/to/audio_1", "path/to/audio_2", ..., "path/to/audio_n"]}).cast_column("audio", Audio())
>>> audio_dataset[0]["audio"]
{'array': array([ 0.        ,  0.00024414, -0.00024414, ..., -0.00024414,
         0.        ,  0.        ], dtype=float32),
 'path': 'path/to/audio_1',
 'sampling_rate': 16000}

Then upload the dataset to the Hugging Face Hub using Dataset.push_to_hub():

audio_dataset.push_to_hub("<username>/my_dataset")

This will create a dataset repository containing your audio dataset:

my_dataset/
├── README.md
└── data/
    └── train-00000-of-00001.parquet

AudioFolder

The AudioFolder is a dataset builder designed to quickly load an audio dataset with several thousand audio files without requiring you to write any code.

AudioFolder automatically infers the class labels of your dataset based on the directory name. Store your dataset in a directory structure like:

folder/train/dog/golden_retriever.mp3
folder/train/dog/german_shepherd.mp3
folder/train/dog/chihuahua.mp3

folder/train/cat/maine_coon.mp3
folder/train/cat/bengal.mp3
folder/train/cat/birman.mp3

If the dataset follows the AudioFolder structure, then you can load it directly with load_dataset():

>>> from datasets import load_dataset

>>> dataset = load_dataset("username/dataset_name")

This is equivalent to passing audiofolder manually in load_dataset() and the directory in data_dir:

>>> dataset = load_dataset("audiofolder", data_dir="/path/to/folder")

You can also use audiofolder to load datasets involving multiple splits. To do so, your dataset directory should have the following structure:

folder/train/dog/golden_retriever.mp3
folder/train/cat/maine_coon.mp3
folder/test/dog/german_shepherd.mp3
folder/test/cat/bengal.mp3

If there is additional information you’d like to include about your dataset, like text captions or bounding boxes, add it as a metadata.csv file in your folder. This lets you quickly create datasets for different computer vision tasks like text captioning or object detection. You can also use a JSONL file metadata.jsonl or a Parquet file metadata.parquet.

folder/train/metadata.csv
folder/train/0001.mp3
folder/train/0002.mp3
folder/train/0003.mp3

You can also zip your audio files, and in this case each zip should contain both the audio files and the metadata

folder/train.zip
folder/test.zip
folder/validation.zip

Your metadata.csv file must have a file_name or *_file_name field which links audio files with their metadata:

file_name,additional_feature
0001.mp3,This is a first value of a text feature you added to your audio files
0002.mp3,This is a second value of a text feature you added to your audio files
0003.mp3,This is a third value of a text feature you added to your audio files

or using metadata.jsonl:

{"file_name": "0001.mp3", "additional_feature": "This is a first value of a text feature you added to your audio files"}
{"file_name": "0002.mp3", "additional_feature": "This is a second value of a text feature you added to your audio files"}
{"file_name": "0003.mp3", "additional_feature": "This is a third value of a text feature you added to your audio files"}

Here the file_name must be the name of the audio file next to the metadata file. More generally, it must be the relative path from the directory containing the metadata to the audio file.

It’s possible to point to more than one audio in each row in your dataset, for example if both your input and output are audio files:

{"input_file_name": "0001.mp3", "output_file_name": "0001_output.mp3"}
{"input_file_name": "0002.mp3", "output_file_name": "0002_output.mp3"}
{"input_file_name": "0003.mp3", "output_file_name": "0003_output.mp3"}

You can also define lists of audio files. In that case you need to name the field file_names or *_file_names. Here is an example:

{"recordings_file_names": ["0001_r0.mp3", "0001_r1.mp3"], label: "same_person"}
{"recordings_file_names": ["0002_r0.mp3", "0002_r1.mp3"], label: "same_person"}
{"recordings_file_names": ["0003_r0.mp3", "0003_r1.mp3"], label: "different_person"}

WebDataset

The WebDataset format is based on TAR archives and is suitable for big audio datasets. Indeed you can group your audio files in TAR archives (e.g. 1GB of audio files per TAR archive) and have thousands of TAR archives:

folder/train/00000.tar
folder/train/00001.tar
folder/train/00002.tar
...

In the archives, each example is made of files sharing the same prefix:

e39871fd9fd74f55.mp3
e39871fd9fd74f55.json
f18b91585c4d3f3e.mp3
f18b91585c4d3f3e.json
ede6e66b2fb59aab.mp3
ede6e66b2fb59aab.json
ed600d57fcee4f94.mp3
ed600d57fcee4f94.json
...

You can put your audio files labels/captions/bounding boxes using JSON or text files for example.

Load your WebDataset and it will create on column per file suffix (here “mp3” and “json”):

>>> from datasets import load_dataset

>>> dataset = load_dataset("webdataset", data_dir="/path/to/folder", split="train")
>>> dataset[0]["json"]
{"transcript": "Hello there !", "speaker": "Obi-Wan Kenobi"}

It’s also possible to have several audio files per example like this:

e39871fd9fd74f55.input.mp3
e39871fd9fd74f55.output.mp3
e39871fd9fd74f55.json
f18b91585c4d3f3e.input.mp3
f18b91585c4d3f3e.output.mp3
f18b91585c4d3f3e.json
...

For more details on the WebDataset format and the python library, please check the WebDataset documentation.

(Legacy) Loading script

Write a dataset loading script to manually create a dataset. It defines a dataset’s splits and configurations, and handles downloading and generating the dataset examples. The script should have the same name as your dataset folder or repository:

my_dataset/
├── README.md
├── my_dataset.py
└── data/

The data folder can be any name you want, it doesn’t have to be data. This folder is optional, unless you’re hosting your dataset on the Hub.

This directory structure allows your dataset to be loaded in one line:

>>> from datasets import load_dataset
>>> dataset = load_dataset("path/to/my_dataset")

This guide will show you how to create a dataset loading script for audio datasets, which is a bit different from creating a loading script for text datasets. Audio datasets are commonly stored in tar.gz archives which requires a particular approach to support streaming mode. While streaming is not required, we highly encourage implementing streaming support in your audio dataset because users without a lot of disk space can use your dataset without downloading it. Learn more about streaming in the Stream guide!

Here is an example using TAR archives:

my_dataset/
├── README.md
├── my_dataset.py
└── data/
    ├── train.tar.gz
    ├── test.tar.gz
    └── metadata.csv

In addition to learning how to create a streamable dataset, you’ll also learn how to:

• Create a dataset builder class.
• Create dataset configurations.
• Add dataset metadata.
• Download and define the dataset splits.
• Generate the dataset.
• Upload the dataset to the Hub.

The best way to learn is to open up an existing audio dataset loading script, like Vivos, and follow along!

Create a dataset builder class

GeneratorBasedBuilder is the base class for datasets generated from a dictionary generator. Within this class, there are three methods to help create your dataset:

• _info stores information about your dataset like its description, license, and features.
• _split_generators downloads the dataset and defines its splits.
• _generate_examples generates the dataset’s samples containing the audio data and other features specified in info for each split.

Start by creating your dataset class as a subclass of GeneratorBasedBuilder and add the three methods. Don’t worry about filling in each of these methods yet, you’ll develop those over the next few sections:

class VivosDataset(datasets.GeneratorBasedBuilder):
    """VIVOS is a free Vietnamese speech corpus consisting of 15 hours of recording speech prepared for
    Vietnamese Automatic Speech Recognition task."""

    def _info(self):

    def _split_generators(self, dl_manager):

    def _generate_examples(self, prompts_path, path_to_clips, audio_files):

Multiple configurations

In some cases, a dataset may have more than one configuration. For example, LibriVox Indonesia dataset has several configurations corresponding to different languages.

To create different configurations, use the BuilderConfig class to create a subclass of your dataset. The only required parameter is the name of the configuration, which must be passed to the configuration’s superclass __init__(). Otherwise, you can specify any custom parameters you want in your configuration class.

class LibriVoxIndonesiaConfig(datasets.BuilderConfig):
    """BuilderConfig for LibriVoxIndonesia."""

    def __init__(self, name, version, **kwargs):
        self.language = kwargs.pop("language", None)
        self.release_date = kwargs.pop("release_date", None)
        self.num_clips = kwargs.pop("num_clips", None)
        self.num_speakers = kwargs.pop("num_speakers", None)
        self.validated_hr = kwargs.pop("validated_hr", None)
        self.total_hr = kwargs.pop("total_hr", None)
        self.size_bytes = kwargs.pop("size_bytes", None)
        self.size_human = size_str(self.size_bytes)
        description = (
            f"LibriVox-Indonesia speech to text dataset in {self.language} released on {self.release_date}. "
            f"The dataset comprises {self.validated_hr} hours of transcribed speech data"
        )
        super(LibriVoxIndonesiaConfig, self).__init__(
            name=name,
            version=datasets.Version(version),
            description=description,
            **kwargs,
        )

Define your configurations in the BUILDER_CONFIGS class variable inside GeneratorBasedBuilder. In this example, the author imports the languages from a separate release_stats.py file from their repository, and then loops through each language to create a configuration:

class LibriVoxIndonesia(datasets.GeneratorBasedBuilder):
    DEFAULT_CONFIG_NAME = "all"

    BUILDER_CONFIGS = [
        LibriVoxIndonesiaConfig(
            name=lang,
            version=STATS["version"],
            language=LANGUAGES[lang],
            release_date=STATS["date"],
            num_clips=lang_stats["clips"],
            num_speakers=lang_stats["users"],
            total_hr=float(lang_stats["totalHrs"]) if lang_stats["totalHrs"] else None,
            size_bytes=int(lang_stats["size"]) if lang_stats["size"] else None,
        )
        for lang, lang_stats in STATS["locales"].items()
    ]

Now if users want to load the Balinese (bal) configuration, they can use the configuration name:

>>> from datasets import load_dataset
>>> dataset = load_dataset("indonesian-nlp/librivox-indonesia", "bal", split="train")

Add dataset metadata

Adding information about your dataset helps users to learn more about it. This information is stored in the DatasetInfo class which is returned by the info method. Users can access this information by:

>>> from datasets import load_dataset_builder
>>> ds_builder = load_dataset_builder("vivos")
>>> ds_builder.info

There is a lot of information you can include about your dataset, but some important ones are:

1. description provides a concise description of the dataset.
2. features specify the dataset column types. Since you’re creating an audio loading script, you’ll need to include the Audio feature and the sampling_rate of the dataset.
3. homepage provides a link to the dataset homepage.
4. license specify the permissions for using a dataset as defined by the license type.
5. citation is a BibTeX citation of the dataset.

def _info(self):
    return datasets.DatasetInfo(
        description=_DESCRIPTION,
        features=datasets.Features(
            {
                "speaker_id": datasets.Value("string"),
                "path": datasets.Value("string"),
                "audio": datasets.Audio(sampling_rate=16_000),
                "sentence": datasets.Value("string"),
            }
        ),
        supervised_keys=None,
        homepage=_HOMEPAGE,
        license=_LICENSE,
        citation=_CITATION,
    )

Download and define the dataset splits

Now that you’ve added some information about your dataset, the next step is to download the dataset and define the splits.

1. Use the download() method to download metadata file at _PROMPTS_URLS and audio TAR archive at _DATA_URL. This method returns the path to the local file/archive. In streaming mode, it doesn’t download the file(s) and just returns a URL to stream the data from. This method accepts:

   • a relative path to a file inside a Hub dataset repository (for example, in the data/ folder)
   • a URL to a file hosted somewhere else
   • a (nested) list or dictionary of file names or URLs

2. After you’ve downloaded the dataset, use the SplitGenerator to organize the audio files and sentence prompts in each split. Name each split with a standard name like: Split.TRAIN, Split.TEST, and SPLIT.Validation.

   In the gen_kwargs parameter, specify the file path to the prompts_path and path_to_clips. For audio_files, you’ll need to use iter_archive() to iterate over the audio files in the TAR archive. This enables streaming for your dataset. All of these file paths are passed onto the next step where you’ll actually generate the dataset.

def _split_generators(self, dl_manager):
    """Returns SplitGenerators."""
    prompts_paths = dl_manager.download(_PROMPTS_URLS)
    archive = dl_manager.download(_DATA_URL)
    train_dir = "vivos/train"
    test_dir = "vivos/test"

    return [
        datasets.SplitGenerator(
            name=datasets.Split.TRAIN,
            gen_kwargs={
                "prompts_path": prompts_paths["train"],
                "path_to_clips": train_dir + "/waves",
                "audio_files": dl_manager.iter_archive(archive),
            },
        ),
        datasets.SplitGenerator(
            name=datasets.Split.TEST,
            gen_kwargs={
                "prompts_path": prompts_paths["test"],
                "path_to_clips": test_dir + "/waves",
                "audio_files": dl_manager.iter_archive(archive),
            },
        ),
    ]

Generate the dataset

The last method in the GeneratorBasedBuilder class actually generates the samples in the dataset. It yields a dataset according to the structure specified in features from the info method. As you can see, generate_examples accepts the prompts_path, path_to_clips, and audio_files from the previous method as arguments.

Files inside TAR archives are accessed and yielded sequentially. This means you need to have the metadata associated with the audio files in the TAR file in hand first so you can yield it with its corresponding audio file.

examples = {}
with open(prompts_path, encoding="utf-8") as f:
    for row in f:
        data = row.strip().split(" ", 1)
        speaker_id = data[0].split("_")[0]
        audio_path = "/".join([path_to_clips, speaker_id, data[0] + ".wav"])
        examples[audio_path] = {
            "speaker_id": speaker_id,
            "path": audio_path,
            "sentence": data[1],
        }

Finally, iterate over files in audio_files and yield them along with their corresponding metadata. iter_archive() yields a tuple of (path, f) where path is a relative path to a file inside TAR archive and f is a file object itself.

inside_clips_dir = False
id_ = 0
for path, f in audio_files:
    if path.startswith(path_to_clips):
        inside_clips_dir = True
        if path in examples:
            audio = {"path": path, "bytes": f.read()}
            yield id_, {**examples[path], "audio": audio}
            id_ += 1
    elif inside_clips_dir:
        break

Put these two steps together, and the whole _generate_examples method looks like:

def _generate_examples(self, prompts_path, path_to_clips, audio_files):
    """Yields examples as (key, example) tuples."""
    examples = {}
    with open(prompts_path, encoding="utf-8") as f:
        for row in f:
            data = row.strip().split(" ", 1)
            speaker_id = data[0].split("_")[0]
            audio_path = "/".join([path_to_clips, speaker_id, data[0] + ".wav"])
            examples[audio_path] = {
                "speaker_id": speaker_id,
                "path": audio_path,
                "sentence": data[1],
            }
    inside_clips_dir = False
    id_ = 0
    for path, f in audio_files:
        if path.startswith(path_to_clips):
            inside_clips_dir = True
            if path in examples:
                audio = {"path": path, "bytes": f.read()}
                yield id_, {**examples[path], "audio": audio}
                id_ += 1
        elif inside_clips_dir:
            break

Upload the dataset to the Hub

Once your script is ready, create a dataset card and upload it to the Hub.

Congratulations, you can now load your dataset from the Hub! 🥳

>>> from datasets import load_dataset
>>> load_dataset("<username>/my_dataset")

(Advanced) Extract TAR archives locally

In the example above downloaded archives are not extracted and therefore examples do not contain information about where they are stored locally. To explain how to do the extraction in a way that it also supports streaming, we will briefly go through the LibriVox Indonesia loading script.

Download and define the dataset splits

1. Use the download() method to download the audio data at _AUDIO_URL.

2. To extract audio TAR archive locally, use the extract(). You can use this method only in non-streaming mode (when dl_manager.is_streaming=False). This returns a local path to the extracted archive directory:

   Copied

   local_extracted_archive = dl_manager.extract(audio_path) if not dl_manager.is_streaming else None

3. Use the iter_archive() method to iterate over the archive at audio_path, just like in the Vivos example above. iter_archive() doesn’t provide any information about the full paths of files from the archive, even if it has been extracted. As a result, you need to pass the local_extracted_archive path to the next step in gen_kwargs, in order to preserve information about where the archive was extracted to. This is required to construct the correct paths to the local files when you generate the examples.

4. Use the download_and_extract() method to download the metadata file specified in _METADATA_URL. This method returns a path to a local file in non-streaming mode. In streaming mode, it doesn’t download file locally and returns the same URL.

5. Now use the SplitGenerator to organize the audio files and metadata in each split. Name each split with a standard name like: Split.TRAIN, Split.TEST, and SPLIT.Validation.

   In the gen_kwargs parameter, specify the file paths to local_extracted_archive, audio_files, metadata_path, and path_to_clips. Remember, for audio_files, you need to use iter_archive() to iterate over the audio files in the TAR archives. This enables streaming for your dataset! All of these file paths are passed onto the next step where the dataset samples are generated.

def _split_generators(self, dl_manager):
    """Returns SplitGenerators."""
    audio_path = dl_manager.download(_AUDIO_URL)
    local_extracted_archive = dl_manager.extract(audio_path) if not dl_manager.is_streaming else None
    path_to_clips = "librivox-indonesia"

    return [
        datasets.SplitGenerator(
            name=datasets.Split.TRAIN,
            gen_kwargs={
                "local_extracted_archive": local_extracted_archive,
                "audio_files": dl_manager.iter_archive(audio_path),
                "metadata_path": dl_manager.download_and_extract(_METADATA_URL + "/metadata_train.csv.gz"),
                "path_to_clips": path_to_clips,
            },
        ),
        datasets.SplitGenerator(
            name=datasets.Split.TEST,
            gen_kwargs={
                "local_extracted_archive": local_extracted_archive,
                "audio_files": dl_manager.iter_archive(audio_path),
                "metadata_path": dl_manager.download_and_extract(_METADATA_URL + "/metadata_test.csv.gz"),
                "path_to_clips": path_to_clips,
            },
        ),
    ]

Generate the dataset

Here _generate_examples accepts local_extracted_archive, audio_files, metadata_path, and path_to_clips from the previous method as arguments.

1. TAR files are accessed and yielded sequentially. This means you need to have the metadata in metadata_path associated with the audio files in the TAR file in hand first so that you can yield it with its corresponding audio file further:

   Copied

   with open(metadata_path, "r", encoding="utf-8") as f:
       reader = csv.DictReader(f)
       for row in reader:
           if self.config.name == "all" or self.config.name == row["language"]:
               row["path"] = os.path.join(path_to_clips, row["path"])
               # if data is incomplete, fill with empty values
               for field in data_fields:
                   if field not in row:
                       row[field] = ""
               metadata[row["path"]] = row

2. Now you can yield the files in audio_files archive. When you use iter_archive(), it yielded a tuple of (path, f) where path is a relative path to a file inside the archive, and f is the file object itself. To get the full path to the locally extracted file, join the path of the directory (local_extracted_path) where the archive is extracted to and the relative audio file path (path):

   Copied

   for path, f in audio_files:
       if path in metadata:
           result = dict(metadata[path])
           # set the audio feature and the path to the extracted file
           path = os.path.join(local_extracted_archive, path) if local_extracted_archive else path
           result["audio"] = {"path": path, "bytes": f.read()}
           result["path"] = path
           yield id_, result
           id_ += 1

Put both of these steps together, and the whole _generate_examples method should look like:

def _generate_examples(
        self,
        local_extracted_archive,
        audio_files,
        metadata_path,
        path_to_clips,
    ):
        """Yields examples."""
        data_fields = list(self._info().features.keys())
        metadata = {}
        with open(metadata_path, "r", encoding="utf-8") as f:
            reader = csv.DictReader(f)
            for row in reader:
                if self.config.name == "all" or self.config.name == row["language"]:
                    row["path"] = os.path.join(path_to_clips, row["path"])
                    # if data is incomplete, fill with empty values
                    for field in data_fields:
                        if field not in row:
                            row[field] = ""
                    metadata[row["path"]] = row
        id_ = 0
        for path, f in audio_files:
            if path in metadata:
                result = dict(metadata[path])
                # set the audio feature and the path to the extracted file
                path = os.path.join(local_extracted_archive, path) if local_extracted_archive else path
                result["audio"] = {"path": path, "bytes": f.read()}
                result["path"] = path
                yield id_, result
                id_ += 1