Streamlining Data Management with Hugging Face and DVC: A Seamless Integration

Community Article Published January 3, 2024

Introduction

In the landscape of machine learning, the marriage of effective data management and seamless integration of tools is pivotal. This article delves into the fusion of Data Version Control (DVC) and the Hugging Face ecosystem, a synergy that transforms how data is managed, accessed, and utilized within projects.

Defining the Integration: DVC and Hugging Face

DVC serves as the backbone for version controlling data and models, enabling reproducibility and collaboration. On the other hand, Hugging Face has established itself as a go-to platform for model sharing, hosting, and facilitating easy access to datasets. The integration between these two powerhouses brings forth a harmonious environment where data from the Hugging Face Hub seamlessly integrates into DVC projects.

Advantages of Integration

Native Hub Support: DVC extends its support to import and download data directly from the Hugging Face Hub. It eliminates the need for additional installations like Git LFS or the Hugging Face CLI, ensuring a hassle-free experience.
Streamlined Setup and Download: Installing DVC via pip and using dvc get simplifies data downloading. This command allows fetching specific files or entire directories from the Hub without cloning the entire repository.
Enhanced Data Handling: The integration goes beyond mere data access. With DVCLive, logging from Hugging Face Transformers becomes possible, facilitating comprehensive experiment tracking and reproducibility.

Code Implementation

DVC seamlessly integrates with Hugging Face, offering streamlined access to Hugging Face Hub's datasets and models while leveraging DVC's robust version control for efficient data management in machine learning projects.

pip install datasets dvc

## Download data
dvc get https://huggingface.co./stabilityai/stable-diffusion-xl-base-1.0 sd_xl_base_1.0.safetensors

### Import data
dvc import https://huggingface.co./stabilityai/stable-diffusion-xl-base-1.0 sd_xl_base_1.0.safetensors

## Load DVC data
from datasets import load_dataset
load_dataset(
    "csv",
    data_files="dvc://workshop/satellite-data/jan_train.csv",
    storage_options={"url": "https://github.com/iterative/dataset-registry.git"}
 )

Output

DatasetDict({
    train: Dataset({
        features: ['id', 'epoch', 'sat_id', 'x', 'y', 'z', 'Vx', 'Vy', 'Vz', 'x_sim', 'y_sim', 'z_sim', 'Vx_sim', 'Vy_sim', 'Vz_sim'],
        num_rows: 503227
    })
})

Conclusion

The collaboration between DVC and Hugging Face brings a new level of efficiency and convenience to the realm of data management in machine learning projects. By facilitating easy access to datasets hosted on the Hugging Face Hub while leveraging the robust version control and data management capabilities of DVC, this integration empowers practitioners with streamlined workflows and enhanced productivity. The seamless interplay between these platforms ensures reproducibility, accessibility, and ease-of-use, marking a significant stride in simplifying complex data handling processes within machine learning endeavors.

“Stay connected and support my work through various platforms:

Medium: You can read my latest articles and insights on Medium at https://medium.com/@andysingal

Paypal: Enjoyed my article? Buy me a coffee! https://paypal.me/alphasingal?country.x=US&locale.x=en_US"

Requests and questions: If you have a project in mind that you’d like me to work on or if you have any questions about the concepts I’ve explained, don’t hesitate to let me know. I’m always looking for new ideas for future Notebooks and I love helping to resolve any doubts you might have.

Resources:

DVC-Hugging-Face

Upvote