Llama 3.2 1B - Synthia-v1.5-II - Redmond - Fine-tuned Model
This model is a fine-tuned version of NousResearch/Llama-3.2-1B on the Synthia-v1.5-II dataset.
Thanks RedmondAI for all the GPU Support!
Model Description
The base model is Llama 3.2 1B, a multilingual large language model developed by Meta. This version has been fine-tuned on the Synthia-v1.5-II instruction dataset to improve its instruction-following capabilities.
Training Data
The model was fine-tuned on Synthia-v1.5-II.
Training Procedure
The model was trained with the following hyperparameters:
- Learning rate: 2e-05
- Train batch size: 1
- Eval batch size: 1
- Seed: 42
- Gradient accumulation steps: 8
- Total train batch size: 8
- Optimizer: Paged AdamW 8bit (betas=(0.9,0.999), epsilon=1e-08)
- LR scheduler: Cosine with 100 warmup steps
- Number of epochs: 3
Framework Versions
- Transformers 4.46.1
- Pytorch 2.3.1+cu121
- Datasets 3.0.1
- Tokenizers 0.20.3
Intended Use
This model is intended for:
- Instruction following tasks
- Conversational AI applications
- Research and development in natural language processing
Training Infrastructure
The model was trained using the Axolotl framework version 0.5.0.
License
This model is subject to the Llama 3.2 Community License Agreement. Users must comply with all terms and conditions specified in the license.
- Downloads last month
- 32
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.