Llama 3.2 mlp pruned
Collection
Created by pruning the MLP (feedforward) layers, reducing the size of Llama models while improving their performance.
•
6 items
•
Updated
Model Card for oopere/pruned20-llama-1b
This model is a pruned version of the Llama-3.2-1b model, with a parameter reduction of 20% in the MLP Layers. The pruning process aims to enhance computational efficiency while maintaining acceptable performance across specific tasks. This model is not intended to be used directly, but rather to be fine-tuned for specific tasks where it can achieve equal or superior performance compared to fine-tuning the base model for the same task.
Model Details
- Model Type: Pruned version of LLaMA-3.2 using structured pruning
- Original Model: meta-llama/Llama-3.2-1B
- Pruning Method: Structured pruning of MLP layers using importance scores based on absolute maximum weights
- Size Reduction: 13.7% (from 1.24B to 1.07B parameters)
- Architecture: Same as original LLaMA but with reduced MLP layer sizes
- Language(s): Same as original model
- License: Same as original model
- Developed by: Pere Martra
Performance on Standard Benchmarks
| Benchmark | Original Model | Pruned Model | Relative Change |
|---|---|---|---|
| ARC-Easy | 65.19% | 53.03% | -18.7% |
| BoolQ | 64.16% | 62.32% | -2.9% |
| LAMBADA-OpenAI | 62.20% | 42.13% | -32.3% |
| LAMBADA-Standard | 53.46% | 41.04% | -23.2% |
Key Findings
- Maintains strong performance on binary classification tasks (BoolQ)
- Moderate degradation on reasoning tasks (ARC-Easy)
- Significant impact on long-range comprehension (LAMBADA)
Limitations
- Reduced performance on tasks requiring complex language understanding
- More significant degradation on tasks requiring long-range dependencies
- May not be suitable for applications requiring high accuracy on language completion tasks
Implementation Details
- Pruning Notebook: Detailed implementation and methodology
- GitHub Repository: LLM Course
Pruning Method
- Technique: Structured pruning targeting MLP layers
- Pruning Ratio: 20% of neurons removed from MLP layers
- Selection Criteria: Importance scoring based on absolute maximum weights
- Architecture Specifics: Maintained GLU structure during pruning
Hardware Requirements
- Reduced memory footprint compared to original model
- Can run on hardware with ~20% less memory than original
Acknowledgments
- Thanks to Mariusz Kurman for creating llama-pruning, a library that extends and improve this pruning methodology.
- Downloads last month
- 37
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.
Model tree for oopere/pruned20-llama-3.2-1B
Base model
meta-llama/Llama-3.2-1B