|
--- |
|
inference: false |
|
license: mit |
|
tags: |
|
- text-generation |
|
- mamba |
|
- long context |
|
--- |
|
|
|
# DeciMamba Checkpoint |
|
|
|
The official checkpoint of DeciMamba-130m, finetuned for Language Modeling over the PG-19 dataset as presented in [DeciMamba: Exploring the Length Extrapolation Potential of Mamba](https://arxiv.org/abs/2406.14528). |
|
|
|
See our [Github Repo](https://github.com/assafbk/DeciMamba) for evalution and training scripts. |
|
|
|
Bibtex: |
|
``` |
|
@misc{benkish2024decimambaexploringlengthextrapolation, |
|
title={DeciMamba: Exploring the Length Extrapolation Potential of Mamba}, |
|
author={Assaf Ben-Kish and Itamar Zimerman and Shady Abu-Hussein and Nadav Cohen and Amir Globerson and Lior Wolf and Raja Giryes}, |
|
year={2024}, |
|
eprint={2406.14528}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.LG}, |
|
url={https://arxiv.org/abs/2406.14528}, |
|
} |
|
``` |
