What is the system compatibility to run ise-uiuc/Magicoder-S-DS-6.7B model in our local machine?
#7
by
aravindasagarvg
- opened
I would like to run ise-uiuc/Magicoder-S-DS-6.7B Model in local machine. Do anyone have know the system compatibility to run the model locally?
During inference, the raw model will roughly occupy <= 20GB GPU memory with bfloat16. Quantization can further relax these constraints. I recommend checking https://huggingface.co./TheBloke/Magicoder-S-DS-6.7B-GGUF. In general, this can run on any system with enough memory.