Would you be willing to do this on a smaller model?
As above, for people like myself who cannot seem to get Llamacpp working on Windows? Say on Mistral small 22b, Qwen 32b or similar around the 20-35b range?
Cheers
Fancy seeing you here ;D
Fancy seeing you here ;D
Hehehe
I do plan to try other models, yeah.
Qwen 32b
I actually gave Qwen-2.5-72b a try, but the result wasn't so good. I suspect the Qwen models haven't been trained on as many books as Mistral and Command-R. So the resulting model couldn't easily write in the style of .
Mistral small 22b
I'm actually working on a 22b at the moment, experimental/custom architecture. If it's coherent enough, I'll release it.
Other than that I was planning to try gemma2-27b as it can be cohered into writing pretty well, though it only has an 8192 context.
cannot seem to get Llamacpp working on Windows
This should be fixable by the way. I haven't got a windows machine myself, but I'm pretty sure llama.cpp can run on pretty much anything. There were some big changes with llama.cpp earlier this month where they deprecated make, and it was a bit unstable for a week after that. But I'm betting the latest main branch will build fine with cmake
I do plan to try other models, yeah.
Qwen 32b
I actually gave Qwen-2.5-72b a try, but the result wasn't so good. I suspect the Qwen models haven't been trained on as many books as Mistral and Command-R. So the resulting model couldn't easily write in the style of .
Mistral small 22b
I'm actually working on a 22b at the moment, experimental/custom architecture. If it's coherent enough, I'll release it.
Other than that I was planning to try gemma2-27b as it can be cohered into writing pretty well, though it only has an 8192 context.
cannot seem to get Llamacpp working on Windows
This should be fixable by the way. I haven't got a windows machine myself, but I'm pretty sure llama.cpp can run on pretty much anything. There were some big changes with llama.cpp earlier this month where they deprecated
make, and it was a bit unstable for a week after that. But I'm betting the latest main branch will build fine withcmake
That sounds great I look forward to seeing whatever you release.
I tried a few months ago and had some luck with LlamaCPP In that I got it to load a model but I just couldn't get it to work with CUDA/CuBLAS no matter what I did. I'll be honest though I am really strapped for time having 2 young kids and working full time. But I'll give it another gom if you know of any decent guides out there (unlike as you don't use Windows) let me know.
Anyway Merry Christmas!