Bartowski's picture

Bartowski PRO

bartowski

·

AI & ML interests

Official model curator for https://lmstudio.ai/

Recent Activity

replied to their post about 5 hours ago

Looks like Q4_0_N_M file types are going away Before you panic, there's a new "preferred" method which is online (I prefer the term on-the-fly) repacking, so if you download Q4_0 and your setup can benefit from repacking the weights into interleaved rows (what Q4_0_4_4 was doing), it will do that automatically and give you similar performance (minor losses I think due to using intrinsics instead of assembly, but intrinsics are more maintainable) You can see the reference PR here: https://github.com/ggerganov/llama.cpp/pull/10446 So if you update your llama.cpp past that point, you won't be able to run Q4_0_4_4 (unless they add backwards compatibility back), but Q4_0 should be the same speeds (though it may currently be bugged on some platforms) As such, I'll stop making those newer model formats soon, probably end of this week unless something changes, but you should be safe to download and Q4_0 quants and use those ! Also IQ4_NL supports repacking though not in as many shapes yet, but should get a respectable speed up on ARM chips, PR for that can be found here: https://github.com/ggerganov/llama.cpp/pull/10541 Remember, these are not meant for Apple silicon since those use the GPU and don't benefit from the repacking of weights

new activity about 13 hours ago

Qwen/QVQ-72B-Preview:GGUF weights?

updated a model about 22 hours ago

bartowski/QVQ-72B-Preview-GGUF

View all activity

Organizations

bartowski's activity

New activity in Qwen/QVQ-72B-Preview about 13 hours ago

GGUF weights?

#1 opened 1 day ago by

New activity in bartowski/Sailor2-20B-Chat-GGUF 2 days ago

Question on target-language quantization

#1 opened 6 days ago by

New activity in Daemontatox/RA_Reasoner 2 days ago

Quant_Request

#2 opened 5 days ago by

New activity in bartowski/Qwen2-VL-2B-Instruct-GGUF 2 days ago

Question about the example run command

#3 opened 4 days ago by

New activity in bartowski/Llama-3_1-Nemotron-51B-Instruct-GGUF 2 days ago

Is it possible?

#1 opened 2 days ago by

New activity in mradermacher/model_requests 2 days ago

Request to bring back Q4_1

#299 opened 3 months ago by

New activity in meta-llama/Llama-3.3-70B-Instruct 3 days ago

Llama 3.1 lumi maid crashing Kobold

#29 opened 14 days ago by

New activity in bartowski/Falcon3-7B-Instruct-GGUF 3 days ago

Tokenizer issues

#1 opened 8 days ago by

New activity in bartowski/Ava-1.0-12B-GGUF 3 days ago

Extra `<s>` in EOG and EOS.

#1 opened 7 days ago by

New activity in bartowski/Qwen2.5-Coder-14B-GGUF 4 days ago

qwen2.5-coder-14b-instruct@iq4_xs Quantized Model Fails to Follow Instructions on LMStudio

#1 opened 4 days ago by

New activity in bartowski/Llama-3.3-70B-Instruct-GGUF 4 days ago

Problem in running with vllm

#4 opened 4 days ago by

New activity in lmstudio-community/DeepSeek-V2.5-1210-GGUF 6 days ago

"Supports a context length of 160k through yarn settings."

#1 opened 6 days ago by

New activity in sthenno/miscii-1218-ckpt350 7 days ago

Quant Request

#1 opened 7 days ago by

New activity in bartowski/Qwen2-VL-2B-Instruct-GGUF 9 days ago

Upload mmproj-Qwen2-VL-2B-Instruct-f16.gguf

#1 opened 10 days ago by

New activity in huihui-ai/c4ai-command-r7b-12-2024-abliterated 9 days ago

GGUF

#1 opened 9 days ago by

New activity in bartowski/Qwen2-VL-2B-Instruct-GGUF 9 days ago

"Error: llama runner process has terminated", when running "ollama run"

#2 opened 9 days ago by

New activity in bartowski/gemma-2-9b-it-GGUF 10 days ago

quant

#13 opened 13 days ago by

New activity in Qwen/Qwen2-VL-2B-Instruct 10 days ago

gguf

#4 opened 4 months ago by

New activity in bartowski/Llama-3.3-70B-Instruct-GGUF 10 days ago

Missing Tensors in Q5_K_S + Q4_K_M

#3 opened 13 days ago by

New activity in bartowski/openbuddy-nemotron-70b-v23.2q-131k-GGUF 12 days ago

Quant Request

#1 opened 12 days ago by