40 26 26

Xuan Son NGUYEN

ngxson

https://blog.ngxson.com

AI & ML interests

Doing AI for fun, not for profit

Recent Activity

updated a model about 6 hours ago

ngxson/Qwen2.5-7B-Instruct-1M-Q4_K_M-GGUF

published a model about 6 hours ago

ngxson/Qwen2.5-7B-Instruct-1M-Q4_K_M-GGUF

reacted to mitkox's post with 🚀 1 day ago

llama.cpp is 26.8% faster than ollama. I have upgraded both, and using the same settings, I am running the same DeepSeek R1 Distill 1.5B on the same hardware. It's an Apples to Apples comparison. Total duration: llama.cpp 6.85 sec <- 26.8% faster ollama 8.69 sec Breakdown by phase: Model loading llama.cpp 241 ms <- 2x faster ollama 553 ms Prompt processing llama.cpp 416.04 tokens/s with an eval time 45.67 ms <- 10x faster ollama 42.17 tokens/s with an eval time of 498 ms Token generation llama.cpp 137.79 tokens/s with an eval time 6.62 sec <- 13% faster ollama 122.07 tokens/s with an eval time 7.64 sec llama.cpp is LLM inference in C/C++; ollama adds abstraction layers and marketing. Make sure you own your AI. AI in the cloud is not aligned with you; it's aligned with the company that owns it.

View all activity

Articles

Code a simple RAG from scratch

Oct 29, 2024

• 18

Organizations

Posts 3

Post

495

Fun fact: you can get any DeepSeek-R1-Qwen **abliterated** by using one of these LoRA adapters (GGUF available!)

ngxson/extracted-lora-mergekit-677d5c3eea0b6a7661201846

Post

2089

Check out my collection of pre-made GGUF LoRA adapters!

This allow you to use both normal + abliterated version of popular models like llama, qwen, etc, without having to double to amount of VRAM usage.

ngxson/gguf_lora_collection

View all posts