File size: 3,198 Bytes
a2d7926 65bc207 a2d7926 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 |
---
license: apache-2.0
base_model: SubtleOne/Rombos-Qwen2.5-Writer-32b
library_name: transformers
tags:
- merge
- mergekit
- llama-cpp
- gguf-my-repo
---
# Triangle104/Rombos-Qwen2.5-Writer-32b-Q5_K_S-GGUF
This model was converted to GGUF format from [`SubtleOne/Rombos-Qwen2.5-Writer-32b`](https://huggingface.co./SubtleOne/Rombos-Qwen2.5-Writer-32b) using llama.cpp via the ggml.ai's [GGUF-my-repo](https://huggingface.co./spaces/ggml-org/gguf-my-repo) space.
Refer to the [original model card](https://huggingface.co./SubtleOne/Rombos-Qwen2.5-Writer-32b) for more details on the model.
---
Model details:
-
This model is a merge using Rombos's top-ranked 32b model, based on
Qwen 2.5, and merging three creative writing finetunes. The creative
content is a serious upgrade over the base it started with, and I
enjoyed it in my DnD RPG campaign.
Merge
This is a merge of pre-trained language models created using mergekit.
Merge Method
This model was merged using the DELLA merge method using Rombos-LLM-V2.5-Qwen-32b as a base.
Models Merged
The following models were included in the merge:
EVA-UNIT-01/EVA-Qwen2.5-32B-v0.2
allura-org/Qwen2.5-32b-RP-Ink
nbeerbower/Qwen2.5-Gutenberg-Doppel-32B
Configuration
The following YAML configuration was used to produce this model:
base_model: rombodawg/Rombos-LLM-V2.5-Qwen-32b
parameters:
int8_mask: true
rescale: false
normalize: true
dtype: bfloat16
tokenizer_source: union
merge_method: dare_ties
models:
- model: EVA-UNIT-01/EVA-Qwen2.5-32B-v0.2
parameters:
weight: [0.4]
density: [0.55]
- model: nbeerbower/Qwen2.5-Gutenberg-Doppel-32B
parameters:
weight: [0.3]
density: [0.55]
- model: allura-org/Qwen2.5-32b-RP-Ink
parameters:
weight: [0.4]
density: [0.55]
---
## Use with llama.cpp
Install llama.cpp through brew (works on Mac and Linux)
```bash
brew install llama.cpp
```
Invoke the llama.cpp server or the CLI.
### CLI:
```bash
llama-cli --hf-repo Triangle104/Rombos-Qwen2.5-Writer-32b-Q5_K_S-GGUF --hf-file rombos-qwen2.5-writer-32b-q5_k_s.gguf -p "The meaning to life and the universe is"
```
### Server:
```bash
llama-server --hf-repo Triangle104/Rombos-Qwen2.5-Writer-32b-Q5_K_S-GGUF --hf-file rombos-qwen2.5-writer-32b-q5_k_s.gguf -c 2048
```
Note: You can also use this checkpoint directly through the [usage steps](https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#usage) listed in the Llama.cpp repo as well.
Step 1: Clone llama.cpp from GitHub.
```
git clone https://github.com/ggerganov/llama.cpp
```
Step 2: Move into the llama.cpp folder and build it with `LLAMA_CURL=1` flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).
```
cd llama.cpp && LLAMA_CURL=1 make
```
Step 3: Run inference through the main binary.
```
./llama-cli --hf-repo Triangle104/Rombos-Qwen2.5-Writer-32b-Q5_K_S-GGUF --hf-file rombos-qwen2.5-writer-32b-q5_k_s.gguf -p "The meaning to life and the universe is"
```
or
```
./llama-server --hf-repo Triangle104/Rombos-Qwen2.5-Writer-32b-Q5_K_S-GGUF --hf-file rombos-qwen2.5-writer-32b-q5_k_s.gguf -c 2048
```
|