Article
KV Caching Explained: Optimizing Transformer Inference Efficiency
By
•
•
36sometimesanotion PRO
sometimesanotion
AI & ML interests
Agentic LLM services, model merging, finetunes, distillation
Recent Activity
new activity
5 days ago
wanlige/li-14b-v0.4-slerp0.1:Fusion vs. SLERP?
updated
a model
5 days ago
sometimesanotion/Lamarck-14B-v0.7
replied to
their
post
6 days ago
I have tracked down a blocker preventing Lamarck releases to a della_linear bug in newer mergekit versions.
If you use slices in della_linear merges that have multiple models - as you'd expect of a merge! - an attempt to load the output model in torch will get you:
```
ValueError: Trying to set a tensor of shape torch.Size([1, 5120]) in "weight" (which has shape torch.Size([5120])), this looks incorrect.
```
This strategy was key to Lamarck v0.6 and v0.7's success. Their merge recipes haven't been working with newer mergekits.
These work:
```yaml
models:
- model: sometimesanotion/Qwen2.5-14B-Vimarckoso-v3
- model: sthenno-com/miscii-14b-0218
```
```yaml
slices:
- sources:
- { layer_range: [ 0, 2 ], model: sometimesanotion/Qwen2.5-14B-Vimarckoso-v3 }
- sources:
- { layer_range: [ 2, 6 ], model: sometimesanotion/Qwen2.5-14B-Vimarckoso-v3 }
```
This does not:
```yaml
slices:
- sources:
- { layer_range: [ 0, 2 ], model: sometimesanotion/Qwen2.5-14B-Vimarckoso-v3 }
- { layer_range: [ 0, 2 ], model: sthenno-com/miscii-14b-0218 }
- sources:
- { layer_range: [ 2, 6 ], model: sometimesanotion/Qwen2.5-14B-Vimarckoso-v3 }
- { layer_range: [ 2, 6 ], model: sthenno-com/miscii-14b-0218 }
```
@Crystalcareai, do you know of any work on this? Will @arcee-ai need a detailed report? These della_linear recipes used to work. Overall, thank you for all the cool work, I hope to get this fixed!
Organizations
sometimesanotion's activity
upvoted
an
article
about 1 month ago
upvoted
a
collection
about 2 months ago
upvoted
a
paper
3 months ago
upvoted
a
paper
4 months ago