Hugging Face's logo

Blackroot
/
Mirai-3.0-70B

Safetensors
llama
roleplay
experimental
indie
Merge
Model card Files Community
Mirai-3.0-70B / README.md
Blackroot's picture
Blackroot
Update README.md
4e00329 verified
preview code
|
raw
history blame contribute delete
13.9 kB
metadata
base_model:
  - huihui-ai/Llama-3.1-Nemotron-70B-Instruct-HF-abliterated
  - Sao10K/L3.3-70B-Euryale-v2.3
  - yentinglin/Llama-3-Taiwan-70B-Instruct
  - PKU-Baichuan-MLSystemLab/Llama3-PBM-Nova-70B
  - rinna/llama-3-youko-70B
  - Bllossom/llama-3-Korean-Bllossom-70B
  - hitachi-nlp/Llama-3.1-70B-FLDx2
  - tokyotech-llm/Llama-3.1-Swallow-70B-v0.1
  - PKU-Alignment/ProgressGym-HistLlama3-70B-C018-pretrain-v0.1
  - PKU-Alignment/ProgressGym-HistLlama3-70B-C013-pretrain-v0.1
  - PKU-Alignment/ProgressGym-HistLlama3-70B-C020-pretrain-v0.1
  - PKU-Alignment/ProgressGym-HistLlama3-70B-C017-pretrain-v0.1
  - PKU-Alignment/ProgressGym-HistLlama3-70B-C014-pretrain-v0.1
  - PKU-Alignment/ProgressGym-HistLlama3-70B-C021-pretrain-v0.1
tags:
  - roleplay
  - experimental
  - indie
  - merge

Mascot Discord

New and different, wow! This one actually expects you to use Llama 3 Instruct format. There's lots to talk about here, and this model is TOTALLY different than previous version for a variety of reasons.

This is evolution 1. Yes, I know it makes no sense, see long rant for details.

Model Architecture

This is a stockmerge and TIES model. Thanks mergekit for making this really easy to do.

Stock for the "True Merge" -- This was a TIES Merge, the reasoning is explained below for using TIES over Model Stock this time. Although, model stock was also used.

  • PKU-Baichuan-MLSystemLab/Llama3-PBM-Nova-70B
  • yentinglin/Llama-3-Taiwan-70B-Instruct
  • Sao10K/L3.3-70B-Euryale-v2.3
  • (Custom Base Model-Stock Soup -- Recipe Below)

Base

  • huihui-ai/Llama-3.1-Nemotron-70B-Instruct-HF-abliterated

One note here, I wasn't really sure how to state this in the huggingface tags. This model is actually THREE different merges. There's a base history merge, which was rolled into a base model merge, and you can see we merged the bases with our instruct models. Whew. I tried to give a thorough overview of model contributions, but not all of them contribute to the "final" merge directly.

Why a different approach?

As some users had noted, particularly thanks to |GodZio| and The-Istar, the previous Mirai's instruct format was very unclear. Infact, when testing Llama-3 instruct format it seemed just broken, and, it was. Why? Well, the issue was with merging multiple models with different stopping tokens. I'll leave a technical explanation below for my assumption about why this happened. The long story short, I changed strategies for this model. It's very different, and expects the Llama-3 format to be used.

Possible cause of the issue (Technical)

Llama-3 instruct alone has two distinct EOS tokens, and models like Hermes have it's own EOS. What appears to have happened is that ALL of the EOS tokens basically lost weighting because the weights spread around different EOS tokens. And, this meant that the model did not know which EOS token to produce. Merge enough different models like this, and you end up with no EOS token ever being generated. There's other issues at play, hermes has a different number of tokens, so the hermes EOS does not actually make it into the merge at all, meaning models like Hermes effectively erase the EOS when merging against smaller heads. The puzzling part of this is why Llama-3 format was apparently so disproportionately affected by the merge. I don't have a clear answer for this at all.

New strategy (Still Evolutionary)

Alright. We're still doing the evolutionary thing, take the previous generation, and either add or remove a model. However, this version is a bit of a rollercoaster, I spent lots of time with modelstock trying to ensure I could preserve the EOS token, and it essentially boiled down to only leaving a handful of models with a very sub-par result. I really was not a fan. So much so, I changed directions.

Base Folding

We have competing objectives for the model. We want model diversity for interesting storytelling. That's really great. However, increasing model diversity also increases EOS diversity. This is a problem. We also want the model to be able to shut up when it wants to. There's a problem, modelstock is using geometric interpolation. In the case of lots of EOS tokens, it's going to act a bit like averaging them. This is a huge problem. We can't disagree about which EOS to use. Hence, we're going to fold the bases together, which all have the same EOS tokens. Specifically, we're going to do a modelstock merge like this:

Model Stock:

  • Bllossom/llama-3-Korean-Bllossom-70B
  • rinna/llama-3-youko-70B
  • hitachi-nlp/Llama-3.1-70B-FLDx2
  • (MergedHistLlama -- A custom model stock merge itself of multiple See below)
  • tokyotech-llm/Llama-3.1-Swallow-70B-v0.1

Base model:

  • meta-llama/Meta-Llama-3-70B

All of these were folded into the base model, and they critically all agreed about the EOS token.

What is the MergedHistLlama?

Well, it's more merging, Specifically of six base models from PKU-Alignment. The selection of the particular base model was done at random.

Model Stock:

  • PKU-Alignment/ProgressGym-HistLlama3-70B-C013-pretrain-v0.1
  • PKU-Alignment/ProgressGym-HistLlama3-70B-C020-pretrain-v0.1
  • PKU-Alignment/ProgressGym-HistLlama3-70B-C017-pretrain-v0.1
  • PKU-Alignment/ProgressGym-HistLlama3-70B-C014-pretrain-v0.1
  • PKU-Alignment/ProgressGym-HistLlama3-70B-C021-pretrain-v0.1

Base Model:

  • PKU-Alignment/ProgressGym-HistLlama3-70B-C018-pretrain-v0.1

Model Stock is Really bad at this EOS thing

I love the interest and diversity of model stock, but after trying many different merges, I realized that I had to try something else. Specifically, TIES. Model stock and TIES are almost polar opposites. TIES acts as an amplifier, when models agree, the task vectors align, TIES strengthens those underlying weights, this means things that are good about the model get amplified, things that are bad get amplified. Model stock smoothes things out, smoothing the weights out between models. If smoothing is an issue, lets try amplifying. I've avoided TIES merges, because I'm specifically trying to avoid some of the bad mannerisms of the base ablation nemo model. However, I tried it anyways. Wouldn't you know it, TIES preserved the EOS, and can actually shut up most of the time. Not only that, but the model result is good. Quite good. The instruct is simply better than prior Mirai's and I don't think it's by a small margin either. There's some quirks, but I'm still able to run inference without any penalties and with the same sampling settings I've been running with. This was really surprising to me, I had not anticipated good results with TIES merging, but I'll eat my shoes now, it's good. The model is by no means perfect, there's some edge areas that end up in strange outputs, and the model occasionally will insert phrases that appeared commonly into the end of responses. However, overall, I like the result.

Ties is Really, Really slow, And also Evolutions or Something

Ties takes something like 10-15x longer than model stock. It has to calculate a bunch of fancy vectors and directions, and this is slow. In practice, this means it's even slower for me to iterate on evolutions. Speaking of, which evolution is this? Well, this is where it gets weird, because the previous evolution was 13, but all of these were done as model stock merges. I just decide to switch to TIES out of the blue. This means that basically none of the other evolutions were relevant in the analysis of this one, since changing the merging strategy basically rebases the whole project. Therefore, this is sort of evolution 1, despite having many models incorporated already. I'll be calling it evolution 1, but just know this was the actual reality of the situation.

Merging is Hard to Direct

Now, I never thought I'd be the one saying this. But here I am, merging is effective. It's not a free lunch, but it's certainly edible. There's issues though, and some of these probably can't be solved by merging. Let me go over some thoughts here in the following sections on what materially is not improving when merging.

Reluctance Redirection

This is a phenomenon that is hard to observe if you aren't looking for it, because the models are sneaky. You'll see this in ablations, where instead of direct refusals, models will skirt the point, redirect the conversation, or use doublespeak. These are also known as "soft refusals" where the model does not explicitly state it won't do something, but it will not in fact do the thing. This is an issue, if the model has a learned guardrail that thinks gore and violence are bad, and you're over here sword in hand slaying a dragon, you might want gore in your fairytale. But the model may disagree with the premise of fantasy, and instead you miss. That way the dragon never bleeds. This is the idea with reluctance redirection. After having put in more hours than I care to admit in testing these models, it's quite clear when this happens now. What's more annoying, this behaviour is very hard to get rid of. In fact, I just can't. There's always some degree of reluctance. Now, this is a bit of a problem, because I could entirely ablate these concepts, but that's actually bad. Abliteration (can) cause overcompliance, which I find to be the least human trait in a model. I try to avoid abliteration when possible, and I have not liked the abliteration results I've tested to get rid of reluctance redirection.

Low Coherence Areas

These models have seen an absolutely ABSURD amount of data. Like truly staggering amounts. However, despite seeing a massive amount of data, not all of the data is balanced, so this causes areas of what I'll call "low coherence". When you get into one of these areas, prose collapses and the model basically has very few logits to choose from, so it ends up very boring, similar, and overall, it's not fun. No matter how many models you merge, if they don't have a strong distribution, they will not increase these areas of low coherence.

"GPT-isms"/Model slop

This is more of a well known thing, but I'll address it here too. Models like certain words, and phrases. This might happen as a result of the other two, or simply be over-represented in the weights of the model for specific conditions. Whatever the case, merging does seem to help reduce model slop, but it does not eliminate it. I still see "fingers drumming a stocatto rhythm" at times. Sadly.

Ghost in the Shell

Do models have a personality? What about a superposition of personalities? People will argue, and as much as i love arguing, I'm after a more practical question. How do we elicit roleplay personalities from the model? As it turns out, prompting sort of works sometimes maybe depending on the prompt, and also depending on the scenarios, and blah blah. Here's the thing, personalities are largely inconsistent, and people notice this. Overall, things feel much more like superposition personalities where you're talking to a mish-mash of various people depending on the circumstances. Is this fixable in other ways? Well, maybe, but merging does not seem to elicit more normalized personalities.

Ghost in the Shell Redux

Now, you're going to say, what are you on about? How could you have two sections for this? Here's the thing, some parts of the model's personality are dictated by the dataset. Others however, are from guard rails in the RLHF steps. RLHF much more strongly changes a model's personality (air quotes or whatever), and there's a "true persona" underneath the roleplay that is essentially, guard rail persona. This is the moralizer, the flowery poet, and the ghost in the shell. If we can make any claims about a model's actual personality, it's this one, the one that resulted from directive RLHF, and often comes out to make totally out of place sentiments like "we should respect equality" in the middle of fighting a dragon. These sentiments are totally misplaced, and my own thoughts here are that RLHF guardrails do not transition well to roleplay settings. Luckily, some of this is reduced by merging, notably I find model stock really reduces these tendencies by regularizing across base models. However, then we have the EOS issue. Overall, this is a hard one to get the tradeoffs correct, but maybe you can.

So What?

All these problems, how to fix? Well, fine tuning can help. Particularly, it will love areas of low coherence. But gathering, forming, and just the dataset work alone is really time consuming. I've done small-scale tunes for smaller models, but doing this on a real scale is just a huge problem. Principal among these issues, is that training huge models takes an insane amount of compute, and you'll likely need to try several times to get it right. That said, I've begin the arduous task of finding data, getting good regularization sets, and trying to isolate the areas of low coherence so I can dig up good examples of these for the models. So, maybe in the future some actual fine tunes will come out, but probably not until the next gen models are released. Data is fickle, getting the right balances are difficult, training over instruct tunes is very destructive, there's lots of real problems with this approach. However, after seeing the results of so many merges, I have some hope a combination of merging and fine tuning might solve some of the more egregious issues I have with the current gen models.

Thinking about things, and actually doing the things

I'm on discord often, if you'd like to talk models, datasets or something and you made it this far, I enjoy discussing all things machine learning not just limited to LLMs, there's a discord link under the mascot banner if you'd like to come chat.