Orion-zhen

/

Athene-V2-Chat-abliterated-bnb-4bit

like 0

I want to, but I am not sure if I can find an affordable cloud server with sufficient memory. Cloud server platforms that I can reach are either poor in memory or way too expensive. But I will keep improving my code so you can make one yourself if you have sufficient hardware ; )

I did it with an H200 and accelerate.
Modified tensor got applied in FP16 because data was locked in RAM. Model is BF16.

I will make it public later and add a description (I'm not at home). Seem to work, but still censored. I used default config with the Chinese arg you provided for Qwen models.

If you want me to try out something else, tell me, I still can use other H200.
Nice job tho, the script don't seem to make the model too braindead and is easy to use (I modified it a little for my usage) but I didn't have lot of time to really try the model.

Here you go : https://huggingface.co./Undi95/Athene-V2-Chat-abliterated

I stubbled randomly upon your profile and wanted to give a little help, in exchange I keep your script in my toolbox hehehe.

Edit: There's a Q4_K_M too I just done with iMatrix if you can run it.

Many thanks @Undi95
You can try scale factor >= 1 as long as it doesn't mess up the model. I have just changed my help message. Model loading strategy of my code is optimized for my local machine, you can change it to fit your pc.
Chinese models are really stubborn. I hardly managed to make qwen2.5 abliterated, sad.
I would be grateful if you have any idea or advice. Just feel free to issue or pr my repo

@Orion-zhen I'm currently toying with Qwen too, I made a fork if that give you some idea : https://github.com/Undi95/abliteration/
I succeeded to get Qwen to answer to everything, but it's still dumb.
I took another path, instead of calculating the tensors of the rejection for one layer and apply it to all of them, I get the refusal for each layer and apply them one by one, I hope it will make less dumb.
For a 72B it take time, dunno if I will have enough credit, for now script is working and save all refusal tensors in a folder for future usage.