The purpose of this is to use the gguf. I am new to uploading models. So I am just uploading what was working here on my pc. I will look over all the files after they upload I am sorry. I have been uploading for over 10 hours. I deeply apologize for any errors and I will fix them after all the uploads have finished.

Here is the code I am using to upload the model. So far only the F16 has finished the other ggufs are uploading now.

from unsloth import FastLanguageModel
import torch
import os

Load the merged model

model, tokenizer = FastLanguageModel.from_pretrained(
model_name="model", # Replace with the actual path where you saved the merged model
max_seq_length=2048,
dtype=torch.float16 # Since it's saved as 16-bit
)

huggingface_model_name = "terrycraddock/Reflection-Llama-3.1-8B" # Replace with your chosen name

model.push_to_hub(huggingface_model_name, token = os.getenv("HF_TOKEN"))
tokenizer.push_to_hub(huggingface_model_name, token = os.getenv("HF_TOKEN"))

Merge to 16bit

if True: model.push_to_hub_merged(huggingface_model_name, tokenizer, save_method = "merged_16bit", token = os.getenv("HF_TOKEN"))

# Save to multiple GGUF options - much faster if you want multiple!

if True:
model.push_to_hub_gguf(
huggingface_model_name, # Change hf to your username!
tokenizer,
quantization_method = ["q4_k_m", "q8_0", "q5_k_m",],
token = os.getenv("HF_TOKEN")
)

@mradermacher This is only the second model I have ever attempted to upload. I will get better at this, as I am training Mistral on this data set now. I plan to upload MANY models using this dataset. I am sorry for the difficulties you are having and I will do what I can to help.

Ah, I understand - you need none of those files to just upload ggufs - the ggufs are self contained, so only they are needed for people to run the model. But it is always nice if the original model is available, so that people can generate newer/better quants later.

Yes you are correct, and I will do better I promise.

I would also suggest renaming the repo to ...-GGUF as well (if the intention is to only provide those), as that's the de-facto standard for ggufs on hf - but of course, it's not required. In general, people keep the transformers repo ands the gguf quants in separate repos.

It's just soft suggestions, of course, there are always exceptions. :)

The original files are there and you can use them. They use unsloth. I just downloaded and tested. you can use my files to make your own quants. Perhaps you are not familiar with unsloth?

if True: model.push_to_hub_merged(huggingface_model_name, tokenizer, save_method = "merged_16bit", token = os.getenv("HF_TOKEN"))

That line pushes the original and non gguf files to the repo.

model.push_to_hub(huggingface_model_name, token = os.getenv("HF_TOKEN"))
tokenizer.push_to_hub(huggingface_model_name, token = os.getenv("HF_TOKEN"))

These also so the tokenizer is even uploaded for your use.

Then I create and upload the gguf:

F16 gguf

if True: model.push_to_hub_gguf(huggingface_model_name, tokenizer, quantization_method = "f16", token = os.getenv("HF_TOKEN"))

# Save to multiple GGUF options - much faster if you want multiple!

if True:
model.push_to_hub_gguf(
huggingface_model_name, # Change hf to your username!
tokenizer,
quantization_method = ["q4_k_m", "q8_0", "q5_k_m",],
token = os.getenv("HF_TOKEN")
)

I am VERY inexperienced with this tho so it is possible I did this non-correctly. I am trying hard to give you all the files tho and I will do my best to make this better for the community. I am new to this. I am sorry.
@mradermacher I will do MUCH more research before I upload the mistral version. All I can do is apologize for the non-professional presentation of my work. I am sorry.

Maybe we have a bit of a miscommunication here, so first: Welcome to Huggingface. We are all inexperienced, more or less, so you should fit in quite well :)

You have done nothing wrong, even if I found it a bit confusing. Just continue as you did, you are doing great, and don't worry about professionalism, we are all very much interested in your work :)

Thank you so very much. :)

Just a heads up; config.json is empty. I subbed in "config.json" from Llama 3.1 Instruct. ; works fine now.
(merging using mergekit)
Thanks @terrycraddock

@DavidAU Thanks! I am re-training the model now to get a lower loss. I will make sure to fix that when I upload the new version. Sorry about that. :)

The new version has the updated config.json now.