upstage

/

SOLAR-10.7B-Instruct-v1.0

like 614

Follow

upstage 225

This is a fantastic model, could you please share how did you build it, maybe a post or a paper.

Thanks

We will be submitting them to arXiv shortly. Thank you for your interest!

Thank you! Excited to see it!

Hi @hunkim , is this basically a merge of Mistral and Llama, trained on more tokens? Were the original Llama weights used, and if so, does the license apply?

@mrfakename

We use the first and last 24 layers and initialize the Mistral weights. (Yes, the license applies.) Then, we continue pre-training the depth-upscaled model.

In some sense, this can be considered depth upscaling, while we see MOE as width upscaling.

@hunkim Thanks a lot for all this info. I have two questions:

1. Did you use the first 24 layers from LLMa and the last 24 layers from the mistral in the final merged model? Is there any logic behind selecting the order?
2. What is the dataset you have used?

Thanks in advanced :).