Hugging Face's logo

PygTesting
/
mystery-merge-nemo

Text Generation
Transformers
Safetensors
mistral
mergekit
Merge
conversational
text-generation-inference
Inference Endpoints
Model card Files Community

evo_model_test

This is a merge of pre-trained language models created using mergekit.

Merge Details

Merge Method

This model was merged using the DARE TIES merge method using ./evolve_merges/input_models/merge-10162024_972739363 as a base.

Models Merged

The following models were included in the merge:

  • ./evolve_merges/input_models/Magnum-Picaro-0.7-v2-12b_3809452655
  • ./evolve_merges/input_models/Chronos-Gold-12B-1.0_1861025797
  • ./evolve_merges/input_models/MN-12B-Mag-Mell-R1_399051020

Configuration

The following YAML configuration was used to produce this model:

base_model: ./evolve_merges/input_models/merge-10162024_972739363
dtype: bfloat16
merge_method: dare_ties
parameters:
  int8_mask: 1.0
  normalize: 1.0
slices:
- sources:
  - layer_range: [0, 4]
    model: ./evolve_merges/input_models/merge-10162024_972739363
    parameters:
      density:
      - filter: self_attn
        value: 0.6617851833521375
      - filter: mlp
        value: 1.0
      - value: 0.7758506135029611
      weight:
      - filter: self_attn
        value: 0.06553850894305135
      - filter: mlp
        value: 0.32372893196093133
      - value: 0.24761893893703177
  - layer_range: [0, 4]
    model: ./evolve_merges/input_models/Magnum-Picaro-0.7-v2-12b_3809452655
    parameters:
      density:
      - filter: self_attn
        value: 0.8619096186212604
      - filter: mlp
        value: 0.9632945037149085
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.5496368676404241
      - filter: mlp
        value: 0.2817627768141395
      - value: 0.2831242003449033
  - layer_range: [0, 4]
    model: ./evolve_merges/input_models/Chronos-Gold-12B-1.0_1861025797
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 0.9238831652008582
      weight:
      - filter: self_attn
        value: 0.6983534009784523
      - filter: mlp
        value: 0.7786486269006042
      - value: 0.3362711484417948
  - layer_range: [0, 4]
    model: ./evolve_merges/input_models/MN-12B-Mag-Mell-R1_399051020
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 0.897712174766424
      weight:
      - filter: self_attn
        value: 0.6494468053120542
      - filter: mlp
        value: 0.11769817501358182
      - value: 0.23745407940550356
- sources:
  - layer_range: [4, 8]
    model: ./evolve_merges/input_models/merge-10162024_972739363
    parameters:
      density:
      - filter: self_attn
        value: 0.768056839478356
      - filter: mlp
        value: 0.7392675781352855
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.4137398667324908
      - filter: mlp
        value: 0.5364761127195374
      - value: -0.06120952450996993
  - layer_range: [4, 8]
    model: ./evolve_merges/input_models/Magnum-Picaro-0.7-v2-12b_3809452655
    parameters:
      density:
      - filter: self_attn
        value: 0.9328263901133284
      - filter: mlp
        value: 1.0
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.512662918449004
      - filter: mlp
        value: 0.8133160093541117
      - value: 0.09518477923218693
  - layer_range: [4, 8]
    model: ./evolve_merges/input_models/Chronos-Gold-12B-1.0_1861025797
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.6534355737222919
      - filter: mlp
        value: -0.2733724467069448
      - value: 0.35896371241039604
  - layer_range: [4, 8]
    model: ./evolve_merges/input_models/MN-12B-Mag-Mell-R1_399051020
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 0.9645408518441749
      - value: 0.9920721804462888
      weight:
      - filter: self_attn
        value: 0.043888879112993606
      - filter: mlp
        value: 0.37533863309727755
      - value: 0.32692015564467836
- sources:
  - layer_range: [8, 12]
    model: ./evolve_merges/input_models/merge-10162024_972739363
    parameters:
      density:
      - filter: self_attn
        value: 0.9340306321054911
      - filter: mlp
        value: 1.0
      - value: 0.7968276665543247
      weight:
      - filter: self_attn
        value: 0.14846986084920036
      - filter: mlp
        value: 0.3955452929300913
      - value: 0.4270837195831495
  - layer_range: [8, 12]
    model: ./evolve_merges/input_models/Magnum-Picaro-0.7-v2-12b_3809452655
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.3649415030710907
      - filter: mlp
        value: 0.16275044387393922
      - value: 0.2758727640654811
  - layer_range: [8, 12]
    model: ./evolve_merges/input_models/Chronos-Gold-12B-1.0_1861025797
    parameters:
      density:
      - filter: self_attn
        value: 0.8295983370283204
      - filter: mlp
        value: 0.7788134370117827
      - value: 0.9398894811483364
      weight:
      - filter: self_attn
        value: 0.28746483121862637
      - filter: mlp
        value: 0.3358374043922244
      - value: 0.2275533582239845
  - layer_range: [8, 12]
    model: ./evolve_merges/input_models/MN-12B-Mag-Mell-R1_399051020
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 0.727821766634972
      weight:
      - filter: self_attn
        value: 0.3081244623443608
      - filter: mlp
        value: 0.45014674558784984
      - value: 0.11047219740073362
- sources:
  - layer_range: [12, 16]
    model: ./evolve_merges/input_models/merge-10162024_972739363
    parameters:
      density:
      - filter: self_attn
        value: 0.6489316039694529
      - filter: mlp
        value: 1.0
      - value: 0.8272372022626591
      weight:
      - filter: self_attn
        value: 0.470708064142626
      - filter: mlp
        value: -0.047129110924588186
      - value: 0.42971949234723295
  - layer_range: [12, 16]
    model: ./evolve_merges/input_models/Magnum-Picaro-0.7-v2-12b_3809452655
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 0.6616234442454084
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.26282202905677127
      - filter: mlp
        value: 0.4448525732857457
      - value: 0.2229765978922556
  - layer_range: [12, 16]
    model: ./evolve_merges/input_models/Chronos-Gold-12B-1.0_1861025797
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 0.6135513085208061
      - value: 0.9581737790930396
      weight:
      - filter: self_attn
        value: 0.24444794214178578
      - filter: mlp
        value: 0.07937992720612315
      - value: -0.05228450555064985
  - layer_range: [12, 16]
    model: ./evolve_merges/input_models/MN-12B-Mag-Mell-R1_399051020
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.1719406804216106
      - filter: mlp
        value: 0.0934880168140769
      - value: 0.35045642161724166
- sources:
  - layer_range: [16, 20]
    model: ./evolve_merges/input_models/merge-10162024_972739363
    parameters:
      density:
      - filter: self_attn
        value: 0.5446785752563841
      - filter: mlp
        value: 0.8810586946591301
      - value: 0.9152297583356134
      weight:
      - filter: self_attn
        value: -0.0016341576761690624
      - filter: mlp
        value: -0.14493024949671152
      - value: 0.26832439639581773
  - layer_range: [16, 20]
    model: ./evolve_merges/input_models/Magnum-Picaro-0.7-v2-12b_3809452655
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 0.5944606032155147
      - value: 0.9302142529770252
      weight:
      - filter: self_attn
        value: 0.35950618403078893
      - filter: mlp
        value: 0.11051887834512175
      - value: 0.42291230769302385
  - layer_range: [16, 20]
    model: ./evolve_merges/input_models/Chronos-Gold-12B-1.0_1861025797
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 0.6546859569496538
      - value: 0.8503723026949942
      weight:
      - filter: self_attn
        value: 0.35331354069135923
      - filter: mlp
        value: 0.11666399796526544
      - value: 0.027977616826786067
  - layer_range: [16, 20]
    model: ./evolve_merges/input_models/MN-12B-Mag-Mell-R1_399051020
    parameters:
      density:
      - filter: self_attn
        value: 0.8237153213010172
      - filter: mlp
        value: 0.7779880619326531
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.7145318763470817
      - filter: mlp
        value: 0.4104048815986916
      - value: 0.07468194955613425
- sources:
  - layer_range: [20, 24]
    model: ./evolve_merges/input_models/merge-10162024_972739363
    parameters:
      density:
      - filter: self_attn
        value: 0.5231923060339636
      - filter: mlp
        value: 1.0
      - value: 0.9856713754180749
      weight:
      - filter: self_attn
        value: 0.4081014822719611
      - filter: mlp
        value: 0.09758488254406042
      - value: 0.3348194266336727
  - layer_range: [20, 24]
    model: ./evolve_merges/input_models/Magnum-Picaro-0.7-v2-12b_3809452655
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.7490383834336071
      - filter: mlp
        value: 0.4662047924812158
      - value: -0.24858277913931304
  - layer_range: [20, 24]
    model: ./evolve_merges/input_models/Chronos-Gold-12B-1.0_1861025797
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 0.8502797089454639
      weight:
      - filter: self_attn
        value: 0.276884170342346
      - filter: mlp
        value: 0.633656940319029
      - value: 0.5235799339573071
  - layer_range: [20, 24]
    model: ./evolve_merges/input_models/MN-12B-Mag-Mell-R1_399051020
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 0.8562223977334964
      - value: 0.9716150483673114
      weight:
      - filter: self_attn
        value: 0.5270260765195226
      - filter: mlp
        value: 0.32711936701658684
      - value: 0.05670152518434478
- sources:
  - layer_range: [24, 28]
    model: ./evolve_merges/input_models/merge-10162024_972739363
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 0.8553635955278736
      weight:
      - filter: self_attn
        value: 0.35406982791511876
      - filter: mlp
        value: -0.11643971781340703
      - value: 0.20075532527415488
  - layer_range: [24, 28]
    model: ./evolve_merges/input_models/Magnum-Picaro-0.7-v2-12b_3809452655
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 0.87297120460794
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.07480839031742999
      - filter: mlp
        value: 0.18311115096539785
      - value: 0.3625508152553395
  - layer_range: [24, 28]
    model: ./evolve_merges/input_models/Chronos-Gold-12B-1.0_1861025797
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.494667527482752
      - filter: mlp
        value: 0.3944202674139632
      - value: -0.19227439649461792
  - layer_range: [24, 28]
    model: ./evolve_merges/input_models/MN-12B-Mag-Mell-R1_399051020
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.06851638816347627
      - filter: mlp
        value: 0.431372227001768
      - value: 0.1747985843980182
- sources:
  - layer_range: [28, 32]
    model: ./evolve_merges/input_models/merge-10162024_972739363
    parameters:
      density:
      - filter: self_attn
        value: 0.9094528371038374
      - filter: mlp
        value: 1.0
      - value: 0.6090545725123906
      weight:
      - filter: self_attn
        value: 0.25309591486694805
      - filter: mlp
        value: -0.263292487608102
      - value: 0.1323202337738385
  - layer_range: [28, 32]
    model: ./evolve_merges/input_models/Magnum-Picaro-0.7-v2-12b_3809452655
    parameters:
      density:
      - filter: self_attn
        value: 0.6494843615875994
      - filter: mlp
        value: 1.0
      - value: 0.7515064103597758
      weight:
      - filter: self_attn
        value: 0.07729701084822604
      - filter: mlp
        value: 0.2170958326731126
      - value: 0.22214702687265422
  - layer_range: [28, 32]
    model: ./evolve_merges/input_models/Chronos-Gold-12B-1.0_1861025797
    parameters:
      density:
      - filter: self_attn
        value: 0.8431056158343985
      - filter: mlp
        value: 0.8838909258744341
      - value: 0.35295455870641634
      weight:
      - filter: self_attn
        value: 0.6551015978225493
      - filter: mlp
        value: 0.016410780482769546
      - value: 0.6370635339121399
  - layer_range: [28, 32]
    model: ./evolve_merges/input_models/MN-12B-Mag-Mell-R1_399051020
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.04318024669287196
      - filter: mlp
        value: 0.7642269685567962
      - value: 0.26850603466331324
- sources:
  - layer_range: [32, 36]
    model: ./evolve_merges/input_models/merge-10162024_972739363
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 0.579520070097527
      weight:
      - filter: self_attn
        value: -0.051737601944818495
      - filter: mlp
        value: 0.3503787657405606
      - value: 0.08607827555366553
  - layer_range: [32, 36]
    model: ./evolve_merges/input_models/Magnum-Picaro-0.7-v2-12b_3809452655
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.28766985337224327
      - filter: mlp
        value: 0.3046959778412749
      - value: -0.0005520428411238121
  - layer_range: [32, 36]
    model: ./evolve_merges/input_models/Chronos-Gold-12B-1.0_1861025797
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 0.915429997855087
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.440410051026902
      - filter: mlp
        value: -0.21574554516791783
      - value: 0.15656972383477347
  - layer_range: [32, 36]
    model: ./evolve_merges/input_models/MN-12B-Mag-Mell-R1_399051020
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.3263876152481672
      - filter: mlp
        value: -0.040618303294953154
      - value: 0.47900376528192473
- sources:
  - layer_range: [36, 40]
    model: ./evolve_merges/input_models/merge-10162024_972739363
    parameters:
      density:
      - filter: self_attn
        value: 0.9171778237104341
      - filter: mlp
        value: 0.7229727777891508
      - value: 0.9122033861491662
      weight:
      - filter: self_attn
        value: 0.6154987734241069
      - filter: mlp
        value: 0.3910860949496661
      - value: 0.5286422728941228
  - layer_range: [36, 40]
    model: ./evolve_merges/input_models/Magnum-Picaro-0.7-v2-12b_3809452655
    parameters:
      density:
      - filter: self_attn
        value: 0.6023409600465159
      - filter: mlp
        value: 1.0
      - value: 1.0
      weight:
      - filter: self_attn
        value: 0.39644253937030505
      - filter: mlp
        value: 0.7570672338863116
      - value: 0.10261227723433294
  - layer_range: [36, 40]
    model: ./evolve_merges/input_models/Chronos-Gold-12B-1.0_1861025797
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 1.0
      - value: 0.8342554461687561
      weight:
      - filter: self_attn
        value: 0.4563403174251752
      - filter: mlp
        value: 0.313992481082509
      - value: 0.022583139471508834
  - layer_range: [36, 40]
    model: ./evolve_merges/input_models/MN-12B-Mag-Mell-R1_399051020
    parameters:
      density:
      - filter: self_attn
        value: 1.0
      - filter: mlp
        value: 0.9211392650515542
      - value: 1.0
      weight:
      - filter: self_attn
        value: -0.17092104595693997
      - filter: mlp
        value: 0.13032109680489912
      - value: -0.03480332269062497
Downloads last month
38
Safetensors
Model size
12.2B params
Tensor type
BF16
·
Inference Examples
Text Generation
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for PygTesting/mystery-merge-nemo

Finetunes
1 model