Q4 gguf, config.json, image processing

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+mmproj-model-f16.gguf filter=lfs diff=lfs merge=lfs -text
+NousResearch_Nous-Hermes-2-Vision-GGUF_Q4_0.gguf filter=lfs diff=lfs merge=lfs -text

NousResearch_Nous-Hermes-2-Vision-GGUF_Q4_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b3fa9f25cb31da1746288f8bf2e88ab110d28d6510e662efd9f4100c2ea83ba4
+size 4108928000

README.md

@@ -0,0 +1,131 @@
+---
+base_model: mistralai/Mistral-7B-v0.1
+tags:
+- mistral
+- instruct
+- finetune
+- chatml
+- gpt4
+- synthetic data
+- distillation
+- multimodal
+- llava
+model-index:
+- name: Nous-Hermes-2-Vision
+  results: []
+license: apache-2.0
+language:
+- en
+---
+
+GGUF Quants by Twobob, Thanks to @jartine and @cmp-nct for the assists
+
+It's vicuna ref: [here](https://github.com/qnguyen3/hermes-llava/blob/173b4ef441b5371c1e7d99da7a2e7c14c77ad12f/llava/conversation.py#L252)
+
+Caveat emptor: There is still some kind of bug in the inference that is likely to get fixed upstream. Just FYI
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/64a22257d3149e05bc6d259f/aF3VQrpwGJQLxbeyj1JOf.png)
+
+
+# Nous-Hermes-2-Vision - Mistral 7B
+
+
+![image/png](https://camo.githubusercontent.com/b09dc35a93b4b70748fa4e2f307b011cd3d548369dd926ec9a2d3a51f7b3721e/68747470733a2f2f66696c65732e6f616975736572636f6e74656e742e636f6d2f66696c652d6b4437565358734f5649576472624b3042353662686644363f73653d323032332d31322d3033543137253341333425334135385a2673703d722673763d323032312d30382d30362673723d6226727363633d6d61782d6167652533443331353336303030253243253230696d6d757461626c6526727363643d6174746163686d656e7425334225323066696c656e616d6525334439643530333039622d356236342d343964302d623832362d6165316638366132396661382e77656270267369673d50396973694b4679654a54435a47424b526d45494b3043586e6e55676c6334704a583071312532425478666a34253344)
+
+*In the tapestry of Greek mythology, Hermes reigns as the eloquent Messenger of the Gods, a deity who deftly bridges the realms through the art of communication. It is in homage to this divine mediator that I name this advanced LLM "Hermes," a system crafted to navigate the complex intricacies of human discourse with celestial finesse.*
+
+## Model description
+
+Nous-Hermes-2-Vision stands as a pioneering Vision-Language Model, leveraging advancements from the renowned **OpenHermes-2.5-Mistral-7B** by teknium. This model incorporates two pivotal enhancements, setting it apart as a cutting-edge solution:
+
+- **SigLIP-400M Integration**: Diverging from traditional approaches that rely on substantial 3B vision encoders, Nous-Hermes-2-Vision harnesses the formidable SigLIP-400M. This strategic choice not only streamlines the model's architecture, making it more lightweight, but also capitalizes on SigLIP's remarkable capabilities. The result? A remarkable boost in performance that defies conventional expectations.
+
+- **Custom Dataset Enriched with Function Calling**: Our model's training data includes a unique feature – function calling. This distinctive addition transforms Nous-Hermes-2-Vision into a **Vision-Language Action Model**. Developers now have a versatile tool at their disposal, primed for crafting a myriad of ingenious automations.
+
+This project is led by [qnguyen3](https://twitter.com/stablequan) and [teknium](https://twitter.com/Teknium1).
+## Training
+### Dataset
+- 220K from **LVIS-INSTRUCT4V**
+- 60K from **ShareGPT4V**
+- 150K Private **Function Calling Data**
+- 50K conversations from teknium's **OpenHermes-2.5**
+
+## Usage
+### Prompt Format
+- Like other LLaVA's variants, this model uses Vicuna-V1 as its prompt template. Please refer to `conv_llava_v1` in [this file](https://github.com/qnguyen3/hermes-llava/blob/main/llava/conversation.py)
+- For Gradio UI, please visit this [GitHub Repo](https://github.com/qnguyen3/hermes-llava)
+
+### Function Calling
+- For functiong calling, the message should start with a `<fn_call>` tag. Here is an example:
+
+```json
+<fn_call>{
+  "type": "object",
+  "properties": {
+    "bus_colors": {
+      "type": "array",
+      "description": "The colors of the bus in the image.",
+      "items": {
+        "type": "string",
+        "enum": ["red", "blue", "green", "white"]
+      }
+    },
+    "bus_features": {
+      "type": "string",
+      "description": "The features seen on the back of the bus."
+    },
+    "bus_location": {
+      "type": "string",
+      "description": "The location of the bus (driving or pulled off to the side).",
+      "enum": ["driving", "pulled off to the side"]
+    }
+  }
+}
+```
+
+Output:
+```json
+{
+  "bus_colors": ["red", "white"],
+  "bus_features": "An advertisement",
+  "bus_location": "driving"
+}
+```
+
+## Example
+
+### Chat
+![image/png](https://i.ibb.co/tMg8h2t/Screenshot-from-2023-12-04-00-13-59.png)
+
+### Function Calling
+Input image:
+
+![image/png](https://www.slcmenu.com/wp-content/uploads/2017/11/In-N-Out-Burger-menu-2020-982x1024.jpg)
+
+Input message:
+```json
+<fn_call>{
+    "type": "object",
+    "properties": {
+      "food_list": {
+        "type": "array",
+        "description": "List of all the food",
+        "items": {
+          "type": "string",
+        }
+      },
+    }
+}
+```
+
+Output:
+```json
+{
+    "food_list": [
+        "Double Burger",
+        "Cheeseburger",
+        "French Fries",
+        "Shakes",
+        "Coffee"
+    ]
+}
+```

config.json

@@ -0,0 +1,37 @@
+{
+  "_name_or_path": "teknium/OpenHermes-2.5-Mistral-7B",
+  "architectures": [
+    "LlavaMistralForCausalLM"
+  ],
+  "bos_token_id": 1,
+  "eos_token_id": 32000,
+  "freeze_mm_mlp_adapter": false,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "image_aspect_ratio": "pad",
+  "image_grid_pinpoints": null,
+  "initializer_range": 0.02,
+  "intermediate_size": 14336,
+  "max_position_embeddings": 32768,
+  "mm_hidden_size": 1152,
+  "mm_projector_type": "mlp2x_gelu",
+  "mm_use_im_patch_token": false,
+  "mm_use_im_start_end": false,
+  "mm_vision_select_feature": "patch",
+  "mm_vision_select_layer": -2,
+  "mm_vision_tower": "ikala/ViT-SO400M-14-SigLIP-384-hf",
+  "model_type": "llava_mistral",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 8,
+  "rms_norm_eps": 1e-05,
+  "rope_theta": 10000.0,
+  "sliding_window": 4096,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.34.1",
+  "tune_mm_mlp_adapter": false,
+  "use_cache": true,
+  "use_mm_proj": true,
+  "vocab_size": 32002
+}

mmproj-model-f16.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5c2a602b13c155db542d26fa5884e98e1ca4f2bb947e18957fc32c8f87492182
+size 839316352