Hugging Face's logo

QuantFactory
/
TIPO-200M-ft-GGUF

Text Generation
Transformers
GGUF
English
Inference Endpoints
Model card Files Community
aashish1904 commited on
Commit
7a6501c
•
1 Parent(s): dcb4d60

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +116 -0
README.md ADDED
@@ -0,0 +1,116 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ ---
3
+
4
+ license: other
5
+ license_name: kohaku-license-1.0
6
+ datasets:
7
+ - laion/conceptual-captions-12m-webdataset
8
+ - CaptionEmporium/coyo-hd-11m-llavanext
9
+ - KBlueLeaf/danbooru2023-metadata-database
10
+ - graph-based-captions/GBC10M
11
+ language:
12
+ - en
13
+ pipeline_tag: text-generation
14
+ library_name: transformers
15
+
16
+ ---
17
+
18
+ [![QuantFactory Banner](https://lh7-rt.googleusercontent.com/docsz/AD_4nXeiuCm7c8lEwEJuRey9kiVZsRn2W-b4pWlu3-X534V3YmVuVc2ZL-NXg2RkzSOOS2JXGHutDuyyNAUtdJI65jGTo8jT9Y99tMi4H4MqL44Uc5QKG77B0d6-JfIkZHFaUA71-RtjyYZWVIhqsNZcx8-OMaA?key=xt3VSDoCbmTY7o-cwwOFwQ)](https://hf.co/QuantFactory)
19
+
20
+
21
+ # QuantFactory/TIPO-200M-ft-GGUF
22
+ This is quantized version of [KBlueLeaf/TIPO-200M-ft](https://huggingface.co/KBlueLeaf/TIPO-200M-ft) created using llama.cpp
23
+
24
+ # Original Model Card
25
+
26
+ # TIPO: Text to Image with text presampling for Prompt Optimization
27
+
28
+ 200M LLaMA arch model trained for TIPO. <br>
29
+ Tech Report: https://kblueleaf.net/document/TIPO-tech-report.pdf
30
+
31
+ ![image/png](https://cdn-uploads.huggingface.co/production/uploads/630593e2fca1d8d92b81d2a1/fc9ovmARapQmgq9DZ7ApJ.png)
32
+
33
+ ## Introduction
34
+
35
+ In this project, we introduce "TIPO" (**T**ext to **I**mage with text presampling for **P**rompt **O**ptimization), an innovative framework designed to significantly enhance the quality and usability of Text-to-Image (T2I) generative models. TIPO utilizes the Large Language Models (LLMs) to perform "Text Presampling" within the inference pipeline of text-to-image generative modeling. By refining and extending user input prompts, TIPO enables generative models to produce superior results with minimal user effort, making T2I systems more accessible and effective for a wider range of users.
36
+
37
+ ## Usage
38
+
39
+ Use updated version of DTG extension (renamed to z-tipo-extension), current version of z-tipo-extension support stable-diffusion-webui, stable-diffusion-webui-forge and ComfyUI. SD-Next haven't been tested.
40
+ https://github.com/KohakuBlueleaf/z-tipo-extension
41
+
42
+ ## Model arch and Training
43
+
44
+ This model is LLaMA arch with 200M parameters, the training data is combined version of Danbooru2023, Coyo-HD-11M. <br>
45
+ The total token seen is around 50B tokens. <br>
46
+ For more information please refer to the tech report and following table.
47
+
48
+ | | TIPO-200M | TIPO-200M-ft | TIPO-500M |
49
+ | ----------------- | ------------------------------------------------------------------------------ | ---------------------------------- | ------------------------------------------------------------------------------ |
50
+ | Arch | LLaMA | LLaMA | LLaMA |
51
+ | Max ctx length | 1024 | 1024 | 1024 |
52
+ | Batch Size | 2048 | 2048 | 3584 |
53
+ | Training dataset | Danbooru, GBC10M, 5epoch<br />Danbooru, GBC10M, Coyo11M, 3epoch | Danbooru(pixtral), Coyo11M, 2epoch | Danbooru, GBC10M, Coyo11M, 5epoch |
54
+ | Real Token Seen* | 40B token | 50B (10B more from TIPO-200M) | 30B token |
55
+ | Training Hardware | RTX 3090 x 4 | RTX 3090 x 4 | H100 x 8 |
56
+ | Training Time | 420 hour` | 120 hour` | 100 hour` |
57
+ | Huggingface | [KBlueLeaf/TIPO-200M · Hugging Face](https://huggingface.co/KBlueLeaf/TIPO-200M) | You Are HERE | [KBlueLeaf/TIPO-500M · Hugging Face](https://huggingface.co/KBlueLeaf/TIPO-500M) |
58
+
59
+ *: We only count "non-padding token" in the token seen, since all the training data have very large length range. <br>
60
+ `: Since the training data is pretty short, it cost more time to reach same token seen than general LLM pretraining. <br>
61
+ As reference, with 4096 as max ctx length and almost all the data have reach that length, you may only need 2days to reach 10B token seen on RTX 3090 x 4 with 200M model.
62
+
63
+ ### Evaluation
64
+ **Evaluation are done on TIPO-200M model** <br>
65
+ We have tested TIPO compared to other Model in several test and metrics:
66
+
67
+ #### Scenery tag test
68
+
69
+ In this test we use single "scenery" tag as input. (With some certain meta) <br>
70
+ To test each prompt gen method to see if they can obtain the desired distribution of outputs while maintain the quality of images.
71
+
72
+ | Scenery Tag Test | Original | GPT4o-mini | Prompt DB | Promptis | TIPO(ours) |
73
+ | ---- | ---- | ---- | ---- | ---- | ---- |
74
+ | FDD ↓ | 0.3558 | 0.5414 | 0.3247 | *0.2350* | **0.2282** |
75
+ | Aesthetic ↑ | 5.0569 | **6.3676** | 6.1609 | 5.9468 | *6.2571* |
76
+ | AI Corrupt ↑ | 0.4257 | *0.7490* | 0.5024 | 0.5669 | **0.9195** |
77
+
78
+ #### Short/Truncated Long test
79
+
80
+ In this test we use short caption or manually truncated caption from GBC10M and CoyoHD11M. <br>
81
+ This test examine the ability of prompt gen method on handling almostly completed prompts.
82
+
83
+ | Short | Original | GPT4o-mini | Prompt DB | Promptis | TIPO(ours) |
84
+ | ---- | ---- | ---- | ---- | ---- | ---- |
85
+ | FDD ↓ | 0.0957 | 0.1668 | *0.0980* | 0.1783 | 0.1168 |
86
+ | Aesthetic ↑ | 5.8370 | **6.0589** | 5.8213 | 5.7963 | *5.8531* |
87
+ | AI Corrupt ↑ | 0.7113 | 0.6985 | 0.7064 | 0.6314 | **0.7131** |
88
+
89
+ | Truncated Long | Original | GPT4o-mini | Prompt DB | Promptis | TIPO(ours) |
90
+ | ---- | ---- | ---- | ---- | ---- | ---- |
91
+ | FDD ↓ | 0.0955 | 0.1683 | *0.1247* | 0.2096 | 0.1210 |
92
+ | Aesthetic ↑ | 5.7497 | **6.0168** | 5.8191 | 5.7759 | *5.8364* |
93
+ | AI Corrupt ↑ | 0.6868 | 0.6712 | 0.6741 | 0.5925 | **0.7130** |
94
+
95
+
96
+
97
+ ## LICENSE
98
+
99
+ This model is released under [Kohaku License 1.0](https://kblueleaf.net/documents/kohaku-license/?[Your%20Organization/Name]=KohakuBlueLeaf&[Year]=2024) <br>
100
+ You can check the above provided URL or check the LICENSE file in this repo.
101
+
102
+ ### Citation
103
+
104
+ ```bibtex
105
+ @misc{yeh2024tipo,
106
+ title = {TIPO: Text to Image with text presampling for Prompt Optimization},
107
+ author = {Yeh, Shih-Ying},
108
+ year = {2024},
109
+ month = {10},
110
+ day = {6},
111
+ note = {Technical report available at \url{https://kblueleaf.net/document/TIPO-tech-report.pdf}.
112
+ Model available at \url{https://huggingface.co/KBlueLeaf/TIPO-500M}.
113
+ Source code available at \url{https://github.com/KohakuBlueleaf/KGen}},
114
+ }
115
+ ```
116
+