MakiPan commited on
Commit
f840f8e
Β·
1 Parent(s): df2aeaf

Update app.py

Browse files
Files changed (1) hide show
  1. app.py +6 -12
app.py CHANGED
@@ -218,19 +218,16 @@ with gr.Blocks(theme='gradio/soft') as demo:
218
  gr.Markdown("This model is a ControlNet model using MediaPipe hand landmarks for control.")
219
  gr.Markdown("""
220
  <style>
221
- ul {padding-left: 50px;}
222
  </style>
223
  <center><h1>Summary</h1></center>
224
 
225
  As Stable diffusion and other diffusion models are notoriously poor at generating realistic hands for our project we decided to train a ControlNet model using MediaPipes landmarks in order to generate more realistic hands avoiding common issues such as unrealistic positions and irregular digits.
226
  <br>
227
  We opted to use the [HAnd Gesture Recognition Image Dataset](https://github.com/hukenovs/hagrid) (HaGRID) and [MediaPipe's Hand Landmarker](https://developers.google.com/mediapipe/solutions/vision/hand_landmarker) to train a control net that could potentially be used independently or as an in-painting tool.
228
- <br>
229
  To preprocess the data there were three options we considered:
230
- <be>
231
  <ul>
232
  <li>The first was to use Mediapipes built-in draw landmarks function. This was an obvious first choice however we noticed with low training steps that the model couldn't easily distinguish handedness and would often generate the wrong hand for the conditioning image.</li>
233
- <br>
234
  <center>
235
  <table><tr>
236
  <td>
@@ -249,9 +246,7 @@ To preprocess the data there were three options we considered:
249
  </td>
250
  </tr></table>
251
  </center>
252
- <br>
253
- <li>To counter this issue we changed the palm landmark colors with the intention to keep the color similar in order to learn that they provide similar information, but different to make the model know which hands were left or right.</li>
254
- <br>
255
  <center>
256
  <table><tr>
257
  <td>
@@ -270,20 +265,19 @@ To preprocess the data there were three options we considered:
270
  </td>
271
  </tr></table>
272
  </center>
273
- <br>
274
  <li>The last option was to use <a href="https://ai.googleblog.com/2020/12/mediapipe-holistic-simultaneous-face.html">MediaPipe Holistic</a> to provide pose face and hand landmarks to the ControlNet. This method was promising in theory, however, the HaGRID dataset was not suitable for this method as the Holistic model performs poorly with partial body and obscurely cropped images.</li>
275
  </ul>
276
- <br>
277
  We anecdotally determined that when trained at lower steps the encoded hand model performed better than the standard MediaPipe model due to implied handedness. We theorize that with a larger dataset of more full-body hand and pose classifications, Holistic landmarks will provide the best images in the future however for the moment the hand-encoded model performs best.
278
  """)
279
  gr.Markdown("""
280
  <center><h2><b>LINKS πŸ”—</b></h2>
 
281
  <h4 style="text-align: center;"><a href="https://huggingface.co/Vincent-luo/controlnet-hands">Standard Model Link</a></h4>
282
  <h4 style="text-align: center;"> <a href="https://huggingface.co/MakiPan/controlnet-encoded-hands-130k/">Model using Hand Encoding</a></h4>
283
- <br>
284
  <h4 style="text-align: center;"> <a href="https://huggingface.co/datasets/MakiPan/hagrid250k-blip2">Dataset Used To Train the Standard Model</a></h4>
285
  <h4 style="text-align: center;"> <a href="https://huggingface.co/datasets/MakiPan/hagrid-hand-enc-250k">Dataset Used To Train the Hand Encoding Model</a></h4>
286
- <br>
287
  <h4 style="text-align: center;"> <a href="https://github.com/Maki-DS/Jax-Controlnet-hand-training/blob/main/normal-preprocessing.py">Standard Data Preprocessing Script</a></h4>
288
  <h4 style="text-align: center;"> <a href="https://github.com/Maki-DS/Jax-Controlnet-hand-training/blob/main/Hand-encoded-preprocessing.py">Hand Encoding Data Preprocessing Script</a></h4></center>
289
  """)
@@ -301,7 +295,7 @@ We anecdotally determined that when trained at lower steps the encoded hand mode
301
 
302
  with gr.Column():
303
  output_image = gr.Gallery(label='Output Image', show_label=False, elem_id="gallery").style(grid=2, height='auto')
304
-
305
  gr.Examples(
306
  examples=[
307
  [
 
218
  gr.Markdown("This model is a ControlNet model using MediaPipe hand landmarks for control.")
219
  gr.Markdown("""
220
  <style>
221
+ ul {padding-left: 5em;}
222
  </style>
223
  <center><h1>Summary</h1></center>
224
 
225
  As Stable diffusion and other diffusion models are notoriously poor at generating realistic hands for our project we decided to train a ControlNet model using MediaPipes landmarks in order to generate more realistic hands avoiding common issues such as unrealistic positions and irregular digits.
226
  <br>
227
  We opted to use the [HAnd Gesture Recognition Image Dataset](https://github.com/hukenovs/hagrid) (HaGRID) and [MediaPipe's Hand Landmarker](https://developers.google.com/mediapipe/solutions/vision/hand_landmarker) to train a control net that could potentially be used independently or as an in-painting tool.
 
228
  To preprocess the data there were three options we considered:
 
229
  <ul>
230
  <li>The first was to use Mediapipes built-in draw landmarks function. This was an obvious first choice however we noticed with low training steps that the model couldn't easily distinguish handedness and would often generate the wrong hand for the conditioning image.</li>
 
231
  <center>
232
  <table><tr>
233
  <td>
 
246
  </td>
247
  </tr></table>
248
  </center>
249
+ <li>To counter this issue we changed the palm landmark colors with the intention to keep the color similar in order to learn that they provide similar information, but different to make the model know which hands were left or right.</li>
 
 
250
  <center>
251
  <table><tr>
252
  <td>
 
265
  </td>
266
  </tr></table>
267
  </center>
 
268
  <li>The last option was to use <a href="https://ai.googleblog.com/2020/12/mediapipe-holistic-simultaneous-face.html">MediaPipe Holistic</a> to provide pose face and hand landmarks to the ControlNet. This method was promising in theory, however, the HaGRID dataset was not suitable for this method as the Holistic model performs poorly with partial body and obscurely cropped images.</li>
269
  </ul>
 
270
  We anecdotally determined that when trained at lower steps the encoded hand model performed better than the standard MediaPipe model due to implied handedness. We theorize that with a larger dataset of more full-body hand and pose classifications, Holistic landmarks will provide the best images in the future however for the moment the hand-encoded model performs best.
271
  """)
272
  gr.Markdown("""
273
  <center><h2><b>LINKS πŸ”—</b></h2>
274
+ <h3 style="text-align: center;">Models πŸš€</h3>
275
  <h4 style="text-align: center;"><a href="https://huggingface.co/Vincent-luo/controlnet-hands">Standard Model Link</a></h4>
276
  <h4 style="text-align: center;"> <a href="https://huggingface.co/MakiPan/controlnet-encoded-hands-130k/">Model using Hand Encoding</a></h4>
277
+ <h3 style="text-align: center;">Datasets πŸ’Ύ</h3>
278
  <h4 style="text-align: center;"> <a href="https://huggingface.co/datasets/MakiPan/hagrid250k-blip2">Dataset Used To Train the Standard Model</a></h4>
279
  <h4 style="text-align: center;"> <a href="https://huggingface.co/datasets/MakiPan/hagrid-hand-enc-250k">Dataset Used To Train the Hand Encoding Model</a></h4>
280
+ <h3 style="text-align: center;">Preprocessing Scripts πŸ“‘</h3>
281
  <h4 style="text-align: center;"> <a href="https://github.com/Maki-DS/Jax-Controlnet-hand-training/blob/main/normal-preprocessing.py">Standard Data Preprocessing Script</a></h4>
282
  <h4 style="text-align: center;"> <a href="https://github.com/Maki-DS/Jax-Controlnet-hand-training/blob/main/Hand-encoded-preprocessing.py">Hand Encoding Data Preprocessing Script</a></h4></center>
283
  """)
 
295
 
296
  with gr.Column():
297
  output_image = gr.Gallery(label='Output Image', show_label=False, elem_id="gallery").style(grid=2, height='auto')
298
+
299
  gr.Examples(
300
  examples=[
301
  [