Color-Canny CantrolNet

These are ControlNet checkpoints trained on runwayml/stable-diffusion-v1-5, using fused color and canny edge as conditioning.

You can find some example images in the following.

Examples

Color examples

prompt: a concept art of by Makoto Shinkai, a girl is standing in the middle of the sea

negative prompt: text, bad anatomy, blurry, (low quality, blurry) images_1)

prompt: a concept art of by Makoto Shinkai, a girl is standing in the middle of the sea

negative prompt: text, bad anatomy, blurry, (low quality, blurry) images_2)

prompt: a concept art of by Makoto Shinkai, a girl is standing in the middle of the grass

negative prompt: text, bad anatomy, blurry, (low quality, blurry) images_3)

Brightness examples

This model also can be used to control image brightness. The following images are generated with different brightness conditioning image and controlnet strength(0.5 ~ 0.7). images_4)

Limitations and Bias

  • No strict control by input color
  • Sometimes generate image with confusion When color description in prompt

Training

Dataset We train this model on laion-art dataset with 2.6m images, the processed dataset can be found in ghoskno/laion-art-en-colorcanny.

Training Details

  • Hardware: Google Cloud TPUv4-8 VM

  • Optimizer: AdamW

  • Train Batch Size: 4 x 4 = 16

  • Learning rate: 0.00001 constant

  • Gradient Accumulation Steps: 4

  • Resolution: 512

  • Train Steps: 36000

Downloads last month
138
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for ghoskno/Color-Canny-Controlnet-model

Finetuned
(599)
this model

Dataset used to train ghoskno/Color-Canny-Controlnet-model

Spaces using ghoskno/Color-Canny-Controlnet-model 4