XLabs ControlNet Error Report

Issue Description

I am encountering persistent issues when using the XLabs Flux model and its associated ControlNet in ComfyUI. After initial successful generations, subsequent attempts result in noise outputs, regardless of input changes or workflow modifications. This issue persists even after restarting ComfyUI and isolating the Flux model and ControlNet in a minimal workflow. The only way to stop Flux from generation noise is to bypass the Xlabs apply controlnet node

Environment

• ComfyUI version: ComfyUI v0.2.2
• Flux model version: Flux dev fp8
• XLabs ControlNet version: flux-depth-controlnet-v3.safetensors
• GPU: GTX 4090
• VRAM: 24gb
• Operating System: Windows

Workflow Description

1. Initial setup includes Flux dev fp8 model with its Xlabs depth ControlNet and Stable Diffusion 1.5 with IC-Light ControlNet.
2. First generation typically succeeds without issues.
3. Subsequent generations, even with input changes, result in noise outputs when using Flux ControlNet.
4. Disabling Flux ControlNet allows for successful generations, but with loss of control.
5. Issue persists even after isolating Flux model and its ControlNet in a new, minimal workflow. (deleting everything else in the workflow)

Observed Behavior

• First generation: Successful output
• Subsequent generations: Noise output
• Behavior persists across ComfyUI restarts
• Problem continues even when other models (e.g., IC-Light) are removed from the workflow

Steps to Reproduce

1. Set up a workflow with Flux model and its ControlNet.
2. Generate an image successfully.
3. Modify any input parameter.
4. Attempt another generation.
5. Observe noise output instead of a proper image.
6. Restart ComfyUI.
7. Set up a minimal workflow with only Flux and its ControlNet.
8. Attempt generation again.
9. Observe persistent noise output.

Logs

These logs are after deleting IC-light from the workflow and only isolating flux and xlabs controlnet and still generating noise

got prompt
Failed to validate prompt for output 151:

• (prompt):
  • Required input is missing: images
• PreviewImage 151:
  • Required input is missing: images
    Output will be ignored
    Failed to validate prompt for output 255:
• (prompt):
  • Required input is missing: images
• PreviewImage 255:
  • Required input is missing: images
    Output will be ignored
    Executed {'node': '133', 'display_node': '133', 'output': {'images': [{'filename': 'ComfyUI_temp_ftgxl_00001_.png', 'subfolder': '', 'type': 'temp'}, {'filename': 'ComfyUI_temp_ftgxl_00002_.png', 'subfolder': '', 'type': 'temp'}, {'filename': 'ComfyUI_temp_ftgxl_00003_.png', 'subfolder': '', 'type': 'temp'}, {'filename': 'ComfyUI_temp_ftgxl_00004_.png', 'subfolder': '', 'type': 'temp'}]}, 'prompt_id': '419dc0a8-94fb-4ece-996d-95f846d9f1f8'}
    Loading model from: C:\Users\trist\Documents\ComfyUI_windows_portable\ComfyUI\models\depthanything\depth_anything_v2_vits_fp16.safetensors
    Executed {'node': '252', 'display_node': '252', 'output': {'images': [{'filename': 'ComfyUI_temp_sylsj_00001_.png', 'subfolder': '', 'type': 'temp'}, {'filename': 'ComfyUI_temp_sylsj_00002_.png', 'subfolder': '', 'type': 'temp'}, {'filename': 'ComfyUI_temp_sylsj_00003_.png', 'subfolder': '', 'type': 'temp'}, {'filename': 'ComfyUI_temp_sylsj_00004_.png', 'subfolder': '', 'type': 'temp'}]}, 'prompt_id': '419dc0a8-94fb-4ece-996d-95f846d9f1f8'}
    using MLP layer as FFN
    Executed {'node': '126', 'display_node': '126', 'output': {'images': [{'filename': 'ComfyUI_temp_pkscp_00001_.png', 'subfolder': '', 'type': 'temp'}, {'filename': 'ComfyUI_temp_pkscp_00002_.png', 'subfolder': '', 'type': 'temp'}, {'filename': 'ComfyUI_temp_pkscp_00003_.png', 'subfolder': '', 'type': 'temp'}, {'filename': 'ComfyUI_temp_pkscp_00004_.png', 'subfolder': '', 'type': 'temp'}]}, 'prompt_id': '419dc0a8-94fb-4ece-996d-95f846d9f1f8'}
    Using pytorch attention in VAE
    Using pytorch attention in VAE
    using sdpa for attention
    Executed {'node': '274', 'display_node': '274', 'output': {'images': [{'filename': 'ComfyUI_temp_fvpsz_00001_.png', 'subfolder': '', 'type': 'temp'}, {'filename': 'ComfyUI_temp_fvpsz_00002_.png', 'subfolder': '', 'type': 'temp'}, {'filename': 'ComfyUI_temp_fvpsz_00003_.png', 'subfolder': '', 'type': 'temp'}, {'filename': 'ComfyUI_temp_fvpsz_00004_.png', 'subfolder': '', 'type': 'temp'}]}, 'prompt_id': '419dc0a8-94fb-4ece-996d-95f846d9f1f8'}
    Executed {'node': '270', 'display_node': '270', 'output': {'images': [{'filename': 'ComfyUI_temp_opuff_00001_.png', 'subfolder': '', 'type': 'temp'}, {'filename': 'ComfyUI_temp_opuff_00002_.png', 'subfolder': '', 'type': 'temp'}, {'filename': 'ComfyUI_temp_opuff_00003_.png', 'subfolder': '', 'type': 'temp'}, {'filename': 'ComfyUI_temp_opuff_00004_.png', 'subfolder': '', 'type': 'temp'}]}, 'prompt_id': '419dc0a8-94fb-4ece-996d-95f846d9f1f8'}
    model weight dtype torch.float8_e4m3fn, manual cast: torch.bfloat16
    model_type FLUX
    Requested to load FluxClipModel_
    Loading 1 new model
    loaded completely 0.0 4777.53759765625 True
    clip missing: ['text_projection.weight']
    UserWarning: 1Torch was not compiled with flash attention. (Triggered internally at ..\aten\src\ATen\native\transformers\cuda\sdp_utils.cpp:455.)
    Requested to load Flux
    Loading 1 new model
    loaded completely 0.0 11350.048889160156 True
    Sampling: 100%|███████████████████████████████████████████████████████████████████████████████| 4/4 [00:10<00:00, 2.51s/it]
    Requested to load AutoencodingEngine
    Loading 1 new model
    loaded completely 0.0 159.87335777282715 True
    Executed {'node': '46', 'display_node': '46', 'output': {'images': [{'filename': 'canny_res_00789_.png', 'subfolder': '', 'type': 'output'}, {'filename': 'canny_res_00790_.png', 'subfolder': '', 'type': 'output'}, {'filename': 'canny_res_00791_.png', 'subfolder': '', 'type': 'output'}, {'filename': 'canny_res_00792_.png', 'subfolder': '', 'type': 'output'}]}, 'prompt_id': '419dc0a8-94fb-4ece-996d-95f846d9f1f8'}
    Executed {'node': '108', 'display_node': '108', 'output': {'images': [{'filename': 'ComfyUI_temp_zfrkp_00001_.png', 'subfolder': '', 'type': 'temp'}, {'filename': 'ComfyUI_temp_zfrkp_00002_.png', 'subfolder': '', 'type': 'temp'}, {'filename': 'ComfyUI_temp_zfrkp_00003_.png', 'subfolder': '', 'type': 'temp'}, {'filename': 'ComfyUI_temp_zfrkp_00004_.png', 'subfolder': '', 'type': 'temp'}]}, 'prompt_id': '419dc0a8-94fb-4ece-996d-95f846d9f1f8'}

Attempted Solutions

1. Restarting ComfyUI
2. Rebooting the system
3. Isolating the Flux model and ControlNet in a minimal workflow

None of these solutions have resolved the issue.

Questions for Developers

1. Are there known incompatibilities between Flux/XLabs ControlNet and other models like IC-Light?
2. Could this be related to GPU memory management or potential memory leaks?
3. Are there specific ComfyUI settings or configurations recommended for using Flux and its ControlNet?
4. Are there any known issues with Flux ControlNet that might cause persistent noise outputs after initial successful generations?

Any insights or suggestions for further troubleshooting would be greatly appreciated. Thank you for your time and assistance in resolving this issue.