Have you tried using a quantized t5 encoder or a quantized version of Flux? Flux doesn't loose much quality when quantized, at least with 4-bit methods and above. 3-bits works fairly well but starts to get more noticeably degraded. The t5-v1_1xxl model can be replaced with the encoder-only version without quality loss, and quantizing it saves a lot of VRAM. If you quantize both, you might have enough VRAM left to do QLoRA training or a LyCORIS PEFT training method.