Batch inference
#25
by
gordonyeh
- opened
Hi
I am trying to do the batch inference.
Is there a plan to support it on this model (microsoft/Phi-3.5-vision-instruct)
sample format like: [prompt1, prompt2], [[img0, img1], [img_a, img_b, img_c]]
Thanks
@gordonyeh I implemented batching for multiple (prompt, img) pairs. I think it can be extended to (prompt, multiple_imgs) pairs.