Mediocreatmybest

/

instructblip-vicuna-13b_8bit

like 0

Hi, I'm trying to use your quantized version but i'm stuck on the following error.
"Input type (float) and bias type (struct c10::Half) should be the same"
In "outputs = model.generate("

I guess some kind of normalization is needed. can you help me? This is my code:
from transformers import InstructBlipProcessor, InstructBlipForConditionalGeneration
import torch
from PIL import Image
import requests

model = InstructBlipForConditionalGeneration.from_pretrained("Mediocreatmybest/instructblip-vicuna-13b_8bit")
processor = InstructBlipProcessor.from_pretrained("Mediocreatmybest/instructblip-vicuna-13b_8bit")

device = "cuda" if torch.cuda.is_available() else "cpu"

url = "https://raw.githubusercontent.com/salesforce/LAVIS/main/docs/_static/Confusing-Pictures.jpg"
image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
prompt = "What is unusual about this image?"
inputs = processor(images=image, text=prompt, return_tensors="pt").to(device)

outputs = model.generate(
**inputs,
do_sample=False,
num_beams=5,
max_length=256,
min_length=1,
top_p=0.9,
repetition_penalty=1.5,
length_penalty=1.0,
temperature=1,
)
generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0].strip()
print(generated_text)