OPEA/Molmo-7B-D-0924-int4-sym-inc

Model Details

This model is an int4 model with group_size 128 and symmetric quantization of allenai/Molmo-7B-D-0924 generated by intel/auto-round. Load the model with revision="e64d453" to use AutoGPTQ format.

How To Use

INT4 Inference

from auto_round import AutoRoundConfig ## must import for auto-round format
from transformers import AutoModelForCausalLM, AutoProcessor, GenerationConfig
from PIL import Image
import requests

quantized_model_path = "OPEA/Molmo-7B-D-0924-int4-sym-inc"

# load the processor
processor = AutoProcessor.from_pretrained(
    quantized_model_path,
    trust_remote_code=True,
    torch_dtype='auto',
    device_map='auto'
)

# load the model
model = AutoModelForCausalLM.from_pretrained(
    quantized_model_path,
    trust_remote_code=True,
    torch_dtype='auto',
    device_map='auto',
    ##revision="e64d453" ##AutoGPTQ format
)

image_url = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"
text = "Describe this image."
# process the image and text
inputs = processor.process(
    images=[Image.open(requests.get(image_url, stream=True).raw)],
    text=text
)

# move inputs to the correct device and make a batch of size 1
inputs = {k: v.to(model.device).unsqueeze(0) for k, v in inputs.items()}
inputs["images"] = inputs["images"].to(model.dtype)

# generate output; maximum 200 new tokens; stop generation when <|endoftext|> is generated
output = model.generate_from_batch(
    inputs,
    GenerationConfig(max_new_tokens=200, stop_strings="<|endoftext|>"),
    tokenizer=processor.tokenizer
)

# only get generated tokens; decode them to text
generated_tokens = output[0,inputs['input_ids'].size(1):]
generated_text = processor.tokenizer.decode(generated_tokens, skip_special_tokens=True)

# print the generated text
print(generated_text)
##INT4:
## In this serene beach scene, a woman with long, dark hair sits on the sandy shore, facing left. She is dressed in a plaid shirt with rolled-up sleeves, black pants, and sandals. Her eyes are closed, and she is smiling warmly as she reaches out to high-five a large, light brown dog. The dog, possibly a Labrador or a similar breed, sits on its hind legs with its front paws raised, eagerly engaging in the friendly gesture. The dog is adorned with a blue harness featuring a pattern of pink and blue flowers, and a red leash lies on the sand beside it. The beach is calm, with gentle waves lapping at the shore, and the sky above is a clear, light blue. The sun is setting, casting a soft, warm glow over the scene, enhancing the tranquil and joyful atmosphere.

##FP32:
## In this serene beach scene, a woman and her dog share a tender moment. The woman, with long dark hair, is seated on the sandy shore, her legs crossed as she faces the ocean. She is wearing a plaid shirt with rolled-up sleeves, black pants, and sandals. Her eyes are closed, and she is smiling warmly at her canine companion. The dog, a light brown Labrador, sits beside her with its front paws raised, eagerly reaching out to touch her hand. The dog is adorned with a blue harness decorated with pink and green flowers, and a red leash lies on the sand nearby. The beach is calm, with gentle waves lapping at the shore, and the sky above is a clear, light blue. The sun is setting, casting a soft, golden glow over the scene, enhancing the peaceful and joyful atmosphere.


image_url = "http://images.cocodataset.org/train2017/000000411975.jpg"
text = "How many people are there on the baseball field in the picture?？"
##INT4:
## Counting the <points x1="46.5" y1="37.1" x2="58.6" y2="48.3" x3="76.5" y3="33.0" alt="people on the baseball field">people on the baseball field</points> shows a total of 3.

##FP32:
## Counting the <points x1="46.5" y1="37.6" x2="58.5" y2="49.0" x3="76.0" y3="33.1" alt="people on the baseball field">people on the baseball field</points> shows a total of 3.


image_url = "https://intelcorp.scene7.com/is/image/intelcorp/processor-overview-framed-badge:1920-1080?wid=480&hei=270"
text = "Which company does this image represent?"
##INT4:
## The image represents Intel, a well-known technology company. The logo features the text "Intel" in white lowercase letters, followed by "INSIDE" in uppercase letters. This iconic logo design is instantly recognizable and has been a symbol of Intel's brand for many years.

##FP32:
## The image represents Intel, a well-known technology company. The logo features the text "Intel" in white lowercase letters, with "INSIDE" in uppercase letters below it. This iconic logo design is instantly recognizable and associated with Intel's brand in the computer industry.

Evaluation the model

pip3 install git+https://github.com/open-compass/VLMEvalKit.git@7de2dcb.

auto-round-mllm --eval --model OPEA/Molmo-7B-D-0924-int4-sym-inc --tasks MMBench_DEV_EN_V11,ScienceQA_VAL,TextVQA_VAL,POPE --output_dir "./eval_result"

Metric	16bits	INT4
avg	69.22	68.90
MMBench_DEV_EN_V11	70.89	69.20
ScienceQA_VAL	35.81	36.34
TextVQA_VAL	81.16	80.64
POPE	89.02	89.40

Generate the model

Here is the sample command to reproduce the model.

pip install auto-round
auto-round-mllm
--model allenai/Molmo-7B-D-0924 \
--device 0 \
--group_size 128 \
--bits 4 \
--iters 1000 \
--nsamples 512 \
--seqlen 2048 \
--format 'auto_gptq,auto_round' \
--output_dir "./tmp_autoround"

Ethical Considerations and Limitations

The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs.

Therefore, before deploying any applications of the model, developers should perform safety testing.

Caveats and Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.

Here are a couple of useful links to learn more about Intel's AI software:

• Intel Neural Compressor link

Disclaimer

The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please consult an attorney before using this model for commercial purposes.

Cite

@article{cheng2023optimize, title={Optimize weight rounding via signed gradient descent for the quantization of llms}, author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi}, journal={arXiv preprint arXiv:2309.05516}, year={2023} }

arxiv github