Qwen

/

Qwen2-VL-2B-Instruct

like 405

Follow

Qwen 20k

What can I do?

Hello,
Make sure you use the latest HF LLM DLC. Here is a code snippet that works.

!pip install sagemaker --upgrade
import json
import sagemaker
import boto3
from sagemaker.huggingface import HuggingFaceModel, get_huggingface_llm_image_uri

try:
    role = sagemaker.get_execution_role()
except ValueError:
    iam = boto3.client("iam")
    role = iam.get_role(RoleName="sagemaker_execution_role")["Role"]["Arn"]
# Hub Model configuration. https://huggingface.co./models
hub = {
    'HF_MODEL_ID':'Qwen/Qwen2-VL-2B-Instruct',
    'SM_NUM_GPUS': json.dumps(1),
    'CUDA_GRAPHS': json.dumps(0)
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
    image_uri=get_huggingface_llm_image_uri("huggingface"),
    env=hub,
    role=role,
)

# deploy model to SageMaker Inference
predictor = huggingface_model.deploy(
    initial_instance_count=1,
    instance_type="ml.g5.2xlarge",
    container_startup_health_check_timeout=300,
)