MONAI

/

VISTA3D-HF

like 1

Follow

MONAI 39

Description:
When we include more than 15 labels in the label prompts, the VISTA-3D pipeline will throw the following error

ValueError: Undefined label prompt detected. Provide point prompts for zero-shot.

Steps to reproduce

1. Download the model

from huggingface_hub import snapshot_download

local_dir = snapshot_download(
    repo_id="MONAI/VISTA3D-HF",
    local_dir=".",
)

2. Download a sample image

URL = "https://developer.download.nvidia.com/assets/Clara/monai/samples/ct_liver_0.nii.gz"

import requests

response = requests.get(URL)
with open("ct_liver_0.nii.gz", "wb") as f:
    f.write(response.content)

3. Inference with list [1..15] will succeed:

import os
import tempfile

import torch
from hugging_face_pipeline import HuggingFacePipelineHelper


FILE_PATH = os.path.dirname(__file__)
with tempfile.TemporaryDirectory() as tmp_dir:
    output_dir = os.path.join(tmp_dir, "output_dir")
    pipeline_helper = HuggingFacePipelineHelper("vista3d")
    pipeline = pipeline_helper.init_pipeline(
        os.path.join(FILE_PATH, "vista3d_pretrained_model"),
        device=torch.device("cuda:0"),
    )
    inputs = [
        {
            "image": "ct_liver_0.nii.gz",
            "label_prompt": [i for i in range(1, 16)],  # 16 will succeed, but 17 will fail
        },
    ]
    pipeline(inputs, output_dir=output_dir)

4. Inference with list [1..16] will fail.
   Change the label_prompt to

"label_prompt": [i for i in range(1, 17)],  # 16 will succeed, but 17 will fail