T4 - bfloat 16 not support

#2
by SylvainV - opened

Hi,
Thanks for your work.
I'd like to know if you plan to release an other version of the model in float16 instead of bfloat16.

Best regards,

+1, I think this can be supported easily by adding a device argument to the chat method call, which is then used here: https://huggingface.co./ucaslcl/GOT-OCR2_0/blob/main/modeling_GOT.py#L566

this notebook works on colab but with the L4 gpu not the T4 https://colab.research.google.com/drive/1J_2SyiGvbBt2ohg_aoMyXpc0xjg9LFJg?usp=sharing

T4 support colab , feel free to like if it's worked fine : https://colab.research.google.com/drive/1oMA9u4M_hT5gxd80TptA5ik2xnERCC8F?usp=sharing

T4 support colab , feel free to like if it's worked fine : https://colab.research.google.com/drive/1oMA9u4M_hT5gxd80TptA5ik2xnERCC8F?usp=sharing

@jeanflop Can you please share access to the colab file you shared?

T4 support colab , feel free to like if it's worked fine : https://colab.research.google.com/drive/1oMA9u4M_hT5gxd80TptA5ik2xnERCC8F?usp=sharing

@jeanflop Can you please share access to the colab file you shared?

sure , done

Why not load it in float32 then? It's just a sub-1B model.

This comment has been hidden

Managed to run on CPU in Windows 10
Updated files in archive link

https://drive.google.com/file/d/1cUMwQyWDtk0XUsdYEKbrFlpUIU-Okl8x/view?usp=sharing

Note that the run.py file is located in the model folder.

Screenshot_2.jpg

/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1150: FutureWarning: resume_download is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use force_download=True.
warnings.warn(
/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1150: FutureWarning: resume_download is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use force_download=True.
warnings.warn(
/usr/local/lib/python3.10/dist-packages/gradio/components/dropdown.py:188: UserWarning: The value passed into gr.Dropdown() is not in the list of choices. Please update the list of choices to include: or set allow_custom_value=True.
warnings.warn(
Running on local URL: http://127.0.0.1:7860

To create a public link, set share=True in launch().

not run in broser

Sign up or log in to comment