Not Found: Model not found
Hello there, I wasn't sure if this was something that needed to be brought to your attention, but the vast majority, if not all of your models, seem to not be working as off the past hour. If one attempts to use the inference API, it produces error such as "error failed to fetch blob" or something like "this model does not exist" or "not found: model not found". In my spaces, the error logs are showing JSONDecoderErrors. I am not sure if this is something on Huggingfaces end, or if it is something you need to fix on your end (in which case if it is, I do not envy you having to update over 3,000 models.)
Thanks for the report!
I think it's probably a problem on the Hugging Face side.😅 There have been reports of abnormalities related to the Inference API in various places over the past few days.
Hello there, according to status.huggingface.co, the inference API service did experience issues recently, but this was only a few days ago, and it has since then been resolved. If you search for models under your name and filter them for text-to-image and running on the HF Inference API only, only one model is currently loaded, and using the inference text box produces "not found: model not found". Interestingly enough, if you use a space such as your Diffusecraft space which has a Zero GPU, a model of yours can be loaded and generate successfully, but the deployment status on the actual model page still state "This model is not currently available via any of the supported Inference Providers." My spaces are also producing this error for every model during the inference API call in the generation process:
Traceback (most recent call last):
File "/home/user/app/app.py", line 63, in gen_fn
result = models_loadmodel_str
File "/usr/local/lib/python3.10/site-packages/gradio/blocks.py", line 1122, in call
outputs = client_utils.synchronize_async(
File "/usr/local/lib/python3.10/site-packages/gradio_client/utils.py", line 540, in synchronize_async
return fsspec.asyn.sync(fsspec.asyn.get_loop(), func, *args, **kwargs) # type: ignore
File "/usr/local/lib/python3.10/site-packages/fsspec/asyn.py", line 103, in sync
raise return_result
File "/usr/local/lib/python3.10/site-packages/fsspec/asyn.py", line 56, in _runner
result[0] = await coro
File "/usr/local/lib/python3.10/site-packages/gradio/blocks.py", line 1553, in process_api
result = await self.call_function(
File "/usr/local/lib/python3.10/site-packages/gradio/blocks.py", line 1191, in call_function
prediction = await anyio.to_thread.run_sync(
File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2461, in run_sync_in_worker_thread
return await future
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 962, in run
result = context.run(func, *args)
File "/usr/local/lib/python3.10/site-packages/gradio/utils.py", line 659, in wrapper
response = f(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/gradio/external.py", line 415, in query_huggingface_api
errors_json = response.json()
File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 978, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Hope this helps.
The “not found” is too mysterious...
Edit:
I reported it to HF Discord.
I wish I had more information, but all else that I have is a screenshot of the only model that is supposedly deployed and the error it produces:
That's a great tip, even better than you think. It's LoRA. And the base model Kivotos is broken...
https://huggingface.co./yodayo-ai/kivotos-xl-2.0?text=test
This particular model is the only model that is supposedly deployed via the HF Inference API that is one of your models. It is not one the models used in my spaces, but the issue persists not only in my spaces, but all CPU based spaces that use your models. This is a very strange issue.
Actually, there are still some models that are causing problems even with the LLM. Some of the models in the Inference API are clearly malfunctioning in ways I've never seen before.
Edit:
The HF staff are currently verifying the matter, including the LLM.
Have the staff been able to determine why the Loras are breaking? I noticed that they added another third-party API, but I wonder if they updated the API in a way that caused everything to break, such as an errant comma where a period should be, or a typo.
I apologize for bothering you once more, but considering you are very active on the huggingface community (and I presume the discord as well), I was wondering if you knew if the staff have figured out how to get the LoRas working on serverless spaces once more. They are being frustratingly opaque and have not made a single post in the posts section of the website that I can see stating they are working on the issue, beyond the single comment from the post linked above. It has been almost 2 weeks, and at this point I'm wondering if this issue is such a low priority for them that they intend to let the situation fester for several months, if they intend to fix it at all.
I haven't seen or heard anything since then, including on Discord. In any case, it seems that the Inference API is undergoing a major overhaul, so it may be unstable for a while.🤔
Well, I think we just have to wait for this time, but we'll put together a list of links we can use to make requests.
https://github.com/huggingface/hub-docs/issues
https://huggingface.co./posts/victor/964839563451127
[email protected]
https://huggingface.co./join/discord
https://huggingface.co./spaces/zero-gpu-explorers/README/discussions # for Zero GPU related issue
Hello there, I ran across something interesting. Within the last 24 hours, a user named Uthar duplicated one of your illustrious models, and interestingly enough, their model is deployed and functional with the HF Inference API. As far as I can tell, they duplicated your model repo and made zero changes other than the model name, yet their model is functional. At the same time, yours are still seemingly unavailable in serverless environments. I cannot give you a screenshot of my test output due to the output of the word prompt "test" being NSFW, I can give you the link to the model. As of the moment I am posting this, the model is deployed via the HF Inference API while not being used in any public spaces. I wonder if there is something they did that restored function to the HF Inference API test widget. and, therefore, core functionality within spaces. https://huggingface.co./Uthar/John6666_zukiillustrious-cyberfixitercomp-cutev10cyberfixitercomp-sdxl
EDIT: Right after I post this, it starts to give the "failed to fetch" errors again without explanation, so maybe the issue is still persistent or is in a state of sometimes it works.
Non NSFW Screenshot:
I find it interesting that a duplicate of your model works, while the original you uploaded doesn't.
Hello once again. I have been spending the last 3 hours using QwQ 32B to try and fix my spaces, and the chatbot mentioned something that I want you to confirm on your end, if possible. The chatbot mentioned trying something on the model page that you, being the owner, are only able to do. I am not sure if this is something that exists due to the recent updates to Inference APIs, but it doesn't hurt to verify.
While I am no longer getting the JSON errors from before with the code that QwQ helped me update, I am now receiving API errors, so maybe this could be the root cause of everything.
EDIT: Nevermind on the JSON errors, they still seem to be occurring, so I still have to work that out assuming the fix above doesn't work or amount to anything.
Hmm, I don't see that option...
The specifications for Hugging Face often change, and there are many cases where there is no explanation, so for example, that option may have existed in the past, and the chatbot may have known about the old information.🤔