Error running the model
I'm following the tutorial here: https://github.com/microsoft/onnxruntime-genai/blob/main/examples/python/phi-3-tutorial.md
Trying to run \cpu_and_mobile\cpu-int4-rtn-block-32-acc-level-4 on my PC and I see the following error after supplying an input:
2024-04-23 15:14:06.4954767 [E:onnxruntime:onnxruntime-genai, sequential_executor.cc:516 onnxruntime::ExecuteKernel] Non-zero status code returned while running Add node. Name:'/model/attn_mask_reformat/Add' Status Message: D:\a_work\1\s\onnxruntime\core/providers/cpu/math/element_wise_ops.h:540 onnxruntime::BroadcastIterator::Init axis == 1 || axis == largest was false. Attempting to broadcast an axis by a dimension other than 1. 7 by 8
Traceback (most recent call last):
File "c:\Phi-3-mini-4k-instruct-onnx\model-qa.py", line 82, in
main(args)
File "c:\Phi-3-mini-4k-instruct-onnx\model-qa.py", line 47, in main
generator.compute_logits()
onnxruntime_genai.onnxruntime_genai.OrtException: Non-zero status code returned while running Add node. Name:'/model/attn_mask_reformat/Add' Status Message: D:\a_work\1\s\onnxruntime\core/providers/cpu/math/element_wise_ops.h:540 onnxruntime::BroadcastIterator::Init axis == 1 || axis == largest was false. Attempting to broadcast an axis by a dimension other than 1. 7 by 8
Thoughts on what could be wrong?
could you share which device your are running this on (OS name, system type, system SKU etc)
@arunraghuram , are you using onnxruntime-genai-directml package to run the cpu model? it is a bug and we have a fix in main branch. Will have a new release to fix it soon.
To work around it, you can either install the onnxruntime-genai cpu package to run it or build from source. Will let you know once a new release is out. Here is the instructions to build from source:
https://onnxruntime.ai/docs/genai/howto/build-from-source.html#option-3-build-from-source
Yes, I was using onnxruntime-genai-directml running it on Windows 10. Okay, will wait for the release.
@arunraghuram , we just published RC3. It should solve the issue. Could you please update your package onnxruntime-genai-directml and try again?
It works now! Thanks a lot.