runtime error
Exit code: 1. Reason: mi/python/lib/python3.11/site-packages/deepspeed/ops/op_builder/fp_quantizer.py", line 35, in is_compatible sys_cuda_major, _ = installed_cuda_version() ^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/bitnami/python/lib/python3.11/site-packages/deepspeed/ops/op_builder/builder.py", line 51, in installed_cuda_version raise MissingCUDAException("CUDA_HOME does not exist, unable to compile CUDA op(s)") deepspeed.ops.op_builder.builder.MissingCUDAException: CUDA_HOME does not exist, unable to compile CUDA op(s) Exception ignored in atexit callback: <function matmul_ext_update_autotune_table at 0x7fb112eb1f80> Traceback (most recent call last): File "/opt/bitnami/python/lib/python3.11/site-packages/deepspeed/ops/transformer/inference/triton/matmul_ext.py", line 480, in matmul_ext_update_autotune_table fp16_matmul._update_autotune_table() File "/opt/bitnami/python/lib/python3.11/site-packages/deepspeed/ops/transformer/inference/triton/matmul_ext.py", line 457, in _update_autotune_table TritonMatmul._update_autotune_table(__class__.__name__ + "_2d_kernel", __class__._2d_kernel) File "/opt/bitnami/python/lib/python3.11/site-packages/deepspeed/ops/transformer/inference/triton/matmul_ext.py", line 186, in _update_autotune_table cache_manager.put(autotune_table) File "/opt/bitnami/python/lib/python3.11/site-packages/deepspeed/ops/transformer/inference/triton/matmul_ext.py", line 102, in put with FileLock(self.lock_path): File "/opt/bitnami/python/lib/python3.11/site-packages/filelock/_api.py", line 376, in __enter__ self.acquire() File "/opt/bitnami/python/lib/python3.11/site-packages/filelock/_api.py", line 332, in acquire self._acquire() File "/opt/bitnami/python/lib/python3.11/site-packages/filelock/_unix.py", line 42, in _acquire fd = os.open(self.lock_file, open_flags, self._context.mode) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PermissionError: [Errno 13] Permission denied: '/opt/bitnami/deepspeed/Fp16Matmul_2d_kernel.pickle.lock'
Container logs:
Fetching error logs...