THUDM

/

chatglm3-6b

like 1.09k

Follow

Knowledge Engineering Group (KEG) & Data Mining at Tsinghua University 1,211

环境：centerOS7.9 ，两张nvida-A10显卡，python版本：3.11，pytorch版本：2.2.1 , CUDA Version: 12.2 Driver Version: 535.129.03
模型启动正常，单人去对话也正常，但是在同时有多人去使用或者连续请求的时候，就会有这个报错，但是模型并没有挂掉。
之前在微软云服务器上使用过同样版本的模型，也是多人使用，却没有问题。
切换到8k，和使用不同的启动方式，都有这样的问题，直接调用/v1/chat/completions没有问题，问题发生时都是在使用langchain的ChatOpenAI方法调用，流式输出时。
在网上找过很多解决方案，包括禁用do_sample等，均没有效果。求助！
Environment: CenterOS7.9, two NVida-A10 graphics cards, Python version: 3.11, PyTorch version: 2.2.1, CUDA version: 12.2 Driver version: 535.129.03
The model starts normally, and it's normal for a single person to have a conversation. However, when multiple people are using it or making continuous requests, this error message may occur, but the model did not crash.
I have previously used the same version of the model on Microsoft Cloud Server and it has been used by multiple people without any issues.
Switching to 8k and using different startup methods, there is no problem with directly calling/v1/chat/completion. When the problem is happening, it is called using the ChatOpenAI method of Langchain and outputted in a streaming manner.
I have found many solutions online, including disabling do_sample, but none of them have been effective. Help!