mosaicml

/

mpt-7b-instruct

like 467

Follow

Mosaic ML, Inc. 352

** Duplicate of https://huggingface.co./mosaicml/mpt-7b/discussions/47 **

This PR adds updates from the LLM Foundry repo as of 06/01/2023.

These include:

• device_map support for multiple GPUs
• faster inference thanks to a refactor of the KV cacheing
• bugfix for returning the last hidden_state
• support for output_attentions when using attn_impl: torch
• a requirements.txt file to make it easier to know what you need to install for MPT
• updated README instructions for fast GPU initialization