Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

max_memory? #10

Open
fabianocastello opened this issue May 5, 2023 · 2 comments
Open

max_memory? #10

fabianocastello opened this issue May 5, 2023 · 2 comments

Comments

@fabianocastello
Copy link

I'm quite new to LLM world. I'm brazilian and decided to start with Cabrita. I managed to deal with several errors but i am stuck with this:

"Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit
the quantized model. If you have set a value for max_memory you should increase that. To have
an idea of the modules that are set on the CPU or RAM you can print model.hf_device_map."

I tried several solutions but none work. Is it a memory problem?

I have a MacBook Pro i7 16Gb (2018)
@feliperafael
Copy link

Probably your issue really is a lack of memory, since the minimum required to run this model (with 7 billion parameters) is around 12gb of VRAM (GPU RAM memory). You can try using GPUs from google colab, or use a computer with at least 32gb of RAM and a video card with at least 12gb of VRAM

@matthsena
Copy link

I was having the same problem trying to fine tune the alpaca-lora, to solve it i had to subscribe to Colab Pro to use more powerful GPUs and more RAM. I didn't try to run the cabrita-lora.ipynb before that, but try to run it in the free version of Colab, if you get an error the solution may be the same one I took

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@feliperafael @fabianocastello @matthsena