-
Notifications
You must be signed in to change notification settings - Fork 961
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
llama-server with cpu device is not working in docker image #2634
Comments
Hi, thanks for reporting the issue. For a workaround, I recommend use the linux binary distribution directly: https://tabby.tabbyml.com/docs/quick-start/installation/linux/#download-the-release |
I also encountered the same error😭 |
The issue seems to be related to the llama-server, the |
updates cuda and documentation regarding running tabby inside docker containers with cuda support
Submitted the pull request #2711 . In the meanwhile you can use my temporary image If you're using docker compose you can use the following snippet: version: '3.8'
services:
tabby:
restart: always
image: tabbyml/tabby
command: serve --model StarCoder-1B --chat-model Qwen2-1.5B-Instruct --device cuda
volumes:
- "$HOME/.tabby:/data"
ports:
- 8080:8080
environment:
- PATH=/usr/local/cuda/bin:$PATH
- LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/compat:$LD_LIBRARY_PATH
- NVIDIA_VISIBLE_DEVICES=all
- NVIDIA_DRIVER_CAPABILITIES=compute,utility
runtime: nvidia
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu] If you're using docker you can use the following snippet: docker run -it --gpus all \
-p 8080:8080 \
-v $HOME/.tabby:/data \
-e PATH=/usr/local/cuda/bin:$PATH \
-e LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/compat:$LD_LIBRARY_PATH \
tabbyml/tabby serve --model StarCoder-1B --chat-model Qwen2-1.5B-Instruct --device cuda |
@0x4139 nope this is not my issue. I want to run it without a GPU. Just CPU Mode. |
Same error here. |
For those experiencing the issue, please refer to the comment at #2634 (comment) to see if it resolves the problem for you. If it doesn't, feel free to share your experiences. Thank you! |
The issues are related, the binary won't start even in cpu mode due to the fact that the cuda libraries are not linked. Just tested it now, and it works also in cpu mode. |
@0x4139 ur docker command and compose uses different images. |
I'm experiencing similar issue, but for me Docker image works fine, it's Linux release that doesn't work. Error:
I'm using command: Adding env as suggested #2634 (comment) doesn't help: EndeavourOS, tabby P.S. Is it fine to discuss it here or should I open new issue? |
Be sure you have installed |
Thank that fixed it. Now I'm getting:
Seems similar to this #2803 |
Could you provide a broader view of the logs, as well as your tabby configuration? |
I'm on EndeavourOS and I've downloaded https://github.com/TabbyML/tabby/releases/tag/v0.14.0 / Command
It just goes on forever. Here's
Where can I find tabby configuration so I can provide it? I looked at |
Seems to be related to some flags passed to the |
tabby build the image with cuda by default, Line 42 in 48d9c08
that's why llama-cpp-server looks for the libcuda, and failed to start if no GPU existed. the libcuda is mounted at runtime by nvidia-container-runtime. maybe we need a cpu Dockerfile to build the CPU image? it can also largely reduce the image size without the cuda dependencies. WDYT @wsxiaoys |
Thank you that fixed the problem. v15 was released but I get the same error on it. Is there any issue/pr that related to it? So I can monitor when it's safe to upgrade. |
There is a merge request here #2711 |
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
which is document here https://tabby.tabbyml.com/docs/quick-start/installation/docker-compose/ wont work
Originally posted by @b-reich in #2082 (comment)
The text was updated successfully, but these errors were encountered: