Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: update run Jan in Docker mode #2150

Merged
merged 1 commit into from
Feb 26, 2024
Merged

docs: update run Jan in Docker mode #2150

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
133 changes: 77 additions & 56 deletions docs/docs/guides/02-installation/05-docker.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 13,13 @@ keywords:
no-subscription fee,
large language model,
docker installation,
cpu mode,
gpu mode,
]
---

# Installing Jan using Docker

## Installation

### Pre-requisites

:::note
Expand All @@ -37,66 37,87 @@ sudo sh ./get-docker.sh --dry-run

- If you intend to run Jan in GPU mode, you need to install `nvidia-driver` and `nvidia-docker2`. Follow the instruction [here](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) for installation.

### Instructions

- Run Jan in Docker mode

- **Option 1**: Run Jan in CPU mode
### Run Jan in Docker Mode

| Docker compose Profile | Description |
| ---------------------- | -------------------------------------------- |
| `cpu-fs` | Run Jan in CPU mode with default file system |
| `cpu-s3fs` | Run Jan in CPU mode with S3 file system |
| `gpu-fs` | Run Jan in GPU mode with default file system |
| `gpu-s3fs` | Run Jan in GPU mode with S3 file system |

| Environment Variable | Description |
| ----------------------- | ------------------------------------------------------------------------------------------------------- |
| `S3_BUCKET_NAME` | S3 bucket name - leave blank for default file system |
| `AWS_ACCESS_KEY_ID` | AWS access key ID - leave blank for default file system |
| `AWS_SECRET_ACCESS_KEY` | AWS secret access key - leave blank for default file system |
| `AWS_ENDPOINT` | AWS endpoint URL - leave blank for default file system |
| `AWS_REGION` | AWS region - leave blank for default file system |
| `API_BASE_URL` | Jan Server URL, please modify it as your public ip address or domain name default http://localhost:1377 |

- **Option 1**: Run Jan in CPU mode

```bash
# cpu mode with default file system
docker compose --profile cpu-fs up -d

# cpu mode with S3 file system
docker compose --profile cpu-s3fs up -d
```

- **Option 2**: Run Jan in GPU mode

- **Step 1**: Check CUDA compatibility with your NVIDIA driver by running `nvidia-smi` and check the CUDA version in the output

```bash
nvidia-smi

# Output
---------------------------------------------------------------------------------------
| NVIDIA-SMI 531.18 Driver Version: 531.18 CUDA Version: 12.1 |
|----------------------------------------- ---------------------- ----------------------
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|========================================= ====================== ======================|
| 0 NVIDIA GeForce RTX 4070 Ti WDDM | 00000000:01:00.0 On | N/A |
| 0% 44C P8 16W / 285W| 1481MiB / 12282MiB | 2% Default |
| | | N/A |
----------------------------------------- ---------------------- ----------------------
| 1 NVIDIA GeForce GTX 1660 Ti WDDM | 00000000:02:00.0 Off | N/A |
| 0% 49C P8 14W / 120W| 0MiB / 6144MiB | 0% Default |
| | | N/A |
----------------------------------------- ---------------------- ----------------------
| 2 NVIDIA GeForce GTX 1660 Ti WDDM | 00000000:05:00.0 Off | N/A |
| 29% 38C P8 11W / 120W| 0MiB / 6144MiB | 0% Default |
| | | N/A |
----------------------------------------- ---------------------- ----------------------

---------------------------------------------------------------------------------------
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
```

- **Step 2**: Visit [NVIDIA NGC Catalog ](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/cuda/tags) and find the smallest minor version of image tag that matches your CUDA version (e.g., 12.1 -> 12.1.0)

- **Step 3**: Update the `Dockerfile.gpu` line number 5 with the latest minor version of the image tag from step 2 (e.g. change `FROM nvidia/cuda:12.2.0-runtime-ubuntu22.04 AS base` to `FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04 AS base`)

- **Step 4**: Run command to start Jan in GPU mode

```bash
docker compose --profile cpu up -d
# GPU mode with default file system
docker compose --profile gpu up -d

# GPU mode with S3 file system
docker compose --profile gpu-s3fs up -d
```

- **Option 2**: Run Jan in GPU mode

- **Step 1**: Check CUDA compatibility with your NVIDIA driver by running `nvidia-smi` and check the CUDA version in the output

```bash
nvidia-smi

# Output
---------------------------------------------------------------------------------------
| NVIDIA-SMI 531.18 Driver Version: 531.18 CUDA Version: 12.1 |
|----------------------------------------- ---------------------- ----------------------
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|========================================= ====================== ======================|
| 0 NVIDIA GeForce RTX 4070 Ti WDDM | 00000000:01:00.0 On | N/A |
| 0% 44C P8 16W / 285W| 1481MiB / 12282MiB | 2% Default |
| | | N/A |
----------------------------------------- ---------------------- ----------------------
| 1 NVIDIA GeForce GTX 1660 Ti WDDM | 00000000:02:00.0 Off | N/A |
| 0% 49C P8 14W / 120W| 0MiB / 6144MiB | 0% Default |
| | | N/A |
----------------------------------------- ---------------------- ----------------------
| 2 NVIDIA GeForce GTX 1660 Ti WDDM | 00000000:05:00.0 Off | N/A |
| 29% 38C P8 11W / 120W| 0MiB / 6144MiB | 0% Default |
| | | N/A |
----------------------------------------- ---------------------- ----------------------

---------------------------------------------------------------------------------------
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
```

- **Step 2**: Visit [NVIDIA NGC Catalog ](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/cuda/tags) and find the smallest minor version of image tag that matches your CUDA version (e.g., 12.1 -> 12.1.0)

- **Step 3**: Update the `Dockerfile.gpu` line number 5 with the latest minor version of the image tag from step 2 (e.g. change `FROM nvidia/cuda:12.2.0-runtime-ubuntu22.04 AS base` to `FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04 AS base`)

- **Step 4**: Run command to start Jan in GPU mode

```bash
# GPU mode
docker compose --profile gpu up -d
```

This will start the web server and you can access Jan at `http://localhost:3000`.
This will start the web server and you can access Jan at `http://localhost:3000`.

:::warning

- Docker mode is currently only suitable for development and localhost. Production is not supported yet, and the RAG feature is not available in Docker mode.
- RAG feature is not supported in Docker mode with s3fs yet.

:::
Loading