Includes free use of the Robusta AI model.

Sign up for Robusta SaaS (Kubernetes cluster required) or contact us about on-premise options.

Investigate Prometheus alerts right from Slack with the official Robusta integration.

Or run HolmesGPT from the cli:

kubectl port-forward alertmanager-robusta-kube-prometheus-st-alertmanager-0 9093:9093 &
holmes investigate alertmanager --alertmanager-url http://localhost:9093

Note - if on Mac OS and using the Docker image, you will need to use http://docker.for.mac.localhost:9093 instead of http://localhost:9093

Via the Holmes CLI or a free UI (video)

holmes ask "what pods are in crashloopbackoff in my cluster and why?"

holmes investigate opsgenie --opsgenie-api-key <PLACEHOLDER_APIKEY>

By default results are displayed in the CLI . Use --update --opsgenie-team-integration-key <PLACEHOLDER_TEAM_KEY> to get the results as a comment in the OpsGenie alerts. Refer to the CLI help for more info.

holmes investigate pagerduty --pagerduty-api-key <PLACEHOLDER_APIKEY>

By default results are displayed in the CLI. Use --update --pagerduty-user-email <PLACEHOLDER_EMAIL> to get the results as a comment in the PagerDuty issue. Refer to the CLI help for more info.

You can add HolmesGPT as a plugin for K9s to investigate why any Kubernetes resource is unhealthy.

Add the following contents to the K9s plugin file, typically ~/.config/k9s/plugins.yaml on Linux and ~/Library/Application Support/k9s/plugins.yaml on Mac. Read more about K9s plugins here and check your plugin path here.

Note: HolmesGPT must be installed and configured for the K9s plugin to work.

Basic plugin to run an investigation on any Kubernetes object, using the shortcut Shift H:

plugins:
  holmesgpt:
    shortCut: Shift-H
    description: Ask HolmesGPT
    scopes:
      - all
    command: bash
    background: false
    confirm: false
    args:
      - -c
      - |
        holmes ask "why is $NAME of $RESOURCE_NAME in -n $NAMESPACE not working as expected"
        echo "Press 'q' to exit"
        while : ; do
        read -n 1 k <&1
        if [[ $k = q ]] ; then
        break
        fi
        done

Advanced plugin that lets you modify the questions HolmesGPT asks about the LLM, using the shortcut Shift O. (E.g. you can change the question to "generate an HPA for this deployment" and the AI will follow those instructions and output an HPA configuration.)

plugins:
  custom-holmesgpt:
    shortCut: Shift-Q
    description: Custom HolmesGPT Ask
    scopes:
      - all
    command: bash

      - |
        INSTRUCTIONS="# Edit the line below. Lines starting with '#' will be ignored."
        DEFAULT_ASK_COMMAND="why is $NAME of $RESOURCE_NAME in -n $NAMESPACE not working as expected"
        QUESTION_FILE=$(mktemp)

        echo "$INSTRUCTIONS" > "$QUESTION_FILE"
        echo "$DEFAULT_ASK_COMMAND" >> "$QUESTION_FILE"

        # Open the line in the default text editor
        ${EDITOR:-nano} "$QUESTION_FILE"

        # Read the modified line, ignoring lines starting with '#'
        user_input=$(grep -v '^#' "$QUESTION_FILE")
        echo running: holmes ask "\"$user_input\""

        holmes ask "$user_input"
        echo "Press 'q' to exit"
        while : ; do
        read -n 1 k <&1
        if [[ $k = q ]] ; then
        break
        fi
        done

You can use Holmes as a library and pass in your own LLM implementation. This is particularly useful if LiteLLM or the default Holmes implementation does not suit you.

See an example implementation here.

1. Add our tap:

brew tap robusta-dev/homebrew-holmesgpt

2. Install holmesgpt:

brew install holmesgpt

3. Check that installation was successful. This will take a few seconds on the first run - wait patiently.:

holmes --help

4. Run holmesgpt:

holmes ask "what issues do I have in my cluster"

Run the prebuilt Docker container docker.pkg.dev/genuine-flight-317411/devel/holmes, with extra flags to mount relevant config files (so that kubectl and other tools can access AWS/GCP resources using your local machine's credentials)

docker run -it --net=host -v ~/.holmes:/root/.holmes -v ~/.aws:/root/.aws -v ~/.config/gcloud:/root/.config/gcloud -v $HOME/.kube/config:/root/.kube/config us-central1-docker.pkg.dev/genuine-flight-317411/devel/holmes ask "what pods are unhealthy and why?"

You can install HolmesGPT from the latest git version with pip or pipx.

We recommend using pipx because it guarantees that HolmesGPT is isolated from other python packages on your system, preventing dependency conflicts.

First Pipx (skip this step if you are using pip).

Then install HolmesGPT from git with either pip or pipx:

pipx install "https://github.com/robusta-dev/holmesgpt/archive/refs/heads/master.zip"

Verify that HolmesGPT was installed by checking the version:

holmes version

To upgrade HolmesGPT with pipx, you can run:

pipx upgrade holmesgpt

First install poetry (the python package manager)

git clone https://github.com/robusta-dev/holmesgpt.git
cd holmesgpt
poetry install --no-root
poetry run python3 holmes.py ask "what pods are unhealthy and why?"

Clone the project from github, and then run:

cd holmesgpt
docker build -t holmes . -f Dockerfile.dev
docker run -it --net=host -v -v ~/.holmes:/root/.holmes -v ~/.aws:/root/.aws -v ~/.config/gcloud:/root/.config/gcloud -v $HOME/.kube/config:/root/.kube/config holmes ask "what pods are unhealthy and why?"

Most users should install Holmes using the instructions in the Robusta docs ↗ and NOT the below instructions.

By using the Robusta integration you’ll benefit from an end-to-end integration that integrates with Prometheus alerts and Slack. Using the below instructions you’ll have to build many of those components yourself.

In this mode, all the parameters should be passed to the HolmesGPT deployment, using environment variables.

We recommend pulling sensitive variables from Kubernetes secrets.

First, you'll need to create your holmes-values.yaml file, for example:

additionalEnvVars:
- name: MODEL
  value: gpt-4o
- name: OPENAI_API_KEY
  value: <your open ai key>

Then, install with helm;

helm repo add robusta https://robusta-charts.storage.googleapis.com && helm repo update
helm install holmes robusta/holmes -f holmes-values.yaml

For all LLMs you need to provide the MODEL environment variable, which specifies which model you are using.

Some LLMs requires additional variables:

additionalEnvVars:
- name: MODEL
  value: gpt-4o
- name: OPENAI_API_KEY
  valueFrom:
    secretKeyRef:
      name: my-holmes-secret
      key: openAiKey

additionalEnvVars:
- name: MODEL
  value: azure/my-azure-deployment         # your azure deployment name
- name: AZURE_API_VERSION
  value: 2024-02-15-preview                # azure openai api version
- name: AZURE_API_BASE
  value: https://my-org.openai.azure.com/  # base azure openai url
- name: AZURE_API_KEY
  valueFrom:
    secretKeyRef:
      name: my-holmes-secret
      key: azureOpenAiKey

enablePostProcessing: true
additionalEnvVars:
- name: MODEL
  value: bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0
- name: AWS_REGION_NAME
  value: us-east-1
- name: AWS_ACCESS_KEY_ID
  valueFrom:
    secretKeyRef:
      name: my-holmes-secret
      key: awsAccessKeyId
- name: AWS_SECRET_ACCESS_KEY
  valueFrom:
    secretKeyRef:
      name: my-holmes-secret
      key: awsSecretAccessKey

To work with OpenAI’s GPT 3.5 or GPT-4 models you need a paid OpenAI API key.

Note: This is different from being a “ChatGPT Plus” subscriber.

Pass your API key to holmes with the --api-key cli argument. Because OpenAI is the default LLM, the --model flag is optional for OpenAI (gpt-4o is the default).

holmes ask --api-key="..." "what pods are crashing in my cluster and why?"

If you prefer not to pass secrets on the cli, set the OPENAI_API_KEY environment variable or save the API key in a HolmesGPT config file.

To work with Azure AI, you need an Azure OpenAI resource and to set the following environment variables:

• AZURE_API_VERSION - e.g. 2024-02-15-preview
• AZURE_API_BASE - e.g. https://my-org.openai.azure.com/
• AZURE_API_KEY (optional) - equivalent to the --api-key cli argument

Set those environment variables and run:

holmes ask "what pods are unhealthy and why?" --model=azure/<DEPLOYMENT_NAME> --api-key=<API_KEY>

Refer LiteLLM Azure docs ↗ for more details.

Before running the below command you must run pip install boto3>=1.28.57 and set the following environment variables:

• AWS_REGION_NAME
• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY

If the AWS cli is already configured on your machine, you may be able to find those parameters with:

cat ~/.aws/credentials ~/.aws/config

Once everything is configured, run:

holmes ask "what pods are unhealthy and why?" --model=bedrock/<MODEL_NAME>

Be sure to replace MODEL_NAME with a model you have access to - e.g. anthropic.claude-3-5-sonnet-20240620-v1:0. To list models your account can access:

aws bedrock list-foundation-models --region=us-east-1

Note that different models are available in different regions. For example, Claude Opus is only available in us-west-2.

Refer to LiteLLM Bedrock docs ↗ for more details.

If you'd like to try using Ollama anyway, see below:

export OLLAMA_API_BASE="http://localhost:11434"
holmes ask "what pods are unhealthy in my cluster?" --model="ollama_chat/llama3.1"

You can also connect to Ollama in the standard OpenAI format (this should be equivalent to the above):

# note the v1 at the end
export OPENAI_API_BASE="http://localhost:11434/v1"
# holmes requires OPENAPI_API_KEY to be set but value does not matter
export OPENAI_API_KEY=123
holmes ask "what pods are unhealthy in my cluster?" --model="openai/llama3.1"

To use Gemini, set the GEMINI_API_KEY environment variable as follows:

export GEMINI_API_KEY="your-gemini-api-key"

Once the environment variable is set, you can run the following command to interact with Gemini:

holmes ask "what pods are unhealthy and why?" --model=gemini/<MODEL_NAME>

Be sure to replace MODEL_NAME with a model you have access to - e.g., gemini-pro,gemini/gemini-1.5-flash, etc.

To use Vertex AI with Gemini models, set the following environment variables:

export VERTEXAI_PROJECT="your-project-id"
export VERTEXAI_LOCATION="us-central1" 
export GOOGLE_APPLICATION_CREDENTIALS="path/to/your/service_account_key.json"

Once the environment variables are set, you can run the following command to interact with Vertex AI Gemini models:

poetry run python holmes.py ask "what pods are unhealthy and why?" --model "vertex_ai/<MODEL_NAME>"

Be sure to replace MODEL_NAME with a model you have access to - e.g., gemini-pro,gemini-2.0-flash-exp, etc. Ensure you have the correct project, location, and credentials for accessing the desired Vertex AI model.

You will need an LLM with support for function-calling (tool-calling).

• Set the environment variable for your URL with OPENAI_API_BASE
• Set the model as openai/<your-model-name> (e.g., llama3.1:latest)
• Set your API key (if your URL doesn't require a key, then add a random value for --api-key)

export OPENAI_API_BASE=<URL_HERE>
holmes ask "what pods are unhealthy and why?" --model=openai/<MODEL_NAME> --api-key=<API_KEY_HERE>

Important: Please verify that your model and inference server support function calling! HolmesGPT is currently unable to check if the LLM it was given supports function-calling or not. Some models that lack function-calling capabilities will hallucinate answers instead of reporting that they are unable to call functions. This behaviour depends on the model.

In particular, note that vLLM does not yet support function calling, whereas llama-cpp does support it.

• CONFLUENCE_BASE_URL - e.g. https://robusta-dev-test.atlassian.net
• CONFLUENCE_USER - e.g. [email protected]
• CONFLUENCE_API_KEY - refer to Atlassian docs on generating API keys

HolmesGPT can pull tickets/alerts from each of these sources and investigate them.

Refer to holmes investigate jira --help etc for details.

HolmesGPT can consult webpages containing runbooks or other relevant information. HolmesGPT uses playwright to scrape webpages and requires playwright to be installed and working through playwright install.

Holmes can use the argocd CLI to get details about the ArgoCD setup like the apps configuration and status, clusters and projects within ArgoCD. To enable ArgoCD, set the ARGOCD_AUTH_TOKEN environment variable as described in the argocd documentation.

LLM uses the built-in Helm toolset to gather information.

holmes ask "what helm value should I change to increase memory request of the my-argo-cd-argocd-server-6864949974-lzp6m pod"

LLM uses the built-in Docker toolset to gather information.

holmes ask "Tell me what layers of my pavangudiwada/robusta-ai docker image consume the most storage and suggest some fixes to it"

The more data you give HolmesGPT, the better it will perform. Give it access to more data by adding custom tools.

New tools are loaded using -t from custom toolset files or by adding them to the ~/.holmes/config.yaml with the setting custom_toolsets: ["/path/to/toolset.yaml"].

HolmesGPT can investigate by following runbooks written in plain English. Add your own runbooks to provided the LLM specific instructions.

New runbooks are loaded using -r from custom runbook files or by adding them to the ~/.holmes/config.yaml with the custom_runbooks: ["/path/to/runbook.yaml"].

You can customize HolmesGPT's behaviour with command line flags, or you can save common settings in config file for re-use.

You can view an example config file with all available settings here.

By default, without specifying --config the agent will try to read ~/.holmes/config.yaml. When settings are present in both config file and cli, the cli option takes precedence.

# Add paths to your custom toolsets here
# Example: ["path/to/your/custom_toolset.yaml"]
#custom_toolsets: ["examples/custom_toolset.yaml"]

# URL for the Alertmanager
#alertmanager_url: "http://localhost:9093"

# Jira credentials and query settings
#jira_username: "[email protected]"
#jira_api_key: "..."
#jira_url: "https://your-company.atlassian.net"
#jira_query: "project = 'Natan Test Project' and Status = 'To Do'"

# GitHub credentials and query settings
#github_owner: "robusta-dev"
#github_pat: "..."
#github_url: "https://api.github.com" (default)
#github_repository: "holmesgpt"
#github_query: "is:issue is:open"

pagerduty_api_key: "..."
pagerduty_user_email: "[email protected]"
pagerduty_incident_key:  "..."

opsgenie_api_key : "..."
opsgenie-team-integration-key: "...."
opsgenie-query: "..."

# Slack token and channel configuration
#slack_token: "..."
#slack_channel: "#general"

# Add paths to your custom runbooks here
# Example: ["path/to/your/custom_runbook.yaml"]
#custom_runbooks: ["examples/custom_runbooks.yaml"]

Choose between OpenAI, Azure, AWS Bedrock, and more. Provide the necessary API keys and endpoints for the selected service.

# Configuration for OpenAI LLM
#api_key: "your-secret-api-key"

# Configuration for Azure LLM
#api_key: "your-secret-api-key"
#model: "azure/<DEPLOYMENT_NAME>"
#you will also need to set environment variables - see above

# Configuration for AWS Bedrock LLM
#model: "bedrock/<MODEL_ID>"
#you will also need to set environment variables - see above

You can contribute knowledge on solving common alerts and HolmesGPT will use this knowledge to solve related issues. To do so, add a new file to ./holmes/plugins/runbooks - or edit an existing runbooks file in that same directory.

Note: if you prefer to keep your runbooks private, you can store them locally and pass them to HolmesGPT with the -r flag. However, if your runbooks relate to common problems that others may encounter, please consider opening a PR and making HolmesGPT better for everyone!

You can add define new tools in YAML and HolmesGPT will use those tools in it's investigation. To do so, add a new file to ./holmes/plugins/toolsets - or edit an existing toolsets file in that same directory.

Note: if you prefer to keep your tools private, you can store them locally and pass them to HolmesGPT with the -t flag. However, please consider contributing your toolsets! At least one other community member will probably find them useful!

The default prompts for HolmesGPT are located in ./holmes/plugins/prompts. Most holmes commands accept a --system-prompt flag that you can use to override this.

If you find a scenario where the default prompts don't work, please consider letting us know by opening a GitHub issue or messaging us on Slack! We have an internal evaluation framework for benchmarking prompts on many troubleshooting scenarios and if you share a case where HolmesGPT doesn't work, we will be able to add it to our test framework and fix the performance on that issue and similar ones.

If you want HolmesGPT to investigate external tickets or alert, you can add a new datasource. This requires modifying the source code and opening a PR. You can see an example PR like that here, which added support for investigating GitHub issues.