Description

Showcasing The Problem

When executing the attached code locally, everything runs perfectly.

When deploying the above, and creating a run from the UI, I get one of the following errors:

**ValueError: Task #b454bc74 ("Provide a detailed analysis of how similar compani...") failed: Encountered HTTP 429 errors while trying to fetch data using the Tavily search tool, indicating rate limits have been exceeded. Unable to complete the task due to technical limitations.**

Failed due to a(n) ValueError Task #2094992e failed: Encountered HTTP 429 errors while attempting to retrieve data for the Athleisure industry. Unable to complete the research task due to request limits being exceeded.

Context - The Most Important Part

The context is that I am creating an agent with the TavilySearchResults tool from langchain-community.

I have a paid account with Tavily, and after contacting their customer support, they are certain that I have adequate tokens and I am not crossing their rate limits.

The fact that the same exact code runs perfectly locally, gives me a hint that when executed from Prefect Cloud, the bug ensues.

An important thing to add, is that the deployment runs on an AWS Push Work Pool.

Cloudwatch Example Logs

  | 2024-12-17T22:59:09.428Z | 22:59:09.427 | INFO | Task run 'Agent turn: Industry Researcher' - Finished in state Completed()
  | 2024-12-17T22:59:09.444Z | ╰──────────────────────────────────────────────────────────────── 10:58:59 PM ─╯The extracted industry is: industry='Fashion'.
  | 2024-12-17T22:59:09.444Z | ╭─ Agent: Industry Researcher ─────────────────────────────────────────────────╮
  | 2024-12-17T22:59:09.444Z | │ │
  | 2024-12-17T22:59:09.444Z | │ ✅ Tool call: "tavily_search_results_json" │
  | 2024-12-17T22:59:09.444Z | │ │
  | 2024-12-17T22:59:09.444Z│ Tool args: {'query': 'current global market performance of fashion │ | │ Tool args: {'query': 'current global market performance of fashion │
  | 2024-12-17T22:59:09.444Z | │ industry 2023'} │
  | 2024-12-17T22:59:09.444Z | │ │
  | 2024-12-17T22:59:09.444Z│ Tool result: ["HTTPError('429 Client Error: Too Many Requests for url: │ | │ Tool result: ["HTTPError('429 Client Error: Too Many Requests for url: │
  | 2024-12-17T22:59:09.444Z | │ https://api.tavily.com/search')",{}] │
  | 2024-12-17T22:59:09.444Z | │ │
  | 2024-12-17T22:59:09.444Z | ╰──────────────────────────────────────────────────────────────── 10:59:01 PM ─╯
  | 2024-12-17T22:59:09.444Z | ╭─ Agent: Industry Researcher ─────────────────────────────────────────────────╮
  | 2024-12-17T22:59:09.444Z | │ │
  | 2024-12-17T22:59:09.444Z | │ ✅ Tool call: "tavily_search_results_json" │
  | 2024-12-17T22:59:09.444Z | │ │
  | 2024-12-17T22:59:09.444Z | │ Tool args: {'query': 'key performance indicators fashion industry │
  | 2024-12-17T22:59:09.444Z | │ 2023'} │
  | 2024-12-17T22:59:09.444Z | │ │
  | 2024-12-17T22:59:09.444Z | │ Tool result: ["HTTPError('429 Client Error: Too Many Requests for url: │
  | 2024-12-17T22:59:09.444Z | │ https://api.tavily.com/search')",{}] │
  | 2024-12-17T22:59:09.444Z | │ │
  | 2024-12-17T22:59:09.444Z | ╰──────────────────────────────────────────────────────────────── 10:59:04 PM ─╯
  | 2024-12-17T22:59:09.444Z | ╭─ Agent: Industry Researcher ─────────────────────────────────────────────────╮
  | 2024-12-17T22:59:09.444Z | │ │
  | 2024-12-17T22:59:09.444Z | │ ✅ Tool call: "tavily_search_results_json" │
  | 2024-12-17T22:59:09.444Z | │ │
  | 2024-12-17T22:59:09.444Z | │ Tool args: {'query': 'fashion industry sentiment 2023'} │
  | 2024-12-17T22:59:09.444Z | │ │
  | 2024-12-17T22:59:09.444Z | │ Tool result: ["HTTPError('429 Client Error: Too Many Requests for url: │
  | 2024-12-17T22:59:09.444Z | │ https://api.tavily.com/search')",{}] │
  | 2024-12-17T22:59:09.444Z | │ │
  | 2024-12-17T22:59:09.444Z | ╰──────────────────────────────────────────────────────────────── 10:59:06 PM ─╯
  | 2024-12-17T22:59:09.444Z | ╭─ Agent: Industry Researcher ─────────────────────────────────────────────────╮
  | 2024-12-17T22:59:09.444Z | │ │
  | 2024-12-17T22:59:09.444Z | │ ✅ Tool call: "mark_task_90892026_failed" │
  | 2024-12-17T22:59:09.444Z | │ │
  | 2024-12-17T22:59:09.444Z | │ Tool args: {'reason': "Received '429 Client Error: Too Many Requests' │
  | 2024-12-17T22:59:09.444Z | │ error when attempting to retrieve data from the Tavily search API for │
  | 2024-12-17T22:59:09.444Z | │ multiple queries related to the fashion industry analysis. Unable to │
  | 2024-12-17T22:59:09.444Z | │ complete the analysis without access to the necessary information."} │
  | 2024-12-17T22:59:09.444Z | │ │
  | 2024-12-17T22:59:09.444Z│ Tool result: Task #90892026 ("Provide a detailed analysis of how │ | │ Tool result: Task #90892026 ("Provide a detailed analysis of how │

Example Code

import controlflow as cf
from prefect.context import FlowRunContext
import requests
from pydantic import BaseModel
from typing import Annotated, Literal
from prefect import task
from prefect.blocks.system import Secret
from prefect_aws.s3 import S3Bucket
from io import BytesIO
from flow_utils import check_if_file_exists_in_bucket, download_from_s3_bucket, tavily_agent
import json 
from dotenv import load_dotenv
load_dotenv()  # take environment variables from .env.
import logging
from langchain_community.tools import TavilySearchResults
from langchain_community.tools import DuckDuckGoSearchRun
from prefect.blocks.system import Secret

logging.basicConfig(level=logging.INFO)

# Variables
cf.defaults.model = "openai/gpt-4o-mini"

# Schemas
class IndustryClass(BaseModel):
    industry: Annotated[str, "Industry in 5 words maximum"]

class IndustryTrends(BaseModel):
    industry: Annotated[str, "Industry in 5 words maximum"]
    industry_report: str
    industry_sentiment: Literal['negative', 'neutral', 'positive']


@cf.flow
def industry_magnification_tavily(
    industry: str = None
):
    
    tavily_agent = cf.Agent(
        name="Industry Researcher",
        description="An AI agent that researches the latest trends and performance of any given industry.",
        #model="anthropic/claude-3-5-sonnet-20240620",
        model="openai/gpt-4o",
        #tools=[DuckDuckGoSearchRun()],
        tools=[TavilySearchResults(tavily_api_key=Secret.load("tavily-api-key", _sync=True).get())],
    )

    ### Task 5
    industry_trends = cf.Task(
        objective="""
        Provide a detailed analysis of how similar companies are faring in the given industry, including:
            1. Current Market Performance: How is the industry faring in the global market?
            2. Key Performance Indicators (KPIs): Present numerical KPIs that indicate the industry's performance globally, including both the KPI names and their current numerical values.
            3. Industry Sentiment: Discuss the current sentiment (both public and investor) about the industry."
        """,
        agents=[tavily_agent],
        context={"industry": industry},
        result_type=IndustryTrends
    )
    
    trends = industry_trends.run()

    return {"result": trends}
    

if __name__ == "__main__":
    xxx = industry_magnification_tavily(
        industry = "Athleisure")

Version Information

ControlFlow version: 0.11.0                                       
       Prefect version: 3.0.11                                       
LangChain Core version: 0.3.13                                       
        Python version: 3.12.7                                       
              Platform: macOS-15.1.1-arm64-arm-64bit                 
                  Path: /opt/anaconda3/envs/cf_magnify/lib/python3.12

Additional Context

The whole thing was working perfectly for months, until today, same code, same environment, same versions of everything.