Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

'NaN' token counts when using streamText with Azure OpenAI models #2543

Open
steveoOn opened this issue Aug 5, 2024 · 5 comments
Open

'NaN' token counts when using streamText with Azure OpenAI models #2543

steveoOn opened this issue Aug 5, 2024 · 5 comments
Labels
ai/provider bug Something isn't working

Comments

@steveoOn
Copy link

steveoOn commented Aug 5, 2024

Description

I am using streamText with the Azure OpenAI provider for the AI SDK and there models. I use createAzure to create the provider instance. When I try to get the token counts, I get NaN values.(both in API endpoint onFinish() and useChat({onFinish:() => {}}))

log

[0] 👉 azureChat: start save data
[0] 👉 azureChat usage: { promptTokens: NaN, completionTokens: NaN, totalTokens: NaN }

Code example

import { createAzure } from '@ai-sdk/azure';

export const azure = createAzure({
  resourceName: 'resource-name',
  apiKey: process.env.AZURE_OPENAI_API_KEY,
});

const result = await streamText({
          model: azure(selectedModel),
          system: systemContent,
          messages: strictPayload.messages,
          maxTokens: maxTokens,
          maxRetries: 3,
          onFinish: async ({ text, usage }) => {
            appendData.close();
            console.log('👉 azureChat: start save data');
            console.log('👉 azureChat usage:', usage);
            const { promptTokens, completionTokens, totalTokens } = usage;

            const lastMessageContent =
              strictPayload.messages[strictPayload.messages.length - 1].content;

            try {
              if (typeof lastMessageContent === 'string') {
                await saveChatMessage(
                  lastMessageContent,
                  text,
                  JSON.stringify(strictPayload.messages),
                  activeConversationId
                );
              } else {
                console.error(
                  'azureChat: lastMessageContent is not a string:',
                  lastMessageContent
                );
              }

              if (data?.userId && data?.sessionId) {
                await updateUserTokenUsage({
                  userId: data.userId,
                  sessionId: data.sessionId,
                  promptTokens,
                  completionTokens,
                  totalTokens,
                });
              } else {
                console.error(
                  'azureChat: Missing userId or sessionId for token update'
                );
              }
            } catch (error) {
              console.error('Error in azureChat onFinish:', error);
            }
          },
        });
        
 // Component 
 const {
    messages,
    input,
    handleInputChange,
    handleSubmit,
    setMessages,
    stop,
    data,
    error,
  } = useChat({
    api: '/api/chat',
    body: {
      activeConversation,
      selectedModel,
      selectedToggle: selectedRole,
      internet: false,
    },
    onResponse: () => {
      responseEnd.current = false;
      if (activeConversation) return;
    },
    onError: () => {
      responseEnd.current = true;
    },
    onFinish: (message, { usage }) => {
      responseEnd.current = true;
      refreshConversationList(data);
      setAIState((prev) => ({
        ...prev,
        usage,
      }));
    },
  });

Additional context

// package.json
"dependencies": {
    "@ai-sdk/anthropic": "^0.0.35",
    "@ai-sdk/azure": "^0.0.17",
    "@ai-sdk/openai": "^0.0.40",
    "ai": "^3.3.0",
    "react": "18.3.1",
    "react-dom": "18.3.1",
    "next": "14.2.5",
    ...
@lgrammel lgrammel added bug Something isn't working ai/provider labels Aug 5, 2024
@lgrammel
Copy link
Collaborator

lgrammel commented Aug 5, 2024

It seems that token counts for streaming are currently not supported by Azure OpenAI: https://learn.microsoft.com/en-us/answers/questions/1805363/azure-openai-streaming-token-usage

@trulymittal
Copy link

@lgrammel Not only related to Azure, but it's also not working for OpenAI/Google/Mistral, fireworks works when using openai with fireworks basepath...
I am sure it is related to the use of a registry for LLM providers.

When I use streamText with the registry...It gives me NaN

model: registry.languageModel('openai:gpt-3.5-turbo'),

{ promptTokens: NaN, completionTokens: NaN, totalTokens: NaN }

But when I use streamText without a registry provider...it provides proper usage

model: openai('gpt-3.5-turbo')

{ promptTokens: 351, completionTokens: 10, totalTokens: 361 }

@lgrammel
Copy link
Collaborator

lgrammel commented Aug 8, 2024

@trulymittal how are you setting up the providers in the registry? createOpenAI needs to be set up with compatibility: "strict" for streaming usage: https://sdk.vercel.ai/providers/ai-sdk-providers/openai#provider-instance

@lgrammel
Copy link
Collaborator

lgrammel commented Aug 8, 2024

Double-checked Mistral and it provides the token usage information, try e.g. https://github.com/vercel/ai/blob/main/examples/ai-core/src/stream-text/mistral.ts

@trulymittal
Copy link

@lgrammel apologies, it was my mistake not to be using the strict compatibility mode. Thanks for this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ai/provider bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants