(draft, not ready for review) feat: add conversation cost tracking #8531
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
ESLint found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
ConstantTime
changed the title
ConstantTime
force-pushed
the
feat/conversation-cost-tracking
branch
3 times, most recently
from
e1e9fd3 to
c5e7993
Compare
|
@ConstantTime hi, how is it going? Are you stuck on ESLint, or are there some more issues you need help resolving? |
Hey @michnovka, sorry for the late response. Planning to wrap this up during the weekend and open up for review. |
ConstantTime
force-pushed
the
feat/conversation-cost-tracking
branch
from
5998d33 to
e364282
Compare
🚨 Unused i18next Keys DetectedThe following translation keys are defined in
|
ConstantTime
force-pushed
the
feat/conversation-cost-tracking
branch
2 times, most recently
from
78fa244 to
a199930
Compare
ConstantTime
force-pushed
the
feat/conversation-cost-tracking
branch
2 times, most recently
from
a52c997 to
068e1ad
Compare
ConstantTime
force-pushed
the
feat/conversation-cost-tracking
branch
from
068e1ad to
2c8df88
Compare
…ting - Add comprehensive ModelPricing service with 100+ models and historical pricing - Create real-time ConversationCost component that displays in chat header - Use actual token counts from model APIs instead of client-side estimation - Fix BaseClient.js to preserve tokenCount in response messages - Add tokenCount, usage, and tokens fields to message schema - Update Header component to include ConversationCost display - Support OpenAI, Anthropic, Google, and other major model providers - Include color-coded cost display based on amount - Add 32 unit tests for pricing calculation logic 🤖 Generated with Claude Code Co-Authored-By: Claude <[email protected]>
ConstantTime
force-pushed
the
feat/conversation-cost-tracking
branch
from
2c8df88 to
3edf6fd
Compare
| // Persist usage metadata on the assistant message if available for accurate costing | ||
| if (this.getStreamUsage != null) { | ||
| const streamUsage = this.getStreamUsage(); | ||
| if (streamUsage && (Number(streamUsage[this.inputTokensKey]) > 0 || Number(streamUsage[this.outputTokensKey]) > 0)) { |
Check failure
Code scanning / ESLint
Ensure code is properly formatted, use insertion, deletion, or replacement to obtain desired formatting. Error
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 5 days ago
The best fix is to reformat the conditional at line 785 so that:
- The logical AND (
&&) and logical OR (||) expressions each begin on a new line with appropriate indentation. - Each operand of the logical operators is clearly separated, following ESLint’s suggestion for multiline formatting of long logical statements.
This change only affects formatting, not logic, and applies only to the conditional in line 785 ofapi/app/clients/BaseClient.js.
| @@ -782,7 +782,13 @@ | ||
| // Persist usage metadata on the assistant message if available for accurate costing | ||
| if (this.getStreamUsage != null) { | ||
| const streamUsage = this.getStreamUsage(); | ||
| if (streamUsage && (Number(streamUsage[this.inputTokensKey]) > 0 || Number(streamUsage[this.outputTokensKey]) > 0)) { | ||
| if ( | ||
| streamUsage && | ||
| ( | ||
| Number(streamUsage[this.inputTokensKey]) > 0 || | ||
| Number(streamUsage[this.outputTokensKey]) > 0 | ||
| ) | ||
| ) { | ||
| responseMessage.usage = { | ||
| prompt_tokens: streamUsage[this.inputTokensKey], | ||
| completion_tokens: streamUsage[this.outputTokensKey], |
| }; | ||
|
|
||
| if (message.usage) { | ||
| currentTokenUsage.promptTokens = message.usage.prompt_tokens || message.usage.input_tokens || 0; |
Check failure
Code scanning / ESLint
Ensure code is properly formatted, use insertion, deletion, or replacement to obtain desired formatting. Error
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 5 days ago
The best way to fix the formatting issue flagged by ESLint is to follow its suggestion and properly format the assignment on line 98. Specifically, split the line so each part of the assignment (when using the logical OR expression) appears on its own line and is indented to the correct level, complying with the project's style and ESLint's expectations. This usually means putting each part of the expression on a new line, indented appropriately relative to the enclosing code. The change should be made in the function calculateConversationCostFromMessages, in the block assigning currentTokenUsage.promptTokens.
| @@ -95,7 +95,10 @@ | ||
| }; | ||
|
|
||
| if (message.usage) { | ||
| currentTokenUsage.promptTokens = message.usage.prompt_tokens || message.usage.input_tokens || 0; | ||
| currentTokenUsage.promptTokens = | ||
| message.usage.prompt_tokens || | ||
| message.usage.input_tokens || | ||
| 0; | ||
| currentTokenUsage.completionTokens = message.usage.completion_tokens || message.usage.output_tokens || 0; | ||
| currentTokenUsage.reasoningTokens = message.usage.reasoning_tokens || 0; | ||
| const write = Number(message.usage?.input_token_details?.cache_creation) || 0; |
|
|
||
| if (message.usage) { | ||
| currentTokenUsage.promptTokens = message.usage.prompt_tokens || message.usage.input_tokens || 0; | ||
| currentTokenUsage.completionTokens = message.usage.completion_tokens || message.usage.output_tokens || 0; |
Check failure
Code scanning / ESLint
Ensure code is properly formatted, use insertion, deletion, or replacement to obtain desired formatting. Error
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 5 days ago
To fix the formatting error reported by ESLint, insert a line break (empty line) at the indicated location (after line 99), which separates the assignment of currentTokenUsage.completionTokens and currentTokenUsage.reasoningTokens. This improves code readability and matches typical formatting standards that make blocks of assignments within conditional branches easier to scan and maintain. The edit only needs to add an empty line at the specified point; no imports, methods, or functional changes are required.
| @@ -97,6 +97,7 @@ | ||
| if (message.usage) { | ||
| currentTokenUsage.promptTokens = message.usage.prompt_tokens || message.usage.input_tokens || 0; | ||
| currentTokenUsage.completionTokens = message.usage.completion_tokens || message.usage.output_tokens || 0; | ||
|
|
||
| currentTokenUsage.reasoningTokens = message.usage.reasoning_tokens || 0; | ||
| const write = Number(message.usage?.input_token_details?.cache_creation) || 0; | ||
| const read = Number(message.usage?.input_token_details?.cache_read) || 0; |
| currentTokenUsage.cacheReadTokens = read; | ||
| } else if (message.tokens) { | ||
| currentTokenUsage.promptTokens = message.tokens.prompt || message.tokens.input || 0; | ||
| currentTokenUsage.completionTokens = message.tokens.completion || message.tokens.output || 0; |
Check failure
Code scanning / ESLint
Ensure code is properly formatted, use insertion, deletion, or replacement to obtain desired formatting. Error
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 5 days ago
To fix the formatting issue on line 107, insert a newline and properly indent the statement assigning to currentTokenUsage.completionTokens. This means that after line 106, you should add a line break so that the assignments to promptTokens and completionTokens are clearly separated and properly formatted. Only modify lines 106 and 107; the logic remains unchanged, ensuring all existing functionality is preserved.
| @@ -104,6 +104,7 @@ | ||
| currentTokenUsage.cacheReadTokens = read; | ||
| } else if (message.tokens) { | ||
| currentTokenUsage.promptTokens = message.tokens.prompt || message.tokens.input || 0; | ||
|
|
||
| currentTokenUsage.completionTokens = message.tokens.completion || message.tokens.output || 0; | ||
| } else if (message.tokenCount) { | ||
| if (inferredRole === 'assistant') { |
| getMultipleConversationCosts, | ||
| }; | ||
|
|
||
| const { calculateTokenCost, getModelProvider } = require('./ModelPricing'); |
Check failure
Code scanning / ESLint
Disallow variable redeclaration Error
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 5 days ago
To fix the variable redeclaration error, remove the duplicated declarations from the bottom of the file (calculateTokenCost, getModelProvider, and the second logger). The correct approach is to keep only the initial import and definition at the top (lines 1–8) and delete the duplicated code blocks at lines 251–258. This change will resolve the ESLint error without impacting functionality, as the variables are already declared and can be referenced anywhere in the file. No additional imports, methods, or definitions are necessary because the relevant identifiers are already available globally in the file after their first declaration.
| @@ -248,14 +248,7 @@ | ||
| getMultipleConversationCosts, | ||
| }; | ||
|
|
||
| const { calculateTokenCost, getModelProvider } = require('./ModelPricing'); | ||
|
|
||
| // Use console for logging to avoid circular dependencies | ||
| const logger = { | ||
| info: (msg, data) => console.log(msg, data || ''), | ||
| warn: (msg) => console.warn(msg), | ||
| error: (msg, error) => console.error(msg, error || ''), | ||
| }; | ||
|
|
||
| /** | ||
| * Calculate the total cost of a conversation from messages |
| * @returns {Array} returns.modelBreakdown - Per-model cost and usage | ||
| * @returns {Date} returns.lastUpdated - Timestamp of the last message | ||
| */ | ||
| function calculateConversationCostFromMessages(messages) { |
Check failure
Code scanning / ESLint
Disallow variable redeclaration Error
Copilot Autofix
AI 5 days ago
Copilot could not generate an autofix suggestion
Copilot could not generate an autofix suggestion for this alert. Try pushing a new commit or if the problem persists contact support.
| * @returns {number} returns.totalTokens - Total token count across all messages | ||
| * @returns {Date} returns.lastUpdated - Timestamp of the last message | ||
| */ | ||
| function getConversationCostDisplayFromMessages(messages) { |
Check failure
Code scanning / ESLint
Disallow variable redeclaration Error
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 5 days ago
To fix this issue, you need to ensure that the function getConversationCostDisplayFromMessages is defined only once in api/server/services/ConversationCostDynamic.js.
- First, search through the whole file for multiple definitions of
getConversationCostDisplayFromMessages. - Remove or rename all but one of the function definitions (choose the definitive implementation to keep, likely the latest or most complete one), or refactor if needed so only one remains.
- Be careful not to change any other functionality or imports; only remove or rename as needed.
- If the file also contains tests or calls to multiple versions of this function, update those usages as necessary.
- As with
calculateConversationCostFromMessages(which appears to be defined twice in this file and should undergo similar deduplication), do not leave both in the file as it leads to confusion about which one is actually used.
| @@ -463,44 +463,7 @@ | ||
| * @returns {number} returns.totalTokens - Total token count across all messages | ||
| * @returns {Date} returns.lastUpdated - Timestamp of the last message | ||
| */ | ||
| function getConversationCostDisplayFromMessages(messages) { | ||
| try { | ||
| if (!messages || messages.length === 0) { | ||
| return null; | ||
| } | ||
|
|
||
| const costSummary = calculateConversationCostFromMessages(messages); | ||
| if (!costSummary) { | ||
| return null; | ||
| } | ||
|
|
||
| // Format cost for display | ||
| const formatCost = (cost) => { | ||
| if (cost < 0.001) { | ||
| return '<$0.001'; | ||
| } | ||
| if (cost < 0.01) { | ||
| return `$${cost.toFixed(4)}`; | ||
| } | ||
| if (cost < 1) { | ||
| return `$${cost.toFixed(3)}`; | ||
| } | ||
| return `$${cost.toFixed(2)}`; | ||
| }; | ||
|
|
||
| return { | ||
| totalCost: formatCost(costSummary.totalCost), | ||
| totalCostRaw: costSummary.totalCost, | ||
| primaryModel: costSummary.modelBreakdown[0]?.model || 'Unknown', | ||
| totalTokens: costSummary.tokenUsage.promptTokens + costSummary.tokenUsage.completionTokens, | ||
| lastUpdated: costSummary.lastUpdated, | ||
| }; | ||
| } catch (error) { | ||
| logger.error('Error getting conversation cost display from messages:', error); | ||
| return null; | ||
| } | ||
| } | ||
|
|
||
| /** | ||
| * Get costs for multiple conversations in batch | ||
| * @param {string[]} conversationIds - Array of conversation IDs |
| * @param {string} userId - User ID | ||
| * @returns {Object} Map of conversationId to cost display data | ||
| */ | ||
| async function getMultipleConversationCosts(conversationIds, userId) { |
Check failure
Code scanning / ESLint
Disallow variable redeclaration Error
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 5 days ago
To fix the variable redeclaration error, we must ensure the function getMultipleConversationCosts is only defined once in this file. The error indicates that a previous definition with this name exists. The cleanest way to resolve this without affecting functionality is to rename one of the functions to a distinct, descriptive name. We can do this by renaming the function starting at line 510 and updating all of its internal references accordingly. Additionally, we need to update any places where it is exported or referenced at the bottom of the file.
The changes required are:
- Rename the function defined at line 510 to a unique name, such as
getMultipleConversationCostsBatch. - Update the export at line 558 accordingly.
Since we've only been shown the definition and export near the bottom of the file, all changes are confined to this region.
| @@ -507,7 +507,7 @@ | ||
| * @param {string} userId - User ID | ||
| * @returns {Object} Map of conversationId to cost display data | ||
| */ | ||
| async function getMultipleConversationCosts(conversationIds, userId) { | ||
| async function getMultipleConversationCostsBatch(conversationIds, userId) { | ||
| try { | ||
| const { getMessages } = require('~/models/Message'); | ||
| const results = {}; | ||
| @@ -555,5 +555,5 @@ | ||
| module.exports = { | ||
| calculateConversationCostFromMessages, | ||
| getConversationCostDisplayFromMessages, | ||
| getMultipleConversationCosts, | ||
| getMultipleConversationCostsBatch, | ||
| }; |
|
|
||
| if (!data || data.totalCostRaw === 0) { | ||
| return ( | ||
| <div className="flex items-center gap-1 rounded-md px-2 py-1 text-xs text-gray-400" title={t('com_ui_conversation_cost')}> |
Check failure
Code scanning / ESLint
Ensure code is properly formatted, use insertion, deletion, or replacement to obtain desired formatting. Error
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 5 days ago
To fix the formatting issue, the relevant JSX element (a div) should be reformatted so that each prop is on its own line, after the opening tag and properly indented. In this case, on line 48, spread the className and title props over separate lines inside the opening <div> tag. The rest of the JSX remains unchanged. Only the indentation and line breaks of this tag need adjustment.
Edit only the affected line (line 48) within client/src/components/Chat/ConversationCost.tsx:
- Break the line after the opening
<div. - Place each prop on its own line, indented.
- Close the tag after the last prop.
No additional imports or logic changes are required.
| @@ -45,7 +45,10 @@ | ||
|
|
||
| if (!data || data.totalCostRaw === 0) { | ||
| return ( | ||
| <div className="flex items-center gap-1 rounded-md px-2 py-1 text-xs text-gray-400" title={t('com_ui_conversation_cost')}> | ||
| <div | ||
| className="flex items-center gap-1 rounded-md px-2 py-1 text-xs text-gray-400" | ||
| title={t('com_ui_conversation_cost')} | ||
| > | ||
| <span>💰</span> | ||
| <span>$0.00</span> | ||
| </div> |
| const tooltipText = `${t('com_ui_conversation_cost')}: ${data.totalCost} | ${t('com_ui_primary_model')}: ${data.primaryModel} | ${t('com_ui_total_tokens')}: ${data.totalTokens.toLocaleString()} | ${t('com_ui_last_updated')}: ${new Date(data.lastUpdated).toLocaleTimeString()}`; | ||
|
|
||
| return ( | ||
| <div className="flex items-center gap-1 rounded-md px-2 py-1 text-xs transition-colors hover:bg-surface-hover" title={tooltipText}> |
Check failure
Code scanning / ESLint
Ensure code is properly formatted, use insertion, deletion, or replacement to obtain desired formatting. Error
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 5 days ago
To fix this formatting issue, split the <div> opening tag on line 58 so that each prop (className and title) appears on its own line, indented to match the other code in the file. This change will improve readability and comply with the linting instructions. Make sure className and title are each on a new line, indented with four spaces to align with existing code style.
Only one code region in client/src/components/Chat/ConversationCost.tsx requires an edit.
| @@ -55,7 +55,10 @@ | ||
| const tooltipText = `${t('com_ui_conversation_cost')}: ${data.totalCost} | ${t('com_ui_primary_model')}: ${data.primaryModel} | ${t('com_ui_total_tokens')}: ${data.totalTokens.toLocaleString()} | ${t('com_ui_last_updated')}: ${new Date(data.lastUpdated).toLocaleTimeString()}`; | ||
|
|
||
| return ( | ||
| <div className="flex items-center gap-1 rounded-md px-2 py-1 text-xs transition-colors hover:bg-surface-hover" title={tooltipText}> | ||
| <div | ||
| className="flex items-center gap-1 rounded-md px-2 py-1 text-xs transition-colors hover:bg-surface-hover" | ||
| title={tooltipText} | ||
| > | ||
| <span className="text-text-tertiary">💰</span> | ||
| <span className={`font-medium ${colorClass}`}>{data.totalCost}</span> | ||
| </div> |
Summary
Adds real-time cost tracking for conversations. Users can now see how much their conversations cost as messages stream, with automatic updates and color-coded indicators.
Features
Implementation
The cost display appears in the conversation header and updates automatically as new messages arrive. Hover over the cost to see detailed breakdown including model used, token count, and last update time.
Testing
Change Type
Checklist