Query scorer span token costs with SQL

Applies to:

Plan -
Deployment -

Summary

Goal: Query scorer LLM spend for experiments or project logs. The estimated_cost() function prices scorer spans, but the built-in UI cost surfaces exclude them, so you need to query scorer costs directly. Features: SQL span queries, span_attributes.purpose, span_attributes.type, project_logs()

Configuration steps

Step 1: Understand how scorer costs are surfaced

The estimated_cost() SQL function prices all LLM spans, including scorer (judge) spans. However, some built-in UI cost surfaces exclude scorer spans by design:

The Estimated cost column on the Experiments page.
The preset cost charts on the Monitor page.

To see scorer costs, query them directly. Sum estimated_cost() over scorer spans for a total, or query token counts and apply your own per-model pricing for a granular breakdown by model. For example, to get total scorer cost for an experiment:

SELECT SUM(estimated_cost()) AS scorer_cost
FROM experiment('<EXPERIMENT_ID>', shape => 'spans')
WHERE span_attributes.purpose = 'scorer'
  AND span_attributes.type = 'llm'

Step 2: Query scorer token usage

Use the SQL sandbox or API to sum tokens by model for scorer spans. For experiments, an example query is:

SELECT
  SUM(metrics.prompt_tokens) AS scorer_prompt_tokens,
  SUM(metrics.completion_tokens) AS scorer_completion_tokens,
  metadata.model
FROM experiment('<EXPERIMENT_ID>', shape => 'spans')
WHERE span_attributes.purpose = 'scorer'
  AND span_attributes.type = 'llm'
GROUP BY metadata.model

For project logs, replace experiment() with project_logs():

SELECT
  SUM(metrics.prompt_tokens) AS scorer_prompt_tokens,
  SUM(metrics.completion_tokens) AS scorer_completion_tokens,
  metadata.model
FROM project_logs('<PROJECT_ID>', shape => 'spans')
WHERE span_attributes.purpose = 'scorer'
  AND span_attributes.type = 'llm'
GROUP BY metadata.model

Calculate costs by multiplying token counts by your per-model pricing (e.g., prompt_tokens * (0.04 / 1000000) + completion_tokens * (0.05 / 1000000)).

Step 3: Surface costs in the Monitor dashboard (optional)

In the Monitor page, create a custom chart with a span filter on span_attributes.purpose = 'scorer' and span_attributes.type = 'llm'. Use sum(metrics.prompt_tokens) as the measure, or create a cost expression like:

sum(metrics.prompt_tokens) * (0.04 / 1000000)

​Summary

​Configuration steps

​Step 1: Understand how scorer costs are surfaced

​Step 2: Query scorer token usage

​Step 3: Surface costs in the Monitor dashboard (optional)

Summary

Configuration steps

Step 1: Understand how scorer costs are surfaced

Step 2: Query scorer token usage

Step 3: Surface costs in the Monitor dashboard (optional)