Skip to main content
Applies to:
  • Plan:
  • Deployment:

​
Summary

Issue: Online scorers configured with time-based filters like created > now() - 1 day fail to score production traces, showing missing scores in the UI. Cause: Time-based filters in online scorer queries can interfere with trace discovery due to timing mismatches between trace creation and scorer evaluation. Resolution: Remove time-based filters from online scorer SQL queries, as scorers automatically apply to new traces without needing time conditions.

​
Resolution Steps

​
Remove Time-Based Filters

​
Step 1: Edit scorer rule

Navigate to the Online Scoring modal and remove any created > now() - X conditions from your SQL filter. 
-- Before
WHERE span_attributes.name IN ('agent_run', 'turn_start')
AND created > now() - 1 day

-- After
WHERE span_attributes.name IN ('agent_run', 'turn_start')

​
Step 2: Set sampling to 100%

Temporarily increase sampling rate to 100% to verify the scorer works on all matching traces.

​
Step 3: Test the rule

Click Test rule to confirm your SQL filter matches recent spans.

​
Step 4: Verify scoring

Create new traces and confirm scores appear. Some lag (seconds to minutes) is normal.

​
If No Results Appear After Testing

​
Step 1: Simplify filter

Temporarily remove span_attributes.name IN (...) to check if any spans match.

​
Step 2: Verify scorer output

Run the scorer against a recent trace in the scorer page to confirm it returns a numeric value.

​
Step 3: Check LLM-judge scores

For LLM-judge scorers, ensure each choice maps to a unique numeric score in [0,1] range. 
# Incorrect - both choices set to 0
clear: 0
showed_frustration: 0

# Correct
clear: 0
showed_frustration: 1

​
Additional Context

Online scorers automatically apply to new traces as they are created. Time-based filters are unnecessary and can cause the scorer to miss traces due to timing issues between trace creation and query evaluation. Some delay in score appearance is expected behavior.