Alerting on scorer errors and aggregated scores

Applies to:

Plan -
Deployment -

Summary

Use a log alert with a span-level error filter to catch scorer failures: span_attributes.name = '<Scorer name>' AND error IS NOT NULL. For aggregated checks, use a Time window alert when that alert type is available. For self-hosted deployments, Time window alerts require data plane v2.10.0 or later. If your deployment does not show Time window as an alert type, run a scheduled SQL check externally and notify from there.

What is happening

Log alerts evaluate individual rows. Any row that matches your filter can trigger an alert. That works well for scorer failures, but per-row score thresholds are noisy when you want an hourly average or another aggregate. Time window alerts evaluate a scalar SQL calculation over a window and compare the numeric result to a threshold. The scorer error field is populated when the scorer throws or encounters an error while running. Filtering on that field targets actual scorer failures instead of yet-to-be-processed or empty logs.

Fix or suggestion

Option 1: alert on scorer errors (recommended)

Action: create a UI alert that only fires when the scorer produced an error.

In the alert filter use:

span_attributes.name = 'Feed Density' AND error IS NOT NULL

Save the alert and set an appropriate notification interval to avoid repeated notifications.

Why: This targets only spans where the scorer explicitly failed.

Option 2: aggregated or rolling-window check (recommended when available)

Action: create a Time window alert that computes an aggregate over a time window and notifies you if the value crosses your threshold.

Go to Settings > Alerts.
Click Alert.
Select Time window as the alert type.
Configure the calculation with the builder or a SQL query.
Set Window length (minutes) to the aggregation window.
Set Alert trigger to the comparison you want. For example, set value is < 0.8.
Use Advanced settings to configure trigger delay, late-data handling, recovery notifications, evaluation schedule, and renotification.

Example raw SQL calculation:

SELECT AVG(scores."Feed Density") AS avg_score
FROM project_logs('<PROJECT_ID>')
WHERE scores."Feed Density" IS NOT NULL

Why: A Time window alert computes the aggregate inside Braintrust, compares the scalar result to your threshold, and handles recovery and no-data behavior.

Option 3: external scheduled aggregate check (fallback)

Action: run a scheduled SQL job that computes an aggregate over a time window and notify externally if it drops below your threshold.

Example query:

SELECT AVG(scores."Feed Density") AS avg_score, COUNT(*) AS scored_count
FROM project_logs('<PROJECT_ID>')
WHERE created >= NOW() - INTERVAL 1 HOUR
  AND scores."Feed Density" IS NOT NULL
HAVING AVG(scores."Feed Density") < 0.8

Schedule this query to run at your desired cadence, such as every 5 to 15 minutes.
Use your external notifier, such as Slack, PagerDuty, email, or another destination, to send alerts when the query returns a row.

Why: Use this fallback only when Time window alerts are not available in your deployment.

Additional guidance to reduce noise

Exclude yet-to-be-processed logs by requiring scores IS NOT NULL in filters.
Add a short buffer to the time window if ingestion latency causes transient nulls. For example, ignore the most recent 5 to 10 minutes.
Tune notification frequency or suppression to avoid alert storms.

How to confirm it worked

Error alert check: run the following query to find recent scorer errors, then confirm the alert fires for matching rows.

SELECT id, created, span_attributes, error
FROM project_logs('<PROJECT_ID>')
WHERE span_attributes.name = 'Feed Density'
  AND error IS NOT NULL
LIMIT 5

Aggregation check: Use the Recent evaluation preview in the Time window alert to confirm the calculation returns the expected value. If you use the external fallback, run the scheduled aggregate query manually. Confirm avg_score is below your threshold and that your external notifier receives the alert.

Notes

For self-hosted deployments, Time window alerts require data plane v2.10.0 or later.

​Summary

​What is happening

​Fix or suggestion

​Option 1: alert on scorer errors (recommended)

​Option 2: aggregated or rolling-window check (recommended when available)

​Option 3: external scheduled aggregate check (fallback)

​Additional guidance to reduce noise

​How to confirm it worked

​Notes

Summary

What is happening

Fix or suggestion

Option 1: alert on scorer errors (recommended)

Option 2: aggregated or rolling-window check (recommended when available)

Option 3: external scheduled aggregate check (fallback)

Additional guidance to reduce noise

How to confirm it worked

Notes