Configure sustained score alerts with external aggregation

Applies to:

Plan -
Deployment -

Summary

Chart visuals cannot be used directly to create time-windowed alerts. Braintrust log alerts match events in evaluation batches and cannot require a condition to persist for N minutes. Use a log alert that mirrors your chart and either tune filters/notify-intervals or send alert notifications to a webhook and have an external service query or aggregate the underlying logs before paging.

What is happening

There is no built-in “sustained for 5 minutes” or sliding-window aggregate (avg/p90 over X minutes) condition. That causes single outliers to generate alerts unless you approximate persistence with stricter filters or external aggregation.

Fix or suggestion

Option 1: mirror chart with a log alert (fast, limited)

Go to Settings > Alerts and create a new Log alert.

scores.factuality IS NOT NULL
AND scores.factuality < 0.8
AND environment = 'prod'

Choose an action (Slack or Webhook).
Set a notify interval long enough to reduce noise (e.g., 30m).
Test the alert from the UI.

This will still fire on a single matching event in an evaluation batch. The notify interval suppresses repeat notifications but does not require persistence.

Option 2: webhook + external 5-minute aggregator (recommended for “sustained”)

Create a Log alert whose SQL filter captures candidate breaches, for example scores.factuality IS NOT NULL AND scores.factuality < 0.8 AND metadata.environment = 'prod'. Set the action to Webhook and point it at your aggregator endpoint.
Use the webhook payload as a trigger. The alert webhook payload is fixed; it includes the alert metadata, count, time window, and related logs URL, but not every matching log row or score value.
In your aggregator, use the alert time window and the same SQL filter to query Braintrust, or consume the same trace data from your own pipeline.
Compute the windowed condition you care about, such as percentage below threshold, average score, p90, or consecutive-count, and only page if that aggregate crosses your threshold.

Minimal Python pseudocode (conceptual):

THRESHOLD = 0.8
PCT_REQUIRED = 0.5  # e.g. 50% of events

def on_webhook_event(payload):
    time_start = payload["details"]["time_start"]
    time_end = payload["details"]["time_end"]

    rows = query_braintrust_logs(
        start=time_start,
        end=time_end,
        filter="scores.factuality IS NOT NULL AND metadata.environment = 'prod'",
    )

    scores = [
        row["scores"]["factuality"]
        for row in rows
        if row.get("scores", {}).get("factuality") is not None
    ]

    if not scores:
        return

    pct_below_threshold = sum(score < THRESHOLD for score in scores) / len(scores)

    if pct_below_threshold >= PCT_REQUIRED:
        page_ops.send_page({
            "time_start": time_start,
            "time_end": time_end,
            "count": len(scores),
            "pct_below_threshold": pct_below_threshold,
        })

Keep the aggregator simple and observable.

How to confirm it worked

For Option 1: Use UI Test alert. Confirm the webhook or Slack receives the expected single-match payload.
For Option 2: Send test events that simulate sustained and transient breaches. Confirm your aggregator only pages for sustained cases and remains silent for single outliers. Check aggregator logs/metrics for window counts and the page history.

Notes

See Alerts for full alert configuration reference.
Native windowed or aggregate alert conditions are a tracked feature request.

​Summary

​What is happening

​Fix or suggestion

​Option 1: mirror chart with a log alert (fast, limited)

​Option 2: webhook + external 5-minute aggregator (recommended for “sustained”)

​How to confirm it worked

​Notes

Summary

What is happening

Fix or suggestion

Option 1: mirror chart with a log alert (fast, limited)

Option 2: webhook + external 5-minute aggregator (recommended for “sustained”)

How to confirm it worked

Notes