Skip to main content
Applies to:
  • Plan -
  • Deployment -

​
Summary

Issue: High-frequency SQL polling through the /btql API across multiple projects can exhaust the query rate limit, resulting in 429 Too Many Requests errors. Cause: Each API query counts toward the limit. Polling many projects individually, especially concurrently or on a short schedule, can exceed the available request budget. Resolution: For continuous log ingestion into external systems like Datadog, use cloud storage export instead of polling. If SQL polling is required, combine project queries where possible, use cursor/high-watermark pagination, select only needed columns, and pace requests.

​
Resolution steps

​
If you are exporting logs to an external system (Datadog, observability pipelines)

​
Step 1: Set up S3 cloud storage export

In Braintrust, go to Settings > Data management > Create export automation. Configure an incremental export to an S3 bucket in your account. This replaces interactive SQL API polling with an incremental export pipeline. It avoids the /btql polling rate limit and preserves full payloads depending on the export shape you choose.

​
Step 2: Ingest from S3 into Datadog

Use the Datadog Forwarder Lambda or Firehose to pick up files from the S3 bucket. Your existing metric logic applies unchanged.

​
If you still require SQL polling

​
Step 1: Batch multiple projects into one query

project_logs() accepts multiple project IDs. Combine projects into a single query instead of one query per project. 
SELECT is_root, error, created
FROM project_logs('project_id_1', 'project_id_2', shape => 'traces')
WHERE created >= now() - interval '15' minute
ORDER BY created DESC

​
Step 2: Use a high-watermark cursor instead of a fixed time window

Store the last created timestamp or _xact_id per project after each successful query. Query only new rows on the next run to avoid re-scanning overlapping windows. 
WHERE created > '2024-01-01T00:00:00Z'  -- last successful timestamp
ORDER BY created ASC

​
Step 3: Select only required columns

Omit input, output, expected, and metadata if not strictly needed. These fields can be very large and increase backend load. 
SELECT is_root, error, created
FROM project_logs('project_id', shape => 'traces')
WHERE created >= now() - interval '15' minute

​
Step 4: Pace queries with a queue

If querying many projects individually, introduce a delay between requests (e.g., one query every 3–5 seconds) to stay within the 20 requests/minute limit.

​
Additional information

Rate limit increases are generally not granted when a workload-appropriate alternative exists. For multi-project log ingestion at scale, S3 export is the recommended architecture. Self-hosted deployments can adjust the limit using the RATELIMIT_BTQL_DEFAULT environment variable. See BTQL rate limits on Starter and Pro plans for details.