Gateway calls fail in bt eval --dev mode

Applies to:

Plan -
Deployment -

Summary

Issue: Running bt eval --dev <file> and connecting to the Playground causes gateway calls to fail with AI_APICallError: Not Found when the AI SDK client reads BRAINTRUST_API_KEY. Cause: bt eval --dev replaces BRAINTRUST_API_KEY with a short-lived Playground session token, which the Braintrust gateway does not accept as a valid credential. Resolution: Pass your real API key to the gateway client via a separate environment variable that bt eval --dev does not overwrite.

Environment variables injected by `bt eval --dev`

When the Playground connects to your local dev server, the following variables are overwritten in your process environment:

Variable	Value injected
`BRAINTRUST_API_KEY`	Short-lived Playground session token
`BRAINTRUST_API_URL`	Your org’s API URL
`BRAINTRUST_APP_URL`	Dev server app URL
`BRAINTRUST_ORG_NAME`	Org you connected from
`BRAINTRUST_DEFAULT_PROJECT`	Project from `--project` flag or config

The session token injected into BRAINTRUST_API_KEY carries your identity but is not accepted by the Braintrust gateway (gateway.braintrust.dev). Only real API keys (sk-...) or service tokens (bt-st-...) authenticate successfully to the gateway.

Resolution steps

Step 1: Set a separate env var for your real API key

Before launching the dev server, set an additional variable in the same shell:

export BRAINTRUST_API_KEY_OVERRIDE=<your-real-api-key>
bt eval --dev <eval-file>

Step 2: Update your gateway client to read the override first

import { createOpenAI } from "@ai-sdk/openai";

const btGateway = createOpenAI({
  baseURL: process.env.OPENAI_BASE_URL ?? "https://gateway.braintrust.dev/v1",
  apiKey: process.env.BRAINTRUST_API_KEY_OVERRIDE ?? process.env.BRAINTRUST_API_KEY,
});

The fallback to BRAINTRUST_API_KEY ensures the client still works in non-dev contexts where the override is not set.

Notes

A 404 Not Found from the AI SDK indicates a bad route or unaccepted credential at the gateway — not a missing org permission. A permission error would surface as 401 or 403.
Any env var name not in the injected list above can serve as the override. Common choices: BRAINTRUST_API_KEY_OVERRIDE, BT_API_KEY, BT_GATEWAY_KEY.
This workaround is required any time your gateway client reads BRAINTRUST_API_KEY directly in dev mode.

​Summary

​Environment variables injected by bt eval --dev

​Resolution steps

​Step 1: Set a separate env var for your real API key

​Step 2: Update your gateway client to read the override first

​Notes

Summary

Environment variables injected by `bt eval --dev`

Resolution steps

Step 1: Set a separate env var for your real API key

Step 2: Update your gateway client to read the override first

Notes