MCP or CLI?
MCP or CLI?
- MCP: Best when your AI tool can connect to Braintrust but has no authenticated shell, which is common in chat applications. It also fits when you want an assistant to reason over your Braintrust data and take several connected actions in one conversation, without installing and maintaining a CLI in its execution environment.
btCLI: Best for repeatable work in scripts, CI, local files, and shell pipelines, where you want deterministic commands instead of an assistant’s judgment. Coding agents with shell access can call those commands too.
Connect your client
Braintrust hosts the MCP server, so there is nothing to install or deploy.Choose authentication
- OAuth (recommended)
- API key
- No Braintrust API key required
- No OAuth client ID or client secret to configure
/.well-known/oauth-authorization-server on the same host as your MCP endpoint (see next step), so a client can register itself.Choose your endpoint
- Self-hosted: Copy the MCP URL from Settings > Data plane. This is your Braintrust API URL with
/mcpappended. - Transport: If your client asks you to choose a transport, choose HTTP (also called streamable HTTP). SSE-only clients can’t connect.
Configure your client
--transport http to claude mcp add. In Gemini CLI’s settings.json, set the server address with httpUrl, not url.| Client | Setup |
|---|---|
If you installed the retired Braintrust MCP plugin, remove it before you add the server. | |
If you installed the retired Braintrust MCP plugin, remove it before you add the server. | |
Verify the connection
What the MCP can do
Your assistant works with the data and objects in your Braintrust organization, and it chains several tools in one turn. Suppose a support chatbot starts making claims about your product that aren’t true. In a single conversation, your assistant can:- Query recent logs to find examples of the behavior.
- Write an evaluator that detects the unsupported claims, and test it against those traces before saving it.
- Add the failing cases to a regression dataset, with the corrected responses as the expected output.
- Run an eval comparing the current prompt against a proposed fix.
Install the Braintrust SDK
Before you can query anything, your application has to send traces to Braintrust. Your assistant handles that setup: it detects your programming language and frameworks, installs the appropriate SDK, and configures auto-instrumentation. Once complete, it runs your app, verifies traces are being logged, and provides a permalink to view them in Braintrust. Example prompts:Set up tracing from scratch
Set up tracing from scratch
Instrument an existing integration
Instrument an existing integration
Confirm traces are landing
Confirm traces are landing
docs://sdk-install.
Explore your data
Your assistant can answer questions about what your application actually did. It queries logs, experiments, and datasets with SQL, discovers the fields and value distributions in a data source before writing a query, and pulls aggregated metrics for an experiment with or without a baseline to compare against. Because it reads the same data the UI shows, you can investigate a production issue without switching to a browser. When a query returns more than 1 MB,sql_query returns a signed URL to the full result instead of inline rows, so your assistant can work with production-scale results without filling its context window. See Tools for how to control that behavior.
Example prompts:
Find recent errors
Find recent errors
See what fields your data has
See what fields your data has
Break down cost
Break down cost
Compare an experiment to its baseline
Compare an experiment to its baseline
sql_query, infer_schema, summarize_experiment.
Find and share objects
Most Braintrust tools take an object ID, so your assistant finds the right project, experiment, or dataset by name and translates between names and IDs on its own. It usually does this as a step inside a larger request rather than as something you ask for. When you want to hand a result to someone else, it produces a direct link to the object. It can also reverse the lookup, finding a person’s traces from their name or email. Example prompts:List recent work in a project
List recent work in a project
Identify a Braintrust URL
Identify a Braintrust URL
Attribute traces to a teammate
Attribute traces to a teammate
list_recent_objects, resolve_object, generate_permalink, lookup_users, lookup_api_keys.
Analyze traces
A long agent trace is hard to follow span by span. Your assistant partitions a single trace into chronological work sections and annotates each one, so you can see what a run actually did and where it spent its time. Example prompts:Break down a trace
Break down a trace
Find the slow steps
Find the slow steps
get_trace_work_items, update_trace_work_report.
Manage patterns
Patterns record the recurring behaviors your traces reveal, so an investigation doesn’t have to start over each time. Your assistant checks what has already been reported, catching duplicates that use different wording, adds a new pattern with supporting evidence, and attaches further evidence as it turns up. Example prompts:Investigate a suspected problem
Investigate a suspected problem
Review active patterns
Review active patterns
Manage duplicate patterns
Manage duplicate patterns
- Find duplicates:
search_patternsacceptsstatus: "duplicate"and returnsduplicate_of, the canonical pattern’s ID. - Close a duplicate: Call
update_patternwithstatus: "duplicate"andduplicate_ofset to an existing canonical pattern in the same project. The target cannot be the pattern itself or another duplicate. - Undo duplicate closure: Call
update_patternwithduplicate_of: nullandstatus: "active"orstatus: "inactive". Both fields are required to undo the closure.
search_patterns, new_pattern, update_pattern.
Configure Topics
Topics preprocesses traces into text, extracts facets from that text, and clusters the results to show what your users actually do. Your assistant builds that pipeline for you: it writes a preprocessor that matches your trace shape, tests it against real traces before saving, defines the facets to extract, and enables the automation that keeps it running. It can also rewind an automation over historical traffic.Set up Topics from scratch
Set up Topics from scratch
Write a preprocessor for an unusual trace shape
Write a preprocessor for an unusual trace shape
Process historical traces
Process historical traces
create_preprocessor, test_preprocessor_on_trace, create_facet, test_facet_on_trace, enable_topics_automation, set_topics_automation, rewind_topics_automation.
Build dashboards
Dashboards collect the charts you check regularly. Your assistant previews a chart against real project logs so you can see it before anything is saved, then puts it into a new or existing dashboard. It can also read back what a dashboard already contains and edit charts in place, one at a time or in bulk. Example prompts:Preview a chart before saving it
Preview a chart before saving it
Create a cost dashboard
Create a cost dashboard
Add a chart to an existing view
Add a chart to an existing view
generate_monitor_chart, list_monitoring_views, get_monitoring_view, create_monitoring_view, update_monitoring_view.
Manage automations and alerts
Automations watch your data so you don’t have to. Your assistant inspects what a project already runs, including online scoring rules, exports, and retention policies, then creates what’s missing: an alert on individual matching logs, an alert on an aggregate threshold over a recent window, an alert on environment updates, or a Loop job that analyzes recent traffic on a schedule. It can also pause and resume any of them. Example prompts:Audit what's configured
Audit what's configured
Alert on an error rate
Alert on an error rate
Schedule recurring analysis
Schedule recurring analysis
Pause an automation
Pause an automation
list_automations, set_automation_status, create_log_alert, create_environment_update_alert, create_threshold_alert, create_scheduled_loop_job, list_slack_channels.
Author prompts and evaluators
Prompts and evaluators are versioned objects that your application, experiments, and online scoring all share. Your assistant drafts one from what it found in your logs, runs it against real traces to see how it behaves before anything is saved, and saves it as a new version when you’re satisfied. It can then attach an evaluator to an online scoring rule so it scores production logs continuously. Example prompts:Write a scorer for errors you found
Write a scorer for errors you found
Create an LLM-as-a-judge
Create an LLM-as-a-judge
Apply a scorer to production logs
Apply a scorer to production logs
Save a prompt as a new version
Save a prompt as a new version
create_prompt, create_evaluator, test_evaluator, update_online_scoring_rule.
Run evals and edit datasets
Evals, and the datasets that feed them, are how you measure whether a change helps. Your assistant curates dataset rows from the failures it finds in your logs, then runs an experiment against them using a saved or inline task and whichever scorers you want. A prior experiment can supply the input data, in which case its outputs become the expected values. Useadd_to_dataset to copy project log rows into a dataset in the same project without loading their full contents into the assistant’s context. Find rows with sql_query against project_logs, then pass their id values in log_row_ids, along with project_id and either dataset_id or dataset_name. Use row IDs, not span_id or root_span_id values. Set create_if_missing to create a new named dataset.
- Copied fields. Each row preserves
input,metadata, andtags, with anoriginlink to the source log. Itsexpectedcomes from the log’sexpected, falling back tooutputwhenexpectedis missing ornull. - Missing rows. If any requested row ID is missing, the tool writes no rows.
- Batch limits. Both
add_to_datasetandedit_dataset_rowsaccept up to 1,000 rows per call on hosted Braintrust. Self-hosted limits depend on deployment configuration, with a fallback of 100. Check the tool’s description for its limit and split larger batches across calls.
save_dataset_custom_view to give reviewers a custom view for labeling a dataset’s rows. The tool saves the view’s React code on the dataset and returns a link that opens the dataset with the view selected. To reuse a view as a template, call list_dataset_custom_views to read its code, adjust it, and save it on another dataset under a new slug. Set if_exists to replace to save a new version of an existing view.
Example prompts:
Build a regression dataset from failures
Build a regression dataset from failures
Compare two prompts
Compare two prompts
Re-run last week's inputs
Re-run last week's inputs
run_eval, edit_dataset_rows, add_to_dataset, list_dataset_custom_views, save_dataset_custom_view.
run_eval creates an experiment and can execute your code or call AI providers, so it incurs compute and model usage.Manage project settings
Project settings hold the defaults that other functions inherit. Your assistant reads a project’s typed settings, including which preprocessor facets and other project functions fall back to, and changes that default when you want a new one to apply everywhere. Example prompts:Check the default preprocessor
Check the default preprocessor
Change the default
Change the default
get_project_settings, set_project_default_preprocessor.
Search docs and load skills
Your assistant grounds its answers in Braintrust documentation rather than guesswork, so it can explain a concept or find the right guide without leaving your editor. For multi-step work, it loads a skill first: a workflow guide that tells it the order to do things in and what to validate at each stage. Example prompts:Ask a product question
Ask a product question
Clarify a concept
Clarify a concept
Follow a guided workflow
Follow a guided workflow
search_docs, load_braintrust_skill.
Reference
Tools
Every tool the server exposes, in the order of the capabilities above.sql_query uploads it to object storage and returns an overflow envelope instead of inline rows. The envelope includes an overflow_url (a signed URL to the JSON result), a byte_length, a row_count (when available), and an instructions field describing how to retrieve the full result. Set return_url: true to request a URL even when the result is below the threshold, which is useful when you want to download or save results without putting them in model context. Field values in the result are truncated to preview_length characters (1024 by default). Set preview_length: -1 to include untruncated field values.
Skills
Skills are workflow guides your assistant loads withload_braintrust_skill and then follows. They cover Braintrust workflows such as Topics, evaluators, patterns, and automations, as well as working with the Braintrust SDK and your application code. Loading a skill is read-only and costs one tool call.
The catalog combines Braintrust’s built-in workflow guides with skills from the public Braintrust skills repository. You can also install the public skills directly into your assistant by following the repository’s installation instructions.
Ask your assistant to inspect load_braintrust_skill for the available names, then load the relevant skill. Each MCP server release includes a pinned revision of the public skills alongside its built-in guides, so its contents can differ from the latest version on GitHub. Follow each skill’s prerequisites, such as access to your application repository or an authenticated Braintrust CLI.
Loop uses a selected subset of this catalog. A skill available to an external MCP client is not necessarily available to Loop. Use the names advertised to the assistant making the request.
Resources
MCP resources provide contextual documentation that AI assistants can read to perform tasks more effectively.docs://sdk-install- Step-by-step guidance for installing the Braintrust SDK into a project, setting up tracing, configuring auto-instrumentation, and running your first eval.docs://sql- Documentation for thesql_querytool, including syntax, available fields, and examples.docs://url-formats- Reference for Braintrust URL patterns, used by theresolve_objecttool.docs://experiments- Background on Braintrust experiments and how to create them.
docs://sdk-install has companion resources for Python, TypeScript, Go, Java, Ruby, and C#. Your assistant reads the one matching your project automatically.
Troubleshooting
Authentication fails: For OAuth, sign in again using your client’s authentication controls. If you are switching from API-key authentication, remove the manually configuredAuthorization header or bearer token first. For API-key authentication, verify that the key is valid and that your client sends the header as Authorization: Bearer YOUR_BRAINTRUST_API_KEY.
Invalid client errors:
Verify that the URL matches your organization’s MCP endpoint, including /mcp and no trailing slash.
Connection timeouts:
Check your internet connection. Corporate networks might need to allowlist *.braintrust.dev and, for self-hosted deployments, your MCP endpoint.
MCP server not appearing:
Restart your client, then check the server configuration and connection status using your client’s MCP documentation.
Request body too large (HTTP 413):
The MCP server accepts request bodies up to 100 MiB. A tool call whose payload exceeds that limit returns an HTTP 413 error. Split the input into smaller requests, or fetch the data incrementally.
Server URL errors on a self-hosted deployment:
The MCP server derives its own address from the forwarding headers your ingress sets. If it reports that it could not determine the server URL, set the MCP_SERVER_URL environment variable on your data plane to your API URL.