Skip to main content
Preview - This is an early preview of Loop’s expanded agentic capabilities. The behavior, tools, and limits described here can change. To try it, reach out to your Braintrust team.

​
What’s new

  • Runs as a backend agent. Loop executes in a Braintrust-managed runtime instead of only your browser, so it’s more capable and your threads persist for you to resume later.
  • Works across your whole project. Loop uses the context it’s opened in, such as a specific experiment or trace, but can also inspect logs, experiments, and datasets across the project.
  • Takes action, not just analyzes. Loop can create and edit Braintrust objects directly, including patterns, prompts, scorers, facets, preprocessors, datasets, custom views, and dashboards.
  • Asks before changing anything. Actions that create or modify objects pause for your approval, unless you turn on auto-accept. Each approval dialog includes a short model-written description of the specific action, so you can see what will change before you allow it.
  • Previews changes in a workspace. Loop renders the objects it creates or edits, including patterns, prompts, scorers, facets, preprocessors, dashboards, datasets, traces, automations, and experiment runs, in a workspace beside the chat, so you can preview changes without leaving Loop.
  • Runs in an isolated sandbox. Each active thread gets its own sandbox, where Loop can run commands to inspect trace files directly.
  • Detects patterns in your project data. Loop searches your project for recurring problems and trends, then saves patterns with evidence from your traces.
  • Runs on a schedule. Loop can work without being asked, on a schedule and an instruction you configure.

​
Get started

There are two ways to use Loop: interactively, or through scheduled automations. To start an interactive Loop thread:
  1. Open Loop. The new dedicated Loop page gives you a full-page experience and saves your threads so you can revisit or resume them. You can also start a thread from the prompt box on the project overview page. Braintrust creates the thread and opens it on the Loop page.
  2. Choose a harness, model, and reasoning effort.
    • Harness. Choose Native for direct use of Loop’s built-in tools or Codex for deeper, code-driven investigation.
    • Model. Pick from the models available to Loop in your project. Models come from your organization and project AI providers.
    • Reasoning effort. For supported models, choose a level from Low to Max. Available levels depend on the model. The menu marks each model’s default, and the selector shows the active effort next to the model name. Loop remembers the effort you choose for each thread.
  3. Add context and attach files. Loop uses the current page’s data automatically. To point it elsewhere, select Add context. To attach a file, select Attach files or drag files into the composer. Loop accepts images, audio, video, PDF, text, and JSON, up to 8 files per message, 20 MB per file, and 40 MB per message.
  4. Ask, approve, and iterate. Type a question and press Enter. Loop runs read-only work without interruption and pauses for approval before it creates or edits anything. Approve or reject each action in the chat, and keep refining. To skip approvals, turn on Auto-accept edits in settings. To show per-response metrics, turn on Show turn metrics in the same menu. The footer includes duration, TTFT, input and output token counts, cache hit rate, and estimated turn and session cost.
As Loop works, results become available in a workspace beside the chat. You can open objects Loop creates, edits, or references, such as a pattern, prompt, scorer, facet, preprocessor, dashboard, dataset, trace, automation, or a set of experiment runs, and keep multiple objects in workspace tabs while the thread continues. Hold Cmd, Ctrl, or Shift while selecting an object to open it in a new browser tab instead.

​
What Loop can do

Loop has its own tools for driving playgrounds, custom views, dataset snapshots, and the workspace, and it can also call the read and write tools that the Braintrust MCP server exposes to coding agents. Loop chains several of them in one turn. For example, it can find a failure pattern in your logs, write a scorer for it, and open that scorer in the workspace for you to review.

​
Explore your data

Loop can query your logs, experiments, and datasets with SQL, inspect the schema of a data source, and read, search, and list files in a sandbox, or run commands there, to dig through large or complex traces. When Loop resolves a trace or generates a permalink, you can open that trace in the workspace and inspect it alongside the thread. Loop can also write a query into the SQL sandbox and run it there, so you keep the query to refine by hand. Across projects, Loop can pull project rows with their usage metrics to compare projects. Example prompts:

​
Analyze traces

Loop can analyze complex traces, making it easier to understand what happened across many LLM and tool calls. On the Logs page, open a trace in the Timeline view and click Analyze trace in the timeline header. Loop groups meaningful LLM and tool spans into a Work view and explains what each part of the trace contributed. If the trace already has a work analysis, click Reanalyze to run it again. The Debugger view on the Logs page focuses on failure analysis. Click Run debugger to have Loop identify likely failure modes, cite evidence from spans, tool calls, and model outputs, and suggest next steps. If a debugger report already exists, click Rerun debugger to refresh it.

​
Find patterns

Most Braintrust projects contain more traces than a team can review by hand. Loop can search logs and raw trace data for recurring problems and trends that are not already captured by a scorer, such as a failure mode that affects one cohort of users or a recurring LLM call whose cost increased. It records each recurring problem or trend as a pattern: a saved result with the evidence behind it, rather than a single trace. Example prompts:
When Loop records a pattern, it appears on the Patterns page alongside patterns found by scheduled automation runs. When new evidence points to a pattern Loop already recorded, it updates that pattern instead of adding a new one.
Patterns are scoped to a single project. To review a pattern and its evidence, go to Patterns, directly below Loop in the left sidebar, and select the pattern from the list.The list shows open patterns by default. Click Filter patterns above the list to switch between Open, Closed as resolved, Closed as not useful, and Closed without reason. Each option shows how many patterns it contains.The pattern detail shows:
  • Description and recommended next step. What Loop found, the context behind it, and the next investigation or fix Loop recommends.
  • Evidence. Monitor charts can track matching traces over time, and trace evidence shows the traces themselves with Loop’s notes quoted inline. Select a trace to open it in the Debugger view on the Logs page. The trace scrolls to the span field the evidence came from and highlights the quoted terms.
  • How to monitor. When Loop cannot attach a useful monitor chart, the pattern can instead recommend a scorer, classification, or facet you could create to measure it later. Click Create classifier or Create scorer next to the recommendation, depending on what Loop recommended, to open Loop with a pre-filled prompt that adds the evaluator to the pattern, determines a backfill strategy with you, and creates a monitor chart for it.
  • Lifecycle. Click Close in the pattern header to mark a pattern inactive. Selecting an outcome is optional: choose Resolved if the problem is fixed, addressed, or no longer happening, or Not useful if the pattern is noise, transient, or unimportant. You can also add free-text feedback, which Loop uses to improve future findings. The header then reads Closed as resolved, Closed as not useful, or Closed without outcome, and a banner on the detail attributes the closure to the person who closed it and shows their feedback. Click Reopen to set a closed pattern back to active. Loop can also update status when it changes an existing pattern. Treat a closed pattern as Loop’s judgment about the available evidence, not as proof that the underlying problem was fixed.
To continue, share, or close a pattern:
  • Keep investigating.
    • On the Patterns page, open a pattern and select Continue in Loop. Braintrust starts a thread with the pattern attached as a context badge showing its name, so you can see what Loop is working from.
    • To work out why a scheduled run did or did not produce a given pattern, open the automation, select the Past runs tab, and select the run. Then select Continue to fork the read-only transcript into a private, editable thread.
  • Hand it off. On the Patterns page, open a pattern and click Copy pattern as prompt in the pattern header. Braintrust copies a Markdown brief with the pattern’s status and close outcome, description, recommended next step, evidence trace IDs, and bt CLI commands for pulling the full traces.
  • Close it. On the Patterns page, open a pattern and click Close. Closed patterns leave the Open list and move to Closed as resolved, Closed as not useful, or Closed without reason, and can be reopened at any time.
Closing and reopening a pattern requires update permission on the project’s patterns dataset.
Loop can also find patterns on a schedule. See Run Loop automatically.

​
Configure Topics

Loop can configure the whole Topics pipeline, not just test individual facets. Given a goal, Loop works through the pipeline in stages:
  • Check the preprocessor. Loop inspects your project’s effective default preprocessor and runs it on real traces to confirm it returns usable conversation text. If the default returns nothing, Loop writes a custom preprocessor for your trace shape, tests it, and can set it as the project default.
  • Confirm the scope. Loop looks for a grouping key such as metadata.conversation_id and recommends whether to analyze a span, a trace, or a group of related traces.
  • Validate the facets. Loop runs a facet on up to ten representative traces and shows you the results inline for review, so you can correct its reading before it commits to a prompt. Build up a set of traces worth checking, then have Loop re-run the facet against that same set as you refine it. For a larger set, Loop runs the facet as an eval instead.
  • Evaluate candidates. Loop runs evals over candidate facet prompts to measure quality before anything is saved.
  • Enable the automation. Loop creates the facets and turns on the Topics automation, then offers to rewind it over a historical window.
Rewinding a Topics automation processes historical traces and draws from your monthly model credits. Loop estimates the volume and asks for approval before it rewinds.
Example prompts:

​
Build views and dashboards

Loop can create and edit custom views for both trace and dataset row data, and it can create dashboards. When Loop creates or updates a dashboard, you can open it in the workspace to review the result alongside the thread. Example prompts:

​
Manage automations and alerts

Loop can set up recurring work and notifications from what it finds, so a problem you just investigated becomes something you get told about, or something Loop rechecks on a schedule.
  • Alerts. Loop can create an alert on individual matching logs, on an aggregate that crosses a threshold over a recent window, or on environment updates.
  • Automations. Loop can create a scheduled Loop job with its own instruction and schedule. It can also list the project’s existing automations, including online scoring rules, alerts, exports, retention policies, and Topics automations, and return each one’s full configuration, which helps when you’re working out why a scheduled run behaves the way it does.
  • Status. Loop can pause an automation or set it back to active.
  • Updates. Loop can change an alert or a Loop automation, including a Loop automation’s instruction and schedule, by rewriting it under its existing name. A rewrite replaces the whole configuration rather than patching a single field, so ask Loop to read the current configuration first, then check what it proposes before you approve. Scheduled exports and retention policies are inspect-only.
You can also edit any automation directly in Settings > Automations. Example prompts:
Alerts and automations Loop creates appear alongside the ones you configure by hand. See Alerts for delivery channels and tuning, and Run Loop automatically for configuring a scheduled run yourself.

​
Author prompts and evaluators

Loop can suggest evaluators grounded in your application’s real behavior, then create and save them, not just describe them. An evaluator is either a scorer, which returns a number, or a classifier, which returns a label. Loop can also create and update prompts directly. New and updated evaluators open in the workspace so you can review them inline. Example prompts:

​
Run evaluations

Loop runs evaluations two ways:
  • In a playground or experiment. Loop can edit the task prompt or code, add or modify scorers, and re-run the evaluation to show the effect of a change.
  • As a new experiment. Loop can run an experiment from a dataset, from rows it supplies inline, or from a prior experiment, using a saved or inline task and any scorers you choose. When a prior experiment supplies the data, its outputs become the expected values.
When Loop starts an evaluation, the runs open in an Experiments panel in the workspace that lists each run’s progress and scores, so you can compare results without leaving the chat.
Loop runs evaluations through Braintrust rather than by writing an Eval() script. To keep evaluations in your own repository and run them in CI, see Run evaluations in code.
Example prompts:

​
Curate datasets

Loop can generate datasets from log patterns and edit rows in an existing dataset. When Loop edits dataset rows, the dataset opens as a read-only table in the workspace panel. Select a row to inspect its trace inline, and the table refreshes as Loop makes further edits. Loop can also manage dataset versions and the environments that point at them:
  • Snapshots. Loop can save a snapshot of a dataset before it changes anything, list the existing snapshots along with their environment tags and current head version, and restore the dataset to a snapshot. Restoring uses the same confirmation flow as the dataset UI, and can save the current state first.
  • Environments. Loop can point an environment such as production or staging at a specific snapshot, or clear that mapping so consumers no longer resolve the dataset by it. It can list the organization’s environments to check available slugs, and create one that doesn’t exist yet.
Restoring a dataset to a snapshot replaces its current rows, and retagging an environment changes which dataset state your applications resolve. Review the snapshot Loop proposes before you approve either action.
Example prompts:

​
Search docs and find objects

Loop can search the Braintrust documentation, resolve object names and URLs, list recent objects, summarize an experiment’s metrics against a baseline, and generate shareable links to objects it creates. Example prompts:

​
What Loop cannot do

Loop works inside your permissions, on one project at a time. Loop cannot:
  • Reach another organization’s data. A thread is scoped to the project you opened it in, and Loop resolves objects only within your organization.
  • Delete Braintrust objects. Loop can create and update prompts, scorers, facets, preprocessors, datasets, views, and automations, but it has no tool to delete them. It can delete dataset rows, and it can close a pattern, which marks the pattern inactive rather than removing it.
  • Change organization settings. Members, permission groups, API keys, AI providers, and billing are out of reach. Loop can list and create environments for dataset versioning, but it can’t delete one.
  • See another user’s threads. Threads are private to the person who created them.
  • Act without approval. Every action that creates or modifies an object pauses for your approval, unless you turn on auto-accept.
  • Change your application. Loop works on the data and objects in Braintrust, not on your codebase or your deployments.

​
Run Loop automatically

Use Loop automations for recurring work that should run without someone starting and managing a chat. A Loop automation is a scheduled Loop run with an instruction, model, write tool permissions, and optional destinations. The instruction defines the job for each run, and the default query range controls how much recent project data Loop inspects. For example, you can use Loop automations to:
  • Find patterns in recent traces.
  • Review cost or latency on a schedule.
  • Check whether quality is degrading.
  • Summarize recent errors.
  • Review logs using the filters from a dashboard.
  • Send a Slack or webhook digest after each run.
Each run creates a read-only thread with the work Loop did. To find it, open the automation and select the Past runs tab, then select a run. Automation run threads stay with their automation and don’t appear in the thread list on the Loop page. To ask follow-up questions from one of those transcripts, select Continue, which forks the run into a private, editable thread. Loop automations use the model selected in the automation, and cost depends on that model and how much data the run inspects. Keep the instruction focused: a run that reads a broad set of traces can cost more than one that reviews a narrower slice. Models served by your configured AI providers bill to those providers. To create a Loop automation, go to Settings > Automations, click + Rule, and select Loop.

​
Configure a Loop automation

Each automation has its own instruction, frequency, model, query range, write permissions, and destinations. Use the configuration panel to control what the run investigates, how often it runs, and what it is allowed to change. To stop a Loop automation, pause or delete it.
To run a saved automation immediately, open the automation and click Run now. Braintrust queues the run and shows a link to the Loop thread when it is ready.
In a Loop thread, actions that change something pause for your approval. A scheduled run has nobody to ask, so it relies on the automation’s Write tool permissions configuration.
  • Read-only tools do not require approval, so the run can inspect project data such as traces, existing patterns, experiments, and datasets.
  • By default, automations created from the Patterns page can create and update pattern records. A pattern record can include trace evidence, monitor chart configurations, and recommendations for how to measure the pattern later. Creating scorers, facets, or other project objects requires the corresponding write tools.
  • To let a Loop automation create or modify project objects, add the corresponding tools. The dropdown lists the available tools and what each one permits.
  • Objects a run creates are not linked back to the pattern. If a pattern’s monitor chart measures a scorer or facet the run created, deleting it stops new data from reaching that chart.
Enabling a tool lets scheduled runs make that kind of change without asking for approval during the run. Choose only the tools the automation needs.
Allowing a run to create or update online scoring rules can start work that continues after the run ends. Online scoring rules score production traffic continuously and can keep costing money.
To send Loop automation results to a destination outside Braintrust, add a destination in the configuration panel. Click + Destination, then choose Send to webhook or Send to Slack.
  • Slack. Select a Slack channel to post a text digest to Slack. Slack destinations require a connected Slack workspace. Use the destination’s Formatting prompt to tell Loop how to format the message.
  • Webhook. Enter a webhook URL to send run results to an external service. Use the destination’s Formatting prompt to describe the JSON payload Loop should produce.
Patterns recorded from an interactive Loop chat do appear in the Patterns list, but they do not trigger Slack or webhook destinations. Destinations belong to automation runs rather than to the Patterns list itself.

​
Pattern discovery

Braintrust provides a shortcut for creating a Loop automation that looks for patterns. On the Patterns page:
  • Enable automatic pattern discovery by clicking Enable patterns. Braintrust opens the Introducing Patterns dialog, where you can optionally choose a Slack workspace and channel to be notified in when Loop finds patterns. Click Enable patterns to confirm. Braintrust creates one default Pattern discovery automation that runs every 24 hours, uses a 30-day default query range, and has permission to create and update patterns. After you enable it, Enable patterns changes to Settings.
  • Configure an existing automation by clicking Settings and selecting the automation you want to edit.
  • Create another automation by selecting Settings > New automation. Braintrust opens a prefilled automation you can review and save.
Loop automations created from the Patterns page are still ordinary Loop automations. You can also view and configure them in Settings > Automations. When you create a pattern automation, keep the instruction framed around finding or updating patterns, even when the automation focuses on a specific area such as latency, cost, quality, or sentiment. Enabling pattern discovery queues an initial analysis run right away, and patterns appear once that run finishes. While a run is in progress, the Patterns page shows an Analyzing traces… indicator. When a project has no patterns yet, click Analyze patterns to queue another run without waiting for the schedule.

​
Manage threads

To delete a thread, select the menu on the thread header or on any thread in the sidebar, select Delete session, then confirm.
Deleting a session is permanent and cannot be undone.
You can’t delete a session while it’s running. Automation run threads are read-only and can’t be deleted this way.

​
Limits and caveats

Loop runs in an isolated, per-user sandbox. When a sandbox starts. Loop starts a sandbox for a thread the first time you send a message in it, not when you open the Loop page or revisit a saved thread. One active thread uses one sandbox. When a sandbox stops counting. After 15 minutes without activity, the sandbox goes to sleep, and a sleeping sandbox doesn’t count toward your concurrent limit. Send another message in that thread and it resumes automatically, with the thread’s history intact. The concurrent limit therefore applies to the threads you’re working in at the same time, not to the total number of threads you’ve created. When you reach the limit. If you send a message while all your sandboxes are running, Loop makes room by reclaiming the thread that has been quiet longest, and your message proceeds. You only see an error if every sandbox is actively working, such as several threads running long investigations at once. Wait for one to finish, then send the message again. When a sandbox reaches its maximum lifetime, Loop starts a fresh one for your next message. Your thread and its history are unaffected. Additional caveats during the preview:
  • This is a preview feature enabled per organization. Capabilities and limits can change.
  • Loop threads are private to the user who created them. Sharing and cross-user access are not yet available.

​
Share feedback

The preview is an active area of development, and your feedback shapes it. Tell your Braintrust team what worked, what didn’t, and what you’d want Loop to do next. You can also ask Loop to file a support ticket directly from the chat.