Tools

Tool functions in Braintrust allow you to define general-purpose code that can be invoked by LLMs to add complex logic or external operations to your workflows. Tools are reusable and composable, making it easy to iterate on assistant-style agents and more advanced applications. You can create tools in TypeScript or Python, and deploy them across the UI and API via prompts.

Creating a tool

Currently, you can only create tools via code. To create a tool, use project.tool.create and pick a name and unique slug for your tool:

import * as braintrust from "braintrust";
import { z } from "zod";
 
const project = braintrust.projects.create({ name: "calculator" });
 
project.tools.create({
  handler: calculator,
  name: "Calculator method",
  slug: "calculator-2",
  description:
    "A simple calculator that can add, subtract, multiply, and divide.",
  parameters: z.object({
    op: z.enum(["add", "subtract", "multiply", "divide"]),
    a: z.number(),
    b: z.number(),
  }),
  returns: z.array(
    z.object({
      definition: z.number(),
      description: z.string(),
      op: z.string(),
    }),
  ),
  ifExists: "replace",
});

Using tools

Most models support tool calling, which allows you to define tools with well-defined input and output types. They are commonly used for two purposes:

• To enable models to "call" tools that perform external tasks, and then use those results to produce a final response
• To coerce a model into production structured outputs that match a given JSON schema

Braintrust supports both use cases.

Calling external tools

Braintrust allows you to define custom tools that can be called securely during prompt execution. You can use tools to create simple and composable agents that perform tasks like web-scraping, retrieval augmented generation (RAG), API execution, and much more.

Custom tools use the OpenAI tool calling format which means they are automatically supported by most models including OpenAI, Anthropic, and modern open-source LLMs, while following well-established industry standards.

Let's walk through an example. The following tool looks up information about the most recent commit in a GitHub repository:

import * as braintrust from "braintrust";
import { z } from "zod";
 
const project = braintrust.projects.create({ name: "github" });
 
project.tools.create({
  handler: async ({ org, repo }: { org: string; repo: string }) => {
    const url = `https://api.github.com/repos/${org}/${repo}/commits?per_page=1`;
    const response = await fetch(url);
 
    if (!response.ok) {
      throw new Error(`HTTP error! status: ${response.status}`);
    }
 
    const data = await response.json();
 
    if (data.length > 0) {
      return data[0];
    } else {
      return null;
    }
  },
  name: "Get latest commit",
  slug: "get-latest-commit",
  description: "Get the latest commit in a repository",
  parameters: z.object({
    org: z.string(),
    repo: z.string(),
  }),
  ifExists: "replace",
});

If you save this file locally to github.ts, you can run

npx braintrust push github.ts

to push the function to Braintrust. Once the command completes, you should see the function listed in the Library's Tools tab.

To use a tool, simply select it in the Tools dropdown in your Prompt window. Braintrust will automatically:

• Include it in the list of available tools to the model
• Invoke the tool if the model calls it, and append the result to the message history
• Call the model again with the tool's result as context
• Continue for up to (default) 5 iterations or until the model produces a non-tool result

Structured outputs

To define a set of tools available to a model, expand the Tools dropdown and select the Raw tab. You can enter an array of tool definitions, following the OpenAI tool format.

By default, if a tool is called, Braintrust will return the arguments of the first tool call as a JSON object. If you use the invoke API, you'll receive a JSON object as the result.