Detecting Remote Eval and CLI Execution Context: SDK Hooks and Environment Variables

Applies to:

Plan -
Deployment -

Summary When running evaluations via remote evals, traces may appear minimal compared to CLI evals, and prompts may not be visible in the experiment. This occurs because remote evals run on your infrastructure without Braintrust’s instrumented runner. You can detect execution context using environment variables and hooks.tags to conditionally adjust tracing, logging, or other behavior between CLI and remote eval runs.

Configuration Steps

Step 1: Set Environment Variable for CLI Execution

When running evaluations via CLI, set an environment variable to mark the execution context.

BRAINTRUST_CLI=true bt eval my_eval.py

Step 2: Add Context Tag in Task Function

Check the environment variable in your task function and add a tag to identify CLI execution.

import os
from braintrust import Eval, EvalCase

def my_task(input, hooks):
    # Add "cli" tag when running via CLI
    if os.getenv("BRAINTRUST_CLI") == "true":
        if hooks.tags is None:
            hooks.tags = ["cli"]
        else:
            hooks.tags = list(hooks.tags) + ["cli"]

    # Your task logic here
    return output

Eval(
    name="my-eval",
    data=lambda: [EvalCase(input="example")],
    task=my_task,
    scores=[...],
)

Step 3: Use Context Detection in Task Logic

Check for the presence of the tag to determine execution context and implement conditional behavior.

def my_task(input, hooks):
    is_cli = "cli" in (hooks.tags or [])

    if is_cli:
        # CLI-specific logic (e.g., verbose logging)
        print("Running in CLI context")
    else:
        # Remote eval logic (e.g., reduced logging)
        print("Running in remote eval context")

    return output

Limitations

This approach requires manually prepending the environment variable to every CLI command. There is no built-in SDK support for execution context detection currently available.

⌘I

​Configuration Steps

​Step 1: Set Environment Variable for CLI Execution

​Step 2: Add Context Tag in Task Function

​Step 3: Use Context Detection in Task Logic

​Limitations

Configuration Steps

Step 1: Set Environment Variable for CLI Execution

Step 2: Add Context Tag in Task Function

Step 3: Use Context Detection in Task Logic

Limitations