Debug hanging eval processes with external dependencies

Applies to:

Plan -
Deployment -

Summary

Issue: Node.js process hangs indefinitely after braintrust eval completes successfully, preventing the process from exiting. The eval finishes but the CLI appears frozen with no output or error messages. Cause: External dependencies (database connection pools, persistent connections, WebSocket clients) keep the Node.js event loop active after eval completion. Resolution: Identify blocking dependencies using Node.js debugger and explicitly close all external connections after eval completion, or use in-memory alternatives during evals.

Resolution Steps

Step 1: Identify the blocking dependency

Run your eval with Node.js inspector to identify which module prevents process exit.

node --inspect-brk ./node_modules/.bin/braintrust eval your-eval.ts

Open chrome://inspect, pause execution when the process hangs, and check the call stack for active connections.

Step 2: Close external connections after eval completion

Explicitly close all database connections, clients, and persistent resources in your eval code.

import { Eval } from "braintrust";
import { MongoClient } from "mongodb";

const client = new MongoClient(process.env.MONGODB_URI);

Eval("my-eval", {
  data: async () => {
    await client.connect();
    // ... eval logic
    return data;
  },
  task: async (input) => {
    // ... task logic
  },
  scores: [/* ... */],
});

// Close connections after eval
process.on('beforeExit', async () => {
  await client.close();
});

Alternative: Use in-memory implementations during evals

Replace blocking integrations with in-memory alternatives to avoid connection pool issues.

const memoryStore = process.env.IS_EVAL
  ? new InMemoryStore()
  : new MongoDBStore(client);

Step 3: Common culprits to check

Verify these dependencies are properly closed:

MongoDB connection pools: client.close()
Redis clients: redis.disconnect()
PostgreSQL/MySQL pools: pool.end()
WebSocket connections: ws.close()
HTTP keep-alive agents: agent.destroy()

​Summary

​Resolution Steps

​Step 1: Identify the blocking dependency

​Step 2: Close external connections after eval completion

​Alternative: Use in-memory implementations during evals

​Step 3: Common culprits to check

Summary

Resolution Steps

Step 1: Identify the blocking dependency

Step 2: Close external connections after eval completion

Alternative: Use in-memory implementations during evals

Step 3: Common culprits to check