Setup With Observability

Guide to run OpenResponses with observability stack including OpenTelemetry, Jaeger, Prometheus, and Grafana

Running Open Responses with Observability Stack

This guide provides step-by-step instructions for running the Open Responses service with the observability stack, which includes OpenTelemetry, Jaeger, Prometheus, and Grafana.

Steps to Run Open Responses with Observability

1. Navigate to the observability infrastructure directory:

   cd observability/infra
   

2. Start the observability stack:

   docker-compose up
   

   This will run the following services:

   • OpenTelemetry Collector
   • Jaeger
   • Prometheus
   • Grafana

3. Navigate back to open-responses directory. Run Open Responses service with OpenTelemetry enabled:

   docker-compose --profile mcp up open-responses-otel
   

4. Note: You might see momentary exceptions like this for spans, logs and metrics.

   ERROR [OkHttp http://otel-collector:4318/...] [unknown] [unknown] i.o.e.internal.http.HttpExporter - Failed to export spans. The request could not be executed. Full error message: otel-collector
   

   This is normal during startup. Note: by default setup do not configure log exporter in otel-collector. Feel free to connect your favourite logging system.

5. Once all services are running properly, these errors should disappear.

6. Access the observability services through your browser:

   • Jaeger UI: http://localhost:16686/
   • Prometheus: http://localhost:9090/
   • Grafana: http://localhost:3000/

7. Grafana comes pre-loaded with production-ready dashboards in the folder ‘Open Responses’ at http://localhost:3000/dashboards. These include:

   • Open-Responses GenAI Stats: Contains generative AI performance and usage metrics
   • Open-Responses Service Stats: Contains service compute level metrics like CPU, memory usage, etc.

8. Run few times any of the curl examples mentioned in the Quickstart guide to generate data. You should see statistics in the Grafana dashboards and traces in Jaeger.

9. You can also run examples from the OpenAI Agent SDK by following the instructions in the Quickstart guide.

10. To generate enough datapoints for meaningful dashboard visualization, you can use the load generation examples available at: OpenAI Agents Python Examples

    Prerequisites:

    • The service should be running as described in step #3
    • At least one model provider key is set (GROQ_API_KEY, OPENAI_API_KEY, or CLAUDE_API_KEY)

    Examples:

    1. Simple agent load generation:

       python -m examples.open_responses.agent_hands_off_for_load_generation
       

       You can choose the model provider of your choice (groq, openai, claude).

    2. Brave search agent load generation (if brave_web_search MCP tool is configured):

       python -m examples.open_responses.brave_search_agent_for_load_generation
       

       You can choose the model provider of your choice (groq, openai, claude).

Observability in Action

The overall Prometheus state during load generation is shown below:

Here are the trace outcomes from the load generation examples: