🚀 TracePilot: Turning AI Agent Executions into Actionable Insights with OpenTelemetry and SigNoz

Building TracePilot

• AI agents are becoming more powerful every day, but understanding how they make decisions is still difficult.
• When an AI workflow fails, developers usually switch between logs, traces, dashboards, and monitoring tools to identify the root cause. This process is slow and frustrating.
• I wanted to build something that makes AI execution easier to understand.
• That's why I built "TracePilot", an AI-powered execution intelligence dashboard that transforms execution telemetry into an interactive visualization.
• The project is built using Next.js and is designed to work alongside SigNoz and OpenTelemetry.

("TracePilot dashboard showing live execution metrics, replay controls, and span details.")

The Problem

• Debugging AI systems is challenging because execution information is spread across multiple places.
• Some common problems include:
  1. Thousands of logs
  2. Complex execution flows
  3. Slow root cause analysis
  4. Difficult trace inspection
  5. No visual execution replay
• Developers spend more time debugging than building.

( " The dashboard immediately highlights execution health using key metrics.")

• A single execution may involve:

User Prompt
│
▼
Planner
│
▼
Memory
│
▼
Tool/API
│
▼
LLM
│
▼
Final Response

My Solution

• TracePilot provides a single dashboard where developers can monitor AI executions in real time.
• Instead of reading raw logs, developers can inspect executions visually.
• The platform focuses on:
  1. - Execution replay
  2. - Performance metrics
  3. - Failure analysis
  4. - Span inspection
• Future SigNoz integration.

Introducing TracePilot

TracePilot is an Interactive AI Execution Explorer designed to make AI workflows easier to understand.

Instead of treating telemetry as isolated traces and logs, TracePilot presents an execution as a connected workflow.

The goal is simple:

Help developers move from telemetry to understanding.

The current MVP focuses on visualizing execution data and demonstrating the developer experience. It uses sample execution data to showcase the interface, with the long-term goal of integrating directly with live OpenTelemetry traces collected by SigNoz.

Architecture

The high-level architecture behind TracePilot is designed to work with standard observability tooling.

User
│
▼
AI Agent
│
▼
OpenTelemetry SDK
│
▼
OTLP Exporter
│
▼
SigNoz
│
▼
TracePilot Dashboard
│
▼
Execution Intelligence

In this prototype, the dashboard demonstrates the intended user experience using mock execution data. The next step is connecting these visualizations to live telemetry.

Technology Stack

Layer Technology
Frontend : Next.js
Language : TypeScript
Styling : Tailwind CSS
Workflow Visualization : React Flow
Animations : Framer Motion
Icons : Lucide React
Observability : OpenTelemetry (planned integration)
Monitoring : SigNoz (planned integration)

🔄 Interactive Execution Graph

One of the core ideas behind TracePilot is representing an AI workflow as an interactive graph.

Instead of reading dozens of spans individually, developers can follow the complete execution path visually.

User Prompt
│
▼
Planner
│
▼
Memory
│
▼
Flight Search API
│
▼
LLM
│
▼
Response

Each node represents a stage in the workflow.

🔍 Span Details

Clicking any execution step reveals additional information, such as:

1. Status
2. Duration
3. Retries
4. Token usage
5. Trace identifier

This allows developers to inspect each stage without leaving the dashboard.

▶️ Execution Replay

Understanding execution order is just as important as understanding execution details.

Replay mode walks through each step sequentially, making the workflow easier to follow.

This concept could become especially valuable for long-running AI agents involving multiple tools.

🤖 AI Investigation (Prototype)

Another idea explored in TracePilot is AI-assisted investigation.

Instead of requiring developers to manually interpret telemetry, the interface can summarize:

Possible root cause
Supporting evidence
Recommendations

For example:

1. Root Cause
2. Flight Search API timed out.
3. Evidence
4. Execution duration exceeded expected latency.
5. Recommendation
6. Increase timeout.
7. Enable retry strategy.
8. Investigate API response times.
9. The current implementation demonstrates the user experience using example data.

Development Journey

One aspect of this project I found especially interesting was using AI-assisted development tools during the prototyping process.

They helped accelerate repetitive UI work, allowing me to focus more on the product concept, user experience, and observability workflow rather than writing every component from scratch.

This made it easier to iterate quickly while exploring different ideas.

Challenges

Building even a prototype raised several interesting questions:

1. How should complex AI workflows be visualized?
2. What information matters most during debugging?
3. How can observability data be presented without overwhelming developers?
4. How should execution replay work for multi-step AI agents?

Designing the interface required balancing technical detail with usability.

Future Improvements

There are several directions I would like to explore:

1. Live OpenTelemetry integration
2. Direct SigNoz trace linking
3. AI-generated root cause analysis from real telemetry
4. Execution comparison across multiple runs
5. Token usage analytics
6. Cost estimation
7. Performance regression detection
8. Multi-agent workflow visualization
9. What I Learned

Working on TracePilot reinforced an important idea:

1. Collecting telemetry is only the first step.
2. Developers also need interfaces that help them interpret that telemetry quickly.
3. Visualizing an execution as a workflow feels far more intuitive than manually reconstructing it from logs and traces.

Conclusion

AI agents are becoming increasingly complex, and understanding their behavior is quickly becoming an observability challenge.

TracePilot explores a simple idea:

Instead of asking developers to reconstruct an execution from raw telemetry, present the execution as an interactive, visual story.

This project is still evolving, but building the prototype helped me think differently about how observability tools can support AI systems.

The next milestone is integrating TracePilot with live OpenTelemetry traces and SigNoz, turning this prototype into a fully functional execution intelligence platform.

Thank You

Thank you to the WeMakeDevs and SigNoz teams for organizing this hackathon and encouraging developers to experiment with modern AI observability. Building TracePilot has been a great opportunity to explore how developer experience and observability can come together in new ways.