Ingero

What is Ingero?

Ingero is an eBPF-based agent that provides production-safe, kernel-level causal tracing for GPU workloads. It answers the question: "Why is my GPU training/inference slow right now?" It is designed for developers and operators debugging GPU performance issues.

How to use Ingero?

Install the single Ingero binary, then start the MCP server with ingero mcp --db ~/.ingero/ingero.db for stdio transport (Claude Code, Cursor) or ingero mcp --db ~/.ingero/ingero.db --http :8090 for HTTPS. No code changes are required. Use the provided MCP tools to diagnose performance.

Key features of Ingero

• eBPF-based causal tracing across 4 layers (CUDA API, host kernel, system context)
• Production-safe with <2% overhead and zero code changes required
• Python source line attribution (3.10/3.11/3.12) via DWARF
• SQLite storage with 10 GB rolling cap
• Kubernetes support (DaemonSet, Helm chart, pod metadata)
• Single binary, no runtime dependencies

Use cases of Ingero

• Identifying the root cause of slow GPU training or inference
• Debugging CUDA kernel launch latencies and memory bottlenecks
• Analyzing system-level events (CPU scheduling, OOM, memory pressure) affecting GPU workloads
• Running synthetic demos to test instrumentation without a GPU or root access

FAQ from Ingero

What data does Ingero collect and where is it stored?

Ingero traces CUDA Runtime and Driver API calls, host kernel tracepoints, and system context from /proc. All event data is stored in a local SQLite database at the configured path (default ~/.ingero/ingero.db) with a 10 GB rolling cap.

Does Ingero require code changes or root access?

No code changes are needed. For production monitoring, root access is required to attach eBPF programs; however, the run_demo tool can be used without a GPU or root privileges.

What GPU hardware and environments are supported?

Ingero is tested on A10, A100, H100, GH200, RTX 3090, and RTX 4090. It supports Kubernetes deployments via a DaemonSet and Helm chart, and it works with Python 3.10/3.11/3.12.

What transport protocols and authentication does Ingero use?

Ingero's MCP server supports stdio (local) and HTTP (HTTPS for remote clients). The README does not specify built-in authentication mechanisms.

How does Ingero differ from other GPU profiling tools?

Ingero provides causal chains that link system context and host-level events to specific CUDA calls, offering root cause analysis with severity ranking and fix recommendations, rather than just aggregate metrics or sampling profiles.