README
¶
Inspektor Gadget is a collection of tools (or gadgets) to debug and inspect Kubernetes resources and applications. It manages the packaging, deployment and execution of eBPF programs in a Kubernetes cluster, including many based on BCC tools, as well as some developed specifically for use in Inspektor Gadget. It automatically maps low-level kernel primitives to high-level Kubernetes resources, making it easier and quicker to find the relevant information.
The Gadgets
Inspektor Gadget tools are known as gadgets. You can deploy one, two or many gadgets.
Explore the following documentation to find out which tools can help you in your investigations.
Installation
Install Inspektor Gadget (client-side):
Use krew plugin manager to install:
$ kubectl krew install gadget
Install Inspektor Gadget on Kubernetes:
$ kubectl gadget deploy
Read the detailed install instructions to find more information.
How to use
kubectl gadget --help will provide you the list of supported commands and their flags.
$ kubectl gadget --help
Collection of gadgets for Kubernetes developers
Usage:
kubectl-gadget [command]
Available Commands:
advise Recommend system configurations based on collected information
audit Audit a subsystem
completion Generate the autocompletion script for the specified shell
deploy Deploy Inspektor Gadget on the cluster
help Help about any command
profile Profile different subsystems
prometheus Expose metrics using prometheus
run Run a containerized gadget (experimental)
script Run a bpftrace-compatible scripts
snapshot Take a snapshot of a subsystem and print it
sync Synchronize gadget information with server
top Gather, sort and periodically report events according to a given criteria
trace Trace and print system events
traceloop Get strace-like logs of a container from the past
undeploy Undeploy Inspektor Gadget from cluster
version Show version
...
You can then get help for each subcommand:
$ kubectl gadget advise --help
Recommend system configurations based on collected information
Usage:
kubectl-gadget advise [command]
Available Commands:
network-policy Generate network policies based on recorded network activity
seccomp-profile Generate seccomp profiles based on recorded syscalls activity
...
$ kubectl gadget audit --help
Audit a subsystem
Usage:
kubectl-gadget audit [command]
Available Commands:
seccomp Audit syscalls according to the seccomp profile
...
$ kubectl gadget profile --help
Profile different subsystems
Usage:
kubectl-gadget profile [command]
Available Commands:
block-io Analyze block I/O performance through a latency distribution
cpu Analyze CPU performance by sampling stack traces
tcprtt Analyze TCP connections through an Round-Trip Time (RTT) distribution
...
$ kubectl gadget snapshot --help
Take a snapshot of a subsystem and print it
Usage:
kubectl-gadget snapshot [command]
Available Commands:
process Gather information about running processes
socket Gather information about TCP and UDP sockets
...
$ kubectl gadget top --help
Gather, sort and periodically report events according to a given criteria
Usage:
kubectl-gadget top [command]
Available Commands:
block-io Periodically report block device I/O activity
ebpf Periodically report ebpf runtime stats
file Periodically report read/write activity by file
tcp Periodically report TCP activity
...
$ kubectl gadget trace --help
Trace and print system events
Usage:
kubectl-gadget trace [command]
Available Commands:
bind Trace socket bindings
capabilities Trace security capability checks
dns Trace DNS requests
exec Trace new processes
fsslower Trace open, read, write and fsync operations slower than a threshold
mount Trace mount and umount system calls
network Trace network streams
oomkill Trace when OOM killer is triggered and kills a process
open Trace open system calls
signal Trace signals received by processes
sni Trace Server Name Indication (SNI) from TLS requests
tcp Trace TCP connect, accept and close
tcpconnect Trace connect system calls
tcpdrop Trace TCP kernel-dropped packets/segments
tcpretrans Trace TCP retransmissions
...
How does it work?
Inspektor Gadget is deployed to each node as a privileged DaemonSet. It uses in-kernel eBPF helper programs to monitor events mainly related to syscalls from userspace programs in a pod. The eBPF programs are run by the kernel and gather the log data. Inspektor Gadget's userspace utilities fetch the log data from ring buffers and display it. What eBPF programs are and how Inspektor Gadget uses them is briefly explained in the architecture document.
ig
Inspektor Gadget can also be used without Kubernetes to trace containers with
the ig tool.
Kernel requirements
The different gadgets shipped with Inspektor Gadget use a variety of eBPF capabilities. The capabilities available depend on the version and configuration of the kernel running in the node. To be able to run all the gadgets, you'll need to have at least 5.10 with BTF enabled.
See requirements for a detailed list of the requirements per gadget.
Code examples
There are some examples in this folder showing the usage
of the Golang packages provided by Inspektor Gadget. These examples are
designed for developers that want to use the Golang packages exposed by
Inspektor Gadget directly. End-users do not need this and can use
kubectl-gadget or ig directly.
Contributing
Contributions are welcome, see CONTRIBUTING.
Community Meeting
We hold community meetings every other Thursday at 15:30 UTC, 7:30 PST, 16:30 CET in this link, check the calendar to have the full schedule of next meetings. Please add any topic you want to discuss to our meeting notes document.
Slack
Join the discussions on the #inspektor-gadget channel in the Kubernetes Slack.
Talks
- Collecting Low-Level Metrics with eBPF, KubeCon + CloudNativeCon North America 2023 (video, slides)
- A (re)introduction of Inspektor Gadget: A Containerized Framework for eBPF Systems Inspection, Cloud Native Rejekts Chicago - November 2023 (video)
- Gaining Linux insights with Inspektor Gadget, an eBPF tool and systems inspection framework, All Systems Go - September 2023 (video)
- Overcoming the Challenges of Debugging Containers, Container Days Hamburg - September 2023 (video)
- Using the EBPF Superpowers To Generate Kubernetes Security Policies, KubeCon + CloudNativeCon North America 2022 (video, slides)
- Debug Your Clusters with eBPF-Powered Tools, Cloud Native eBPF Day North America 2022 (video, slides)
- Who Needs an API Server to Debug a Kubernetes Cluster?, Cloud Native eBPF Day North America 2022 (video, slides)
- Inspektor Gadget, introduction and demos, eCHO Livestream - September 2021 (video)
- OpenShift Commons Briefing: Unleash eBPF Superpowers with Kubectl Gadget, Openshift Commons 2020 (video)
- Tutorial: Understanding What Happens Inside Kubernetes Clusters Using BPF Tools, Open Source Summit EU 2020 (video)
- Inspektor Gadget and traceloop: Tracing containers syscalls using BPF, FOSDEM 2020 (video, slides)
- Traceloop for systemd and Kubernetes + Inspektor Gadget, All Systems Go 2019 (video)
Thanks
- BPF Compiler Collection (BCC): some of the gadgets are based on BCC tools.
- kubectl-trace: the Inspektor Gadget architecture was inspired from kubectl-trace.
- cilium/ebpf: the gadget tracer manager and some other gadgets use the cilium/ebpf library.
License
The Inspektor Gadget user space components are licensed under the Apache License, Version 2.0. The BPF code templates are licensed under the General Public License, Version 2.0, with the Linux-syscall-note.
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
examples
|
|
|
gadget-container
|
|
|
internal
|
|
|
pkg
|
|
|
apis/gadget/v1alpha1
Package v1alpha1 contains API Schema definitions for the gadget v1alpha1 API group +kubebuilder:object:generate=true +groupName=gadget.kinvolk.io
|
Package v1alpha1 contains API Schema definitions for the gadget v1alpha1 API group +kubebuilder:object:generate=true +groupName=gadget.kinvolk.io |
|
btfgen
Package btfgen provides a way to load BTF information generated with btfgen.
|
Package btfgen provides a way to load BTF information generated with btfgen. |
|
client/clientset/versioned
This package has the automatically generated clientset.
|
This package has the automatically generated clientset. |
|
client/clientset/versioned/fake
This package has the automatically generated fake clientset.
|
This package has the automatically generated fake clientset. |
|
client/clientset/versioned/scheme
This package contains the scheme of the automatically generated clientset.
|
This package contains the scheme of the automatically generated clientset. |
|
client/clientset/versioned/typed/gadget/v1alpha1
This package has the automatically generated typed clients.
|
This package has the automatically generated typed clients. |
|
client/clientset/versioned/typed/gadget/v1alpha1/fake
Package fake has the automatically generated clients.
|
Package fake has the automatically generated clients. |
|
columns
Package columns is a library that helps to carry data structs in a more generic way using a combination of reflection and generics.
|
Package columns is a library that helps to carry data structs in a more generic way using a combination of reflection and generics. |
|
columns/ellipsis
Package ellipsis helps to truncate text at a specific width and adding an optional ellipsis ("…") to indicate that the text has been truncated.
|
Package ellipsis helps to truncate text at a specific width and adding an optional ellipsis ("…") to indicate that the text has been truncated. |
|
columns/filter
Package filter helps filtering an array of structs that were analyzed by the columns library.
|
Package filter helps filtering an array of structs that were analyzed by the columns library. |
|
columns/formatter/textcolumns
Package textcolumns helps to output structs (and events of structs) using metadata from a `Columns` instance in a tabular way suitable for consoles or other frontends using fixed-width characters / fonts.
|
Package textcolumns helps to output structs (and events of structs) using metadata from a `Columns` instance in a tabular way suitable for consoles or other frontends using fixed-width characters / fonts. |
|
columns/group
Package group can group the entries of an array by one or more columns.
|
Package group can group the entries of an array by one or more columns. |
|
columns/sort
Package sort can be used to sort an array by their columns in either ascending or descending order.
|
Package sort can be used to sort an array by their columns in either ascending or descending order. |
|
container-collection
Package containercollection provides the ContainerCollection struct to keep track of the set of running containers and primitives to query that set with various criteria.
|
Package containercollection provides the ContainerCollection struct to keep track of the set of running containers and primitives to query that set with various criteria. |
|
container-hook
Package containerhook detects when a container is created or terminated.
|
Package containerhook detects when a container is created or terminated. |
|
environment
Package environment is a temporary workaround to have a simple means of knowing what environment we're running in.
|
Package environment is a temporary workaround to have a simple means of knowing what environment we're running in. |
|
environment/k8s
Package k8s sets the environnment to Kubernetes.
|
Package k8s sets the environnment to Kubernetes. |
|
environment/local
Package local sets the environnment to Local.
|
Package local sets the environnment to Local. |
|
gadget-context
Package gadgetcontext handles initializing gadgets and installed operators before handing them over to a specified runtime.
|
Package gadgetcontext handles initializing gadgets and installed operators before handing them over to a specified runtime. |
|
histogram
Package histogram provides a Histogram struct that represents a histogram of the number of events that occurred in each interval.
|
Package histogram provides a Histogram struct that represents a histogram of the number of events that occurred in each interval. |
|
kallsyms
Package kallsyms provides functions to read /proc/kallsyms.
|
Package kallsyms provides functions to read /proc/kallsyms. |
|
logger
Package logger provides a logger interface that is available to gadgets and runtimes.
|
Package logger provides a logger interface that is available to gadgets and runtimes. |
|
networktracer
Package networktracer installs the dispatcher ebpf program in each network namespace of interest.
|
Package networktracer installs the dispatcher ebpf program in each network namespace of interest. |
|
operators/kubeipresolver
Package kubeipresolver provides an operator that enriches events by looking up IP addresses in Kubernetes resources such as pods and services.
|
Package kubeipresolver provides an operator that enriches events by looking up IP addresses in Kubernetes resources such as pods and services. |
|
operators/kubenameresolver
Package kubenameresolver provides an operator that enriches events by looking up the pod name and namespace and enriches it with its ip information.
|
Package kubenameresolver provides an operator that enriches events by looking up the pod name and namespace and enriches it with its ip information. |
|
operators/socketenricher
Package socketenricher creates an eBPF map exposing processes owning each socket.
|
Package socketenricher creates an eBPF map exposing processes owning each socket. |
|
params
Package params provides a generic way to describe parameters used by gadgets, operators and runtimes including validation.
|
Package params provides a generic way to describe parameters used by gadgets, operators and runtimes including validation. |
|
parser
Package parser wraps a couple of helper libraries with the intention of hiding type information and simplifying data handling outside the gadgets.
|
Package parser wraps a couple of helper libraries with the intention of hiding type information and simplifying data handling outside the gadgets. |
|
tchandler
Package tchandler handles how SchedCLS programs are attached to containers and network interfaces.
|
Package tchandler handles how SchedCLS programs are attached to containers and network interfaces. |
|
utils/bpf-iter-ns
Package bpfiterns reads a ebpf iterator in a different namespace.
|
Package bpfiterns reads a ebpf iterator in a different namespace. |
|
utils/host
Package host provides ways to access the host filesystem.
|
Package host provides ways to access the host filesystem. |
|
utils/netns
Package netns is a small wrapper around github.com/vishvananda/netns that provides GetFromPidWithAltProcfs() and GetFromThreadWithAltProcfs().
|
Package netns is a small wrapper around github.com/vishvananda/netns that provides GetFromPidWithAltProcfs() and GetFromThreadWithAltProcfs(). |
|
tools
|
|
|
dnstester
Module
|
|
|
testjson2md
Module
|
Click to show internal directories.
Click to hide internal directories.