README
¶
Linux Observability Primitives
Very grand title for a humble start. This repository contains my experiments with low level Linux observability, in Go.
If you are interested in the domain, you should probably check out bcc and gobpf. They support eBPF. I may integrate eBPF support at some point, either through gobpf of directly, but that's a whole different story.
Tracepoints
Tracepoints are static probes placed at critical points in the Linux kernel. There are some tracepoints for many interesting events:
- Scheduler: when does a process starts & exits, context switches.
- I/O: follow block requests.
- Filesystem: intimate details about filesystem operations (ext4, btrfs, ... are instrumented).
- KVM: watch what KVM does, eg. VM exits.
- Drivers: eg. on can follow Intel GPU execution requests and display events (i915 driver).
- Syscall entry/exit.
- Many other events and Various drivers.
To see the full list of tracepoints:
$ sudo perf list tracepoint
[...]
sched:sched_kthread_stop [Tracepoint event]
sched:sched_kthread_stop_ret [Tracepoint event]
sched:sched_migrate_task [Tracepoint event]
sched:sched_move_numa [Tracepoint event]
sched:sched_process_exec [Tracepoint event]
sched:sched_process_exit [Tracepoint event]
sched:sched_process_fork [Tracepoint event]
sched:sched_process_free [Tracepoint event]
sched:sched_process_hang [Tracepoint event]
sched:sched_process_wait [Tracepoint event]
[...]
Example
// Listen for the sched_process_exec, triggered when a process does an exec(),
// and display the process pid and the path of binary being exec'ed. Error
// handling is omitted.
observer := obs.NewObserver()
observer.AddTracepoint("sched:sched_process_exec")
observer.Open()
for {
event, _ := observer.ReadEvent()
tp := event.(*obs.TracepointEvent)
fmt.Printf("exec\t%d\t%s\n", tp.GetInt("pid"), tp.GetString("filename"))
}
The output below is the result of running docker run busybox sleep 3600 and
shows the (surprisingly big) list programs that end up being executed up to
runc and the sleep process in the container.
exec 7364 /usr/bin/docker
exec 7370 /sbin/auplink
exec 7371 /sbin/auplink
exec 7378 /lib/udev/ifupdown-hotplug
exec 7380 /bin/grep
exec 7381 /sbin/ifquery
exec 7379 /sbin/ifquery
exec 7382 /lib/udev/ifupdown-hotplug
exec 7383 /lib/systemd/systemd-sysctl
exec 7385 /bin/grep
exec 7386 /sbin/ifquery
exec 7384 /sbin/ifquery
exec 7387 /lib/systemd/systemd-sysctl
exec 7389 /usr/bin/docker-containerd-shim
exec 7397 /usr/bin/docker-runc
exec 7404 /proc/self/exe
exec 7413 /usr/bin/dockerd
exec 7426 /proc/self/exe
exec 7433 /lib/udev/ifupdown-hotplug
exec 7434 /usr/bin/docker-runc
exec 7407 /bin/sleep
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type NamespaceKind ¶
type NamespaceKind int
NamespaceKind is the type of Linux namespaces.
const ( // NetworkNS is the network namespace. NetworkNS NamespaceKind = iota // IPCNS is the SysV IPC namespace. IPCNS // UTSNS is the UTS namespace. UTSNS // MountNS is the mount namespace. MountNS // PIDNS is the PID namespace. PIDNS // UserNS is the user namespace. UserNS // CgroupNS is the cgroup namespace. CgroupNS )
type Observer ¶
type Observer struct {
// contains filtered or unexported fields
}
Observer is the object that will observe the system. An observer is first configured to listen to various events such as tracepoints or kprobes being hit. Then events can be read from the observer when they occur.
func (*Observer) AddTracepoint ¶
func (o *Observer) AddTracepoint(name string) EventSource
AddTracepoint adds a tracepoint to watch for.
func (*Observer) Close ¶
func (o *Observer) Close()
Close frees precious resources acquired during Open.
type Process ¶
type Process struct {
// contains filtered or unexported fields
}
Process provides mechanisms to retrieve information about a process.
type TracepointEvent ¶
type TracepointEvent struct {
// contains filtered or unexported fields
}
TracepointEvent is fired when a Tracepoint is hit.
func (*TracepointEvent) Data ¶
func (e *TracepointEvent) Data() []byte
Data returns the tracepoint raw data.
func (*TracepointEvent) GetInt ¶
func (e *TracepointEvent) GetInt(name string) int
GetInt retrieves an integer corresponding to the field named 'name' from the tracepoint data. If 'name' isn't a valid field name, GetInt returns -1.
One can consult the list of per-event fields in its format file:
# cat /sys/kernel/debug/tracing/events/sched/sched_process_exec/format name: sched_process_exec ID: 266 format: field:unsigned short common_type; offset:0; size:2; signed:0; field:unsigned char common_flags; offset:2; size:1; signed:0; field:unsigned char common_preempt_count; offset:3; size:1; signed:0; field:int common_pid; offset:4; size:4; signed:1; field:__data_loc char[] filename; offset:8; size:4; signed:1; field:pid_t pid; offset:12; size:4; signed:1; field:pid_t old_pid; offset:16; size:4; signed:1;
func (TracepointEvent) GetSource ¶
func (e TracepointEvent) GetSource() EventSource
GetSource returns the source ID of the object that has emitted e.
func (*TracepointEvent) GetString ¶
func (e *TracepointEvent) GetString(name string) string
GetString retrieves an integer corresponding to the field named 'name' from the tracepoint data. If 'name' isn't a valid field name, GetString returns "".
Source Files
Directories