rcoredump
rcoredump is meant to be a toolbox for aggregating, indexing, and searching
core dumps. Think ELK for core dumps.
rcoredump is composed of two main softwares:
rcoredumpd
is the indexer. It store, analyze, and index the core
dumps and the associated binaries, and must be installed on the destination
server.
rcoredump
is the forwarder. It collect core dumps and the associated
binaries, and must be installed on the source servers.
Status: Alpha
The project is in its infancy stage. While it can be used already for some
usages, many things may change between now and the v1, and no feature is
guaranteed to stay the same or even be present in future releases. Of course,
we will try to stay compatible as much as possible, but if we can't, you've
been warned.
Installation
The server requires both gdb and
delve to be installed to be able to
extract the stack traces.
From releases
The simplest way to install rcoredump is to use the repository's release
page. The binaries are
currently built for Linux only, in 32 and 64 bits architectures. New systems
and architectures can be added as long as they are supported by Golang. Feel
free to open an issue if you would like to see binaries for your platform.
Note The current list can be listed using the command go tool dist list
with any go installation.
From source
Installing from source require only a working Golang installation. Your system
probably provides a Golang package, you can also read the Getting
Started page on Golang's website. It is
recommended to use the very last version of Golang, but any module-enabled
version should do.
Then, you can either run the following commands to get it using golang's
package manager:
go get github.com/elwinar/rcoredump/bin/rcoredumpd
go get github.com/elwinar/rcoredump/bin/rcoredump
Or clone the repository and build it by hand:
git clone https://github.com/elwinar/rcoredump
cd rcoredump
go build ./bin/rcoredumpd
go build ./bin/rcoredump
Configuration
Both binaries can be configured using either the CLI or a configuration file
with an ini-like syntax (see rcoredump.conf.example
and rcoredumpd.conf.example). Both usage message
can be printed on screen using the -help
flag.
Note The flag syntax follow the Golang's standard package convention,
described here.
The same convention is applied for the configuration files, minus the dashes
before the flag names.
rcoredumpd
Usage of rcoredumpd: rcoredumpd [options]
-bind string
address to listen to (default "localhost:1105")
-c.analyzer string
gdb command to run to generate the stack trace for C coredumps (default "bt")
-conf string
configuration file to load (default "/etc/rcoredump/rcoredumpd.conf")
-data-dir string
directory to store server's data (default "/var/lib/rcoredumpd")
-filelog string
path of the file to log into ("-" for stdout) (default "-")
-go.analyzer string
delve command to run to generate the stack trace for Go coredumps (default "bt")
-index-type string
type of index to use (values: bleve) (default "bleve")
-retention-duration duration
duration to keep an indexed coredump (e.g: "168h"), 0 to disable
-size-buckets string
buckets report the coredump sizes for (default "1MB,10MB,100MB,1GB,10GB")
-store-type string
type of store to use (values: file) (default "file")
-syslog
output logs to syslog
-version
print the version of rcoredumpd
rcoredump
Usage of rcoredump: rcoredump [options] <executable path> <timestamp of dump>
-conf string
configuration file to load (default "/etc/rcoredump/rcoredump.conf")
-dest string
address of the destination host (default "http://localhost:1105")
-filelog string
path of the file to log into ("-" for stdout) (default "-")
-metadata value
list of metadata to send alongside the coredump (key=value, can be specified multiple times or separated by ';')
-src string
path of the coredump to send to the host ("-" for stdin) (default "-")
-syslog
output logs to syslog
-version
print the version of rcoredump
By default, the forwarder expect the core dump to be passed on the standard
input. You can use sysctl's kernel.core_pattern
tunable (on linux) to have
the kernel invoke rcoredump everytime a dump is generated. For example:
kernel.core_pattern=|/path/to/rcoredump %E %t
.
Note No space between the |
and the binary's path. Also, no environment
variable, so no PATH
, you must use an absolute path here.
The forwarder can also be invoked by hand using the -src
flag and a file
path. This is mostly used for development and to test an installation.
Logging
By default, all logging is done on stdout using the logfmt format. For
convenience, both softwares also accept a -syslog
flag to log using syslog,
and a -filelog
flag to log to a file.
Storage
Right now, the indexer stores every core dump and binary on disk. As both of
them can be quite large, it is recommended to closely monitor the size of the
data directory (set by the server's -dir
flag), and the free space on the
disk.
If non-zero, the -retention-duration
flag of the server can be used to
automatically remove coredumps older than the value, eventually removing the
executable if it is not linked to another coredump.
Need Help?
Feel free to open an issue, or contact me by mail at
romain.baugue@elwinar.com
. I won't guarantee and instant solution, but I'm
always happy to help someone get things working.
Contributing
The project is open to contribution, but no guidelines or requirements are set
for now, as this is mostly a one-man project. Feel free to contact me if you
want to help, and we will define this part together.
License
The Unlicence