memr

README

memr: Linux memory reader

Overview

memr is inspired by AVML, with the added goal of extensibility.

• Provides a golang io.ReadCloser interface, as memr.Reader, over the memory source
• Callers can do whatever they please with the data that is read through the memr.Reader
  • write to a local file
  • stream to an S3 bucket
  • compress with a desired algorithm before writing, and so on

The major benefit of memr is that it supports streaming memory data, enabling writing off-host without first copying to the local disk.

Often times, particularly in cloud environments, hosts may have memory sizes that far exceed space available on disk. In these cases, it is infeasible to first write data locally.

Features

memr also adds some additional features:

• Custom page headers can be provided using memr.PageHeaderProviderFunc
  • By default page headers will be written in the LiME format
• Custom handling of page data using memr.PageWriterFunc
  • This is meant to replicate AVML's custom format, or (version 2 by AVML's specification), where page-level compression is performed with snappy. However, in my opinion, this should be avoided and compression should be done at the stream level, not the page level (see the compression example for more on this approach).
• Debug logging using memr.SetLogLevel(memr.LogDebug) (or -vvv using the provided CLI)
• Progress reporting

Examples

See here for various examples leveraging the features outlined above.

Install

With a proper Go installation, the package is installable using:

go get github.com/ryandeivert/memr

If running Linux, a sample CLI tool is included in this repo that can be installed using the below (requires go 1.16+). Note that this method should only be used on Linux systems, as it will compile a binary for your local system's GOOS and GOARCH:

go install github.com/ryandeivert/memr/cmd/memr@latest

Or download binaries directly from GitHub Releases.

CLI Usage

The memr CLI tool included in this repo supports a of couple use cases.

It supports writing to either a local file (with the --local-file flag) or an S3 bucket (with the --bucket/--key flag combination). Either method supports compression (the default), but can be disabled using --compression=false. Other basic sample CLIs are included in the examples directory.

Usage:
  memr [flags]

Examples:

Writing to local file:
memr --local-file <FILE>

Streaming directly to S3 bucket:
memr --bucket <BUCKET> --key <KEY>

Targeting a specific device:
memr /dev/mem --local-file <FILE>

Skipping compression:
memr --compress=false  --local-file <FILE>

Flags:
  -b, --bucket string       S3 bucket to which output should be sent
  -c, --compress            compress the output with snappy (default true)
  -t, --concurrency int     number of threads to use for S3 upload (default 5)
  -h, --help                help for memr
  -k, --key string          key to use for uploading to S3 bucket
  -f, --local-file string   local file to write to, instead of S3
  -p, --progress            show progress (default true)
  -r, --region string       AWS region to use with S3 client (default "us-east-1")
  -v, --verbose count       enable verbose logging
      --version             version for memr

Quick Start API Example

package main

import (
	"io"
	"log"
	"os"

	"github.com/ryandeivert/memr"
)

func main() {
	// Use memr.Probe() to enumerate all available types, attempting to find a valid reader
	// Alternatively use memr.NewReader(memr.SourceKcore) to target a specific type,
	// in this case /proc/kcore
	reader, err := memr.Probe()
	if err != nil {
		log.Fatalf("failed to load memory reader: %s", err)
	}
	defer reader.Close()

	// Open the local file for writing
	localFile := "output.mem"
	writer, err := os.Create(localFile)
	if err != nil {
		log.Fatalf("failed to open local file %q for writing %s", localFile, err)
	}
	defer writer.Close()

	// Use io.Copy to copy the memory to a file
	_, err = io.Copy(writer, reader)
	if err != nil {
		log.Fatalf("failed to copy memory to local file %q: %s", localFile, err)
	}
}

Documentation

Index

• func HeaderLime(start, end uint64) interface{}
• func SetLogLevel(lvl LogLvl)
• type DefaultHeader
• type LogLvl
  • func (s LogLvl) String() string
• type MemSource
• type PageHeaderProviderFunc
• type PageWriterFunc
• type Reader
  • func NewReader(source MemSource, options ...func(*Reader)) (reader *Reader, err error)
  • func Probe(options ...func(*Reader)) (*Reader, error)
  • func (r *Reader) Close() error
  • func (r *Reader) Read(p []byte) (int, error)
  • func (r *Reader) Reset() (err error)
  • func (r *Reader) Size() uint64
  • func (r *Reader) Source() MemSource

Functions

func HeaderLime

func HeaderLime(start, end uint64) interface{}

HeaderLime is a PageHeaderProviderFunc that returns a DefaultHeader struct

func SetLogLevel

func SetLogLevel(lvl LogLvl)

SetupLogging sets logging level on the logger using the specified LogLvl value

Types

type DefaultHeader

type DefaultHeader struct {
	Magic     uint32 // magic (0x4C694D45 for LiME)
	Version   uint32 // version (always 1 for LiME)
	StartAddr uint64 // start address
	EndAddr   uint64 // end address
	// contains filtered or unexported fields
}

DefaultHeader is the default header struct value. This header can be used in conjunction with a PageHeaderProviderFunc to provide a custom header format to be used by the reader

type LogLvl

type LogLvl int

const (
	LogUnknown LogLvl = iota
	LogWarn
	LogInfo
	LogDebug
)

func (LogLvl) String

func (s LogLvl) String() string

type MemSource

type MemSource string

MemSource type used for memory sources

const (
	// SourceKcore disignates /proc/kcore as the memory source
	SourceKcore MemSource = "/proc/kcore"
	// SourceCrash disignates /dev/crash as the memory source
	SourceCrash MemSource = "/dev/crash"
	// SourceMem disignates /dev/mem as the memory source
	SourceMem MemSource = "/dev/mem"
)

type PageHeaderProviderFunc

type PageHeaderProviderFunc func(start, end uint64) interface{}

PageHeaderProviderFunc should be used to provide a page header

type PageWriterFunc

type PageWriterFunc func(io.Writer) io.WriteCloser

PageWriterFunc can be used to add special handling of page contents, such as page-level compression. This applies to a Reader in-line as data is read. Note: use of this should nearly always be avoided.

type Reader

type Reader struct {
	// PageHeaderProvider is a function that returns a custom header structure to be applied
	// prepended before each memory "page" that is read
	PageHeaderProvider PageHeaderProviderFunc

	// PageHandler allows for page-level compression, or other custom handling of pages
	// Note: this should really not be used, and exists only for compatibility with other
	// tools. In most cases, the returned io.Reader should be used for compression instead
	PageHandler PageWriterFunc

	// WithProgress should be set to false if progress should not be reported during
	// the reading of memory. The default when calling NewReader is true.
	WithProgress bool

	// ByteOrder should be either binary.LittleEndian or binary.BigEndian.
	// This determines the byte order applied to pager headers. The default
	// when calling NewReader is binary.LittleEndian.
	ByteOrder binary.ByteOrder
	// contains filtered or unexported fields
}

Reader implements the io.ReadCloser interface Its values can also used as "options" for the resulting Reader returned by the Probe() and NewReader() functions.

func NewReader

func NewReader(source MemSource, options ...func(*Reader)) (reader *Reader, err error)

NewReader tries to open the specified memory source for reading. Source should be one of: SourceKcore (/proc/kcore), SourceCrash (/dev/crash), or SourceMem (/dev/mem). Optional options can be provided for the resulting Reader.

Example:

// Omit headers from the resulting image (raw) and suppress progress
reader, err := memr.NewReader(memr.SourceKcore, func(m *memr.Reader) {
	m.WithProgress = false
	m.PageHeaderProvider = nil
})

func Probe

func Probe(options ...func(*Reader)) (*Reader, error)

Probe enumerates all available memory sources and returns the first valid reader. Current sources are: /proc/kcore, /dev/crash, /dev/mem. Optional options can be provided for the resulting Reader. See NewReader for usage of custom options.

func (*Reader) Close

func (r *Reader) Close() error

Close satisfies the io.Closer interface. This should be called after all reading is complete to close the underlying input file. It also signals the progress bar to flush its output. Without calling this, you will likely see incorrect progress output upon completion.

func (*Reader) Read

func (r *Reader) Read(p []byte) (int, error)

Read satisfies the io.Reader interface.

func (*Reader) Reset

func (r *Reader) Reset() (err error)

func (*Reader) Size

func (r *Reader) Size() uint64

Size returns the expected size of the memory to be read by the reader

func (*Reader) Source

func (r *Reader) Source() MemSource

Source returns the MemSource for this reader (one of: /proc/kcore, /dev/crash, /dev/mem)