mtree

package module
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 16, 2016 License: BSD-3-Clause Imports: 24 Imported by: 0

README

go-mtree

mtree is a filesystem hierarchy validation tooling and format. This is a library and simple cli tool for mtree(8) support.

While the traditional mtree cli utility is primarily on BSDs (FreeBSD, openBSD, etc), even broader support for the mtree specification format is provided with libarchive (libarchive-formats(5)).

There is also an mtree port for Linux though it is not widely packaged for Linux distributions.

Format

The format of hierarchy specification is consistent with the # mtree v2.0 format. Both the BSD mtree and libarchive ought to be interoperable with it with only one definite caveat. On Linux, extended attributes (xattr) on files are often a critical aspect of the file, holding ACLs, capabilities, etc. While FreeBSD filesystem do support extattr, this feature has not made its way into their mtree.

This implementation of mtree supports a few non-upstream "keyword"s, such as: xattr and tar_time. If you include these keywords, the FreeBSD mtree will fail, as they are unknown keywords to that implementation.

To have go-mtree produce specifications that will be strictly compatible with the BSD mtree, use the -bsd-keywords flag when creating a manifest. This will make sure that only the keywords supported by BSD mtree are used in the program.

Typical form

With the standard keywords, plus say sha256digest, the hierarchy specification looks like:

# .
/set type=file nlink=1 mode=0664 uid=1000 gid=100
. size=4096 type=dir mode=0755 nlink=6 time=1459370393.273231538
    LICENSE size=1502 mode=0644 time=1458851690.0 sha256digest=ef4e53d83096be56dc38dbf9bc8ba9e3068bec1ec37c179033d1e8f99a1c2a95
    README.md size=2820 mode=0644 time=1459370256.316148361 sha256digest=d9b955134d99f84b17c0a711ce507515cc93cd7080a9dcd50400e3d993d876ac

[...]

See the directory presently in, and the files present. Along with each path, is provided the keywords and the unique values for each path. Any common keyword and values are established in the /set command.

Extended attributes form
# .
/set type=file nlink=1 mode=0664 uid=1000 gid=1000
. size=4096 type=dir mode=0775 nlink=6 time=1459370191.11179595 xattr.security.selinux=dW5jb25maW5lZF91Om9iamVjdF9yOnVzZXJfaG9tZV90OnMwAA==
    LICENSE size=1502 time=1458851690.583562292 xattr.security.selinux=dW5jb25maW5lZF91Om9iamVjdF9yOnVzZXJfaG9tZV90OnMwAA==
    README.md size=2366 mode=0644 time=1459369604.0 xattr.security.selinux=dW5jb25maW5lZF91Om9iamVjdF9yOnVzZXJfaG9tZV90OnMwAA==

[...]

See the keyword prefixed with xattr. followed by the extended attribute's namespace and keyword. This setup is consistent for use with Linux extended attributes as well as FreeBSD extended attributes.

Since extended attributes are an unordered hashmap, this approach allows for checking each <namespace>.<key> individually.

The value is the base64 encoded of the value of the particular extended attribute. Since the values themselves could be raw bytes, this approach avoids issues with encoding.

Tar form
# .
/set type=file mode=0664 uid=1000 gid=1000
. type=dir mode=0775 tar_time=1468430408.000000000

# samedir
samedir type=dir mode=0775 tar_time=1468000972.000000000
    file2 size=0 tar_time=1467999782.000000000
    file1 size=0 tar_time=1467999781.000000000
    
[...]

While go-mtree serves mainly as a library for upstream mtree support, go-mtree is also compatible with tar archives (which is not an upstream feature). This means that we can now create and validate a manifest by specifying a tar file. More interestingly, this also means that we can create a manifest from an archive, and then validate this manifest against a filesystem hierarchy that's on disk, and vice versa.

Notice that for the output of creating a validation manifest from a tar file, the default behavior for evaluating a notion of time is to use the tar_time keyword. In the "filesystem hierarchy" format of mtree, time is being evaluated with nanosecond precision. However, GNU tar truncates a file's modification time to 1-second precision. That is, if a file's full modification time is 123456789.123456789, the "tar time" equivalent would be 123456789.000000000. This way, if you validate a manifest created using a tar file against an actual root directory, there will be no complaints from go-mtree so long as the 1-second precision time of a file in the root directory is the same.

Usage

To use the Go programming language library, see the docs.

To use the command line tool, first build it, then the following.

Create a manifest

This will also include the sha512 digest of the files.

gomtree -c -K sha512digest -p . > /tmp/root.mtree

With a tar file:

gomtree -c -K sha512digest -T sometarfile.tar > /tmp/tar.mtree
Validate a manifest
gomtree -p . -f /tmp/root.mtree

With a tar file:

gomtree -T sometarfile.tar -f /tmp/root.mtree
See the supported keywords
gomtree -list-keywords
Available keywords:
 uname
 sha1
 sha1digest
 sha256digest
 xattrs (not upstream)
 link (default)
 nlink (default)
 md5digest
 rmd160digest
 mode (default)
 cksum
 md5
 rmd160
 type (default)
 time (default)
 uid (default)
 gid (default)
 sha256
 sha384
 sha512
 xattr (not upstream)
 tar_time (not upstream)
 size (default)
 ripemd160digest
 sha384digest
 sha512digest

Building

Either:

go get github.com/vbatts/go-mtree/cmd/gomtree

or

git clone git://github.com/vbatts/go-mtree.git
cd ./go-mtree/cmd/gomtree
go build .

Documentation

Index

Examples

Constants

View Source 
const (
	// VersionMajor is for an API incompatible changes
	VersionMajor = 0
	// VersionMinor is for functionality in a backwards-compatible manner
	VersionMinor = 2
	// VersionPatch is for backwards-compatible bug fixes
	VersionPatch = 0

	// VersionDev indicates development branch. Releases will be empty string.
	VersionDev = ""
)

Variables

View Source 
var (
	// DefaultKeywords has the several default keyword producers (uid, gid,
	// mode, nlink, type, size, mtime)
	DefaultKeywords = []string{
		"size",
		"type",
		"uid",
		"gid",
		"mode",
		"link",
		"nlink",
		"time",
	}

	// DefaultTarKeywords has keywords that should be used when creating a manifest from
	// an archive. Currently, evaluating the # of hardlinks has not been implemented yet
	DefaultTarKeywords = []string{
		"size",
		"type",
		"uid",
		"gid",
		"mode",
		"link",
		"tar_time",
	}

	// BsdKeywords is the set of keywords that is only in the upstream FreeBSD mtree
	BsdKeywords = []string{
		"cksum",
		"device",
		"flags",
		"ignore",
		"gid",
		"gname",
		"link",
		"md5",
		"md5digest",
		"mode",
		"nlink",
		"nochange",
		"optional",
		"ripemd160digest",
		"rmd160",
		"rmd160digest",
		"sha1",
		"sha1digest",
		"sha256",
		"sha256digest",
		"sha384",
		"sha384digest",
		"sha512",
		"sha512digest",
		"size",
		"tags",
		"time",
		"type",
		"uid",
		"uname",
	}

	// SetKeywords is the default set of keywords calculated for a `/set` SpecialType
	SetKeywords = []string{
		"uid",
		"gid",
	}
	// KeywordFuncs is the map of all keywords (and the functions to produce them)
	KeywordFuncs = map[string]KeywordFunc{
		"size":            sizeKeywordFunc,
		"type":            typeKeywordFunc,
		"time":            timeKeywordFunc,
		"link":            linkKeywordFunc,
		"uid":             uidKeywordFunc,
		"gid":             gidKeywordFunc,
		"nlink":           nlinkKeywordFunc,
		"uname":           unameKeywordFunc,
		"mode":            modeKeywordFunc,
		"cksum":           cksumKeywordFunc,
		"md5":             hasherKeywordFunc("md5digest", md5.New),
		"md5digest":       hasherKeywordFunc("md5digest", md5.New),
		"rmd160":          hasherKeywordFunc("ripemd160digest", ripemd160.New),
		"rmd160digest":    hasherKeywordFunc("ripemd160digest", ripemd160.New),
		"ripemd160digest": hasherKeywordFunc("ripemd160digest", ripemd160.New),
		"sha1":            hasherKeywordFunc("sha1digest", sha1.New),
		"sha1digest":      hasherKeywordFunc("sha1digest", sha1.New),
		"sha256":          hasherKeywordFunc("sha256digest", sha256.New),
		"sha256digest":    hasherKeywordFunc("sha256digest", sha256.New),
		"sha384":          hasherKeywordFunc("sha384digest", sha512.New384),
		"sha384digest":    hasherKeywordFunc("sha384digest", sha512.New384),
		"sha512":          hasherKeywordFunc("sha512digest", sha512.New),
		"sha512digest":    hasherKeywordFunc("sha512digest", sha512.New),

		"tar_time": tartimeKeywordFunc,

		"xattr":  xattrKeywordFunc,
		"xattrs": xattrKeywordFunc,
	}
)
View Source 
var DebugOutput = os.Stderr

DebugOutput is the where DEBUG output is written

View Source 
var Version = fmt.Sprintf("%d.%d.%d%s", VersionMajor, VersionMinor, VersionPatch, VersionDev)

Version is the specification version that the package types support.

Functions

func CollectUsedKeywords 

func CollectUsedKeywords(dh *DirectoryHierarchy) []string

CollectUsedKeywords collects and returns all the keywords used in a a DirectoryHierarchy

func Debugf 

func Debugf(format string, a ...interface{}) (n int, err error)

Debugf does formatted output to DebugOutput, only if DEBUG environment variable is set

func Unvis 

func Unvis(src string) (string, error)

Unvis is a wrapper for the C implementation of unvis, which decodes a string that potentially has characters that are encoded with Vis

func Vis 

func Vis(src string) (string, error)

Vis is a wrapper of the C implementation of the function vis, which encodes a character with a particular format/style

Types

type DirectoryHierarchy 

type DirectoryHierarchy struct {
	Entries []Entry
}

DirectoryHierarchy is the mapped structure for an mtree directory hierarchy spec

func ParseSpec 

func ParseSpec(r io.Reader) (*DirectoryHierarchy, error)

ParseSpec reads a stream of an mtree specification, and returns the DirectoryHierarchy

func Walk 

func Walk(root string, exlcudes []ExcludeFunc, keywords []string) (*DirectoryHierarchy, error)

Walk from root directory and assemble the DirectoryHierarchy. excludes provided are used to skip paths. keywords are the set to collect from the walked paths. The recommended default list is DefaultKeywords.

func (DirectoryHierarchy) WriteTo 

func (dh DirectoryHierarchy) WriteTo(w io.Writer) (n int64, err error)

WriteTo simplifies the output of the resulting hierarchy spec

type Entry 

type Entry struct {
	Parent     *Entry   // up
	Children   []*Entry // down
	Prev, Next *Entry   // left, right
	Set        *Entry   // current `/set` for additional keywords
	Pos        int      // order in the spec
	Raw        string   // file or directory name
	Name       string   // file or directory name
	Keywords   []string // TODO(vbatts) maybe a keyword typed set of values?
	Type       EntryType
}

Entry is each component of content in the mtree spec file

func (Entry) Ascend 

func (e Entry) Ascend() *Entry

Ascend gets the parent of an Entry. Serves mainly to maintain readability when traversing up and down an Entry tree

func (Entry) Descend 

func (e Entry) Descend(filename string) *Entry

Descend searches thru an Entry's children to find the Entry associated with `filename`. Directories are stored at the end of an Entry's children so do a traverse backwards. If you descend to a "."

func (Entry) Find 

func (e Entry) Find(filepath string) *Entry

Find is a wrapper around Descend that takes in a whole string path and tries to find that Entry

func (Entry) Path 

func (e Entry) Path() (string, error)

Path provides the full path of the file, despite RelativeType or FullType. It will be in Unvis'd form.

func (Entry) String 

func (e Entry) String() string

String joins a file with its associated keywords. The file name will be the Vis'd encoded version so that it can be parsed appropriately when Check'd.

type EntryType 

type EntryType int

EntryType are the formats of lines in an mtree spec file 

const (
	SignatureType EntryType = iota // first line of the file, like `#mtree v2.0`
	BlankType                      // blank lines are ignored
	CommentType                    // Lines beginning with `#` are ignored
	SpecialType                    // line that has `/` prefix issue a "special" command (currently only /set and /unset)
	RelativeType                   // if the first white-space delimited word does not have a '/' in it. Options/keywords are applied.
	DotDotType                     // .. - A relative path step. keywords/options are ignored
	FullType                       // if the first word on the line has a `/` after the first character, it interpretted as a file pathname with options
)

The types of lines to be found in an mtree spec file

func (EntryType) String 

func (et EntryType) String() string

String returns the name of the EntryType

type ExcludeFunc 

type ExcludeFunc func(path string, info os.FileInfo) bool

ExcludeFunc is the type of function called on each path walked to determine whether to be excluded from the assembled DirectoryHierarchy. If the func returns true, then the path is not included in the spec.

type Failure 

type Failure struct {
	Path     string `json:"path"`
	Keyword  string `json:"keyword"`
	Expected string `json:"expected"`
	Got      string `json:"got"`
}

Failure of a particular keyword for a path

func (Failure) String 

func (f Failure) String() string

String returns a "pretty" formatting for a Failure

type KeyVal 

type KeyVal string

KeyVal is a "keyword=value"

func (KeyVal) ChangeValue 

func (kv KeyVal) ChangeValue(newval string) string

ChangeValue changes the value of a KeyVal

func (KeyVal) Keyword 

func (kv KeyVal) Keyword() string

Keyword is the mapping to the available keywords

func (KeyVal) KeywordSuffix 

func (kv KeyVal) KeywordSuffix() string

KeywordSuffix is really only used for xattr, as the keyword is a prefix to the xattr "namespace.key"

func (KeyVal) Value 

func (kv KeyVal) Value() string

Value is the data/value portion of "keyword=value"

type KeyVals 

type KeyVals []KeyVal

KeyVals is a list of KeyVal

func MergeSet 

func MergeSet(setKeyVals, entryKeyVals []string) KeyVals

MergeSet takes the current setKeyVals, and then applies the entryKeyVals such that the entry's values win. The union is returned.

func NewKeyVals 

func NewKeyVals(keyvals []string) KeyVals

NewKeyVals constructs a list of KeyVal from the list of strings, like "keyword=value"

func (KeyVals) Has 

func (kvs KeyVals) Has(keyword string) KeyVal

Has the "keyword" present in the list of KeyVal, and returns the corresponding KeyVal, else an empty string.

type Keyword 

type Keyword string

Keyword is the string name of a keyword, with some convenience functions for determining whether it is a default or bsd standard keyword.

func (Keyword) Bsd 

func (k Keyword) Bsd() bool

Bsd returns whether this keyword is in the upstream FreeBSD mtree(8)

func (Keyword) Default 

func (k Keyword) Default() bool

Default returns whether this keyword is in the default set of keywords

type KeywordFunc 

type KeywordFunc func(path string, info os.FileInfo, r io.Reader) (string, error)

KeywordFunc is the type of a function called on each file to be included in a DirectoryHierarchy, that will produce the string output of the keyword to be included for the file entry. Otherwise, empty string. io.Reader `r` is to the file stream for the file payload. While this function takes an io.Reader, the caller needs to reset it to the beginning for each new KeywordFunc

type Result 

type Result struct {
	// list of any failures in the Check
	Failures []Failure `json:"failures"`
	Missing  []Entry   `json:"missing,omitempty"`
	Extra    []Entry   `json:"extra,omitempty"`
}

Result of a Check

func Check 

func Check(root string, dh *DirectoryHierarchy, keywords []string) (*Result, error)

Check a root directory path against the DirectoryHierarchy, regarding only the available keywords from the list and each entry in the hierarchy. If keywords is nil, the check all present in the DirectoryHierarchy

Example 
dh, err := Walk(".", nil, append(DefaultKeywords, "sha1"))
if err != nil {
	// handle error ...
}

res, err := Check(".", dh, nil)
if err != nil {
	// handle error ...
}
if len(res.Failures) > 0 {
	// handle failed validity ...
}

Output:

func TarCheck 

func TarCheck(tarDH, dh *DirectoryHierarchy, keywords []string) (*Result, error)

TarCheck is the tar equivalent of checking a file hierarchy spec against a tar stream to determine if files have been changed.

type Streamer 

type Streamer interface {
	io.ReadCloser
	Hierarchy() (*DirectoryHierarchy, error)
}

Streamer creates a file hierarchy out of a tar stream

Example 
fh, err := os.Open("./testdata/test.tar")
if err != nil {
	// handle error ...
}
str := NewTarStreamer(fh, nil)
if err := extractTar("/tmp/dir", str); err != nil {
	// handle error ...
}

dh, err := str.Hierarchy()
if err != nil {
	// handle error ...
}

res, err := Check("/tmp/dir/", dh, nil)
if err != nil {
	// handle error ...
}
if len(res.Failures) > 0 {
	// handle validation issue ...
}

Output:

func NewTarStreamer 

func NewTarStreamer(r io.Reader, keywords []string) Streamer

NewTarStreamer streams a tar archive and creates a file hierarchy based off of the tar metadata headers

Source Files

View all Source files

Directories

Path Synopsis
cmd
gomtree

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.