squashfs

package
v1.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 9, 2024 License: MIT Imports: 21 Imported by: 5

Documentation

Overview

Package squashfs provides support for reading and creating squashfs filesystems references: 

https://www.kernel.org/doc/Documentation/filesystems/squashfs.txt
https://dr-emann.github.io/squashfs/
https://elinux.org/images/3/32/Squashfs-elce.pdf

Index

Constants

View Source 
const (
	// KB represents one KB
	KB int64 = 1024
	// MB represents one MB
	MB int64 = 1024 * KB
	// GB represents one GB
	GB int64 = 1024 * MB
	// TB represents one TB
	TB int64 = 1024 * GB
)

Variables

This section is empty.

Functions

This section is empty.

Types

type Compressor 

type Compressor interface {
	// contains filtered or unexported methods
}

Compressor defines a compressor. Fulfilled by various implementations in this package

type CompressorGzip 

type CompressorGzip struct {
	CompressionLevel uint32
	WindowSize       uint16
	Strategies       map[GzipStrategy]bool
}

CompressorGzip gzip compression

type CompressorLz4 

type CompressorLz4 struct {
	// contains filtered or unexported fields
}

CompressorLz4 lz4 compression

type CompressorLzma 

type CompressorLzma struct {
}

CompressorLzma lzma compression

type CompressorXz 

type CompressorXz struct {
	DictionarySize    uint32
	ExecutableFilters map[XzFilter]bool
}

CompressorXz xz compression

type CompressorZstd 

type CompressorZstd struct {
	// contains filtered or unexported fields
}

CompressorZstd zstd compression

type File 

type File struct {
	// contains filtered or unexported fields
}

File represents a single file in a squashfs filesystem 

it is NOT used when working in a workspace, where we just use the underlying OS
note that the inode for a file can be the basicFile or extendedFile. We just use extendedFile to
include all of the data

func (*File) Close 

func (fl *File) Close() error

Close close the file

func (*File) Read 

func (fl *File) Read(b []byte) (int, error)

Read reads up to len(b) bytes from the File. It returns the number of bytes read and any error encountered. At end of file, Read returns 0, io.EOF reads from the last known offset in the file from last read or write use Seek() to set at a particular point

func (*File) Seek 

func (fl *File) Seek(offset int64, whence int) (int64, error)

Seek set the offset to a particular point in the file

func (*File) Write 

func (fl *File) Write(p []byte) (int, error)

Write writes len(b) bytes to the File. 

you cannot write to a finished squashfs, so this returns an error

type FileStat 

type FileStat = *directoryEntry

FileStat is the extended data underlying a single file, similar to https://golang.org/pkg/syscall/#Stat_t

type FileSystem 

type FileSystem struct {
	// contains filtered or unexported fields
}

FileSystem implements the FileSystem interface

func Create 

func Create(b backend.Storage, size, start, blocksize int64) (*FileSystem, error)

Create creates a squashfs filesystem in a given directory

requires the backend.Storage where to create the filesystem, size is the size of the filesystem in bytes, start is how far in bytes from the beginning of the backend.Storage to create the filesystem, and blocksize is is the logical blocksize to use for creating the filesystem

note that you are *not* required to create the filesystem on the entire disk. You could have a disk of size 20GB, and create a small filesystem of size 50MB that begins 2GB into the disk. This is extremely useful for creating filesystems on disk partitions.

Note, however, that it is much easier to do this using the higher-level APIs at github.com/diskfs/go-diskfs which allow you to work directly with partitions, rather than having to calculate (and hopefully not make any errors) where a partition starts and ends.

If the provided blocksize is 0, it will use the default of 128 KB.

func Read 

func Read(b backend.Storage, size, start, blocksize int64) (*FileSystem, error)

Read reads a filesystem from a given disk.

requires the backend.Storage where to read the filesystem, size is the size of the filesystem in bytes, start is how far in bytes from the beginning of the backend.Storage the filesystem is expected to begin, and blocksize is is the logical blocksize to use for creating the filesystem

note that you are *not* required to read a filesystem on the entire disk. You could have a disk of size 20GB, and a small filesystem of size 50MB that begins 2GB into the disk. This is extremely useful for working with filesystems on disk partitions.

Note, however, that it is much easier to do this using the higher-level APIs at github.com/diskfs/go-diskfs which allow you to work directly with partitions, rather than having to calculate (and hopefully not make any errors) where a partition starts and ends.

If the provided blocksize is 0, it will use the default of 2K bytes.

This will use a cache for the decompressed blocks of 128 MB by default. (You can set this with the SetCacheSize method and read its size with the GetCacheSize method). A block cache is essential for performance when reading. This implements a cache for the fragments (tail ends of files) and the metadata (directory listings) which otherwise would be read, decompressed and discarded many times.

Unpacking a 3 GB squashfs made from the tensorflow docker image like this: 

docker export $(docker create tensorflow/tensorflow:latest-gpu-jupyter) -o tensorflow.tar.gz
mkdir -p tensorflow && tar xf tensorflow.tar.gz -C tensorflow
[ -f tensorflow.sqfs ] && rm tensorflow.sqfs
mksquashfs tensorflow tensorflow.sqfs  -comp zstd -Xcompression-level 3 -b 1M -no-xattrs -all-root

Gives these timings with and without cache:

- no caching: 206s - 256 MB cache: 16.7s - 128 MB cache: 17.5s (the default) - 64 MB cache: 23.4s - 32 MB cache: 54.s

The cached versions compare favourably to the C program unsquashfs which takes 12.0s to unpack the same archive.

These tests were done using rclone and the archive backend which uses this library like this: 

rclone -P --transfers 16 --checkers 16 copy :archive:/path/to/tensorflow.sqfs /tmp/tensorflow

func (*FileSystem) Chmod added in v1.5.0 

func (fs *FileSystem) Chmod(name string, mode os.FileMode) error

Chmod changes the mode of the named file to mode. If the file is a symbolic link, it changes the mode of the link's target.

func (*FileSystem) Chown added in v1.5.0 

func (fs *FileSystem) Chown(name string, uid, gid int) error

Chown changes the numeric uid and gid of the named file. If the file is a symbolic link, it changes the uid and gid of the link's target. A uid or gid of -1 means to not change that value

func (*FileSystem) Equal 

func (fs *FileSystem) Equal(a *FileSystem) bool

Equal compare if two filesystems are equal

func (*FileSystem) Finalize 

func (fs *FileSystem) Finalize(options FinalizeOptions) error

Finalize finalize a read-only filesystem by writing it out to a read-only format

func (*FileSystem) GetCacheSize added in v1.4.1 

func (fs *FileSystem) GetCacheSize() int

GetCacheSize get the maximum memory used by the block cache in bytes.

func (*FileSystem) Label 

func (fs *FileSystem) Label() string

Label return the filesystem label 

func (fs *FileSystem) Link(oldpath, newpath string) error

creates a new link (also known as a hard link) to an existing file.

func (*FileSystem) Mkdir 

func (fs *FileSystem) Mkdir(p string) error

Mkdir make a directory at the given path. It is equivalent to `mkdir -p`, i.e. idempotent, in that:

* It will make the entire tree path if it does not exist * It will not return an error if the path already exists

if readonly and not in workspace, will return an error

func (*FileSystem) Mknod added in v1.5.0 

func (fs *FileSystem) Mknod(pathname string, mode uint32, dev int) error

creates a filesystem node (file, device special file, or named pipe) named pathname, with attributes specified by mode and dev

func (*FileSystem) OpenFile 

func (fs *FileSystem) OpenFile(p string, flag int) (filesystem.File, error)

OpenFile returns an io.ReadWriter from which you can read the contents of a file or write contents to the file

accepts normal os.OpenFile flags

returns an error if the file does not exist

func (*FileSystem) ReadDir 

func (fs *FileSystem) ReadDir(p string) ([]os.FileInfo, error)

ReadDir return the contents of a given directory in a given filesystem.

Returns a slice of os.FileInfo with all of the entries in the directory.

Will return an error if the directory does not exist or is a regular file and not a directory

func (*FileSystem) Remove added in v1.5.0 

func (fs *FileSystem) Remove(p string) error

func (*FileSystem) Rename added in v1.5.0 

func (fs *FileSystem) Rename(oldpath, newpath string) error

Rename renames (moves) oldpath to newpath. If newpath already exists and is not a directory, Rename replaces it.

func (*FileSystem) SetCacheSize added in v1.4.1 

func (fs *FileSystem) SetCacheSize(cacheSize int)

SetCacheSize set the maximum memory used by the block cache to cacheSize bytes.

The default is 128 MB.

If this is <= 0 then the cache will be disabled.

func (*FileSystem) SetLabel added in v1.3.0 

func (fs *FileSystem) SetLabel(string) error
func (fs *FileSystem) Symlink(oldpath, newpath string) error

creates a symbolic link named linkpath which contains the string target.

func (*FileSystem) Type 

func (fs *FileSystem) Type() filesystem.Type

Type returns the type code for the filesystem. Always returns filesystem.TypeFat32

func (*FileSystem) Workspace 

func (fs *FileSystem) Workspace() string

Workspace get the workspace path

type FinalizeOptions 

type FinalizeOptions struct {
	// Compressor which compressor to use, including, where relevant, options. Defaults ot CompressorGzip
	Compression Compressor
	// NonExportable prevent making filesystem NFS exportable. Defaults to false, i.e. make it exportable
	NonExportable bool
	// NonSparse prevent detecting sparse files. Defaults to false, i.e. detect sparse files
	NonSparse bool
	// Xattrs whether or not to store extended attributes. Defaults to false
	Xattrs bool
	// NoCompressInodes whether or not to compress inodes. Defaults to false, i.e. compress inodes
	NoCompressInodes bool
	// NoCompressData whether or not to compress data blocks. Defaults to false, i.e. compress data
	NoCompressData bool
	// NoCompressFragments whether or not to compress fragments. Defaults to false, i.e. compress fragments
	NoCompressFragments bool
	// NoCompressXattrs whether or not to compress extended attrbutes. Defaults to false, i.e. compress xattrs
	NoCompressXattrs bool
	// NoFragments do not use fragments, but rather dedicated data blocks for all files. Defaults to false, i.e. use fragments
	NoFragments bool
	// NoPad do not pad filesystem so it is a multiple of 4K. Defaults to false, i.e. pad it
	NoPad bool
	// FileUID set all files to be owned by the UID provided, default is to leave as in filesystem
	FileUID *uint32
	// FileGID set all files to be owned by the GID provided, default is to leave as in filesystem
	FileGID *uint32
}

FinalizeOptions options to pass to finalize

type GzipStrategy 

type GzipStrategy uint16
const (
	GzipDefault          GzipStrategy = 0x1
	GzipFiltered         GzipStrategy = 0x2
	GzipHuffman          GzipStrategy = 0x4
	GzipRunLengthEncoded GzipStrategy = 0x8
	GzipFixed            GzipStrategy = 0x10
)

gzip strategy options

type XzFilter 

type XzFilter uint32

XzFilter filter for xz compression 

const (
	XzFilterX86      XzFilter = 0x1
	XzFilterPowerPC  XzFilter = 0x2
	XzFilterIA64     XzFilter = 0x4
	XzFilterArm      XzFilter = 0x8
	XzFilterArmThumb XzFilter = 0x10
	XzFilterSparc    XzFilter = 0x20
)

xz filter options

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.