jsonfs

package module
v0.0.0-...-e9a6ec6 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 6, 2022 License: MIT Imports: 19 Imported by: 0

README

jsonfs - absolutely NOT production ready.

This was a thought experiment on a tool for doing JSON discovery where you don't have a schema.

It treats the JSON as a filesystem. Directories represents JSON dictionaries and arrays. Files represents JSON basic values.

These tie in with an in memory fs.FS filesystem that will allow you to use filesystem tools to work with the data. This seems nicer to use than a map[string]any or []any that you might otherwise deal with from the stdlib.

There is a diskfs that also implements fs.FS . I have not put it through its paces. The idea was to allow you to copy whatever you want from the memfs to the diskfs and then you could use all the filesystem tools that Linux gives you.

At this point, what I guarantee is that the Marshal() and Unmarshal() do work. The Unmarshal() is slower because we provide more structures, which means it takes longer.

Marshal() on the other hand is faster and much less memory intensive.

The statemachines for marshal and unmarshal are pretty good. At some point I may decouple those into their own library to allow building anyone to build JSON tooling.

Here are the benchmarks I referred to:

BenchmarkMarshalJSONSmall-10          	  256087	      4573 ns/op	       0 B/op	       0 allocs/op
BenchmarkMarshalJSONStdlibSmall-10    	  241816	      4981 ns/op	    2304 B/op	      51 allocs/op
BenchmarkMarshalJSONLarge-10          	       7	 143744732 ns/op	 9633560 B/op	   11255 allocs/op
BenchmarkMarshalJSONStdlibLarge-10    	       7	 143856131 ns/op	101807982 B/op	 1571144 allocs/op
BenchmarkUnmarshalSmall-10            	  134758	      8933 ns/op	    7281 B/op	     173 allocs/op
BenchmarkStdlibUnmarshalSmall-10    	  209151	      5903 ns/op	    2672 B/op	      66 allocs/op
BenchmarkUnmarshalLarge-10            	       4	 250373635 ns/op	240508578 B/op	 5336230 allocs/op
BenchmarkStdlibUnmarshalLarge-10    	       6	 187222674 ns/op	99413457 B/op	 1750078 allocs/op
BenchmarkObjectWalk-10                	       4	 284417094 ns/op	145693142 B/op	 3442589 allocs/op
BenchmarkStdlibObjectWalk-10          	      10	 104694233 ns/op	58166933 B/op	 1551677 allocs/op

Documentation

Overview

Package jsonfs provides a JSON marshal/unmarshaller that treats JSON objects as directories and JSON values (bools, numbers or strings) as files.

This is an alternative to various other structures for dealing with JSON that either use maps or structs to represent data. This is particularly great for doing discovery on JSON data or manipulating JSON.

Each file is read-only. You can switch out files in a directory in order to make updates.

A File represents a JSON basic value of bool, integer, float, null or string.

A Directory represents a JSON object or array. Because a directory can be an object or array, a directory that represents an array has _.array._ appened to the name in some filesystems. If creating an array using filesytem tools, use ArrayDirName("name_of_your_array"), which will append the correct suffix. You always opened the file with simply the name. This also means that naming an object (not array) _.array._ will cause unexpected results.

This is a thought experiment. However, it is quite performant, but does have the unattractive nature of being more verbose. Also, you may find problems if your JSON dict keys have invalid characters for the filesystem you are running on AND you use the diskfs filesystem. For the memfs, this is mostly not a problem, except for /. You cannot use / in your keys.

Benchmarks

We are slower and allocate more memory. I'm going to have to spend some time optimising the memory use. I had some previous benchmarks that showed this was faster. But I had a mistake that became obvious with using a large file, (unless I was 700,000x faster on unmarshal, and I'm not that good). 

BenchmarkUnmarshalSmall-10            	  132848	      8384 ns/op	   11185 B/op	     167 allocs/op
BenchmarkStandardUnmarshalSmall-10    	  215502	      5484 ns/op	    2672 B/op	      66 allocs/op
BenchmarkUnmarshalLarge-10            	       5	 227486309 ns/op	318321257 B/op	 4925295 allocs/op
BenchmarkStandardUnmarshalLarge-10    	       6	 185993493 ns/op	99390094 B/op	 1749996 allocs/op

Important notes

  • This doesn't support numbers larger than an Int64. If you need that, you need to use a string.
  • This doesn't support anything other than decimal notation, but the JSON standard does. If someone needs it I'll add it.
  • This does not have []byte conversion to string as the standard lib provides.
  • There are likely bugs in here.

Examples

Example of unmarshalling a JSON file: 

f, err := os.Open("some/file/path.json")
if err != nil {
	// Do something
}
dir, err := UnmarshalJSON(ctx, f)
if err != nil {
	// Do something
}

Example of creating a JSON object via the library: 

dir := MustNewDir(
	"",
	MustNewFile("First Name", "John"),
	MustNewFile("Last Name", "Doak"),
	MustNewDir(
		"Identities",
		MustNewFile("EmployeeID", 10),
		MustNewFile("SSNumber", "999-99-9999"),
	),
)

Example of marshaling a Directory: 

f, err := os.OpenFile("some/file/path.json",  os.O_CREATE+os.O_RDWR, 0700)
if err != nil {
	// Do something
}
defer f.Close()

if err := MarshalJSON(f, dir); err != nil {
	// Do something
}

Example of getting a JSON value by field name: 

f, err := dir.GetFile("Identities/EmployeeID")
if err != nil {
	// Do something
}

Same example, but we are okay with zero values if the field doesn't exist: 

f, _ := dir.GetFile("Identities/EmployeeID")

Get the type a File holds: 

t := f.JSONType()

Get a value the easiest way when you aren't sure what it is: 

v := f.Any()
// Now you have to switch on types nil, string, bool, int64 or float
// In order to use it.

Get a value from a Directory when you care about all the details (uck): 

f, err := dir.GetFile("Identities/EmployeeID")
if err != nil {
	fmt.Println("EmployeeID was not set")
} else if f.Type() == FTNull {
	fmt.Println("EmployeeID was explicitly not set")
}else {
	id, err := f.Int()
	if err != nil {
		// Do something
	}
	fmt.Println("EmployeeID is ", id)
}

Get a value from a Directory when zero values will do if set to null or doesn't exist: 

f, _ := dir.GetFile("Identities/EmployeeID")
fmt.Println(f.StringOrZV())
// There is also BoolOrZV(), IntOrZV(), ...

Put the value in an fs.FS and walk the JSON: 

// Note: this example can be found in examples/dirwalk
fsys := NewFSFromDir(dir)

fs.WalkDir(fsys, ".", func(p string, d fs.DirEntry, err error) error {
	if err != nil {
		log.Fatal(err)
	}
	p = path.Clean(p)

	switch x := d.(type) {
	case jsonfs.File:
		fmt.Printf("%s:%s:%v\n", p, x.JSONType(), x.Any())
	case jsonfs.Directory:
		fmt.Printf("%s/\n", p)
	}
	return nil
})

Index

Constants

View Source 
const (
	OTUnknown = 0
	OTFile    = 1
	OTDir     = 2
)

Variables

This section is empty.

Functions

func Append 

func Append[FD FileOrDir](array Directory, filesOrDirs ...FD) error

Append appends to the Directory array all filesOrDirs passed. A nil FileOrDir value will append a JSON null.

func ArrayDirName 

func ArrayDirName(name string) string

func ArraySet 

func ArraySet[FD FileOrDir](array Directory, index int, fd FD) error

ArraySet sets the value at index to fd. A nil value passed as fd will result in a null value being set.

func ByteSlice2String 

func ByteSlice2String(bs []byte) string

func CP 

func CP[FD FileOrDir](fileOrDir FD) FD

CP will make a copy of the File or Directory and return it. The modtime of the new directory and its files is the same as the old one.

func DirNameFromArray 

func DirNameFromArray(name string) string

func IsArray 

func IsArray(fsys fs.ReadDirFS, path string) (bool, error)

IsArray reads a directory in fsys at path to determine if it is an array. Note: I wish there was a better way, but it would have to be portable across filesystems and has to deal with people doing adhoc writes to the filesystem.

func MarshalJSON 

func MarshalJSON(w io.Writer, d Directory) error

MarshalJSON takes a Directory and outputs it as JSON to a file writer.

func SkipSpace 

func SkipSpace(b *bufio.Reader)

SskipSpace skips all spaces in the reader.

func UnmarshalStream 

func UnmarshalStream(ctx context.Context, r io.Reader) chan Stream

UnmarshalStream unmarshals a stream of JSON objects from a reader. This will handle both streams of objects.

func UnsafeGetBytes 

func UnsafeGetBytes(s string) []byte

UnsafeGetBytes extracts the []byte from a string. Use cautiously.

func ValueCheck 

func ValueCheck(b *bufio.Reader) (next, error)

ValueCheck tells what the next JSON value in a *bufio.Reader is.

func WriteOut 

func WriteOut[S Writeable](w io.Writer, values ...S) error

WriteOut writes to "w" all "values".

Types

type Directory 

type Directory struct {
	// contains filtered or unexported fields
}

Directory represents an object or array in JSON nomenclature.

func MustNewArray 

func MustNewArray(name string, filesOrDirs ...any) Directory

MustNewArray is like NewArray, but errors cause a panic.

func MustNewDir 

func MustNewDir(name string, filesOrDirs ...any) Directory

MustNewDir is like NewDirectory, but errors cause a panic.

func NewArray 

func NewArray(name string, filesOrDirs ...any) (Directory, error)

NewArray creates a new Directory that represents a JSON array. filesOrDirs that have names will have them overridden.

func NewDir 

func NewDir(name string, filesOrDirs ...any) (Directory, error)

NewDir creates a new Directory with name and includes the files and directories passed. All passed Directories and Files must have a name. A top level directory does not have to have a name.

func UnmarshalJSON 

func UnmarshalJSON(r io.Reader) (Directory, error)

UnmarshalJSON unmarshals a single JSON object from an io.Reader. This object must be an object inside {}. This should only be used for reading a file or single object contained in an io.Reader. We will use a bufio.Reader underneath, so this reader is not usable after.

func (Directory) Close 

func (d Directory) Close() error

Close implememnts fs.ReadDirFile.Close().

func (Directory) EncodeJSON 

func (d Directory) EncodeJSON(w io.Writer) error

EncodeJSON encodes the Directory as JSON into the io.Writer passed.

func (Directory) GetDir 

func (d Directory) GetDir(name string) (Directory, error)

GetDir gets a sub directory with the path "name".

func (Directory) GetFile 

func (d Directory) GetFile(name string) (File, error)

GetFile gets a file located at path "name".

func (Directory) GetObjects 

func (d Directory) GetObjects() chan Object

func (Directory) Info 

func (d Directory) Info() (fs.FileInfo, error)

Info implements fs.DirEntry.Info().

func (Directory) IsDir 

func (d Directory) IsDir() bool

IsDir implements fs.DirEntry.IsDir().

func (Directory) Len 

func (d Directory) Len() int

Len is how many items in the Directory.

func (Directory) Name 

func (d Directory) Name() string

Name implements fs.DirEntry.Name().

func (Directory) Read 

func (d Directory) Read([]byte) (int, error)

Read implements fs.File.Read(). This will panic as it does on a filesystem.

func (Directory) ReadDir 

func (d Directory) ReadDir(n int) ([]fs.DirEntry, error)

ReadDir implememnts fs.ReadDirFile.ReadDir().

func (Directory) Remove 

func (d Directory) Remove(name string) error

Remove removes a file or directory (empty) in this directory.

func (Directory) RemoveAll 

func (d Directory) RemoveAll(name string) error

RemoveAll removes a file or directory contained in this directory.

func (Directory) Set 

func (d Directory) Set(filesOrDirs ...any) error

Set will set sub directories or files in the Directory. If a file or Directory already exist, it will be overwritten. This does not work if the Directory is an array.

func (Directory) Stat 

func (d Directory) Stat() (fs.FileInfo, error)

Stat implements fs.File.Stat().

func (Directory) Type 

func (d Directory) Type() fs.FileMode

type implements fs.DirEntry.Type().

func (Directory) WriteFile 

func (d Directory) WriteFile(name string, data []byte) error

WriteFile writes file "name" with "data" to this Directory.

type DirectoryFS 

type DirectoryFS interface {
	// Directory retrieves a Directory from an FS.
	Directory(path string) (Directory, error)
}

DirectoryFS provides methods for reading and writing a Directory to a Filesystem.

type DiskFS 

type DiskFS struct {
	// contains filtered or unexported fields
}

DiskFS provides an FS for OS access. It will treat everything in the filesystem from the root down as JSON. So directories represent JSON objects or arrays while files represent JSON values. Note: I haven't tested this or done much with this.

func NewDiskFS 

func NewDiskFS(path string) (DiskFS, error)

NewDiskFS creates a new DiskFS rooted at path string. The path should represent a new directory and must not exist. MkdirAll() will be called with perms 0700. If you wish to open an existing path, use OpenDiskFS().

func OpenDiskFS 

func OpenDiskFS(path string) (DiskFS, error)

OpenDiskFS opens a DiskFS that exists on the filesystem at path.

func (DiskFS) Directory 

func (f DiskFS) Directory(name string) (Directory, error)

Directory retrieves a Directory from path. Changes to the Directory will not make changes to disk. If you wish to sync this copy of the Directory to disk, use WriteDir().

func (DiskFS) Mkdir 

func (f DiskFS) Mkdir(p string, perm fs.FileMode) error

Mkdir creates a directory named "p" with permissions "perm". perm must be 0700, 0770, 0707 or 0777.

func (DiskFS) MkdirAll 

func (f DiskFS) MkdirAll(p string, perm fs.FileMode) error

MkdirAll creates a directory named path, along with any necessary parents, and returns nil, or else returns an error. The permission bits perm (before umask) are used for all directories that MkdirAll creates. If path is already a directory, MkdirAll does nothing and returns nil. perm must be 0700, 0770, 0707, 0777. This implements github.com/gopherfs/fs.MkdirAllFS.MkdirAll.

func (DiskFS) Open 

func (f DiskFS) Open(name string) (fs.File, error)

Open implements fs.FS.Open().

func (DiskFS) OpenFile 

func (f DiskFS) OpenFile(name string, perms fs.FileMode, options ...gopherfs.OFOption) (fs.File, error)

OpenFile represents github.com/gopherfs/fs.OpenFiler. A file can only be opened in 0400, 0440, 0404, 0444. A Directory in 0700, 0770, 0707, 0777.

func (DiskFS) ReadDir 

func (f DiskFS) ReadDir(name string) ([]fs.DirEntry, error)

ReadDir implements fs.ReadDirFS.Read().

func (DiskFS) ReadFile 

func (f DiskFS) ReadFile(name string) ([]byte, error)

ReadFile implemnts fs.ReadFileFS.ReadFile().

func (DiskFS) Remove 

func (f DiskFS) Remove(name string) error

Remove removes a file or directory (empty) at path "name". This implements github.com/gopherfs/fs.Remove.Remove .

func (DiskFS) RemoveAll 

func (f DiskFS) RemoveAll(path string) error

RemoveAll removes path and any children it contains. It removes everything it can but returns the first error it encounters. If the path does not exist, RemoveAll returns nil (no error). If there is an error, it will be of type *fs.PathError.

func (DiskFS) Stat 

func (f DiskFS) Stat(name string) (fs.FileInfo, error)

func (DiskFS) Sub 

func (f DiskFS) Sub(dir string) (fs.FS, error)

Sub implements fs.SubFS.Sub().

func (DiskFS) WriteFile 

func (f DiskFS) WriteFile(name string, data []byte, perm fs.FileMode) error

WriteFile writes file with path "name" to the filesystem. If the file already exists, this will overwrite the existing file. data must not be mutated after it is passed here. perm must be 0400, 0440, 0404 or 0444. This implements github.com/gopherfs/fs.Writer .

type FS 

type FS interface {
	fs.FS
	fs.ReadDirFS
	fs.ReadFileFS
	fs.StatFS
	fs.SubFS

	gopherfs.MkdirAllFS
	gopherfs.Remove
	gopherfs.Writer
}

FS details the interfaces that a filesytem must have in order to be used by jsonfs purposes. We do not honor filesytems interfaces outside this package at this time.

type File 

type File struct {
	// contains filtered or unexported fields
}

File represents a value in JSON. This can be a string, bool or number. All files are readonly.

func MustNewFile 

func MustNewFile(name string, value any) File

MustNewFile is like NewFile except any error panics.

func NewFile 

func NewFile(name string, value any) (File, error)

NewFile creates a new file named "name" with value []byte. Files created with NewFile cannot have .Read() called, as this only works when opened from FS or a Directory. This simply is used to help construct a JSON value. value can be any type of int, string, bool or float. A nil value stands for a JSON null.

func (File) Any 

func (f File) Any() any

func (File) Bool 

func (f File) Bool() (bool, error)

Bool returns a file's value if it is a bool.

func (File) BoolorZV 

func (f File) BoolorZV() bool

func (File) Close 

func (f File) Close() error

Close closes the file.

func (File) EncodeJSON 

func (f File) EncodeJSON(w io.Writer) error

EncodeJSON outputs the file data as into the writer.

func (File) Float 

func (f File) Float() (float64, error)

Float returns a file's value if it is a float.

func (File) FloatOrZV 

func (f File) FloatOrZV() float64

func (File) Info 

func (f File) Info() (fs.FileInfo, error)

Info implements fs.DirEntry.Info().

func (File) Int 

func (f File) Int() (int64, error)

Int returns a file's value if it is a int.

func (File) IntOrZV 

func (f File) IntOrZV() int64

func (File) IsDir 

func (f File) IsDir() bool

IsDir implements fs.DirEntry.IsDir().

func (File) JSONType 

func (f File) JSONType() FileType

JSONType indicates the JSON type of the file.

func (File) Name 

func (f File) Name() string

Name implements fs.DirEntry.Name().

func (File) Read 

func (f File) Read(dst []byte) (int, error)

Read implements fs.File.Read(). It is not thread-safe.

func (File) Stat 

func (f File) Stat() (fs.FileInfo, error)

Stat implements fs.File.Stat().

func (File) String 

func (f File) String() (string, error)

String returns a file's value if it is a string.

func (File) StringOrZV 

func (f File) StringOrZV() string

func (File) Type 

func (f File) Type() fs.FileMode

Type implements fs.DirEntry.Type().

type FileInfo 

type FileInfo struct {
	// contains filtered or unexported fields
}

FileInfo implements fs.FileInfo.

func (FileInfo) IsDir 

func (f FileInfo) IsDir() bool

func (FileInfo) ModTime 

func (f FileInfo) ModTime() time.Time

func (FileInfo) Mode 

func (f FileInfo) Mode() fs.FileMode

func (FileInfo) Name 

func (f FileInfo) Name() string

func (FileInfo) Size 

func (f FileInfo) Size() int64

func (FileInfo) Sys 

func (f FileInfo) Sys() any

type FileOrDir 

type FileOrDir interface {
	// contains filtered or unexported methods
}

FileOrDir stands can hold a File or Directory.

type FileType 

type FileType uint8

FileType represents the data stored in a File. 

const (
	FTNull   FileType = 0
	FTBool   FileType = 1
	FTInt    FileType = 2
	FTFloat  FileType = 3
	FTString FileType = 4
)

func DataType 

func DataType(data []byte) (FileType, error)

DataType is a more expensive version of ValueCheck for just file data.

func (FileType) String 

func (i FileType) String() string

type MemFS 

type MemFS struct {
	// contains filtered or unexported fields
}

MemFS represents an inmemory filesystem for storing JSON data. It can be used with tools that work on fs.FS to do introspection on the JSON data or make data modifications. The normal way to use MemFS is for a single JSON entry.

func NewMemFS 

func NewMemFS(dir Directory) MemFS

NewMemFS creates a new MemFS from a Directory that will act as the root.

func (MemFS) MkdirAll 

func (f MemFS) MkdirAll(p string, perm fs.FileMode) error

MkdirAll creates a directory named path, along with any necessary parents, and returns nil, or else returns an error. The permission bits perm (before umask) are used for all directories that MkdirAll creates, which must be 2147483940 (fs.ModeDir + 0444). If path is already a directory, MkdirAll does nothing and returns nil. This implements github.com/gopherfs/fs.MkdirAllFS.MkdirAll. TODO(jdoak): Move logic to Directory.MkdirAll and take a lock.

func (MemFS) Open 

func (f MemFS) Open(name string) (fs.File, error)

Open implements fs.FS.Open().

func (MemFS) OpenFile 

func (m MemFS) OpenFile(name string, perms fs.FileMode, options ...gopherfs.OFOption) (fs.File, error)

OpenFile implements gopherfs.OpenFiler. Perms are ignored except for the IsDir directive.

func (MemFS) ReadDir 

func (f MemFS) ReadDir(name string) ([]fs.DirEntry, error)

ReadDir implements fs.ReadDirFS.Read().

func (MemFS) ReadFile 

func (f MemFS) ReadFile(name string) ([]byte, error)

ReadFile implemnts fs.ReadFileFS.ReadFile().

func (MemFS) Remove 

func (f MemFS) Remove(name string) error

Remove removes a file or directory (empty) at path "name". This implements github.com/gopherfs/fs.Remove.Remove .

func (MemFS) RemoveAll 

func (f MemFS) RemoveAll(path string) error

RemoveAll removes path and any children it contains. It removes everything it can but returns the first error it encounters. If the path does not exist, RemoveAll returns nil (no error). If there is an error, it will be of type *fs.PathError.

func (MemFS) Stat 

func (f MemFS) Stat(name string) (fs.FileInfo, error)

func (MemFS) Sub 

func (f MemFS) Sub(dir string) (fs.FS, error)

Sub implements fs.SubFS.Sub().

func (MemFS) WriteFile 

func (f MemFS) WriteFile(name string, data []byte, perm fs.FileMode) error

WriteFile writes file with path "name" to the filesystem. If the file already exists, this will overwrite the existing file. data must not be mutated after it is passed here. perm must be 0444. This implements github.com/gopherfs/fs.Writer .

type ObjType 

type ObjType uint8

type Object 

type Object struct {
	Type ObjType
	File File
	Dir  Directory
}

type Stream 

type Stream struct {
	// Dir is the JSON object as a Directory.
	Dir Directory
	// Err indicates that there was an error in the stream.
	Err error
}

Stream is a stream object from UnmarshalStream().

type Writeable 

type Writeable interface {
	rune | string | []byte
}

Source Files

View all Source files

Directories

Path Synopsis
examples
dirwalk

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.