shop

package module
v0.0.0-...-2b69279 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 14, 2024 License: BSD-3-Clause Imports: 17 Imported by: 0

README

Shop

A CIPD inspired package registry/deployer.

Shop is a package registry/deployment system for development tools.

Just like in CIPD A package has a name and list of content-addressed instances, where slashes in package names form a hierarchy of packages and an instance is a zip file with the package contents.

Shop is different from apt, brew, pip, npm, etc. in that it is not tied to a specific OS or language. Shop is different from CIPD in that it is not tied to chromium project or google infrastructure.

Versions

A package instance can be referenced by a tuple (package name, version). A version is one of:

  • A hash of the instance contents, e.g. bec8e88201949be06b06174178c2f62b81e4008e. This is also called instance id.
  • A key-value tag: e.g. git_revision:deadbeef, if it's unique among all instances of the package.
  • A ref, e.g. latest.

Tags

A package instance can be marked with tags, where a tag is a colon-separated key-value pair, e.g. git_revision:deadbeef. If some tag points to only one instance, such tag can be used as version identifier.

Refs

A package can have a git-like refs, where a ref of package points to one of the instances of the package by id.

Platforms

If a package is platform-specific, the package name should have a /os-arch suffix.

Access-Control

Unlike CIPD, Shop does not have a fine-granular builtin access-control mechanism. If you have access to registry, you have same access to all packages inside that registry. Shop supports working with multiple registries and allows granting different access to different registries.

In most common use-case, users only need to have read-only access to registry. And only registry owners need read-write access to upload new packages. With that in mind, it's not really a hard requirement to have an ACL support.

Shop aims to have no backend service and allows client to talk directly to the storage. With that in mind it's really difficult to built any sort of ACL, unless storage provides some kind of ACL on it's own.

Why?

I like writing C++ code. And I want modern llvm toolchain/SDK to do that. And the only good way to have them for some reason - building them myself. And it's too large to build from scratch for any ocasion. Aim is to create a convenient storage for that kind of tools and a set of automated scripts to build and automatically publish new versions.

Backend

Shop is made to use any S3 or WebDAV compatible storage as a backend. All the API endpoints are saved as JSON objects into the storage so any client with read only access can have the data. Updating those objects is a bit cumbersome but could be easily done automatically. Upside of this approach is: you don't need to have a client to access registry and download packgaes if you have access to the storage. You can even bootstrap the client from that same registry.

Official registry & packages

It would be weird to create a registry and not to run it, I intend to create a set of scripts to build & publish packages for all the tools I consider useful. And I will publish most of them into the public registry. However, due to some licensing issues with some of the packages (some software is illegal to redistribute especially in a modified form), some of the packages would only be uploaded to my private registry. You're free to use the scripts to build your own.

TODOs

  1. Implement high-level commands of the client using local file repository.
  2. Implement a driver for WebDAV repository.
  3. Implement an interface for external repository drivers by running driver process in background and using it as WebDAV proxy.
  4. Implement an external driver for ftp repository.
  5. Implement an external driver for sftp repository.
  6. Implement an external driver for S3 repository.
  7. Prepare collection of scripts to build artifacts and upload them to my registry.

Architecture

Shop aims to get rid of server-side code and work on top of existing file storages. With existence of the broad range of file storage technologies it's hard to make a choice and only support one of them. But adding support for every single one into the client is a huge task which I don't want to do. So one of the key ideas is separating support of specific file storages into separate driver executables. That also allows to move away from the problem of many methods of authenticating into the remote storages.

The simplest abstraction of many different storage systems would have quite limited support for access control. Some storages could have no support of fine-granular access control at all. For that reason the client assumes that it has either read-only or read-write access to the whole storage. Since that could be unwanted behavior from registry admin perspective. The registry will be based on any number of storages. So each storage could have it's own specific access settings.

Terms
  • Repository (repo) - A single instance of the single storage. Like S3 Bucket or directory on WebDAV server. MUST be identified by single URL.
  • Driver - An external program which implements specific API and only provides a way for client to access repositories with specific URL schema.
  • Registry - A collection of packages hosted on top of several (at least one) repositories. Also MUST be identified by a single URL.
  • Root repo - A single repo which is used by registry to keep all metadata. It could also have package contents, however package content could be kept in any other repo, connected to the registry.
  • Client - Program executable which works with registries.

Documentation

Index

Constants

View Source 
const (
	LatestVersion                      = "1.0.0"
	RegistryManifestKey                = "shop.json"
	RegistryPackagesPrefix             = "/packages/"
	RegistryPackageManifestKey         = "package.json"
	RegistryPackageReferencesPrefix    = "/refs/"
	RegistryPackageInstancesPrefix     = "/instances/"
	RegistryPackageInstanceManifestKey = "instance.json"
	RegistryPackageTagsPrefix          = "/tags/"
	RegistryPackageInstanceTagsPrefix  = "/tags/"
	RegistryPackageInstanceIdLen       = sha1.Size * 2
	RegistryCASPrefix                  = "/cas/"
	RegistryCASArchiveExtension        = ".tgz"
)
View Source 
const (
	DefaultRegistryName = "default"
)
View Source 
const (
	RepositoryManifestKey = "shop-repository.json"
)

Variables

View Source 
var (
	ErrRegistryConfigExists    = errors.New("Registry already exists in configuration")
	ErrRegistryConfigNotExists = errors.New("Registry does not exist in configuration")
)
View Source 
var (
	ErrUnimplemented             = errors.New("Unimplemented")
	ErrRegistryAdminIsNotAllowed = errors.New("Admin action on the registry is not enabled in configuration")
	ErrRegistryWriteIsNotAllowed = errors.New("Write action on the registry is not enabled in configuration")
	ErrUnknownRepo               = errors.New("Registry does not have repo")
	ErrInvalidPackageName        = errors.New("Invalid package name")
	ErrInvalidInstanceId         = errors.New("Invalid instance id")
	ErrInvalidReferenceName      = errors.New("Invalid reference name")
	ErrInvalidTagName            = errors.New("Invalid tag name")
	ErrInvalidTagValue           = errors.New("Invalid tag value")
)
View Source 
var (
	RepositoryFactories      = map[string]func(context.Context, RepositoryConfig) (RepositoryFS, error){}
	ErrRepoWriteIsNotAllowed = errors.New("Write to the repository is not enabled in configuration")
	ErrRepoAdminIsNotAllowed = errors.New("Admin action on the repository is not enabled in configuration")
)

Functions

func FindConfigFile 

func FindConfigFile() (path string, err error)

Find location of the config file. Should be $XDG_CONFIG_HOME/shop/config.toml

func IsValidInstanceId 

func IsValidInstanceId(id string) bool

func IsValidPackageName 

func IsValidPackageName(name string) bool

func IsValidRefName 

func IsValidRefName(v string) bool

func IsValidTagName 

func IsValidTagName(v string) bool

func IsValidTagValue 

func IsValidTagValue(v string) bool

func MakeArchive 

func MakeArchive(dst io.Writer, fs fs.FS) (id string, err error)

func NewConfigLoadError 

func NewConfigLoadError(err error, path string) error

func NewConfigSaveError 

func NewConfigSaveError(err error, path string) error

func SaveConfig 

func SaveConfig(cfg Config, path string) (err error)

Types

type Config 

type Config struct {
	DefaultRegistry string `toml:"default_registry,omitempty" comment:"Default registry to use."`
	Cache           string `toml:"cache,omitempty" comment:"Path to the local file cache."`

	Registries map[string]RegistryConfig `toml:"registry,omitempty"`
}

func LoadConfig 

func LoadConfig(path string) (cfg Config, err error)

Load config from file at path. Returns empty config if file does not exist.

func (*Config) AddRegistry 

func (c *Config) AddRegistry(name string, registryCfg RegistryConfig) error

func (*Config) UpdateRegistry 

func (c *Config) UpdateRegistry(name string, registryCfg RegistryConfig) error

type ConfigLoadError 

type ConfigLoadError struct {
	Path string
	// contains filtered or unexported fields
}

func (ConfigLoadError) Error 

func (e ConfigLoadError) Error() string

type ConfigSaveError 

type ConfigSaveError struct {
	Path string
	// contains filtered or unexported fields
}

func (ConfigSaveError) Error 

func (e ConfigSaveError) Error() string

type Cursor 

type Cursor[T any] interface {
	GetNext(context.Context) (*T, error)
}

func NewErrorCursor 

func NewErrorCursor[T any](err error) Cursor[T]

func NewSliceCursor 

func NewSliceCursor[T any](items []T) Cursor[T]

type Entry 

type Entry struct {
	Key      string
	IsPrefix bool
}

type ErrorCursor 

type ErrorCursor[T any] struct {
	// contains filtered or unexported fields
}

func (ErrorCursor[T]) GetNext 

func (c ErrorCursor[T]) GetNext(context.Context) (*T, error)

type FileFS 

type FileFS struct {
	// contains filtered or unexported fields
}

func (FileFS) Create 

func (f FileFS) Create(ctx context.Context, path string) (io.WriteCloser, error)

func (FileFS) Exists 

func (f FileFS) Exists(ctx context.Context, path string) (ok bool, err error)

func (FileFS) ListDir 

func (f FileFS) ListDir(ctx context.Context, path string) Cursor[Entry]

func (FileFS) MakeDir 

func (f FileFS) MakeDir(ctx context.Context, path string) error

func (FileFS) Open 

func (f FileFS) Open(ctx context.Context, path string) (io.ReadCloser, error)

func (FileFS) Read 

func (f FileFS) Read(ctx context.Context, path string) ([]byte, error)

func (FileFS) Remove 

func (f FileFS) Remove(ctx context.Context, path string) error

func (FileFS) Write 

func (f FileFS) Write(ctx context.Context, path string, data []byte) error

type HTTPAccessConfig 

type HTTPAccessConfig struct {
	User       string `toml:"user,omitempty"`
	Password   string `toml:"password,omitempty"`
	ClientCert string `toml:"client_cert,omitempty" comment:"Client Certificate in PEM format."`
}

type Instance 

type Instance struct {
	ApiVersion string        `json:"api_version"`
	Package    string        `json:"package"`
	Id         string        `json:"id"`
	UploadedAt UnixTimestamp `json:"uploaded_at"`
	UpdatedAt  UnixTimestamp `json:"updated_at"`
}

func NewInstance 

func NewInstance(pkg, id string) (instance Instance, err error)

type Package 

type Package struct {
	ApiVersion  string        `json:"api_version"`
	Name        string        `json:"name"`
	Description string        `json:"description,omitempty"`
	Repo        string        `json:"repo,omitempty"`
	UpdatedAt   UnixTimestamp `json:"package"`
}

func NewPackage 

func NewPackage(name, description, repo string) (pkg Package, err error)

type PackageOrPrefix 

type PackageOrPrefix struct {
	Package *Package
	Prefix  string
}

type PackageTag 

type PackageTag struct {
	Package string
	Key     string
}

type PackageTagValue 

type PackageTagValue struct {
	PackageTag
	Value string
}

type Reference 

type Reference struct {
	ApiVersion string        `json:"api_version"`
	Package    string        `json:"package"`
	Name       string        `json:"name"`
	Id         string        `json:"id"`
	UpdatedAt  UnixTimestamp `json:"updated_at"`
}

func NewReference 

func NewReference(pkg, name, id string) (ref Reference, err error)

type Registry 

type Registry interface {
	GetConfig() RegistryConfig

	Initialize(ctx context.Context, name string) error

	GetManifest(ctx context.Context) (*RegistryManifest, error)
	PutManifest(context.Context, RegistryManifest) error

	GetPackage(ctx context.Context, name string) (*Package, error)
	ListPackages(ctx context.Context, prefix string) Cursor[PackageOrPrefix]
	PutPackage(ctx context.Context, pkg Package) error

	UploadPackageInstance(ctx context.Context, name, id string, reader io.Reader) (*Instance, error)
	ListPackageInstances(ctx context.Context, name string) Cursor[Instance]
	GetPackageInstanceInfo(ctx context.Context, name, id string) (*Instance, error)
	PutPackageInstanceInfo(ctx context.Context, instance Instance) error
	DeletePackageInstanceInfo(ctx context.Context, instance Instance) error
	ListPackageInstanceTags(ctx context.Context, instance Instance) Cursor[Tag]

	ListPackageReferences(ctx context.Context, name string) Cursor[Reference]
	GetPackageReference(ctx context.Context, pkg, name string) (*Reference, error)
	PutPackageReference(ctx context.Context, ref Reference) error
	DeletePackageReference(ctx context.Context, ref Reference) error

	ListPackageTags(ctx context.Context, names string) Cursor[PackageTag]
	ListPackageTagValues(ctx context.Context, tag PackageTag) Cursor[PackageTagValue]
	ListPackageInstancesByTag(ctx context.Context, tag PackageTagValue) Cursor[Tag]
	PutPackageInstanceTag(ctx context.Context, tag Tag) error
	DeletePackageInstanceTag(ctx context.Context, tag Tag) error
}

func NewRegistry 

func NewRegistry(ctx context.Context, cfg RegistryConfig) (Registry, error)

type RegistryConfig 

type RegistryConfig struct {
	URL      string                      `toml:"url" comment:"Manifest url."`
	RootRepo RepositoryConfig            `toml:"root_repository" comment:"Main repository settings."`
	Repos    map[string]RepositoryConfig `toml:"repo,omitempty" comment:"Secondar repositories settings."`

	// Local tool configuration
	Admin bool `toml:"admin,omitempty" comment:"Enable admin commands for this registry."`
	Write bool `toml:"write,omitempty" comment:"Enable write commands for this registry."`
}

type RegistryImpl 

type RegistryImpl struct {
	// contains filtered or unexported fields
}

func (*RegistryImpl) DeletePackageInstanceInfo 

func (c *RegistryImpl) DeletePackageInstanceInfo(ctx context.Context, instance Instance) error

func (*RegistryImpl) DeletePackageInstanceTag 

func (c *RegistryImpl) DeletePackageInstanceTag(ctx context.Context, tag Tag) error

func (*RegistryImpl) DeletePackageReference 

func (c *RegistryImpl) DeletePackageReference(ctx context.Context, ref Reference) error

func (*RegistryImpl) GetConfig 

func (c *RegistryImpl) GetConfig() RegistryConfig

func (*RegistryImpl) GetManifest 

func (c *RegistryImpl) GetManifest(ctx context.Context) (manifest *RegistryManifest, err error)

func (*RegistryImpl) GetPackage 

func (c *RegistryImpl) GetPackage(ctx context.Context, name string) (manifest *Package, err error)

func (*RegistryImpl) GetPackageInstanceInfo 

func (c *RegistryImpl) GetPackageInstanceInfo(ctx context.Context, name, id string) (instance *Instance, err error)

func (*RegistryImpl) GetPackageReference 

func (c *RegistryImpl) GetPackageReference(ctx context.Context, pkg, name string) (ref *Reference, err error)

func (*RegistryImpl) Initialize 

func (c *RegistryImpl) Initialize(ctx context.Context, name string) error

func (*RegistryImpl) ListPackageInstanceTags 

func (c *RegistryImpl) ListPackageInstanceTags(ctx context.Context, instance Instance) Cursor[Tag]

func (*RegistryImpl) ListPackageInstances 

func (c *RegistryImpl) ListPackageInstances(ctx context.Context, name string) Cursor[Instance]

func (*RegistryImpl) ListPackageInstancesByTag 

func (c *RegistryImpl) ListPackageInstancesByTag(ctx context.Context, tag PackageTagValue) Cursor[Tag]

func (*RegistryImpl) ListPackageReferences 

func (c *RegistryImpl) ListPackageReferences(ctx context.Context, name string) Cursor[Reference]

func (*RegistryImpl) ListPackageTagValues 

func (c *RegistryImpl) ListPackageTagValues(ctx context.Context, tag PackageTag) Cursor[PackageTagValue]

func (*RegistryImpl) ListPackageTags 

func (c *RegistryImpl) ListPackageTags(ctx context.Context, name string) Cursor[PackageTag]

func (*RegistryImpl) ListPackages 

func (c *RegistryImpl) ListPackages(ctx context.Context, prefix string) Cursor[PackageOrPrefix]

func (*RegistryImpl) PutManifest 

func (c *RegistryImpl) PutManifest(ctx context.Context, manifest RegistryManifest) error

func (*RegistryImpl) PutPackage 

func (c *RegistryImpl) PutPackage(ctx context.Context, pkg Package) error

func (*RegistryImpl) PutPackageInstanceInfo 

func (c *RegistryImpl) PutPackageInstanceInfo(ctx context.Context, instance Instance) error

func (*RegistryImpl) PutPackageInstanceTag 

func (c *RegistryImpl) PutPackageInstanceTag(ctx context.Context, tag Tag) error

func (*RegistryImpl) PutPackageReference 

func (c *RegistryImpl) PutPackageReference(ctx context.Context, ref Reference) error

func (*RegistryImpl) UploadPackageInstance 

func (c *RegistryImpl) UploadPackageInstance(ctx context.Context, name, id string, reader io.Reader) (*Instance, error)

type RegistryManifest 

type RegistryManifest struct {
	ApiVersion string                        `json:"api_version"`
	Name       string                        `json:"name"`
	RootRepo   RepositoryManifest            `json:"root_repo"`
	Repos      map[string]RepositoryManifest `json:"repos"`
	UpdatedAt  UnixTimestamp                 `json:"updated_at"`
}

type Repository 

type Repository interface {
	GetConfig() RepositoryConfig

	Get(ctx context.Context, key string) (io.ReadCloser, error)
	Put(ctx context.Context, key string, body io.Reader) error

	GetJSON(ctx context.Context, key string, output any) error
	PutJSON(ctx context.Context, key string, input any) error

	List(ctx context.Context, prefix string) Cursor[Entry]

	EnsurePrefix(ctx context.Context, key string) error
	Delete(ctx context.Context, key string) error

	GetManifest(ctx context.Context) (RepositoryManifest, error)
	PutManifest(ctx context.Context, manifest RepositoryManifest) error

	ResourceExists(ctx context.Context, key string) (bool, error)
}

func NewRepository 

func NewRepository(ctx context.Context, cfg RepositoryConfig) (repository Repository, err error)

type RepositoryConfig 

type RepositoryConfig struct {
	URL   string `toml:"url" comment:"Repository URL"`
	Admin bool   `toml:"admin,omitempty" comment:"Enable admin access for this repository."`
	Write bool   `toml:"write,omitempty" comment:"Enable write access for this repository."`
}

type RepositoryFS 

type RepositoryFS interface {
	Read(context.Context, string) ([]byte, error)
	Write(context.Context, string, []byte) error
	Open(context.Context, string) (io.ReadCloser, error)
	Create(context.Context, string) (io.WriteCloser, error)
	MakeDir(context.Context, string) error
	ListDir(context.Context, string) Cursor[Entry]
	Remove(context.Context, string) error
	Exists(context.Context, string) (bool, error)
}

func NewFileFS 

func NewFileFS(ctx context.Context, cfg RepositoryConfig) (RepositoryFS, error)

type RepositoryManifest 

type RepositoryManifest struct {
	ApiVersion  string        `json:"api_version"`
	URL         string        `json:"url"`
	Name        string        `json:"name"`
	ReadOnlyURL string        `json:"readonly_url,omitempty"`
	UpdatedAt   UnixTimestamp `json:"updated_at"`
}

type S3AccessConfig 

type S3AccessConfig struct {
	// S3 Bucket settings.
	EndpointURL string `toml:"endpoint_url,omitempty" comment:"S3 Endpoint url."`
	Region      string `toml:"region" comment:"AWS region."`
	Bucket      string `toml:"bucket" comment:"S3 Bucket name."`

	// S3 Auth information.
	AWSProfile      string `toml:"aws_profile,omitempty" comment:"AWS profile name."`
	AccessKeyId     string `toml:"access_key_id,omitempty" comment:"AWS Access Key ID."`
	SecretAccessKey string `toml:"secret_access_key,omitempty" comment:"AWS Secret Access Key."`
}

type SliceCursor 

type SliceCursor[T any] struct {
	// contains filtered or unexported fields
}

func (*SliceCursor[T]) GetNext 

func (c *SliceCursor[T]) GetNext(context.Context) (item *T, err error)

type Tag 

type Tag struct {
	ApiVersion string        `json:"api_version"`
	Package    string        `json:"package"`
	Key        string        `json:"key"`
	Value      string        `json:"value"`
	Id         string        `json:"id"`
	UpdatedAt  UnixTimestamp `json:"updated_at"`
}

func NewTag 

func NewTag(pkg, key, value, id string) (tag Tag, err error)

type TeeWriter 

type TeeWriter []io.Writer

func (TeeWriter) Write 

func (w TeeWriter) Write(data []byte) (n int, err error)

type UnixTimestamp 

type UnixTimestamp struct {
	time.Time
}

func (UnixTimestamp) MarshalJSON 

func (ut UnixTimestamp) MarshalJSON() ([]byte, error)

func (*UnixTimestamp) UnmarshalJSON 

func (ut *UnixTimestamp) UnmarshalJSON(d []byte) (err error)

Source Files

View all Source files

Directories

Path Synopsis
cmd
shop

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.