buildkit

package
v0.12.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 18, 2024 License: Apache-2.0 Imports: 97 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	InternalPrefix = "[internal] "

	// from buildkit, cannot change
	EntitlementsJobKey = "llb.entitlements"

	// OCIStoreName is the name of the OCI content store used for OCI tarball
	// imports.
	OCIStoreName = "dagger-oci"

	// BuiltinContentOCIStoreName is the name of the OCI content store used for
	// builtins like SDKs that we package with the engine container but still use
	// in LLB.
	BuiltinContentOCIStoreName = "dagger-builtin-content"
)
View Source 
const (
	DaggerSessionIDEnv       = "_DAGGER_SESSION_ID"
	DaggerClientIDEnv        = "_DAGGER_NESTED_CLIENT_ID"
	DaggerCallDigestEnv      = "_DAGGER_CALL_DIGEST"
	DaggerEngineVersionEnv   = "_DAGGER_ENGINE_VERSION"
	DaggerRedirectStdoutEnv  = "_DAGGER_REDIRECT_STDOUT"
	DaggerRedirectStderrEnv  = "_DAGGER_REDIRECT_STDERR"
	DaggerHostnameAliasesEnv = "_DAGGER_HOSTNAME_ALIASES"

	DaggerSessionPortEnv  = "DAGGER_SESSION_PORT"
	DaggerSessionTokenEnv = "DAGGER_SESSION_TOKEN"

	// this is set by buildkit, we cannot change
	BuildkitSessionIDHeader = "x-docker-expose-session-uuid"

	OTelTraceParentEnv      = "TRACEPARENT"
	OTelExporterProtocolEnv = "OTEL_EXPORTER_OTLP_PROTOCOL"
	OTelExporterEndpointEnv = "OTEL_EXPORTER_OTLP_ENDPOINT"
	OTelTracesProtocolEnv   = "OTEL_EXPORTER_OTLP_TRACES_PROTOCOL"
	OTelTracesEndpointEnv   = "OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"
	OTelTracesLiveEnv       = "OTEL_EXPORTER_OTLP_TRACES_LIVE"
	OTelLogsProtocolEnv     = "OTEL_EXPORTER_OTLP_LOGS_PROTOCOL"
	OTelLogsEndpointEnv     = "OTEL_EXPORTER_OTLP_LOGS_ENDPOINT"
	OTelMetricsProtocolEnv  = "OTEL_EXPORTER_OTLP_METRICS_PROTOCOL"
	OTelMetricsEndpointEnv  = "OTEL_EXPORTER_OTLP_METRICS_ENDPOINT"
)
View Source 
const (
	// Exec errors will only include the last this number of bytes of output.
	MaxExecErrorOutputBytes = 100 * 1024

	// TruncationMessage is the message that will be prepended to truncated output.
	TruncationMessage = "[omitting %d bytes]..."

	// MaxFileContentsChunkSize sets the maximum chunk size for ReadFile calls
	// Equals around 95% of the max message size (4MB) in
	// order to keep space for any Protocol Buffers overhead:
	MaxFileContentsChunkSize = 3984588

	// MaxFileContentsSize sets the limit of the maximum file size
	// that can be retrieved using File.Contents, currently set to 128MB:
	MaxFileContentsSize = 128 << 20

	// MetaMountDestPath is the special path that the shim writes metadata to.
	MetaMountDestPath     = "/.dagger_meta_mount"
	MetaMountExitCodePath = "exitCode"
	MetaMountStdinPath    = "stdin"
	MetaMountStdoutPath   = "stdout"
	MetaMountStderrPath   = "stderr"
)
View Source 
const InstrumentationLibrary = "dagger.io/engine.buildkit"
View Source 
const TelemetryComponent = "buildkit"

Variables

View Source 
var SkipInputs = fmt.Errorf("skip inputs") //nolint:stylecheck // Err prefix isn't convention for Walk control errors

Functions

func AsWorkerController added in v0.11.8 

func AsWorkerController(w worker.Worker) (*worker.Controller, error)

Buildkit's worker.Controller is a bit odd; it exists to manage multiple workers because that was a planned feature years ago, but it never got implemented. So it exists to manage a single worker, which doesn't really add much.

We still need to provide a worker.Controller value to a few places though, which this method enables.

func ConvertToWorkerCacheResult 

func ConvertToWorkerCacheResult(ctx context.Context, res *solverresult.Result[*ref]) (*solverresult.Result[bkcache.ImmutableRef], error)

func IgnoreErrs added in v0.11.8 

func IgnoreErrs(fn func() error, ignored ...error) func() error

func Infallible added in v0.11.8 

func Infallible(fn func()) func() error

func NewSecretScrubReader added in v0.11.5 

func NewSecretScrubReader(
	r io.Reader,
	env []string,
	secretEnvs []string,
	secretFiles []string,
) (io.Reader, error)

func RunInNetNS added in v0.11.5 

func RunInNetNS[T any](
	ctx context.Context,
	c *Client,
	ns Namespaced,
	fn func() (T, error),
) (T, error)

func ToEntitlementStrings added in v0.11.8 

func ToEntitlementStrings(ents entitlements.Set) []string

Types

type BlobOp added in v0.8.8 

type BlobOp struct {
	*OpDAG
	*pb.SourceOp
}

func (*BlobOp) OCIDescriptor added in v0.8.8 

func (op *BlobOp) OCIDescriptor() (ocispecs.Descriptor, error)

type CleanupFunc added in v0.11.8 

type CleanupFunc struct {
	// contains filtered or unexported fields
}

type Cleanups added in v0.11.8 

type Cleanups struct {
	// contains filtered or unexported fields
}

func (*Cleanups) Add added in v0.11.8 

func (c *Cleanups) Add(msg string, f func() error) CleanupFunc

func (*Cleanups) ReAdd added in v0.11.8 

func (c *Cleanups) ReAdd(f CleanupFunc) CleanupFunc

ReAdd allows you to decide to run an already added cleanup function at a later time. Once readded, it will only be run at this time rather than both times. This is occasionally needed when you want to ensure some state is cleaned up right after it's created, but if more state is created later you ned to run this cleanup at that later time (e.g. closing a network connection in all cases).

func (*Cleanups) Run added in v0.11.8 

func (c *Cleanups) Run() error

type Client 

type Client struct {
	*Opts
	// contains filtered or unexported fields
}

Client is dagger's internal interface to buildkit APIs

func NewClient 

func NewClient(ctx context.Context, opts *Opts) (*Client, error)

func (*Client) BytesToBlob added in v0.11.7 

func (c *Client) BytesToBlob(
	ctx context.Context,
	fileName string,
	perms fs.FileMode,
	bs []byte,
	compressionType compression.Type,
) (_ *bksolverpb.Definition, desc specs.Descriptor, _ error)

func (*Client) CombinedResult 

func (c *Client) CombinedResult(ctx context.Context) (*Result, error)

CombinedResult returns a buildkit result with all the refs solved by this client so far. This is useful for constructing a result for upstream remote caching.

func (*Client) ContainerImageToTarball added in v0.9.0 

func (c *Client) ContainerImageToTarball(
	ctx context.Context,
	engineHostPlatform specs.Platform,
	fileName string,
	inputByPlatform map[string]ContainerExport,
	opts map[string]string,
) (*bksolverpb.Definition, error)

func (*Client) DefToBlob added in v0.9.8 

func (c *Client) DefToBlob(
	ctx context.Context,
	pbDef *bksolverpb.Definition,
	compressionType compression.Type,
) (_ *bksolverpb.Definition, desc specs.Descriptor, _ error)

DefToBlob converts the given llb definition to a content addressed blob valid for the duration of the current session. It's useful for converting unstable sources like local dir imports into stable, content-defined sources. NOTE: it's currently assumed that the provided definition is a single layer. Definitions can be squashed into a single layer by copying from them to scratch.

func (*Client) EngineContainerLocalImport added in v0.9.0 

func (c *Client) EngineContainerLocalImport(
	ctx context.Context,
	platform specs.Platform,
	srcPath string,
	excludePatterns []string,
	includePatterns []string,
) (*bksolverpb.Definition, specs.Descriptor, error)

Import a directory from the engine container, as opposed to from a client

func (*Client) ExportContainerImage 

func (c *Client) ExportContainerImage(
	ctx context.Context,
	inputByPlatform map[string]ContainerExport,
	destPath string,
	opts map[string]string,
) (map[string]string, error)

func (*Client) GetSessionCaller 

func (c *Client) GetSessionCaller(ctx context.Context, wait bool) (_ bksession.Caller, rerr error)

func (*Client) ID 

func (c *Client) ID() string

func (*Client) IOReaderExport added in v0.8.8 

func (c *Client) IOReaderExport(ctx context.Context, r io.Reader, destPath string, destMode os.FileMode) (rerr error)

IOReaderExport exports the contents of an io.Reader to the caller's local fs as a file TODO: de-dupe this with the above method to extent possible

func (*Client) ListenHostToContainer added in v0.9.0 

func (c *Client) ListenHostToContainer(
	ctx context.Context,
	hostListenAddr, proto, upstream string,
) (*session.ListenResponse, func() error, error)

func (*Client) LocalDirExport 

func (c *Client) LocalDirExport(
	ctx context.Context,
	def *bksolverpb.Definition,
	destPath string,
	merge bool,
) (rerr error)

func (*Client) LocalFileExport 

func (c *Client) LocalFileExport(
	ctx context.Context,
	def *bksolverpb.Definition,
	destPath string,
	filePath string,
	allowParentDirPath bool,
) (rerr error)

func (*Client) LocalImport added in v0.8.8 

func (c *Client) LocalImport(
	ctx context.Context,
	platform specs.Platform,
	srcPath string,
	excludePatterns []string,
	includePatterns []string,
) (*bksolverpb.Definition, specs.Descriptor, error)

func (*Client) NewContainer 

func (c *Client) NewContainer(ctx context.Context, req NewContainerRequest) (*Container, error)

func (*Client) NewNetworkNamespace added in v0.11.5 

func (c *Client) NewNetworkNamespace(ctx context.Context, hostname string) (Namespaced, error)

func (*Client) OpenTerminal added in v0.12.0 

func (c *Client) OpenTerminal(
	ctx context.Context,
) (*TerminalClient, error)

func (*Client) PublishContainerImage 

func (c *Client) PublishContainerImage(
	ctx context.Context,
	inputByPlatform map[string]ContainerExport,
	opts map[string]string,
) (map[string]string, error)

func (*Client) ReadCallerHostFile 

func (c *Client) ReadCallerHostFile(ctx context.Context, path string) ([]byte, error)

func (*Client) ResolveImageConfig 

func (c *Client) ResolveImageConfig(ctx context.Context, ref string, opt sourceresolver.Opt) (string, digest.Digest, []byte, error)

func (*Client) ResolveSourceMetadata added in v0.11.1 

func (c *Client) ResolveSourceMetadata(ctx context.Context, op *bksolverpb.SourceOp, opt sourceresolver.Opt) (*sourceresolver.MetaResponse, error)

func (*Client) Solve 

func (c *Client) Solve(ctx context.Context, req bkgw.SolveRequest) (_ *Result, rerr error)

func (*Client) StatCallerHostPath added in v0.9.9 

func (c *Client) StatCallerHostPath(ctx context.Context, path string, returnAbsPath bool) (*fsutiltypes.Stat, error)

func (*Client) UpstreamCacheExport 

func (c *Client) UpstreamCacheExport(ctx context.Context, cacheExportFuncs []ResolveCacheExporterFunc) error

type Container added in v0.11.5 

type Container struct {
	bkgw.Container
	// contains filtered or unexported fields
}

func (*Container) NamespaceID added in v0.11.5 

func (ctr *Container) NamespaceID() string

type ContainerExport 

type ContainerExport struct {
	Definition *bksolverpb.Definition
	Config     specs.ImageConfig
}

type ContainerMount added in v0.12.0 

type ContainerMount struct {
	*bkgw.Mount
	WorkerRef *bkworker.WorkerRef
}

type DiffOp added in v0.8.8 

type DiffOp struct {
	*OpDAG
	*pb.DiffOp
}

type ExecError 

type ExecError struct {
	Cmd      []string
	ExitCode int
	Stdout   string
	Stderr   string
	// contains filtered or unexported fields
}

ExecError is an error that occurred while executing an `Op_Exec`.

func (*ExecError) Error 

func (e *ExecError) Error() string

func (*ExecError) Extensions 

func (e *ExecError) Extensions() map[string]interface{}

func (*ExecError) Unwrap 

func (e *ExecError) Unwrap() error

type ExecOp added in v0.8.8 

type ExecOp struct {
	*OpDAG
	*pb.ExecOp
}

func (*ExecOp) Input added in v0.8.8 

func (exec *ExecOp) Input(i pb.InputIndex) *OpDAG

func (*ExecOp) OutputMount added in v0.8.8 

func (exec *ExecOp) OutputMount() *pb.Mount

func (*ExecOp) OutputMountBase added in v0.8.8 

func (exec *ExecOp) OutputMountBase() *OpDAG

type ExecutionMetadata added in v0.11.3 

type ExecutionMetadata struct {
	ClientID    string
	SessionID   string
	SecretToken string
	Hostname    string

	// Unique (random) ID for this execution.
	// This is used to deduplicate the same execution that gets evaluated multiple times.
	ExecID string

	// Internal execution initiated by dagger and not the user.
	// Used when executing the module runtime itself.
	Internal bool

	// TODO: can rm EncodedModuleID now
	CallID              *call.ID
	EncodedModuleID     string
	EncodedFunctionCall json.RawMessage
	CallerClientID      string

	CachePerSession bool

	// hostname -> list of aliases
	HostAliases map[string][]string

	RedirectStdoutPath string
	RedirectStderrPath string

	SecretEnvNames  []string
	SecretFilePaths []string

	SystemEnvNames []string

	EnabledGPUs []string

	SpanContext propagation.MapCarrier
}

func ExecutionMetadataFromDescription added in v0.11.5 

func ExecutionMetadataFromDescription(desc map[string]string) (*ExecutionMetadata, bool, error)

func (ExecutionMetadata) AsConstraintsOpt added in v0.11.3 

func (md ExecutionMetadata) AsConstraintsOpt() (llb.ConstraintsOpt, error)

type FileOp added in v0.8.8 

type FileOp struct {
	*OpDAG
	*pb.FileOp
}

type GitOp added in v0.8.8 

type GitOp struct {
	*OpDAG
	*pb.SourceOp
}

type HTTPOp added in v0.8.8 

type HTTPOp struct {
	*OpDAG
	*pb.SourceOp
}

type ImageOp added in v0.8.8 

type ImageOp struct {
	*OpDAG
	*pb.SourceOp
}

type LocalOp added in v0.8.8 

type LocalOp struct {
	*OpDAG
	*pb.SourceOp
}

type MergeOp added in v0.8.8 

type MergeOp struct {
	*OpDAG
	*pb.MergeOp
}

type Namespaced added in v0.11.5 

type Namespaced interface {
	NamespaceID() string
	Release(context.Context) error
}

Namespaced is something that has Linux namespaces set up. Currently this is either a full-blown container or just a raw network namespace that's setns'd into to support service tunneling and similar.

type NewContainerRequest added in v0.11.5 

type NewContainerRequest struct {
	Mounts   []ContainerMount
	Platform *bksolverpb.Platform
	Hostname string
	ExecutionMetadata
}

type NewWorkerOpts added in v0.11.3 

type NewWorkerOpts struct {
	WorkerRoot       string
	ExecutorRoot     string
	BaseWorker       *base.Worker
	TelemetryPubSub  *telemetry.PubSub
	BKSessionManager *bksession.Manager
	SessionHandler   sessionHandler

	Runc                *runc.Runc
	DefaultCgroupParent string
	ProcessMode         oci.ProcessMode
	IDMapping           *idtools.IdentityMapping
	DNSConfig           *oci.DNSConfig
	ApparmorProfile     string
	SELinux             bool
	Entitlements        entitlements.Set
	NetworkProviders    map[pb.NetMode]network.Provider
	ParallelismSem      *semaphore.Weighted
	WorkerCache         bkcache.Manager
}

type OCIOp added in v0.8.8 

type OCIOp struct {
	*OpDAG
	*pb.SourceOp
}

type OpDAG added in v0.8.8 

type OpDAG struct {
	*pb.Op                  // the root of the DAG
	OpDigest *digest.Digest // the digest of this root, common across all outputIndexes for this root
	Metadata *pb.OpMetadata // metadata for the root
	Inputs   []*OpDAG       // the inputs to the root
	// contains filtered or unexported fields
}

func DefToDAG added in v0.8.8 

func DefToDAG(def *pb.Definition) (*OpDAG, error)

func (*OpDAG) AsBlob added in v0.8.8 

func (dag *OpDAG) AsBlob() (*BlobOp, bool)

func (*OpDAG) AsDiff added in v0.8.8 

func (dag *OpDAG) AsDiff() (*DiffOp, bool)

func (*OpDAG) AsExec added in v0.8.8 

func (dag *OpDAG) AsExec() (*ExecOp, bool)

func (*OpDAG) AsFile added in v0.8.8 

func (dag *OpDAG) AsFile() (*FileOp, bool)

func (*OpDAG) AsGit added in v0.8.8 

func (dag *OpDAG) AsGit() (*GitOp, bool)

func (*OpDAG) AsHTTP added in v0.8.8 

func (dag *OpDAG) AsHTTP() (*HTTPOp, bool)

func (*OpDAG) AsImage added in v0.8.8 

func (dag *OpDAG) AsImage() (*ImageOp, bool)

func (*OpDAG) AsLocal added in v0.8.8 

func (dag *OpDAG) AsLocal() (*LocalOp, bool)

func (*OpDAG) AsMerge added in v0.8.8 

func (dag *OpDAG) AsMerge() (*MergeOp, bool)

func (*OpDAG) AsOCI added in v0.8.8 

func (dag *OpDAG) AsOCI() (*OCIOp, bool)

func (*OpDAG) BlobDependencies added in v0.8.8 

func (dag *OpDAG) BlobDependencies() (map[digest.Digest]*ocispecs.Descriptor, error)

func (*OpDAG) Marshal added in v0.8.8 

func (dag *OpDAG) Marshal() (*pb.Definition, error)

Marshal will convert the dag back to a flat pb.Definition, updating all digests based on any modifications made to the dag.

func (*OpDAG) String added in v0.8.8 

func (dag *OpDAG) String() string

func (*OpDAG) Walk added in v0.8.8 

func (dag *OpDAG) Walk(f func(*OpDAG) error) error

type Opts 

type Opts struct {
	Worker               *Worker
	SessionManager       *bksession.Manager
	BkSession            *bksession.Session
	LLBBridge            bkfrontend.FrontendLLBBridge
	Dialer               *net.Dialer
	GetMainClientCaller  func() (bksession.Caller, error)
	Entitlements         entitlements.Set
	UpstreamCacheImports []bkgw.CacheOptionsEntry
	Frontends            map[string]bkfrontend.Frontend

	Refs         map[Reference]struct{}
	RefsMu       *sync.Mutex
	Containers   map[bkgw.Container]struct{}
	ContainersMu *sync.Mutex

	Interactive bool
}

Opts for a Client that are shared across all instances for a given DaggerServer

type Reference added in v0.11.8 

type Reference interface {
	bkgw.Reference
	Release(context.Context) error
}

type ResolveCacheExporterFunc 

type ResolveCacheExporterFunc func(ctx context.Context, g bksession.Group) (remotecache.Exporter, error)

type Result 

type Result = solverresult.Result[*ref]

type SpanProcessor added in v0.12.0 

type SpanProcessor struct{}

SpanProcessor modifies spans coming from the Buildkit component to integrate them with Dagger's telemetry stack.

It must be used in combination with the buildkitTraceProvider.

func (SpanProcessor) ForceFlush added in v0.12.0 

func (sp SpanProcessor) ForceFlush(context.Context) error

func (SpanProcessor) OnEnd added in v0.12.0 

func (sp SpanProcessor) OnEnd(sdktrace.ReadOnlySpan)

func (SpanProcessor) OnStart added in v0.12.0 

func (sp SpanProcessor) OnStart(ctx context.Context, span sdktrace.ReadWriteSpan)

func (SpanProcessor) Shutdown added in v0.12.0 

func (sp SpanProcessor) Shutdown(context.Context) error

type TerminalClient added in v0.12.0 

type TerminalClient struct {
	Stdin    io.ReadCloser
	Stdout   io.WriteCloser
	Stderr   io.WriteCloser
	ResizeCh chan bkgw.WinSize
	ErrCh    chan error
	Close    func(exitCode int) error
}

type Trie added in v0.11.5 

type Trie struct {
	// contains filtered or unexported fields
}

Trie is a simple implementation of a compressed trie (or radix tree). In essence, it's a key-value store that allows easily selecting all entries that have a given prefix.

Why not an off-the-shelf implementation? Well, most of those don't allow navigating character-by-character through the tree, like we do with Step.

func (*Trie) Insert added in v0.11.5 

func (t *Trie) Insert(key []byte, value []byte)

func (*Trie) Step added in v0.11.5 

func (t *Trie) Step(ch byte) *Trie

Step selects a node that was previously inserted.

func (Trie) String added in v0.11.5 

func (t Trie) String() string

String prints a debuggable representation of the trie.

func (*Trie) Value added in v0.11.5 

func (t *Trie) Value() []byte

Value gets the value previously inserted at this node.

type Worker added in v0.11.3 

type Worker struct {
	// contains filtered or unexported fields
}

Worker is Dagger's custom worker. Most of the buildkit Worker interface methods are just inherited from buildkit's base.Worker, with the exception of methods involving executor.Executor (most importantly ResolveOp).

We need a custom Executor implementation for setting up containers (currently, just for providing SessionID, but in the future everything the shim does will be migrated here). For simplicity, this Worker struct also implements that Executor interface (in executor.go) since Worker+Executor are so tightly bound together anyways.

func NewWorker added in v0.11.3 

func NewWorker(opts *NewWorkerOpts) *Worker

func (*Worker) Exec added in v0.11.3 

func (w *Worker) Exec(ctx context.Context, id string, process executor.ProcessInfo) (err error)

func (*Worker) Executor added in v0.11.3 

func (w *Worker) Executor() executor.Executor

func (*Worker) ResolveOp added in v0.11.3 

func (w *Worker) ResolveOp(vtx solver.Vertex, s frontend.FrontendLLBBridge, sm *bksession.Manager) (solver.Op, error)

func (*Worker) Run added in v0.11.3 

func (w *Worker) Run(
	ctx context.Context,
	id string,
	rootMount executor.Mount,
	mounts []executor.Mount,
	procInfo executor.ProcessInfo,
	started chan<- struct{},
) (_ resourcestypes.Recorder, rerr error)

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.