schema

README

Core API proposal

This directory contains a proposal for a complete GraphQL API for Dagger. It is written with the following goals in mind:

1. Feature parity with Dagger 0.2
2. Close to feature parity with Buildkit, with an incremental path to reaching full parity in the future
3. Follow established graphql best practices
4. Sove as many outstanding DX problems as possible

Reference

• Environment
• Container
• Directory
• File
• Git
• HTTP
• Secret

DX problems solved

Some problems in the DX that are not yet resolved, and this proposal would help solve, include:

• Uncertainty as to how to express uploads in a graphql-friendly way (deployment, image push, git push, etc)
• Chaining of FS operations greatly reduces verbosity, but cannot be applied all the time
• Transitioning a script to an extension requires non-trivial refactoring, to tease apart the script-specific code from the underlying "API".
• The API sandbox is amazing for prototyping small queries, but of limited use when multiple queries must reference each other. This is because the native code doing the stitching cannot be run by the playground, so filesystem IDs must be manually copy-pasted between queries.

Design highlights

withX and withoutX

To avoid the use of rpc-style verbs (a graphql best practice) and maximize chaining (a strength of our DX), we use the terminology withX and withoutX.

A field of the form withX returns the same object with content X added or changed.

Example:

"An empty directory with a README copied to it"
query readmeDir($readme: FileID!) {
  directory {
    withFile(source: $readme, path: "README.md") {
      id
    }
}

"An empty container with an app directory mounted into it"
query appContainer($app: DirectoryID!) {
  container {
    withMountedDirectory(source: $app, path: "/app") {
      id
    }
  }
}

A field of the form withoutX returns the same object with content X removed.

"Remove node_modules from a JS project"
query removeNodeModules($dir: DirectoryID!) {
  directory(id: $dir) {
    withoutDirectory(path: "node_modules") {
      id
    }
  }
}

Secrets

Secret handling has been simplified and made more consistent with Directory handling.

• Secrets have an ID, and can be loaded by ID in the standard graphql manner
• Secrets can be created in one of two ways:
  1. From an environment variable: type Environment { secret }
  2. From a file: type Directory { secret }

Embrace the llb / Dockerfile model

The Container type proposes an expansive definition of the container, similar to the Buildkit/Dockerfile model. A Container is:

1. A filesystem state
2. An OCI artifact which can be pulled from, and pushed to a repository at any time
3. A persistent configuration to be applied to all commands executed inside it

This is similar to how buildkit models llb state, and how the Dockerfile language models stage state. Note that Dagger extends this model to include even mount configuration (which are scoped to exec in buildkit, but scoped to container in dagger).

Examples:

"""
Download a file over HTTP in a very convoluted way:

1. Download a base linux container
2. Install curl
3. Download the file into the container
4. Load and return the file
"""
query convolutedDownload($url: String!) {
  container {
    from(address: "index.docker.io/alpine:latest") {
      exec(args: ["apk", "add", "curl"]) {
        exec(args: ["curl", "-o", "/tmp/download", $url) {
          file(path: "/tmp/download") {
            id
         }
      }
    }
  }
}

"""
Specialize two containers from a common base
"""
query twoContainers {
  container {
    from(address: "alpine") {
      debug: withVariable(name: "DEBUG", value: "1") {
        id
        exec(args: ["env"]) {
          stdout
        }
      }
      noDebug: withVariable(name: "DEBUG", value: "0") {
        id
        exec(args: ["env"]) {
          stdout
        }
      }
    }
  }
}

Documentation

Index

• Constants
• Variables
• func ErrResolver(err error) graphql.FieldResolveFn
• func PassthroughResolver(p graphql.ResolveParams) (any, error)
• func ResolveIDable[T any](rs Resolvers, name string, obj ObjectResolver)
• func ToResolver[P any, A any, R any](f func(*core.Context, P, A) (R, error)) graphql.FieldResolveFn
• func ToVoidResolver[P any, A any](f func(*core.Context, P, A) error) graphql.FieldResolveFn
• type EnvVariable
• type ExecutableSchema
  • func StaticSchema(p StaticSchemaParams) ExecutableSchema
• type ExposedPort
• type FieldResolvers
• type FunctionContext
  • func (fnCtx *FunctionContext) Digest() (digest.Digest, error)
• type FunctionContextCache
  • func NewFunctionContextCache() *FunctionContextCache
  • func (cache *FunctionContextCache) FunctionContextFrom(ctx context.Context) (*FunctionContext, error)
  • func (cache *FunctionContextCache) WithFunctionContext(ctx *core.Context, fnCtx *FunctionContext) (*core.Context, error)
• type Handler
  • func NewHandler(p *HandlerConfig) *Handler
  • func (h *Handler) ContextHandler(ctx context.Context, w http.ResponseWriter, r *http.Request)
  • func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request)
• type HandlerConfig
  • func NewConfig() *HandlerConfig
• type IDableObjectResolver
  • func ToIDableObjectResolver[T any, I ~string](idToObject func(I) (*T, error), r ObjectResolver) IDableObjectResolver
• type InitializeArgs
• type InvalidInputError
  • func (e InvalidInputError) Error() string
  • func (e InvalidInputError) Unwrap() error
• type Label
• type MergedSchemas
  • func New(params InitializeArgs) (*MergedSchemas, error)
  • func (s *MergedSchemas) HTTPHandler(moduleDigest digest.Digest) (http.Handler, error)
  • func (s *MergedSchemas) MuxEndpoint(path string, handler http.Handler, moduleDigest digest.Digest) error
  • func (s *MergedSchemas) Schema(moduleDigest digest.Digest) (*graphql.Schema, error)
  • func (s *MergedSchemas) ShutdownClient(ctx context.Context, client *engine.ClientMetadata) error
• type ObjectResolver
  • func (r ObjectResolver) Fields() map[string]graphql.FieldResolveFn
  • func (r ObjectResolver) SetField(name string, fn graphql.FieldResolveFn)
• type RequestOptions
  • func NewRequestOptions(r *http.Request) *RequestOptions
• type Resolver
• type Resolvers
• type ResultCallbackFn
• type RootObjectFn
• type SDK
• type ScalarResolver
• type SecretPlaintext
  • func (s SecretPlaintext) MarshalText() ([]byte, error)
• type StaticSchemaParams

Constants

const (
	ContentTypeJSON           = "application/json"
	ContentTypeGraphQL        = "application/graphql"
	ContentTypeFormURLEncoded = "application/x-www-form-urlencoded"
)

Variables

var (
	ErrMergeTypeConflict   = errors.New("object type re-defined")
	ErrMergeFieldConflict  = errors.New("field re-defined")
	ErrMergeScalarConflict = errors.New("scalar re-defined")
)

var Cache string

var Container string

var Directory string

var File string

var Function string

var Git string

var HTTP string

var Host string

var InternalSDK string

var Module string

var Platform string

var Query string

var Secret string

var Service string

var Socket string

Functions

func ErrResolver

func ErrResolver(err error) graphql.FieldResolveFn

func PassthroughResolver

func PassthroughResolver(p graphql.ResolveParams) (any, error)

func ResolveIDable

func ResolveIDable[T any](rs Resolvers, name string, obj ObjectResolver)

func ToResolver

func ToResolver[P any, A any, R any](f func(*core.Context, P, A) (R, error)) graphql.FieldResolveFn

ToResolver transforms any function f with a *Context, a parent P and some args A that returns a Response R and an error into a graphql resolver graphql.FieldResolveFn.

func ToVoidResolver

func ToVoidResolver[P any, A any](f func(*core.Context, P, A) error) graphql.FieldResolveFn

Types

type EnvVariable

type EnvVariable struct {
	Name  string `json:"name"`
	Value string `json:"value"`
}

type ExecutableSchema

type ExecutableSchema interface {
	Name() string
	Schema() string
	Resolvers() Resolvers
	Dependencies() []ExecutableSchema
}

func StaticSchema

func StaticSchema(p StaticSchemaParams) ExecutableSchema

type ExposedPort

type ExposedPort struct {
	Port        int     `json:"port"`
	Protocol    string  `json:"protocol"`
	Description *string `json:"description,omitempty"`
}

NB(vito): we have to use a different type with a regular string Protocol field so that the enum mapping works.

type FieldResolvers

type FieldResolvers interface {
	Resolver
	Fields() map[string]graphql.FieldResolveFn
	SetField(string, graphql.FieldResolveFn)
}

type FunctionContext

type FunctionContext struct {
	Module      *core.Module
	CurrentCall *core.FunctionCall
}

FunctionContext holds the metadata of a function call. Used to support the currentModule and currentFunctionCall APIs.

func (*FunctionContext) Digest

func (fnCtx *FunctionContext) Digest() (digest.Digest, error)

type FunctionContextCache

type FunctionContextCache core.CacheMap[digest.Digest, *FunctionContext]

FunctionContextCache stores the mapping of FunctionContext's digest -> FunctionContext. This enables us to pass just the digest along the client metadata rather than massive serialized objects.

func NewFunctionContextCache

func NewFunctionContextCache() *FunctionContextCache

func (*FunctionContextCache) FunctionContextFrom

func (cache *FunctionContextCache) FunctionContextFrom(ctx context.Context) (*FunctionContext, error)

func (*FunctionContextCache) WithFunctionContext

func (cache *FunctionContextCache) WithFunctionContext(ctx *core.Context, fnCtx *FunctionContext) (*core.Context, error)

type Handler

type Handler struct {
	Schema *graphql.Schema
	// contains filtered or unexported fields
}

func NewHandler

func NewHandler(p *HandlerConfig) *Handler

func (*Handler) ContextHandler

func (h *Handler) ContextHandler(ctx context.Context, w http.ResponseWriter, r *http.Request)

ContextHandler provides an entrypoint into executing graphQL queries with a user-provided context.

func (*Handler) ServeHTTP

func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP provides an entrypoint into executing graphQL queries.

type HandlerConfig

type HandlerConfig struct {
	Schema           *graphql.Schema
	Pretty           bool
	RootObjectFn     RootObjectFn
	ResultCallbackFn ResultCallbackFn
	FormatErrorFn    func(err error) gqlerrors.FormattedError
}

func NewConfig

func NewConfig() *HandlerConfig

type IDableObjectResolver

type IDableObjectResolver interface {
	FromID(id string) (any, error)
	ToID(any) (string, error)
	Resolver
}

func ToIDableObjectResolver

func ToIDableObjectResolver[T any, I ~string](idToObject func(I) (*T, error), r ObjectResolver) IDableObjectResolver

type InitializeArgs

type InitializeArgs struct {
	BuildkitClient *buildkit.Client
	Platform       specs.Platform
	ProgSockPath   string
	OCIStore       content.Store
	LeaseManager   *leaseutil.Manager
	Auth           *auth.RegistryAuthProvider
	Secrets        *core.SecretStore
}

type InvalidInputError

type InvalidInputError struct {
	Err error
}

func (InvalidInputError) Error

func (e InvalidInputError) Error() string

func (InvalidInputError) Unwrap

func (e InvalidInputError) Unwrap() error

type Label

type Label struct {
	Name  string `json:"name"`
	Value string `json:"value"`
}

type MergedSchemas

type MergedSchemas struct {
	// contains filtered or unexported fields
}

func New

func New(params InitializeArgs) (*MergedSchemas, error)

func (*MergedSchemas) HTTPHandler

func (s *MergedSchemas) HTTPHandler(moduleDigest digest.Digest) (http.Handler, error)

func (*MergedSchemas) MuxEndpoint

func (s *MergedSchemas) MuxEndpoint(path string, handler http.Handler, moduleDigest digest.Digest) error

func (*MergedSchemas) Schema

func (s *MergedSchemas) Schema(moduleDigest digest.Digest) (*graphql.Schema, error)

func (*MergedSchemas) ShutdownClient

func (s *MergedSchemas) ShutdownClient(ctx context.Context, client *engine.ClientMetadata) error

type ObjectResolver

type ObjectResolver map[string]graphql.FieldResolveFn

func (ObjectResolver) Fields

func (r ObjectResolver) Fields() map[string]graphql.FieldResolveFn

func (ObjectResolver) SetField

func (r ObjectResolver) SetField(name string, fn graphql.FieldResolveFn)

type RequestOptions

type RequestOptions struct {
	Query         string                 `json:"query" url:"query" schema:"query"`
	Variables     map[string]interface{} `json:"variables" url:"variables" schema:"variables"`
	OperationName string                 `json:"operationName" url:"operationName" schema:"operationName"`
}

func NewRequestOptions

func NewRequestOptions(r *http.Request) *RequestOptions

RequestOptions Parses a http.Request into GraphQL request options struct

type Resolver

type Resolver interface {
	// contains filtered or unexported methods
}

type Resolvers

type Resolvers map[string]Resolver

type ResultCallbackFn

type ResultCallbackFn func(ctx context.Context, params *graphql.Params, result *graphql.Result, responseBody []byte)

type RootObjectFn

type RootObjectFn func(ctx context.Context, r *http.Request) map[string]interface{}

RootObjectFn allows a user to generate a RootObject per request

type SDK

type SDK interface {
	/* Codegen generates code for the module at the given source directory and subpath.

	The Code field of the returned GeneratedCode object should be the generated contents of the module sourceDirSubpath,
	in the case where that's different than the root of the sourceDir.
	*/
	Codegen(ctx *core.Context, sourceDir *core.Directory, sourceDirSubpath string) (*core.GeneratedCode, error)

	// Runtime returns a container that is used to execute module code at runtime in the Dagger engine.
	Runtime(ctx *core.Context, sourceDir *core.Directory, sourceDirSubpath string) (*core.Container, error)
}

An SDK is an implementation of the functionality needed to generate code for and execute a module.

There is one special SDK, the Go SDK, which is implemented in `goSDK` below. It's used as the "seed" for all other SDK implementations.

All other SDKs are themselves implemented as Modules, with Functions matching the two defined in this SDK interface.

An SDK Module needs to choose its own SDK for its implementation. This can be "well-known" built-in SDKs like "go", "python", etc. Or it can be any external module as specified with a module ref.

You can thus think of SDK Modules as a DAG of dependencies, with each SDK using a different SDK to implement its Module, with the Go SDK as the root of the DAG and the only one without any dependencies.

Built-in SDKs are also a bit special in that they come bundled w/ the engine container image, which allows them to be used without hard dependencies on the internet. They are loaded w/ the `loadBuiltinSDK` function below, which loads them as modules from the engine container.

type ScalarResolver

type ScalarResolver struct {
	Serialize    graphql.SerializeFn
	ParseValue   graphql.ParseValueFn
	ParseLiteral graphql.ParseLiteralFn
}

type SecretPlaintext

type SecretPlaintext string

func (SecretPlaintext) MarshalText

func (s SecretPlaintext) MarshalText() ([]byte, error)

This method ensures that the progrock vertex info does not display the plaintext.

type StaticSchemaParams

type StaticSchemaParams struct {
	Name         string
	Schema       string
	Resolvers    Resolvers
	Dependencies []ExecutableSchema
}