config

package
v0.52.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 19, 2024 License: Apache-2.0 Imports: 32 Imported by: 26

README

Authentication

Auth flow

sequenceDiagram
    ClustersAPI->>+DatabricksClient: GET .../clusters/list
    DatabricksClient->>+databricks.Config: Authenticate(HttpRequest)

    databricks.Config-->>+DefaultCredentials: Configure(databricks.Config)
    DefaultCredentials-->>+FirstCredentials: try configure
    FirstCredentials-->>-DefaultCredentials: try next
    DefaultCredentials->>+NextCredentials: try configure
    NextCredentials->>RequestVisitor: configured auth
    NextCredentials->>-DefaultCredentials: authenticated
    DefaultCredentials->>-databricks.Config: set AuthType & request visitor

    databricks.Config->>+RequestVisitor: visit HTTP request
    RequestVisitor-->>+IdentityProvider: ensure fresh token
    IdentityProvider-->>-RequestVisitor: access token
    RequestVisitor->>-databricks.Config: added HTTP headers
    
    databricks.Config->>-DatabricksClient: added HTTP headers

    DatabricksClient->>+API: authenticated request
    API->>-DatabricksClient: JSON payload
    DatabricksClient->>-ClustersAPI: ClustersList or error

Client configuration

classDiagram
    Loader "0..n" <-- Config: Configure(self)
    Credentials "0..1" <-- Config: Configure(self)
    RequestVisitor --* Config: configured auth
    class Config {
        * Host string
        * Token string
        * Profile string
        * Username string
        * Password string
        * AzureResourceID string
        * AzureEnvironment string
        * AzureClientID string
        * AzureSecretID string
        * AzureTenantID string
        * GoogleServiceAccount string

        Credentials: DefaultCredentials
        Loaders: Loader

        Authenticate(HttpRequest) error
    }

    class Loader {
        <<interface>>
        Name() string
        Configure(Config) error
    }

    KnownConfigLoader ..|> Loader
    class KnownConfigLoader

    ConfigAttributes ..|> Loader
    class ConfigAttributes {
        Configure(Config) error
        DebugString(Config) string
        Validate(Config) error
        ResolveFromStringMap(Config, map) error
        ResolveFromAnyMap(Config, map) error
    }

    Config --* DatabricksClient
    class DatabricksClient {
        Config
        - retryPolicy
        
        Get(path, query) T
        Post(path, body) T
        Put(path, body) T
        Patch(path, body) T
        Delete(path, query) T
    }

    Credentials --> "0..1" RequestVisitor: creates
    class Credentials {
        <<interface>>
        Name() string
        Configure(Config) RequestVisitor
    }

    class RequestVisitor {
        <<interface>>
        Visit(HttpRequest) error
    }

    AzureSpnCredentials --* authProviders
    AzureSpnCredentials ..|> Credentials
    class AzureSpnCredentials

    AzureCliCredentials --* authProviders
    AzureCliCredentials ..|> Credentials

    GoogleCredentials --* authProviders
    GoogleCredentials ..|> Credentials
    class GoogleCredentials
    
    DatabricksOauthCredentials --* authProviders
    DatabricksOauthCredentials ..|> Credentials
    class DatabricksOauthCredentials {
        []Scopes
    }

    PatCredentials --* authProviders
    PatCredentials ..|> Credentials
    class PatCredentials

    BasicCredentials --* authProviders
    BasicCredentials ..|> Credentials
    class BasicCredentials

    authProviders --> DefaultCredentials: for reach ConfigAttributes()
    DefaultCredentials ..|> Credentials
    class DefaultCredentials

Documentation

Index

Examples

Constants

View Source
const MetadataServiceHostHeader = "X-Databricks-Host"
View Source
const MetadataServiceVersion = "1"
View Source
const MetadataServiceVersionHeader = "X-Databricks-Metadata-Version"

Variables

View Source
var ConfigAttributes = loadAttrs()
View Source
var ConfigFile = configFileLoader{}
View Source
var ErrCannotConfigureAuth = errors.New(errorMessage)

ErrCannotConfigureAuth (experimental) is returned when no auth is configured

View Source
var ErrNoHostConfigured = fmt.Errorf("no host configured")

ErrNoHostConfigured is the error returned when a user tries to authenticate without a host configured. Applications can check for this error to provide more user-friendly error messages.

Functions

func NewAzureCliTokenSource added in v0.26.2

func NewAzureCliTokenSource(ctx context.Context, resource, azureTenantId string) oauth2.TokenSource

NewAzureCliTokenSource returns oauth2.TokenSource for a passwordless authentication via Azure CLI (`az login`)

func NewAzureMsiTokenSource added in v0.26.2

func NewAzureMsiTokenSource(client *httpclient.ApiClient, resource, clientId string) oauth2.TokenSource

NewAzureMsiTokenSource returns oauth2.TokenSource for a passwordless authentication via Azure Managed identity

Types

type AttrConfig added in v0.35.0

type AttrConfig struct {
	Value            string `json:"value"`
	Source           Source `json:"source"`
	AuthTypeMismatch bool   `json:"auth_type_mismatch"`
}

func (*AttrConfig) String added in v0.35.0

func (a *AttrConfig) String() string

type AuthConfiguration added in v0.35.0

type AuthConfiguration map[string]*AttrConfig

AuthConfiguration is a map of attribute name to its configuration.

func (AuthConfiguration) String added in v0.35.0

func (c AuthConfiguration) String() string

type AuthDetails added in v0.35.0

type AuthDetails struct {
	AuthType      string            `json:"auth_type"`
	Host          string            `json:"host,omitempty"`
	Configuration AuthConfiguration `json:"configuration"`
}

AuthDetails contains the details of the authentication configuration.

func (*AuthDetails) String added in v0.35.0

func (a *AuthDetails) String() string

type AuthDetailsOptions added in v0.35.0

type AuthDetailsOptions int
const (
	ShowSensitive AuthDetailsOptions = iota
)

type AuthType added in v0.38.0

type AuthType struct{}

type AzureCliCredentials

type AzureCliCredentials struct {
}

func (AzureCliCredentials) Configure

func (AzureCliCredentials) Name

func (c AzureCliCredentials) Name() string

type AzureClientSecretCredentials

type AzureClientSecretCredentials struct {
}

func (AzureClientSecretCredentials) Configure

TODO: We need to expose which authentication mechanism is used to Terraform, as we cannot create AKV backed secret scopes when authenticated as SP. If we are authenticated as SP and wish to create one we want to fail early. Also see https://github.com/databricks/terraform-provider-databricks/issues/1490.

func (AzureClientSecretCredentials) Name

type AzureGithubOIDCCredentials added in v0.43.1

type AzureGithubOIDCCredentials struct{}

AzureGithubOIDCCredentials provides credentials for GitHub Actions that use an Azure Active Directory Federated Identity to authenticate with Azure.

func (AzureGithubOIDCCredentials) Configure added in v0.43.1

Configure implements [CredentialsStrategy.Configure].

func (AzureGithubOIDCCredentials) Name added in v0.43.1

Name implements [CredentialsStrategy.Name].

type AzureMsiCredentials added in v0.3.0

type AzureMsiCredentials struct {
}

func (AzureMsiCredentials) Configure added in v0.3.0

func (AzureMsiCredentials) Name added in v0.3.0

func (c AzureMsiCredentials) Name() string

type BasicCredentials

type BasicCredentials struct {
}

func (BasicCredentials) Configure

func (BasicCredentials) Name

func (c BasicCredentials) Name() string

type Config

type Config struct {
	// Credentials holds an instance of Credentials Strategy to authenticate with Databricks REST APIs.
	// If no credentials strategy is specified, `DefaultCredentials` are implicitly used.
	Credentials CredentialsStrategy

	// Databricks host (either of workspace endpoint or Accounts API endpoint)
	Host string `name:"host" env:"DATABRICKS_HOST"`

	ClusterID           string `name:"cluster_id" env:"DATABRICKS_CLUSTER_ID"`
	WarehouseID         string `name:"warehouse_id" env:"DATABRICKS_WAREHOUSE_ID"`
	ServerlessComputeID string `name:"serverless_compute_id" env:"DATABRICKS_SERVERLESS_COMPUTE_ID"`

	// URL of the metadata service that provides authentication credentials.
	MetadataServiceURL string `name:"metadata_service_url" env:"DATABRICKS_METADATA_SERVICE_URL" auth:"metadata-service,sensitive"`

	// Databricks Account ID for Accounts API. This field is used in dependencies.
	AccountID string `name:"account_id" env:"DATABRICKS_ACCOUNT_ID"`

	Token    string `name:"token" env:"DATABRICKS_TOKEN" auth:"pat,sensitive"`
	Username string `name:"username" env:"DATABRICKS_USERNAME" auth:"basic"`
	Password string `name:"password" env:"DATABRICKS_PASSWORD" auth:"basic,sensitive"`

	// Connection profile specified within ~/.databrickscfg.
	Profile string `name:"profile" env:"DATABRICKS_CONFIG_PROFILE"`

	// Location of the Databricks CLI credentials file, that is created
	// by `databricks configure --token` command. By default, it is located
	// in ~/.databrickscfg.
	ConfigFile string `name:"config_file" env:"DATABRICKS_CONFIG_FILE"`

	GoogleServiceAccount string `name:"google_service_account" env:"DATABRICKS_GOOGLE_SERVICE_ACCOUNT" auth:"google" auth_types:"google-id"`
	GoogleCredentials    string `name:"google_credentials" env:"GOOGLE_CREDENTIALS" auth:"google,sensitive" auth_types:"google-credentials"`

	// Azure Resource Manager ID for Azure Databricks workspace, which is exhanged for a Host
	AzureResourceID string `name:"azure_workspace_resource_id" env:"DATABRICKS_AZURE_RESOURCE_ID" auth:"azure" auth_types:"azure-cli,azure-msi"`

	AzureUseMSI       bool   `name:"azure_use_msi" env:"ARM_USE_MSI" auth:"azure" auth_types:"azure-msi"`
	AzureClientSecret string `name:"azure_client_secret" env:"ARM_CLIENT_SECRET" auth:"azure,sensitive" auth_types:"azure-client-secret"`
	AzureClientID     string `name:"azure_client_id" env:"ARM_CLIENT_ID" auth:"azure" auth_types:"azure-client-secret,azure-msi"`
	AzureTenantID     string `name:"azure_tenant_id" env:"ARM_TENANT_ID" auth:"azure" auth_types:"azure-cli,azure-client-secret"`

	// Parameters to request Azure OIDC token on behalf of Github Actions.
	// Ref: https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-cloud-providers
	ActionsIDTokenRequestURL   string `name:"actions_id_token_request_url" env:"ACTIONS_ID_TOKEN_REQUEST_URL"`
	ActionsIDTokenRequestToken string `name:"actions_id_token_request_token" env:"ACTIONS_ID_TOKEN_REQUEST_TOKEN"`

	// AzureEnvironment (PUBLIC, USGOVERNMENT, CHINA) has specific set of API endpoints. Starting from v0.26.0,
	// the environment is determined based on the workspace hostname, if it's specified.
	AzureEnvironment string `name:"azure_environment" env:"ARM_ENVIRONMENT"`

	// Azure Login Application ID. Must be set if authenticating for non-production workspaces. Starting from v0.26.0,
	// the correct Azure Login App ID is determined based on the Azure Databricks Workspace hostname.
	//
	// Deprecated: this configuration property no longer has any effect and will be removed in the future
	// versions of Go SDK.
	AzureLoginAppID string `name:"azure_login_app_id" env:"DATABRICKS_AZURE_LOGIN_APP_ID" auth:"azure"`

	ClientID     string `name:"client_id" env:"DATABRICKS_CLIENT_ID" auth:"oauth" auth_types:"oauth-m2m"`
	ClientSecret string `name:"client_secret" env:"DATABRICKS_CLIENT_SECRET" auth:"oauth,sensitive" auth_types:"oauth-m2m"`

	// Path to the Databricks CLI (version >= 0.100.0).
	DatabricksCliPath string `name:"databricks_cli_path" env:"DATABRICKS_CLI_PATH" auth_types:"databricks-cli"`

	// When multiple auth attributes are available in the environment, use the auth type
	// specified by this argument. This argument also holds currently selected auth.
	AuthType string `name:"auth_type" env:"DATABRICKS_AUTH_TYPE" auth:"-"`

	// Skip SSL certificate verification for HTTP calls.
	// Use at your own risk or for unit testing purposes.
	InsecureSkipVerify bool `name:"skip_verify" auth:"-"`

	// Number of seconds for HTTP timeout. Default is 60 (1 minute).
	HTTPTimeoutSeconds int `name:"http_timeout_seconds" auth:"-"`

	// Truncate JSON fields in JSON above this limit. Default is 96.
	DebugTruncateBytes int `name:"debug_truncate_bytes" env:"DATABRICKS_DEBUG_TRUNCATE_BYTES" auth:"-"`

	// Debug HTTP headers of requests made by the provider. Default is false.
	DebugHeaders bool `name:"debug_headers" env:"DATABRICKS_DEBUG_HEADERS" auth:"-"`

	// Maximum number of requests per second made to Databricks REST API. Default is 15 RPS.
	RateLimitPerSecond int `name:"rate_limit" env:"DATABRICKS_RATE_LIMIT" auth:"-"`

	// Number of seconds to keep retrying HTTP requests. Default is 300 (5 minutes).
	// If negative, the client will retry on retriable errors indefinitely.
	RetryTimeoutSeconds int `name:"retry_timeout_seconds" auth:"-"`

	// HTTPTransport can be overriden for unit testing and together with tooling like https://github.com/google/go-replayers
	HTTPTransport http.RoundTripper

	// Environment override to return when resolving the current environment.
	DatabricksEnvironment *environment.DatabricksEnvironment

	Loaders []Loader
	// contains filtered or unexported fields
}

Config represents configuration for Databricks Connectivity

Example (Accounts)
databricks.Must(databricks.NewWorkspaceClient(&databricks.Config{
	Host:      "https://accounts.cloud.databricks.com", // env: DATABRICKS_HOST
	AccountID: "00000000-0000-0000-0000-111122223333",  // env: DATABRICKS_ACCOUNT_ID
	Username:  "me@example.com",                        // env: DATABRICKS_USERNAME
	Password:  "som3thing!S@cret",                      // env: DATABRICKS_PASSWORD
}))
Output:

Example (AzureActiveDirectoryServicePrincipal)
databricks.Must(databricks.NewWorkspaceClient(&databricks.Config{
	Host:              "https://adb-123.4.azuredatabricks.net", // env: DATABRICKS_HOST
	AzureResourceID:   "/subscriptions/../resourceGroups/...",  // env: DATABRICKS_AZURE_RESOURCE_ID
	AzureTenantID:     "00000000-0000-0000-0000-111122223334",  // env: ARM_TENANT_ID
	AzureClientID:     "00000000-0000-0000-0000-111122223335",  // env: ARM_CLIENT_ID
	AzureClientSecret: "som3thing!S@cret",                      // env: ARM_CLIENT_SECRET
}))
Output:

Example (Basic)
databricks.Must(databricks.NewWorkspaceClient(&databricks.Config{
	Host:     "https://abc.cloud.databricks.com", // env: DATABRICKS_HOST
	Username: "me@example.com",                   // env: DATABRICKS_USERNAME
	Password: "som3thing!S@cret",                 // env: DATABRICKS_PASSWORD
}))
Output:

Example (CustomConfigFile)
databricks.Must(databricks.NewWorkspaceClient(&databricks.Config{
	ConfigFile: "/path/to/.databrickscfg", // env: DATABRICKS_CONFIG_FILE
}))
Output:

Example (CustomProfile)
databricks.Must(databricks.NewWorkspaceClient(&databricks.Config{
	Profile: "production", // env: DATABRICKS_CONFIG_PROFILE
}))
Output:

Example (Debugging)
databricks.Must(databricks.NewWorkspaceClient(&databricks.Config{
	HTTPTimeoutSeconds:  60,
	DebugTruncateBytes:  96,    // env: DATABRICKS_DEBUG_TRUNCATE_BYTES
	DebugHeaders:        false, // env: DATABRICKS_DEBUG_HEADERS
	RateLimitPerSecond:  15,    // env: DATABRICKS_RATE_LIMIT
	RetryTimeoutSeconds: 300,
}))
Output:

Example (ForceAzureActiveDirectoryServicePrincipal)
databricks.Must(databricks.NewWorkspaceClient(&databricks.Config{
	Host:              "https://adb-123.4.azuredatabricks.net", // env: DATABRICKS_HOST
	AzureResourceID:   "/subscriptions/../resourceGroups/...",  // env: DATABRICKS_AZURE_RESOURCE_ID
	AzureTenantID:     "00000000-0000-0000-0000-111122223334",  // env: ARM_TENANT_ID
	AzureClientID:     "00000000-0000-0000-0000-111122223335",  // env: ARM_CLIENT_ID
	AzureClientSecret: "som3thing!S@cret",                      // env: ARM_CLIENT_SECRET
	Credentials:       config.AzureClientSecretCredentials{},
}))
Output:

Example (Pat)
databricks.Must(databricks.NewWorkspaceClient(&databricks.Config{
	Host:  "https://abc.cloud.databricks.com", // env: DATABRICKS_HOST
	Token: "dapi0c2a3f4e...",                  // env: DATABRICKS_TOKEN
}))
Output:

func (*Config) Authenticate

func (c *Config) Authenticate(r *http.Request) error

Authenticate adds special headers to HTTP request to authorize it to work with Databricks REST API

func (*Config) CanonicalHostName added in v0.8.1

func (c *Config) CanonicalHostName() string

func (*Config) EnsureResolved

func (c *Config) EnsureResolved() error

func (*Config) Environment added in v0.26.0

func (c *Config) Environment() environment.DatabricksEnvironment

func (*Config) GetAuthDetails added in v0.35.0

func (c *Config) GetAuthDetails(opts ...AuthDetailsOptions) AuthDetails

func (*Config) GetToken added in v0.41.0

func (c *Config) GetToken() (*oauth2.Token, error)

Authenticate returns an OAuth token for the current configuration. It will return an error if the CredentialsStrategy does not support OAuth tokens.

func (*Config) IsAccountClient added in v0.2.0

func (c *Config) IsAccountClient() bool

IsAccountClient returns true if client is configured for Accounts API

func (*Config) IsAws

func (c *Config) IsAws() bool

IsAws returns if the client is configured for Databricks on AWS.

func (*Config) IsAzure

func (c *Config) IsAzure() bool

IsAzure returns if the client is configured for Azure Databricks.

func (*Config) IsGcp

func (c *Config) IsGcp() bool

IsGcp returns if the client is configured for Databricks on Google Cloud.

func (*Config) NewApiClient added in v0.32.0

func (c *Config) NewApiClient() (*httpclient.ApiClient, error)

func (*Config) NewWithWorkspaceHost added in v0.31.0

func (c *Config) NewWithWorkspaceHost(host string) (*Config, error)

NewWithWorkspaceHost returns a new instance of the Config with the host set to the workspace host. Fields that are not relevant to workspace-level config, like account ID, are omitted. Workspace-level attributes that cannot be computed from the host alone, like Azure Resource ID, are also omitted.

func (*Config) SetAttrSource added in v0.35.0

func (c *Config) SetAttrSource(attr *ConfigAttribute, source Source)

func (*Config) WithTesting added in v0.7.0

func (c *Config) WithTesting() *Config

type ConfigAttribute

type ConfigAttribute struct {
	Name      string
	Kind      reflect.Kind
	EnvVars   []string
	Auth      string
	AuthTypes []string
	Sensitive bool
	Internal  bool
	// contains filtered or unexported fields
}

ConfigAttribute provides generic way to work with Config configuration attributes and parses `name`, `env`, and `auth` field tags.

Internal: this field can become unexported in the future

func (*ConfigAttribute) GetString

func (a *ConfigAttribute) GetString(cfg *Config) string

func (*ConfigAttribute) HasAuthAttribute added in v0.35.0

func (a *ConfigAttribute) HasAuthAttribute() bool

func (*ConfigAttribute) IsZero

func (a *ConfigAttribute) IsZero(cfg *Config) bool

func (*ConfigAttribute) ReadEnv

func (a *ConfigAttribute) ReadEnv() (string, string)

func (*ConfigAttribute) Set

func (a *ConfigAttribute) Set(cfg *Config, i interface{}) error

func (*ConfigAttribute) SetS

func (a *ConfigAttribute) SetS(cfg *Config, v string) error

type CredentialsStrategy added in v0.41.0

type CredentialsStrategy interface {
	// Name returns human-addressable name of this credentials provider strategy
	Name() string

	// Configure creates CredentialsProvider or returns nil if a given credentials
	// strategy are not configured. It returns an error if credentials are misconfigured.
	// Takes a context and a pointer to a Config instance, that holds auth mutex.
	Configure(context.Context, *Config) (credentials.CredentialsProvider, error)
}

CredentialsStrategy responsible for configuring static or refreshable authentication credentials for Databricks REST APIs

type DatabricksCliCredentials added in v0.9.0

type DatabricksCliCredentials struct {
}

func (DatabricksCliCredentials) Configure added in v0.9.0

func (DatabricksCliCredentials) Name added in v0.9.0

type DefaultCredentials

type DefaultCredentials struct {
	// contains filtered or unexported fields
}

func (*DefaultCredentials) Configure

func (*DefaultCredentials) Name

func (c *DefaultCredentials) Name() string

type File added in v0.6.0

type File struct {
	*ini.File
	// contains filtered or unexported fields
}

File represents the contents of a databrickscfg file.

func LoadFile added in v0.6.0

func LoadFile(path string) (*File, error)

LoadFile loads the databrickscfg file at the specified path. The function loads ~/.databrickscfg if the specified path is an empty string. The function expands ~ to the user's home directory.

func (*File) Path added in v0.6.0

func (f *File) Path() string

Path returns the path of the loaded databrickscfg file.

type GoogleCredentials

type GoogleCredentials struct {
}

func (GoogleCredentials) Configure

func (GoogleCredentials) Name

func (c GoogleCredentials) Name() string

type GoogleDefaultCredentials

type GoogleDefaultCredentials struct {
	// contains filtered or unexported fields
}

func (GoogleDefaultCredentials) Configure

func (GoogleDefaultCredentials) Name

type Loader

type Loader interface {
	// Name is human-addressable representation of this config resolver
	Name() string
	Configure(*Config) error
}

type M2mCredentials added in v0.3.0

type M2mCredentials struct {
}

func (M2mCredentials) Configure added in v0.3.0

func (M2mCredentials) Name added in v0.3.0

func (c M2mCredentials) Name() string

type MetadataServiceCredentials added in v0.7.0

type MetadataServiceCredentials struct{}

Credentials provider that fetches a token from a locally running HTTP server

The credentials provider will perform a GET request to the configured URL.

The MUST return 4xx response if the "X-Databricks-Metadata-Version" header is not set or set to a version that the server doesn't support.

The server MUST guarantee stable sessions per URL path. That is, if the server returns a token for a Host on a given URL path, it MUST continue to return tokens for the same Host.

The server MUST return a 4xx response if the Host passed in the "X-Databricks-Host" header doesn't match the token.

The server is expected to return a JSON response with the following fields:

- access_token: The requested access token. - token_type: The type of token, which is a "Bearer" access token. - expires_on: Unix timestamp when the access token expires.

func (MetadataServiceCredentials) Configure added in v0.7.0

func (MetadataServiceCredentials) Name added in v0.7.0

type PatCredentials

type PatCredentials struct {
}

func (PatCredentials) Configure

func (PatCredentials) Name

func (c PatCredentials) Name() string

type Source added in v0.35.0

type Source struct {
	Type SourceType `json:"type"`
	Name string     `json:"name,omitempty"`
}

func (*Source) String added in v0.35.0

func (s *Source) String() string

type SourceType added in v0.35.0

type SourceType string
const (
	SourceEnv           SourceType = "environment variable"
	SourceFile          SourceType = "config file"
	SourceDynamicConfig SourceType = "dynamic configuration"
)

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL