config

package
v0.32.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 14, 2024 License: Apache-2.0 Imports: 29 Imported by: 26

README

Authentication

Auth flow

sequenceDiagram
    ClustersAPI->>+DatabricksClient: GET .../clusters/list
    DatabricksClient->>+databricks.Config: Authenticate(HttpRequest)

    databricks.Config-->>+DefaultCredentials: Configure(databricks.Config)
    DefaultCredentials-->>+FirstCredentials: try configure
    FirstCredentials-->>-DefaultCredentials: try next
    DefaultCredentials->>+NextCredentials: try configure
    NextCredentials->>RequestVisitor: configured auth
    NextCredentials->>-DefaultCredentials: authenticated
    DefaultCredentials->>-databricks.Config: set AuthType & request visitor

    databricks.Config->>+RequestVisitor: visit HTTP request
    RequestVisitor-->>+IdentityProvider: ensure fresh token
    IdentityProvider-->>-RequestVisitor: access token
    RequestVisitor->>-databricks.Config: added HTTP headers
    
    databricks.Config->>-DatabricksClient: added HTTP headers

    DatabricksClient->>+API: authenticated request
    API->>-DatabricksClient: JSON payload
    DatabricksClient->>-ClustersAPI: ClustersList or error

Client configuration

classDiagram
    Loader "0..n" <-- Config: Configure(self)
    Credentials "0..1" <-- Config: Configure(self)
    RequestVisitor --* Config: configured auth
    class Config {
        * Host string
        * Token string
        * Profile string
        * Username string
        * Password string
        * AzureResourceID string
        * AzureEnvironment string
        * AzureClientID string
        * AzureSecretID string
        * AzureTenantID string
        * GoogleServiceAccount string

        Credentials: DefaultCredentials
        Loaders: Loader

        Authenticate(HttpRequest) error
    }

    class Loader {
        <<interface>>
        Name() string
        Configure(Config) error
    }

    KnownConfigLoader ..|> Loader
    class KnownConfigLoader

    ConfigAttributes ..|> Loader
    class ConfigAttributes {
        Configure(Config) error
        DebugString(Config) string
        Validate(Config) error
        ResolveFromStringMap(Config, map) error
        ResolveFromAnyMap(Config, map) error
    }

    Config --* DatabricksClient
    class DatabricksClient {
        Config
        - retryPolicy
        
        Get(path, query) T
        Post(path, body) T
        Put(path, body) T
        Patch(path, body) T
        Delete(path, query) T
    }

    Credentials --> "0..1" RequestVisitor: creates
    class Credentials {
        <<interface>>
        Name() string
        Configure(Config) RequestVisitor
    }

    class RequestVisitor {
        <<interface>>
        Visit(HttpRequest) error
    }

    AzureSpnCredentials --* authProviders
    AzureSpnCredentials ..|> Credentials
    class AzureSpnCredentials

    AzureCliCredentials --* authProviders
    AzureCliCredentials ..|> Credentials

    GoogleCredentials --* authProviders
    GoogleCredentials ..|> Credentials
    class GoogleCredentials
    
    DatabricksOauthCredentials --* authProviders
    DatabricksOauthCredentials ..|> Credentials
    class DatabricksOauthCredentials {
        []Scopes
    }

    PatCredentials --* authProviders
    PatCredentials ..|> Credentials
    class PatCredentials

    BasicCredentials --* authProviders
    BasicCredentials ..|> Credentials
    class BasicCredentials

    authProviders --> DefaultCredentials: for reach ConfigAttributes()
    DefaultCredentials ..|> Credentials
    class DefaultCredentials

Documentation

Index

Examples

Constants

View Source 
const MetadataServiceHostHeader = "X-Databricks-Host"
View Source 
const MetadataServiceVersion = "1"
View Source 
const MetadataServiceVersionHeader = "X-Databricks-Metadata-Version"

Variables

View Source 
var (
	AzurePublicCloud = azureEnvironment{
		Name:                      "PUBLIC",
		ServiceManagementEndpoint: "https://management.core.windows.net/",
		ResourceManagerEndpoint:   "https://management.azure.com/",
		ActiveDirectoryEndpoint:   "https://login.microsoftonline.com/",
	}

	AzureUsGovernmentCloud = azureEnvironment{
		Name:                      "USGOVERNMENT",
		ServiceManagementEndpoint: "https://management.core.usgovcloudapi.net/",
		ResourceManagerEndpoint:   "https://management.usgovcloudapi.net/",
		ActiveDirectoryEndpoint:   "https://login.microsoftonline.us/",
	}

	AzureChinaCloud = azureEnvironment{
		Name:                      "CHINA",
		ServiceManagementEndpoint: "https://management.core.chinacloudapi.cn/",
		ResourceManagerEndpoint:   "https://management.chinacloudapi.cn/",
		ActiveDirectoryEndpoint:   "https://login.chinacloudapi.cn/",
	}
)

based on github.com/Azure/go-autorest/autorest/azure/azureEnvironments.go

View Source 
var ConfigAttributes = loadAttrs()
View Source 
var ConfigFile = configFileLoader{}
View Source 
var ErrCannotConfigureAuth = errors.New(errorMessage)

ErrCannotConfigureAuth (experimental) is returned when no auth is configured

Functions

func NewAzureCliTokenSource added in v0.26.2 

func NewAzureCliTokenSource(resource string) oauth2.TokenSource

NewAzureCliTokenSource returns oauth2.TokenSource for a passwordless authentication via Azure CLI (`az login`)

func NewAzureMsiTokenSource added in v0.26.2 

func NewAzureMsiTokenSource(client *httpclient.ApiClient, resource, clientId string) oauth2.TokenSource

NewAzureMsiTokenSource returns oauth2.TokenSource for a passwordless authentication via Azure Managed identity

Types

type AzureCliCredentials 

type AzureCliCredentials struct {
}

func (AzureCliCredentials) Configure 

func (c AzureCliCredentials) Configure(ctx context.Context, cfg *Config) (func(*http.Request) error, error)

func (AzureCliCredentials) Name 

func (c AzureCliCredentials) Name() string

type AzureClientSecretCredentials 

type AzureClientSecretCredentials struct {
}

func (AzureClientSecretCredentials) Configure 

func (c AzureClientSecretCredentials) Configure(ctx context.Context, cfg *Config) (func(*http.Request) error, error)

TODO: We need to expose which authentication mechanism is used to Terraform, as we cannot create AKV backed secret scopes when authenticated as SP. If we are authenticated as SP and wish to create one we want to fail early. Also see https://github.com/databricks/terraform-provider-databricks/issues/1490.

func (AzureClientSecretCredentials) Name 

func (c AzureClientSecretCredentials) Name() string

type AzureMsiCredentials added in v0.3.0 

type AzureMsiCredentials struct {
}

func (AzureMsiCredentials) Configure added in v0.3.0 

func (c AzureMsiCredentials) Configure(ctx context.Context, cfg *Config) (func(*http.Request) error, error)

func (AzureMsiCredentials) Name added in v0.3.0 

func (c AzureMsiCredentials) Name() string

type BasicCredentials 

type BasicCredentials struct {
}

func (BasicCredentials) Configure 

func (c BasicCredentials) Configure(ctx context.Context, cfg *Config) (func(*http.Request) error, error)

func (BasicCredentials) Name 

func (c BasicCredentials) Name() string

type Cloud added in v0.26.0 

type Cloud string
const (
	CloudAWS   Cloud = "AWS"
	CloudAzure Cloud = "Azure"
	CloudGCP   Cloud = "GCP"
)

type Config 

type Config struct {
	// Credentials holds an instance of Credentials Provider to authenticate with Databricks REST APIs.
	// If no credentials provider is specified, `DefaultCredentials` are implicitly used.
	Credentials CredentialsProvider

	// Databricks host (either of workspace endpoint or Accounts API endpoint)
	Host string `name:"host" env:"DATABRICKS_HOST"`

	ClusterID   string `name:"cluster_id" env:"DATABRICKS_CLUSTER_ID"`
	WarehouseID string `name:"warehouse_id" env:"DATABRICKS_WAREHOUSE_ID"`

	// URL of the metadata service that provides authentication credentials.
	MetadataServiceURL string `name:"metadata_service_url" env:"DATABRICKS_METADATA_SERVICE_URL" auth:"metadata-service,sensitive"`

	// Databricks Account ID for Accounts API. This field is used in dependencies.
	AccountID string `name:"account_id" env:"DATABRICKS_ACCOUNT_ID"`

	Token    string `name:"token" env:"DATABRICKS_TOKEN" auth:"pat,sensitive"`
	Username string `name:"username" env:"DATABRICKS_USERNAME" auth:"basic"`
	Password string `name:"password" env:"DATABRICKS_PASSWORD" auth:"basic,sensitive"`

	// Connection profile specified within ~/.databrickscfg.
	Profile string `name:"profile" env:"DATABRICKS_CONFIG_PROFILE"`

	// Location of the Databricks CLI credentials file, that is created
	// by `databricks configure --token` command. By default, it is located
	// in ~/.databrickscfg.
	ConfigFile string `name:"config_file" env:"DATABRICKS_CONFIG_FILE"`

	GoogleServiceAccount string `name:"google_service_account" env:"DATABRICKS_GOOGLE_SERVICE_ACCOUNT" auth:"google"`
	GoogleCredentials    string `name:"google_credentials" env:"GOOGLE_CREDENTIALS" auth:"google,sensitive"`

	// Azure Resource Manager ID for Azure Databricks workspace, which is exhanged for a Host
	AzureResourceID string `name:"azure_workspace_resource_id" env:"DATABRICKS_AZURE_RESOURCE_ID" auth:"azure"`

	AzureUseMSI       bool   `name:"azure_use_msi" env:"ARM_USE_MSI" auth:"azure"`
	AzureClientSecret string `name:"azure_client_secret" env:"ARM_CLIENT_SECRET" auth:"azure,sensitive"`
	AzureClientID     string `name:"azure_client_id" env:"ARM_CLIENT_ID" auth:"azure"`
	AzureTenantID     string `name:"azure_tenant_id" env:"ARM_TENANT_ID" auth:"azure"`

	// AzureEnvironment (PUBLIC, USGOVERNMENT, CHINA) has specific set of API endpoints. Starting from v0.26.0,
	// the environment is determined based on the workspace hostname, if it's specified.
	AzureEnvironment string `name:"azure_environment" env:"ARM_ENVIRONMENT"`

	// Azure Login Application ID. Must be set if authenticating for non-production workspaces. Starting from v0.26.0,
	// the correct Azure Login App ID is determined based on the Azure Databricks Workspace hostname.
	//
	// Deprecated: this configuration property no longer has any effect and will be removed in the future
	// versions of Go SDK.
	AzureLoginAppID string `name:"azure_login_app_id" env:"DATABRICKS_AZURE_LOGIN_APP_ID" auth:"azure"`

	ClientID     string `name:"client_id" env:"DATABRICKS_CLIENT_ID" auth:"oauth"`
	ClientSecret string `name:"client_secret" env:"DATABRICKS_CLIENT_SECRET" auth:"oauth,sensitive"`

	// Path to the Databricks CLI (version >= 0.100.0).
	DatabricksCliPath string `name:"databricks_cli_path" env:"DATABRICKS_CLI_PATH"`

	// When multiple auth attributes are available in the environment, use the auth type
	// specified by this argument. This argument also holds currently selected auth.
	AuthType string `name:"auth_type" env:"DATABRICKS_AUTH_TYPE" auth:"-"`

	// Skip SSL certificate verification for HTTP calls.
	// Use at your own risk or for unit testing purposes.
	InsecureSkipVerify bool `name:"skip_verify" auth:"-"`

	// Number of seconds for HTTP timeout
	HTTPTimeoutSeconds int `name:"http_timeout_seconds" auth:"-"`

	// Truncate JSON fields in JSON above this limit. Default is 96.
	DebugTruncateBytes int `name:"debug_truncate_bytes" env:"DATABRICKS_DEBUG_TRUNCATE_BYTES" auth:"-"`

	// Debug HTTP headers of requests made by the provider. Default is false.
	DebugHeaders bool `name:"debug_headers" env:"DATABRICKS_DEBUG_HEADERS" auth:"-"`

	// Maximum number of requests per second made to Databricks REST API.
	RateLimitPerSecond int `name:"rate_limit" env:"DATABRICKS_RATE_LIMIT" auth:"-"`

	// Number of seconds to keep retrying HTTP requests. Default is 300 (5 minutes)
	RetryTimeoutSeconds int `name:"retry_timeout_seconds" auth:"-"`

	// HTTPTransport can be overriden for unit testing and together with tooling like https://github.com/google/go-replayers
	HTTPTransport http.RoundTripper

	// Environment override to return when resolving the current environment.
	DatabricksEnvironment *DatabricksEnvironment

	Loaders []Loader
	// contains filtered or unexported fields
}

Config represents configuration for Databricks Connectivity

Example (Accounts) 
databricks.Must(databricks.NewWorkspaceClient(&databricks.Config{
	Host:      "https://accounts.cloud.databricks.com", // env: DATABRICKS_HOST
	AccountID: "00000000-0000-0000-0000-111122223333",  // env: DATABRICKS_ACCOUNT_ID
	Username:  "me@example.com",                        // env: DATABRICKS_USERNAME
	Password:  "som3thing!S@cret",                      // env: DATABRICKS_PASSWORD
}))

Output:
Example (AzureActiveDirectoryServicePrincipal) 
databricks.Must(databricks.NewWorkspaceClient(&databricks.Config{
	Host:              "https://adb-123.4.azuredatabricks.net", // env: DATABRICKS_HOST
	AzureResourceID:   "/subscriptions/../resourceGroups/...",  // env: DATABRICKS_AZURE_RESOURCE_ID
	AzureTenantID:     "00000000-0000-0000-0000-111122223334",  // env: ARM_TENANT_ID
	AzureClientID:     "00000000-0000-0000-0000-111122223335",  // env: ARM_CLIENT_ID
	AzureClientSecret: "som3thing!S@cret",                      // env: ARM_CLIENT_SECRET
}))

Output:
Example (Basic) 
databricks.Must(databricks.NewWorkspaceClient(&databricks.Config{
	Host:     "https://abc.cloud.databricks.com", // env: DATABRICKS_HOST
	Username: "me@example.com",                   // env: DATABRICKS_USERNAME
	Password: "som3thing!S@cret",                 // env: DATABRICKS_PASSWORD
}))

Output:
Example (CustomConfigFile) 
databricks.Must(databricks.NewWorkspaceClient(&databricks.Config{
	ConfigFile: "/path/to/.databrickscfg", // env: DATABRICKS_CONFIG_FILE
}))

Output:
Example (CustomProfile) 
databricks.Must(databricks.NewWorkspaceClient(&databricks.Config{
	Profile: "production", // env: DATABRICKS_CONFIG_PROFILE
}))

Output:
Example (Debugging) 
databricks.Must(databricks.NewWorkspaceClient(&databricks.Config{
	HTTPTimeoutSeconds:  60,
	DebugTruncateBytes:  96,    // env: DATABRICKS_DEBUG_TRUNCATE_BYTES
	DebugHeaders:        false, // env: DATABRICKS_DEBUG_HEADERS
	RateLimitPerSecond:  15,    // env: DATABRICKS_RATE_LIMIT
	RetryTimeoutSeconds: 300,
}))

Output:
Example (ForceAzureActiveDirectoryServicePrincipal) 
databricks.Must(databricks.NewWorkspaceClient(&databricks.Config{
	Host:              "https://adb-123.4.azuredatabricks.net", // env: DATABRICKS_HOST
	AzureResourceID:   "/subscriptions/../resourceGroups/...",  // env: DATABRICKS_AZURE_RESOURCE_ID
	AzureTenantID:     "00000000-0000-0000-0000-111122223334",  // env: ARM_TENANT_ID
	AzureClientID:     "00000000-0000-0000-0000-111122223335",  // env: ARM_CLIENT_ID
	AzureClientSecret: "som3thing!S@cret",                      // env: ARM_CLIENT_SECRET
	Credentials:       config.AzureClientSecretCredentials{},
}))

Output:
Example (Pat) 
databricks.Must(databricks.NewWorkspaceClient(&databricks.Config{
	Host:  "https://abc.cloud.databricks.com", // env: DATABRICKS_HOST
	Token: "dapi0c2a3f4e...",                  // env: DATABRICKS_TOKEN
}))

Output:

func (*Config) Authenticate 

func (c *Config) Authenticate(r *http.Request) error

Authenticate adds special headers to HTTP request to authorize it to work with Databricks REST API

func (*Config) CanonicalHostName added in v0.8.1 

func (c *Config) CanonicalHostName() string

func (*Config) EnsureResolved 

func (c *Config) EnsureResolved() error

func (*Config) Environment added in v0.26.0 

func (c *Config) Environment() DatabricksEnvironment

func (*Config) IsAccountClient added in v0.2.0 

func (c *Config) IsAccountClient() bool

IsAccountClient returns true if client is configured for Accounts API

func (*Config) IsAws 

func (c *Config) IsAws() bool

IsAws returns if the client is configured for Databricks on AWS.

func (*Config) IsAzure 

func (c *Config) IsAzure() bool

IsAzure returns if the client is configured for Azure Databricks.

func (*Config) IsGcp 

func (c *Config) IsGcp() bool

IsGcp returns if the client is configured for Databricks on Google Cloud.

func (*Config) NewApiClient added in v0.32.0 

func (c *Config) NewApiClient() (*httpclient.ApiClient, error)

func (*Config) NewWithWorkspaceHost added in v0.31.0 

func (c *Config) NewWithWorkspaceHost(host string) (*Config, error)

NewWithWorkspaceHost returns a new instance of the Config with the host set to the workspace host. Fields that are not relevant to workspace-level config, like account ID, are omitted. Workspace-level attributes that cannot be computed from the host alone, like Azure Resource ID, are also omitted.

func (*Config) WithTesting added in v0.7.0 

func (c *Config) WithTesting() *Config

type ConfigAttribute 

type ConfigAttribute struct {
	Name      string
	Kind      reflect.Kind
	EnvVars   []string
	Auth      string
	Sensitive bool
	Internal  bool
	// contains filtered or unexported fields
}

ConfigAttribute provides generic way to work with Config configuration attributes and parses `name`, `env`, and `auth` field tags.

Internal: this field can become unexported in the future

func (*ConfigAttribute) GetString 

func (a *ConfigAttribute) GetString(cfg *Config) string

func (*ConfigAttribute) IsZero 

func (a *ConfigAttribute) IsZero(cfg *Config) bool

func (*ConfigAttribute) ReadEnv 

func (a *ConfigAttribute) ReadEnv() string

func (*ConfigAttribute) Set 

func (a *ConfigAttribute) Set(cfg *Config, i interface{}) error

func (*ConfigAttribute) SetS 

func (a *ConfigAttribute) SetS(cfg *Config, v string) error

type CredentialsProvider 

type CredentialsProvider interface {
	// Name returns human-addressable name of this credentials provider name
	Name() string

	// Configure creates HTTP Request Visitor or returns nil if a given credetials
	// are not configured. It returns an error if credentials are misconfigured.
	// Takes a context and a pointer to a Config instance, that holds auth mutex.
	Configure(context.Context, *Config) (func(*http.Request) error, error)
}

CredentialsProvider responsible for configuring static or refreshable authentication credentials for Databricks REST APIs

type DatabricksCliCredentials added in v0.9.0 

type DatabricksCliCredentials struct {
}

func (DatabricksCliCredentials) Configure added in v0.9.0 

func (c DatabricksCliCredentials) Configure(ctx context.Context, cfg *Config) (func(*http.Request) error, error)

func (DatabricksCliCredentials) Name added in v0.9.0 

func (c DatabricksCliCredentials) Name() string

type DatabricksEnvironment added in v0.26.0 

type DatabricksEnvironment struct {
	Cloud              Cloud
	DnsZone            string
	AzureApplicationID string
	AzureEnvironment   *azureEnvironment
}

func (DatabricksEnvironment) AzureActiveDirectoryEndpoint added in v0.26.0 

func (de DatabricksEnvironment) AzureActiveDirectoryEndpoint() string

func (DatabricksEnvironment) AzureResourceManagerEndpoint added in v0.26.0 

func (de DatabricksEnvironment) AzureResourceManagerEndpoint() string

func (DatabricksEnvironment) AzureServiceManagementEndpoint added in v0.26.0 

func (de DatabricksEnvironment) AzureServiceManagementEndpoint() string

func (DatabricksEnvironment) DeploymentURL added in v0.26.0 

func (de DatabricksEnvironment) DeploymentURL(name string) string

type DefaultCredentials 

type DefaultCredentials struct {
	// contains filtered or unexported fields
}

func (*DefaultCredentials) Configure 

func (c *DefaultCredentials) Configure(ctx context.Context, cfg *Config) (func(*http.Request) error, error)

func (*DefaultCredentials) Name 

func (c *DefaultCredentials) Name() string

type File added in v0.6.0 

type File struct {
	*ini.File
	// contains filtered or unexported fields
}

File represents the contents of a databrickscfg file.

func LoadFile added in v0.6.0 

func LoadFile(path string) (*File, error)

LoadFile loads the databrickscfg file at the specified path. The function loads ~/.databrickscfg if the specified path is an empty string. The function expands ~ to the user's home directory.

func (*File) Path added in v0.6.0 

func (f *File) Path() string

Path returns the path of the loaded databrickscfg file.

type GoogleCredentials 

type GoogleCredentials struct {
}

func (GoogleCredentials) Configure 

func (c GoogleCredentials) Configure(ctx context.Context, cfg *Config) (func(*http.Request) error, error)

func (GoogleCredentials) Name 

func (c GoogleCredentials) Name() string

type GoogleDefaultCredentials 

type GoogleDefaultCredentials struct {
	// contains filtered or unexported fields
}

func (GoogleDefaultCredentials) Configure 

func (c GoogleDefaultCredentials) Configure(ctx context.Context, cfg *Config) (func(*http.Request) error, error)

func (GoogleDefaultCredentials) Name 

func (c GoogleDefaultCredentials) Name() string

type Loader 

type Loader interface {
	// Name is human-addressable representation of this config resolver
	Name() string
	Configure(*Config) error
}

type M2mCredentials added in v0.3.0 

type M2mCredentials struct {
}

func (M2mCredentials) Configure added in v0.3.0 

func (c M2mCredentials) Configure(ctx context.Context, cfg *Config) (func(*http.Request) error, error)

func (M2mCredentials) Name added in v0.3.0 

func (c M2mCredentials) Name() string

type MetadataServiceCredentials added in v0.7.0 

type MetadataServiceCredentials struct{}

Credentials provider that fetches a token from a locally running HTTP server

The credentials provider will perform a GET request to the configured URL.

The MUST return 4xx response if the "X-Databricks-Metadata-Version" header is not set or set to a version that the server doesn't support.

The server MUST guarantee stable sessions per URL path. That is, if the server returns a token for a Host on a given URL path, it MUST continue to return tokens for the same Host.

The server MUST return a 4xx response if the Host passed in the "X-Databricks-Host" header doesn't match the token.

The server is expected to return a JSON response with the following fields:

- access_token: The requested access token. - token_type: The type of token, which is a "Bearer" access token. - expires_on: Unix timestamp when the access token expires.

func (MetadataServiceCredentials) Configure added in v0.7.0 

func (c MetadataServiceCredentials) Configure(ctx context.Context, cfg *Config) (func(*http.Request) error, error)

func (MetadataServiceCredentials) Name added in v0.7.0 

func (c MetadataServiceCredentials) Name() string

type PatCredentials 

type PatCredentials struct {
}

func (PatCredentials) Configure 

func (c PatCredentials) Configure(ctx context.Context, cfg *Config) (func(*http.Request) error, error)

func (PatCredentials) Name 

func (c PatCredentials) Name() string

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.