libraries

package
v0.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 5, 2023 License: Apache-2.0 Imports: 11 Imported by: 0

Documentation

Overview

The Libraries API allows you to install and uninstall libraries and get the status of libraries on a cluster.

Cluster Libraries API

Libraries can only be installed on a running cluster. Please use the high-level [LibrariesAPI.Update] instead of direct calls to LibrariesAPI.Install, LibrariesAPI.Uninstall, and LibrariesAPI.ClusterStatus. 

err := w.Libraries.UpdateAndWait(ctx, libraries.Update{
	ClusterId: clusterId,
	Install: []libraries.Library{
		{
			Pypi: &libraries.PythonPyPiLibrary{
				Package: "dbl-tempo",
			},
		},
	},
})

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type ClusterLibraryStatuses 

type ClusterLibraryStatuses struct {
	// Unique identifier for the cluster.
	ClusterId string `json:"cluster_id,omitempty"`
	// Status of all libraries on the cluster.
	LibraryStatuses []LibraryFullStatus `json:"library_statuses,omitempty"`
}

func (ClusterLibraryStatuses) IsRetryNeeded 

func (cls ClusterLibraryStatuses) IsRetryNeeded(w Wait) (bool, error)

IsRetryNeeded returns first bool if there needs to be retry. If there needs to be retry, error message will explain why. If retry does not need to happen and error is not nil - it failed.

func (ClusterLibraryStatuses) ToLibraryList 

func (cls ClusterLibraryStatuses) ToLibraryList() InstallLibraries

ToLibraryList convert to envity for convenient comparison

type ClusterStatus 

type ClusterStatus struct {
	// Unique identifier of the cluster whose status should be retrieved.
	ClusterId string `json:"-" url:"cluster_id"`
}

Get status

type InstallLibraries 

type InstallLibraries struct {
	// Unique identifier for the cluster on which to install these libraries.
	ClusterId string `json:"cluster_id"`
	// The libraries to install.
	Libraries []Library `json:"libraries"`
}

func (*InstallLibraries) Sort 

func (cll *InstallLibraries) Sort()

type LibrariesAPI 

type LibrariesAPI struct {
	// contains filtered or unexported fields
}

The Libraries API allows you to install and uninstall libraries and get the status of libraries on a cluster.

To make third-party or custom code available to notebooks and jobs running on your clusters, you can install a library. Libraries can be written in Python, Java, Scala, and R. You can upload Java, Scala, and Python libraries and point to external packages in PyPI, Maven, and CRAN repositories.

Cluster libraries can be used by all notebooks running on a cluster. You can install a cluster library directly from a public repository such as PyPI or Maven, using a previously installed workspace library, or using an init script.

When you install a library on a cluster, a notebook already attached to that cluster will not immediately see the new library. You must first detach and then reattach the notebook to the cluster.

When you uninstall a library from a cluster, the library is removed only when you restart the cluster. Until you restart the cluster, the status of the uninstalled library appears as Uninstall pending restart.

func NewLibraries 

func NewLibraries(client *client.DatabricksClient) *LibrariesAPI

func (*LibrariesAPI) AllClusterStatuses 

func (a *LibrariesAPI) AllClusterStatuses(ctx context.Context) (*ListAllClusterLibraryStatusesResponse, error)

Get all statuses.

Get the status of all libraries on all clusters. A status will be available for all libraries installed on this cluster via the API or the libraries UI as well as libraries set to be installed on all clusters via the libraries UI.

func (*LibrariesAPI) ClusterStatus 

func (a *LibrariesAPI) ClusterStatus(ctx context.Context, request ClusterStatus) (*ClusterLibraryStatuses, error)

Get status.

Get the status of libraries on a cluster. A status will be available for all libraries installed on this cluster via the API or the libraries UI as well as libraries set to be installed on all clusters via the libraries UI. The order of returned libraries will be as follows.

1. Libraries set to be installed on this cluster will be returned first. Within this group, the final order will be order in which the libraries were added to the cluster.

2. Libraries set to be installed on all clusters are returned next. Within this group there is no order guarantee.

3. Libraries that were previously requested on this cluster or on all clusters, but now marked for removal. Within this group there is no order guarantee.

func (*LibrariesAPI) ClusterStatusByClusterId 

func (a *LibrariesAPI) ClusterStatusByClusterId(ctx context.Context, clusterId string) (*ClusterLibraryStatuses, error)

Get status.

Get the status of libraries on a cluster. A status will be available for all libraries installed on this cluster via the API or the libraries UI as well as libraries set to be installed on all clusters via the libraries UI. The order of returned libraries will be as follows.

1. Libraries set to be installed on this cluster will be returned first. Within this group, the final order will be order in which the libraries were added to the cluster.

2. Libraries set to be installed on all clusters are returned next. Within this group there is no order guarantee.

3. Libraries that were previously requested on this cluster or on all clusters, but now marked for removal. Within this group there is no order guarantee.

func (*LibrariesAPI) Impl 

func (a *LibrariesAPI) Impl() LibrariesService

Impl returns low-level Libraries API implementation

func (*LibrariesAPI) Install 

func (a *LibrariesAPI) Install(ctx context.Context, request InstallLibraries) error

Add a library.

Add libraries to be installed on a cluster. The installation is asynchronous; it happens in the background after the completion of this request.

**Note**: The actual set of libraries to be installed on a cluster is the union of the libraries specified via this method and the libraries set to be installed on all clusters via the libraries UI.

func (*LibrariesAPI) Uninstall 

func (a *LibrariesAPI) Uninstall(ctx context.Context, request UninstallLibraries) error

Uninstall libraries.

Set libraries to be uninstalled on a cluster. The libraries won't be uninstalled until the cluster is restarted. Uninstalling libraries that are not installed on the cluster will have no impact but is not an error.

func (*LibrariesAPI) UpdateAndWait 

func (a *LibrariesAPI) UpdateAndWait(ctx context.Context, update Update,
	options ...retries.Option[ClusterLibraryStatuses]) error

func (*LibrariesAPI) Wait 

func (a *LibrariesAPI) Wait(ctx context.Context, wait Wait,
	options ...retries.Option[ClusterLibraryStatuses]) (*ClusterLibraryStatuses, error)

clusterID string, timeout time.Duration, isActive bool, refresh bool

func (*LibrariesAPI) WithImpl 

func (a *LibrariesAPI) WithImpl(impl LibrariesService) *LibrariesAPI

WithImpl could be used to override low-level API implementations for unit testing purposes with github.com/golang/mock or other mocking frameworks.

type LibrariesService 

type LibrariesService interface {

	// Get all statuses.
	//
	// Get the status of all libraries on all clusters. A status will be
	// available for all libraries installed on this cluster via the API or the
	// libraries UI as well as libraries set to be installed on all clusters via
	// the libraries UI.
	AllClusterStatuses(ctx context.Context) (*ListAllClusterLibraryStatusesResponse, error)

	// Get status.
	//
	// Get the status of libraries on a cluster. A status will be available for
	// all libraries installed on this cluster via the API or the libraries UI
	// as well as libraries set to be installed on all clusters via the
	// libraries UI. The order of returned libraries will be as follows.
	//
	// 1. Libraries set to be installed on this cluster will be returned first.
	// Within this group, the final order will be order in which the libraries
	// were added to the cluster.
	//
	// 2. Libraries set to be installed on all clusters are returned next.
	// Within this group there is no order guarantee.
	//
	// 3. Libraries that were previously requested on this cluster or on all
	// clusters, but now marked for removal. Within this group there is no order
	// guarantee.
	ClusterStatus(ctx context.Context, request ClusterStatus) (*ClusterLibraryStatuses, error)

	// Add a library.
	//
	// Add libraries to be installed on a cluster. The installation is
	// asynchronous; it happens in the background after the completion of this
	// request.
	//
	// **Note**: The actual set of libraries to be installed on a cluster is the
	// union of the libraries specified via this method and the libraries set to
	// be installed on all clusters via the libraries UI.
	Install(ctx context.Context, request InstallLibraries) error

	// Uninstall libraries.
	//
	// Set libraries to be uninstalled on a cluster. The libraries won't be
	// uninstalled until the cluster is restarted. Uninstalling libraries that
	// are not installed on the cluster will have no impact but is not an error.
	Uninstall(ctx context.Context, request UninstallLibraries) error
}

The Libraries API allows you to install and uninstall libraries and get the status of libraries on a cluster.

To make third-party or custom code available to notebooks and jobs running on your clusters, you can install a library. Libraries can be written in Python, Java, Scala, and R. You can upload Java, Scala, and Python libraries and point to external packages in PyPI, Maven, and CRAN repositories.

Cluster libraries can be used by all notebooks running on a cluster. You can install a cluster library directly from a public repository such as PyPI or Maven, using a previously installed workspace library, or using an init script.

When you install a library on a cluster, a notebook already attached to that cluster will not immediately see the new library. You must first detach and then reattach the notebook to the cluster.

When you uninstall a library from a cluster, the library is removed only when you restart the cluster. Until you restart the cluster, the status of the uninstalled library appears as Uninstall pending restart.

type Library 

type Library struct {
	// Specification of a CRAN library to be installed as part of the library
	Cran *RCranLibrary `json:"cran,omitempty"`
	// URI of the egg to be installed. Currently only DBFS and S3 URIs are
	// supported. For example: `{ "egg": "dbfs:/my/egg" }` or `{ "egg":
	// "s3://my-bucket/egg" }`. If S3 is used, please make sure the cluster has
	// read access on the library. You may need to launch the cluster with an
	// IAM role to access the S3 URI.
	Egg string `json:"egg,omitempty"`
	// URI of the jar to be installed. Currently only DBFS and S3 URIs are
	// supported. For example: `{ "jar": "dbfs:/mnt/databricks/library.jar" }`
	// or `{ "jar": "s3://my-bucket/library.jar" }`. If S3 is used, please make
	// sure the cluster has read access on the library. You may need to launch
	// the cluster with an IAM role to access the S3 URI.
	Jar string `json:"jar,omitempty"`
	// Specification of a maven library to be installed. For example: `{
	// "coordinates": "org.jsoup:jsoup:1.7.2" }`
	Maven *MavenLibrary `json:"maven,omitempty"`
	// Specification of a PyPi library to be installed. For example: `{
	// "package": "simplejson" }`
	Pypi *PythonPyPiLibrary `json:"pypi,omitempty"`
	// URI of the wheel to be installed. For example: `{ "whl": "dbfs:/my/whl"
	// }` or `{ "whl": "s3://my-bucket/whl" }`. If S3 is used, please make sure
	// the cluster has read access on the library. You may need to launch the
	// cluster with an IAM role to access the S3 URI.
	Whl string `json:"whl,omitempty"`
}

func (Library) String 

func (library Library) String() string

type LibraryFullStatus 

type LibraryFullStatus struct {
	// Whether the library was set to be installed on all clusters via the
	// libraries UI.
	IsLibraryForAllClusters bool `json:"is_library_for_all_clusters,omitempty"`
	// Unique identifier for the library.
	Library *Library `json:"library,omitempty"`
	// All the info and warning messages that have occurred so far for this
	// library.
	Messages []string `json:"messages,omitempty"`
	// Status of installing the library on the cluster.
	Status LibraryFullStatusStatus `json:"status,omitempty"`
}

type LibraryFullStatusStatus 

type LibraryFullStatusStatus string

Status of installing the library on the cluster. 

const LibraryFullStatusStatusFailed LibraryFullStatusStatus = `FAILED`
const LibraryFullStatusStatusInstalled LibraryFullStatusStatus = `INSTALLED`
const LibraryFullStatusStatusInstalling LibraryFullStatusStatus = `INSTALLING`
const LibraryFullStatusStatusPending LibraryFullStatusStatus = `PENDING`
const LibraryFullStatusStatusResolving LibraryFullStatusStatus = `RESOLVING`
const LibraryFullStatusStatusSkipped LibraryFullStatusStatus = `SKIPPED`
const LibraryFullStatusStatusUninstallOnRestart LibraryFullStatusStatus = `UNINSTALL_ON_RESTART`

func (*LibraryFullStatusStatus) Set added in v0.2.0 

func (lfss *LibraryFullStatusStatus) Set(v string) error

Set raw string value and validate it against allowed values

func (*LibraryFullStatusStatus) String added in v0.2.0 

func (lfss *LibraryFullStatusStatus) String() string

String representation for fmt.Print

func (*LibraryFullStatusStatus) Type added in v0.2.0 

func (lfss *LibraryFullStatusStatus) Type() string

Type always returns LibraryFullStatusStatus to satisfy [pflag.Value] interface

type ListAllClusterLibraryStatusesResponse 

type ListAllClusterLibraryStatusesResponse struct {
	// A list of cluster statuses.
	Statuses []ClusterLibraryStatuses `json:"statuses,omitempty"`
}

type MavenLibrary 

type MavenLibrary struct {
	// Gradle-style maven coordinates. For example: "org.jsoup:jsoup:1.7.2".
	Coordinates string `json:"coordinates"`
	// List of dependences to exclude. For example: `["slf4j:slf4j",
	// "*:hadoop-client"]`.
	//
	// Maven dependency exclusions:
	// https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html.
	Exclusions []string `json:"exclusions,omitempty"`
	// Maven repo to install the Maven package from. If omitted, both Maven
	// Central Repository and Spark Packages are searched.
	Repo string `json:"repo,omitempty"`
}

type PythonPyPiLibrary 

type PythonPyPiLibrary struct {
	// The name of the pypi package to install. An optional exact version
	// specification is also supported. Examples: "simplejson" and
	// "simplejson==3.8.0".
	Package string `json:"package"`
	// The repository where the package can be found. If not specified, the
	// default pip index is used.
	Repo string `json:"repo,omitempty"`
}

type RCranLibrary 

type RCranLibrary struct {
	// The name of the CRAN package to install.
	Package string `json:"package"`
	// The repository where the package can be found. If not specified, the
	// default CRAN repo is used.
	Repo string `json:"repo,omitempty"`
}

type UninstallLibraries 

type UninstallLibraries struct {
	// Unique identifier for the cluster on which to uninstall these libraries.
	ClusterId string `json:"cluster_id"`
	// The libraries to uninstall.
	Libraries []Library `json:"libraries"`
}

type Update 

type Update struct {
	ClusterId string
	// The libraries to install.
	Install []Library
	// The libraries to install.
	Uninstall []Library
}

type Wait 

type Wait struct {
	ClusterID string
	Libraries []Library
	IsRunning bool
	IsRefresh bool
}

func (*Wait) IsNotInScope 

func (w *Wait) IsNotInScope(lib *Library) bool

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.