datalake

Documentation

Index

• type Conn
  • func NewConn() *Conn
  • func (conn *Conn) Get(ctx context.Context, path string) (*minio.Object, error)
  • func (conn *Conn) List(ctx context.Context, path string) <-chan minio.ObjectInfo
  • func (conn *Conn) Put(ctx context.Context, path string, data []byte, metadata map[string]string) (err error)

Types

type Conn

type Conn struct {
	// contains filtered or unexported fields
}

Conn is a wrapper around a datalake connection, which uses S3 compatible storage.

func NewConn

func NewConn() *Conn

NewConn creates a new connection to the S3-compatible storage. It initializes the Minio client with the provided credentials and creates a default bucket if it doesn't exist.

Returns:

• *Conn: A new connection instance with initialized client and bucket

func (*Conn) Get

func (conn *Conn) Get(ctx context.Context, path string) (*minio.Object, error)

Get retrieves an object from the specified path in the datalake. It verifies the object's existence before returning it.

Parameters:

• ctx: Context for the operation
• path: The path to the object to retrieve

Returns:

• *minio.Object: The retrieved object
• error: Any error that occurred during retrieval

func (*Conn) List

func (conn *Conn) List(ctx context.Context, path string) <-chan minio.ObjectInfo

List returns a channel of ObjectInfo for all objects in the specified path. It recursively lists all objects and includes their metadata.

Parameters:

• ctx: Context for the operation
• path: The path prefix to list objects from

Returns:

• <-chan minio.ObjectInfo: Channel streaming object information

func (*Conn) Put

func (conn *Conn) Put(ctx context.Context, path string, data []byte, metadata map[string]string) (err error)

Put stores data at the specified path in the datalake with optional metadata. It creates a new object or overwrites an existing one.

Parameters:

• ctx: Context for the operation
• path: The path where the object should be stored
• data: The byte data to store
• metadata: Optional metadata to attach to the object

Returns:

• error: Any error that occurred during the operation