caramlstore

README

caramlstore

Extractor for Machine Learning(ML) Features from caraml-store.

The extractor uses the CoreService.proto service exposed by caraml-store to extract feature tables.

Usage

source:
  name: caramlstore
  scope: caramlstore-stg
  config:
    url: staging.caraml-store.com:80
    max_size_in_mb: 10
    request_timeout: 30s

Inputs

Key	Value	Example	Description	Required?
url	string	caraml-store.com:80	caraml-store's host URL	✅
max_size_in_mb	int	10	Max MB for gRPC client to receive message. Default is 45.	✘
request_timeout	string	10s	Timeout for gRPC requests to caraml-store. Default is 10s.	✘

Outputs

The feature tables are mapped to an Asset with model specific metadata stored using FeatureTable. Please refer the proto definitions for more information.

Field	Value	Sample Value
resource.urn	urn:caramlstore:{scope}:feature_table:{feature_table.project}.{feature_table.name}	urn:caramlstore:caramlstore-stg:feature_table:my_project.merchant_uuid_t2_discovery
resource.name	{feature_table.name}	merchant_uuid_t2_discovery
resource.service	caramlstore	caramlstore
resource.type	feature_table	feature_table
namespace	{feature_table.project}	my_project
entities	[]Entity
features	[]Feature
create_time	{feature_table.created_timestamp}	2022-08-08T03:17:54Z
update_time	{feature_table.updated_timestamp}	2022-08-08T03:57:54Z

Entity

Field	Value	Sample Value
name	{entity.name}	service_type
labels	map[string]string	{"description":"merchant uuid","value_type":"STRING"}

Feature

Field	Value	Sample Value
name	{feature.name}	avg_gmv_merchant_1d
data_type	{features.value_type}	INT64

Contributing

Refer to the contribution guidelines for information on contributing to this module.

Documentation

Index

• type Client
• type Config
• type Extractor
  • func New(logger log.Logger, client Client) *Extractor
  • func (e *Extractor) Close() error
  • func (e *Extractor) Extract(ctx context.Context, emit plugins.Emit) error
  • func (e *Extractor) Init(ctx context.Context, config plugins.Config) error

Types

type Client

type Client interface {
	Connect(ctx context.Context, host string, maxSizeInMB int, timeout time.Duration) error
	Projects(ctx context.Context) ([]string, error)
	Entities(ctx context.Context, project string) (map[string]*core.Entity, error)
	FeatureTables(ctx context.Context, project string) ([]*core.FeatureTable, error)
	Close() error
}

type Config

type Config struct {
	URL            string        `json:"url" yaml:"url" mapstructure:"url" validate:"required"`
	MaxSizeInMB    int           `json:"max_size_in_mb" yaml:"max_size_in_mb" mapstructure:"max_size_in_mb"`
	RequestTimeout time.Duration `json:"request_timeout" yaml:"request_timeout" mapstructure:"request_timeout" validate:"min=1ms" default:"10s"`
}

Config holds the set of configuration for the CaraML Store extractor

type Extractor

type Extractor struct {
	plugins.BaseExtractor
	// contains filtered or unexported fields
}

Extractor manages the communication with the CaraML Store service

func New

func New(logger log.Logger, client Client) *Extractor

func (*Extractor) Close

func (e *Extractor) Close() error

func (*Extractor) Extract

func (e *Extractor) Extract(ctx context.Context, emit plugins.Emit) error

Extract checks if the table is valid and extracts the table schema

func (*Extractor) Init

func (e *Extractor) Init(ctx context.Context, config plugins.Config) error

Init initializes the extractor