Documentation
¶
Overview ¶
Package source provides functionality for dealing with data sources.
Index ¶
- Constants
- func Contains[S *Source | ~string](srcs []*Source, s S) bool
- func Handle2SafePath(handle string) string
- func IsValidGroup(group string) bool
- func IsValidHandle(handle string) bool
- func ParseTableHandle(input string) (handle, table string, err error)
- func RedactGroup(g *Group)
- func ReservedHandles() []string
- func Sort(srcs []*Source)
- func SortGroups(groups []*Group)
- func SuggestHandle(coll *Collection, typ drivertype.Type, loc string) (string, error)
- func Target(src *Source, tbl string) string
- func ValidGroup(group string) error
- func ValidHandle(handle string) error
- func VerifyIntegrity(coll *Collection) (repaired bool, err error)
- type Collection
- func (c *Collection) Active() *Source
- func (c *Collection) ActiveGroup() string
- func (c *Collection) ActiveHandle() string
- func (c *Collection) Add(src *Source) error
- func (c *Collection) Clone() *Collection
- func (c *Collection) Data() any
- func (c *Collection) Get(handle string) (*Source, error)
- func (c *Collection) Groups() []string
- func (c *Collection) Handles() []string
- func (c *Collection) HandlesInGroup(group string) ([]string, error)
- func (c *Collection) IsExistingGroup(group string) bool
- func (c *Collection) IsExistingSource(handle string) bool
- func (c *Collection) MarshalJSON() ([]byte, error)
- func (c *Collection) MarshalYAML() (any, error)
- func (c *Collection) MoveHandleToGroup(handle, toGroup string) (*Source, error)
- func (c *Collection) Remove(handle string) error
- func (c *Collection) RemoveGroup(group string) ([]*Source, error)
- func (c *Collection) RenameGroup(oldGroup, newGroup string) ([]*Source, error)
- func (c *Collection) RenameSource(oldHandle, newHandle string) (*Source, error)
- func (c *Collection) Scratch() *Source
- func (c *Collection) SetActive(handle string, force bool) (*Source, error)
- func (c *Collection) SetActiveGroup(group string) error
- func (c *Collection) SetScratch(handle string) (*Source, error)
- func (c *Collection) Sources() []*Source
- func (c *Collection) SourcesInGroup(group string) ([]*Source, error)
- func (c *Collection) String() string
- func (c *Collection) Tree(fromGroup string) (*Group, error)
- func (c *Collection) UnmarshalJSON(b []byte) error
- func (c *Collection) UnmarshalYAML(unmarshal func(any) error) error
- func (c *Collection) Visit(fn func(src *Source) error) error
- type Group
- type Source
Constants ¶
const ( // StdinHandle is the reserved handle for stdin pipe input. StdinHandle = "@stdin" // ActiveHandle is the reserved handle for the active source. // // TODO: it should be possible to use "@0" as the active handle, but // the SLQ grammar doesn't currently allow it. Possibly change this // value to "@0" after modifying the SLQ grammar. ActiveHandle = "@active" // ScratchHandle is the reserved handle for the scratch source. ScratchHandle = "@scratch" // JoinHandle is the reserved handle for the join db source. JoinHandle = "@join" // MonotableName is the table name used for "mono-table" drivers // such as CSV. Thus a source @address_csv will have its // data accessible via @address_csv.data. MonotableName = "data" )
const (
// RootGroup is the identifier for the default root group.
RootGroup = "/"
)
Variables ¶
This section is empty.
Functions ¶
func Contains ¶ added in v0.33.0
Contains returns true if srcs contains s, where s is a Source or a source handle.
func Handle2SafePath ¶ added in v0.47.0
Handle2SafePath returns a string derived from handle that is safe to use as a file path.
func IsValidGroup ¶ added in v0.33.0
IsValidGroup returns true if group is a valid group. Examples:
/ prod prod/customer prod/customer/pg
Note that "/" is a special case, representing the root group.
func IsValidHandle ¶ added in v0.33.0
IsValidHandle returns false if handle is not a valid handle.
See also: ValidHandle.
func ParseTableHandle ¶
ParseTableHandle attempts to parse a SLQ source handle and/or table name. Surrounding whitespace is trimmed. Examples of valid input values are:
@handle.tblName @handle .tblName
func RedactGroup ¶ added in v0.34.0
func RedactGroup(g *Group)
RedactGroup modifies g, cloning each descendant Source, and setting the Source.Location field of each contained source to its redacted value.
func ReservedHandles ¶
func ReservedHandles() []string
ReservedHandles returns a slice of the handle names that are reserved for sq use.
func SortGroups ¶ added in v0.33.0
func SortGroups(groups []*Group)
SortGroups sorts a slice of groups by name.
func SuggestHandle ¶
func SuggestHandle(coll *Collection, typ drivertype.Type, loc string) (string, error)
SuggestHandle suggests a handle based on location and type. If typ is TypeNone, the type will be inferred from loc. The takenFn is used to determine if a suggested handle is free to be used (e.g. "@csv/sakila" -> "@csv/sakila1", etc).
If the base name (derived from loc) contains illegal handle runes, those are replaced with underscore. If the handle starts with a number or underscore, it will be prefixed with "h" (for "handle"). Thus "123.xlsx" becomes "@h123_xlsx".
func ValidGroup ¶ added in v0.33.0
ValidGroup returns an error if group is not a valid group name.
func ValidHandle ¶ added in v0.33.0
ValidHandle returns an error if handle is not an acceptable source handle value. Valid input must match:
\A@([a-zA-Z][a-zA-Z0-9_]*)(/[a-zA-Z][a-zA-Z0-9_]*)*$
Examples:
@handle @group/handle @group/sub/sub2/handle
See also: IsValidHandle.
func VerifyIntegrity ¶ added in v0.34.0
func VerifyIntegrity(coll *Collection) (repaired bool, err error)
VerifyIntegrity verifies the internal state of coll. Typically this func is invoked after coll has been loaded from config, verifying that the config is not corrupt. If err is returned non-nil, repaired may be returned true to indicate that coll has been repaired and modified. The caller should save the config to persist the repair.
Types ¶
type Collection ¶ added in v0.34.0
type Collection struct {
// contains filtered or unexported fields
}
Collection is a set of sources. Typically it is loaded from config at a start of a run. Collection's methods are safe for concurrent use.
func (*Collection) Active ¶ added in v0.34.0
func (c *Collection) Active() *Source
Active returns the active source, or nil if no active source.
func (*Collection) ActiveGroup ¶ added in v0.34.0
func (c *Collection) ActiveGroup() string
ActiveGroup returns the active group, which may be the root group, represented by "/".
func (*Collection) ActiveHandle ¶ added in v0.34.0
func (c *Collection) ActiveHandle() string
ActiveHandle returns the handle of the active source, or empty string if no active src.
func (*Collection) Add ¶ added in v0.34.0
func (c *Collection) Add(src *Source) error
Add adds src to s.
func (*Collection) Clone ¶ added in v0.34.0
func (c *Collection) Clone() *Collection
Clone returns a deep copy of s. If s is nil, nil is returned.
func (*Collection) Data ¶ added in v0.34.0
func (c *Collection) Data() any
Data returns the internal representation of the set data. This is a filthy hack so that the internal data can be passed directly to sq's colorizing json encoder (it can't handle colorization of values that implement json.Marshaler).
There are two long-term solutions here:
- The color encoder needs to be able to handle json.RawMessage.
- Refactor source.Collection so that it doesn't have this weird internal representation.
func (*Collection) Get ¶ added in v0.34.0
func (c *Collection) Get(handle string) (*Source, error)
Get gets the src with handle, or returns an error.
func (*Collection) Groups ¶ added in v0.34.0
func (c *Collection) Groups() []string
Groups returns the sorted set of groups, as defined via the handle names.
Given a set of handles:
@handle1 @group1/handle2 @group1/handle3 @group2/handle4 @group2/sub1/handle5 @group2/sub1/sub2/sub3/handle6
Then these groups will be returned.
/ group1 group2 group2/sub1 group2/sub1/sub2 group2/sub1/sub2/sub3
Note that default or root group is represented by "/".
func (*Collection) Handles ¶ added in v0.34.0
func (c *Collection) Handles() []string
Handles returns a new slice containing the set of all source handles.
func (*Collection) HandlesInGroup ¶ added in v0.34.0
func (c *Collection) HandlesInGroup(group string) ([]string, error)
HandlesInGroup returns the set of handles in the active group.
func (*Collection) IsExistingGroup ¶ added in v0.34.0
func (c *Collection) IsExistingGroup(group string) bool
IsExistingGroup returns false if group does not exist.
func (*Collection) IsExistingSource ¶ added in v0.34.0
func (c *Collection) IsExistingSource(handle string) bool
IsExistingSource returns true if handle already exists in the set.
func (*Collection) MarshalJSON ¶ added in v0.34.0
func (c *Collection) MarshalJSON() ([]byte, error)
MarshalJSON implements json.Marshaler.
func (*Collection) MarshalYAML ¶ added in v0.34.0
func (c *Collection) MarshalYAML() (any, error)
MarshalYAML implements yaml.Marshaler.
func (*Collection) MoveHandleToGroup ¶ added in v0.34.0
func (c *Collection) MoveHandleToGroup(handle, toGroup string) (*Source, error)
MoveHandleToGroup moves renames handle to be in toGroup.
$ sq mv @prod/db production @production/db $ sq mv @prod/db / @db
func (*Collection) Remove ¶ added in v0.34.0
func (c *Collection) Remove(handle string) error
Remove removes from the set the src having handle.
func (*Collection) RemoveGroup ¶ added in v0.34.0
func (c *Collection) RemoveGroup(group string) ([]*Source, error)
RemoveGroup removes all sources that are children of group. The removed sources are returned. If group was the active group, the active group is set to "/" (root group).
func (*Collection) RenameGroup ¶ added in v0.34.0
func (c *Collection) RenameGroup(oldGroup, newGroup string) ([]*Source, error)
RenameGroup renames oldGroup to newGroup. Each affected source is returned. This effectively "moves" sources in oldGroup to newGroup, by renaming those sources.
func (*Collection) RenameSource ¶ added in v0.34.0
func (c *Collection) RenameSource(oldHandle, newHandle string) (*Source, error)
RenameSource renames oldHandle to newHandle. If the source was the active source, it remains so (under the new handle). If the source's group was the active group and oldHandle was the only member of the group, newHandle's group becomes the new active group.
func (*Collection) Scratch ¶ added in v0.34.0
func (c *Collection) Scratch() *Source
Scratch returns the scratch source, or nil.
func (*Collection) SetActive ¶ added in v0.34.0
func (c *Collection) SetActive(handle string, force bool) (*Source, error)
SetActive sets the active src, or unsets any active src if handle is empty (and thus returns nil,nil). If handle does not exist, an error is returned, unless arg force is true. In which case, the returned *Source may be nil.
TODO: Revisit SetActive(force) mechanism. It's a hack that we shouldn't need.
func (*Collection) SetActiveGroup ¶ added in v0.34.0
func (c *Collection) SetActiveGroup(group string) error
SetActiveGroup sets the active group, returning an error if group does not exist.
func (*Collection) SetScratch ¶ added in v0.34.0
func (c *Collection) SetScratch(handle string) (*Source, error)
SetScratch sets the scratch src to handle. If handle is empty string, the scratch src is unset, and nil,nil is returned.
func (*Collection) Sources ¶ added in v0.34.0
func (c *Collection) Sources() []*Source
Sources returns a new slice containing the set's sources. It is safe to mutate the returned slice, but note that changes to the *Source elements themselves do take effect in the set's backing data.
func (*Collection) SourcesInGroup ¶ added in v0.34.0
func (c *Collection) SourcesInGroup(group string) ([]*Source, error)
SourcesInGroup returns all sources that are descendants of group. If group is "" or "/", all sources are returned.
func (*Collection) String ¶ added in v0.34.0
func (c *Collection) String() string
String returns a log/debug friendly representation.
func (*Collection) Tree ¶ added in v0.34.0
func (c *Collection) Tree(fromGroup string) (*Group, error)
Tree returns a new Group representing the structure of the set starting at fromGroup downwards. If fromGroup is empty, RootGroup is used. The Group structure is a snapshot of the Collection at the time Tree is invoked. Thus, any change to Collection structure is not reflected in the Group. However, the Source elements of Group are pointers back to the Collection elements, and thus changes to the fields of a Source are reflected in the Collection.
func (*Collection) UnmarshalJSON ¶ added in v0.34.0
func (c *Collection) UnmarshalJSON(b []byte) error
UnmarshalJSON implements json.Unmarshaler.
func (*Collection) UnmarshalYAML ¶ added in v0.34.0
func (c *Collection) UnmarshalYAML(unmarshal func(any) error) error
UnmarshalYAML implements yaml.Unmarshaler.
type Group ¶ added in v0.33.0
type Group struct { // Name is the group name. For the root group, this is source.RootGroup ("/"). Name string `json:"name" yaml:"name"` // Active is true if this is the active group in the set. Active bool `json:"active" yaml:"active"` // Sources are the direct members of the group. Sources []*Source `json:"sources,omitempty" yaml:"sources,omitempty"` // Groups holds any subgroups. Groups []*Group `json:"groups,omitempty" yaml:"groups,omitempty"` }
Group models the hierarchical group structure of a set.
func (*Group) AllGroups ¶ added in v0.33.0
AllGroups returns a new flattened slice of Groups containing g and any subgroups.
func (*Group) AllSources ¶ added in v0.33.0
AllSources returns a new flattened slice of *Source containing all the sources in g and its descendants.
type Source ¶
type Source struct { // Handle is used to refer to a source, e.g. "@sakila". Handle string `yaml:"handle" json:"handle"` // Type is the driver type, e.g. postgres.Type. Type drivertype.Type `yaml:"driver" json:"driver"` // Location is the source location, such as a DB connection URI, // or a file path. Location string `yaml:"location" json:"location"` // Catalog, when non-empty, specifies a catalog (database) name // override to use when constructing table references. // // For example, if the SLQ table selector is "actor", the query // might be rendered as 'SELECT * FROM actor'. But if Catalog is // set to "sakila" and Schema to "dbo", then that same query would // be rendered as 'SELECT * FROM sakila.dbo.actor'. // // Note that although Source.Catalog is not exposed to the end user, // this field is used by the SQL renderer to construct table references, // especially for SQL Server. Because SQL Server doesn't permit setting the // default schema on a per-connection basis, we construct a fully-qualified // table reference using the catalog and schema, e.g. "sakila.dbo.actor". // // See also: Source.Schema. Catalog string `yaml:"catalog,omitempty" json:"catalog,omitempty"` // Schema, when non-empty, specifies a schema name // override to use when constructing table references. // // See also: Source.Catalog. Schema string `yaml:"schema,omitempty" json:"schema,omitempty"` // Options are additional params, typically empty. Options options.Options `yaml:"options,omitempty" json:"options,omitempty"` }
Source describes a data source.
func RedactSources ¶ added in v0.33.0
RedactSources returns a new slice, where each element is a clone of the input *Source with its location field redacted. This is useful for printing.
func (*Source) Clone ¶ added in v0.23.0
Clone returns a deep copy of s. If s is nil, nil is returned.
func (*Source) Group ¶ added in v0.33.0
Group returns the source's group. If s is in the root group, the empty string is returned.
REVISIT: For root group, should "/" be returned instead of empty string?
func (*Source) RedactedLocation ¶
RedactedLocation returns s.Location, with the password component of the location masked.
func (*Source) ShortLocation ¶
ShortLocation returns a short location string. For example, the base name (data.xlsx) for a file or for a DSN, user@host[:port]/db.
Directories
¶
Path | Synopsis |
---|---|
Package drivertype defines drivertype.Type, which is the type of a driver, e.g.
|
Package drivertype defines drivertype.Type, which is the type of a driver, e.g. |
Package location contains functionality related to source location.
|
Package location contains functionality related to source location. |
Package metadata contains types that model source metadata.
|
Package metadata contains types that model source metadata. |