Documentation
¶
Overview ¶
Package configure maintains all related structures for the Jacamar CI configuration through the supplied TOML file. All interactions with the config should be accomplished using the supported Configurer interface.
Index ¶
Constants ¶
View Source
const (
// EnvVariable is expected in the job's environment for the stateful configuration.
EnvVariable = "JACAMAR_CI_CONFIG_STR"
)
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Auth ¶
type Auth struct {
// UserAllowlist is an authoritative list of users who can execute CI jobs.
UserAllowlist []string `toml:"user_allowlist"`
// UserBlocklist is a list of usernames that are not allowed to run CI jobs. More
// authoritative than group lists, but can be overridden by UserAllowlist.
UserBlocklist []string `toml:"user_blocklist"`
// GroupAllowlist is a list of groups that are allowed to run CI jobs. Least authoritative.
GroupAllowlist []string `toml:"groups_allowlist"`
// GroupBlocklist is a list of groups that are not allowed to run CI jobs.
GroupBlocklist []string `toml:"groups_blocklist"`
// ShellAllowlist is an authoritative list of acceptable shells that are defined in the
// user database.
ShellAllowlist []string `toml:"shell_allowlist"`
// PipelineSourceAllowlist is an authoritative list of acceptable CI pipeline sources.
PipelineSourceAllowlist []string `toml:"pipeline_source_allowlist"`
// RunStageAllowlist is an authoritative list of sub-stages within Run
// (see: https://docs.gitlab.com/runner/executors/custom.html#run) that can be executed.
// When configured any stage encountered that is not on this list will be skipped
// with a message to the user.
RunStageAllowlist []string `toml:"run_stage_allowlist"`
// Downscope target mechanism for execution of all CI scripts and generated
// commands through the auth mechanisms (e.g. setuid).
Downscope string `toml:"downscope"`
// JacamarPath full path to the Jacamar application, used in constructing the
// command for job execution.
JacamarPath string `toml:"jacamar_path"`
// DownscopeEnv is an array of "key=value" pairs that will be used when constructing the
// environment for the downscoped command. Care must be given as values provided here take
// the highest priority and will override any identified by Jacamar.
DownscopeEnv []string `toml:"downscope_env"`
// DownscopeCmdDir is a development/debug only command allow direct manipulation of the
// working directory for any downscope command generate (see: https://golang.org/pkg/os/exec/#Cmd).
DownscopeCmdDir string `toml:"downscope_cmd_dir"`
// MaxEnvChars is the maximum number of characters one can have in the environment
// (default: 10000)
MaxEnvChars int `toml:"max_env_chars"`
// ListsPreValidation indicates if the allow/block list rules should be observed
// prior to the execution of the RunAS validate script.
ListsPreValidation bool `toml:"lists_pre_validation"`
// RootDirCreation indicates if the trusted root level directory (e.g. 0700 permissions)
// will be created and chown called.
RootDirCreation bool `toml:"root_dir_creation"`
// AllowBotAccounts indicates if project/group bot accounts (e.g., project_{number}_bot)
// should be allowed in downscoping workflows. By default, encountering these results
// in job failure if a valid downscoping method is defined.
AllowBotAccounts bool `toml:"allow_bot_accounts"`
// JWTExpDelay a configurable delay allowed in a JWTs expiration in select cases to allow
// for automated cleanup related actions even after a job has expired. This will only be
// allowed in stages related to artifacts, caching, and cleanup_exec (when stateful variables
// are present). By default, this will be set to "15m".
JWTExpDelay string `toml:"jwt_exp_delay"`
// JWTRequiredAud defined a required 'aud' when checking the claim.
JWTRequiredAud string `toml:"jwt_required_aud"`
// DevMode allows select rules/configurations to be bypassed for the purpose of development
// and testing. Not designed for production deployment and may lead to failing jobs.
DevMode bool `toml:"dev_mode"`
// NoNewPrivs enforces PR_SET_NO_NEW_PRIVS, functional without seccomp filters.
NoNewPrivs bool `toml:"no_new_privs"`
// EnforceNoLogin indicates that jobs should be blocked during configuration if a pam_nologin
// file (/etc/nologin or /var/run/nologin) is encountered. The contents of this file will
// be presented to the user in their job log.
EnforceNoLogin bool `toml:"enforce_nologin"`
// TokenScopeEnforced (deprecated in server version 17+) ensures all incoming jobs have the
// 'ci_job_token_scoped_enabled' at time of job authorization, obtained via the /jobs endpoint
// using the CI_JOB_TOKEN.
TokenScopeEnforced bool `toml:"job_token_scope_enforced"`
RunAs RunAs `toml:"runas"`
Logging Logging `toml:"logging"`
Seccomp Seccomp `toml:"seccomp"`
}
Auth represents authorization process configuration for establishing a local target account that is associated with the GitLab user who trigger the job.
type Batch ¶
type Batch struct {
// ArgumentsVariable potential CI variables for user provided arguments for job submission
// (default (always enforced): SCHEDULER_PARAMETERS).
ArgumentsVariable []string `toml:"arguments_variable"`
// CommandDelay is the sleep time for all command line interactions with the underlying
// scheduler, this is a duration string (default: 30s).
CommandDelay string `toml:"command_delay"`
// NFSTimeout is the largest possible delay expected from NFS for files, this is a
// duration string (e.g. 1m).
NFSTimeout string `toml:"nfs_timeout"`
// SchedulerBin a directory path used as prefix for all jop submission commands
// (e.g. /usr/scheduler/bin).
SchedulerBin string `toml:"scheduler_bin"`
// SkipCobaltLog identifies that the job status found in the CobaltLog should be skipped
// in favor of an echo in the output file. This should only be used for test/debug purposes.
SkipCobaltLog bool `toml:"skip_cobalt_log"`
// EnvVars list of key=value pairs that are used when building job submission command.
EnvVars []string `toml:"env_vars"`
// AllowIllegalArgs enabled will accept all arguments from a user's variable regardless
// of potential for collision with default workflows.
AllowIllegalArgs bool `toml:"allow_illegal_args"`
// LSFJobCancellation enables the use of 'bkill' to signal a running job it's time to stop
// based upon a runner generated SIGTERM. This also enforces that "-J" will be an illegal argument.
// Note that future releases should seek to enforce this by default as it provide a stronger/safer
// mechanism to ensure jobs are canceled; however, we need to ensure all currently supported
// deployed will not be affected by the change first with the new 'bsub' argument.
LSFJobCancellation bool `toml:"lsf_job_cancellation"`
// FFSlurmSacct [Feature Flag] enables the secondary job verification via sacct upon completion
// of sbatch. This exists to verify edge cases do not result in pending/running jobs or CI job
// status from incorrectly account for canceled job.
FFSlurmSacct bool `toml:"ff_slurm_sacct"`
}
Batch represents configuration related to HPC schedulers.
type Config ¶
type Config struct {
// contains filtered or unexported fields
}
Config maintains all identified options applicable to the running job.
func NewConfig ¶
NewConfig accepts a target configuration, either a complete file path or configure.EnvVariable. The appropriate contents of the associated target will be loaded and made available through the returned interface.
func (Config) PrepareNotification ¶
PrepareNotification generates and returns a valid message with key aspects of the configuration safe to convey to the user.
type Configurer ¶
type Configurer interface {
Options() Options
General() General
Auth() Auth
Batch() Batch
PrepareNotification(string) string
PrepareState(bool) (string, error)
}
Configurer implements an interface for loading and interacting with the executor's configuration and subsequent Options structure.
type General ¶
type General struct {
// Name option description, associated with config in system logs.
Name string `toml:"name"`
// Executor type, eg. shell, cobalt, slurm, etc.
Executor string `toml:"executor"`
// DataDir a directory where builds and cache are stored. If set to $HOME, each will
// be stored in the given users home directory, otherwise each will have their
// own user-specific workspace below this directory.
DataDir string `toml:"data_dir"`
// RetainLogs indicated all logs generated by the underlying system should be kept
// (default: false, removed upon job completion).
RetainLogs bool `toml:"retain_logs"`
// CustomBuildDir allow users to specify their CI_BUILDS_DIR for CI jobs, the runner will
// ensure unique paths and fail jobs if permissions are invalid.
CustomBuildDir bool `toml:"custom_build_dir"`
// KillTimeout represents the maximum timeout the Jacamar-Auth application will wait
// before sending a SIGKILL to the underlying Jacamar process.
KillTimeout string `toml:"kill_timeout"`
// Shell path/application to be used for all command execution by Jacamar and Jacamar-Auth
// only. Please note that only Bash is supported at this time.
Shell string `toml:"shell_path"`
// VariableDataDir no longer supported, error with warning if encountered.
VariableDataDir bool `toml:"variable_data_dir"`
// JobMessage is custom message that will be conveyed at the start of every prepare_exec
// stage to the user. It will appear in the CI job log.
JobMessage string `toml:"job_message"`
// GitLabServer trusted URL for GitLab server used in all web interactions, takes
// priority over any values identified in the job response.
GitLabServer string `toml:"gitlab_server"`
// TLSCAFile is the location to file containing the required certificates for HTTPS actions.
// If establish both application will leverage this for requests thar originate from the
// application. Please note runner configuration will likely be required to support its job
// script requirements.
TLSCAFile string `toml:"tls-ca-file"`
// UnrestrictedCmdline allows for unfettered usages of tokens via the command line in
// previously identified locations (git, artifact-uploader, and artifact-downloader).
// Enabling this will allow these identified commands to run without restriction and
// should only be done in control environments or when proc(5) has been mounted
// to restrict access. Removing these restrictions will also remove the GIT_ASKPASS option.
UnrestrictedCmdline bool `toml:"unrestricted_cmd_line"`
// StaticBuildsDir the builds_dir should be replaced with a static directory with job specif sub-folders
// are not re-used (differentiated with the jobid).
StaticBuildsDir bool `toml:"static_builds_dir"`
// StaticMinDays minimum numbers of days any job specific folder can remain before automated cleanup.
// Note this cleanup will only be attempted during the cleanup_exec of a job on the same data_dir.
// A value of 0 skips any cleanup (default: 7).
StaticMinDays int `toml:"static_min_days"`
// StaticCleanupLimit limits the number of folders that can be removed during cleanup at any given time.
// This is meant to avoid situations where a single job becomes delayed for an extended period of time
// during the cleanup_exec stage (default: 1).
StaticCleanupLimit int `toml:"static_cleanup_limit"`
// GroupPermissions sets base permissions on Jacamar generated data directories to allow read and execute
// access for groups (ie, 0750 permission).
GroupPermissions bool `toml:"group_permissions"`
// JWTEnvVariable defined the environment variable to be checked for the id_token
// (default: CI_JOB_JWT).
JWTEnvVariable string `toml:"jwt_env_variable"`
// OptionalLoginShell allows the users, via the JACAMAR_NO_BASH_PROFILE variable, to request that the shell
// created locally is no longer a login shell but instead use the "--norc --noprofile" arguments. This
// can (and in the case of some executors) break job in unexpected ways. Its highly advised you only
// enable this is proper documentation/support is available to users and ideally only for shell executors.
OptionalLoginShell bool `toml:"optional_login_shell"`
// RunMechanism defines a proposed run_mechanism to executor user/runner scripts (default: undefined).
RunMechanism string `toml:"run_mechanism"`
// ForceMechanism requires the defined mechanism is used for all jobs, ignoring user setting/behaviors
// that normally trigger its usage.
ForceMechanism bool `toml:"force_mechanism"`
Podman Podman `toml:"podman"`
// FFCustomDataDir [Feature Flag] allow users to specify their own data_dir via CI variables.
FFCustomDataDir bool `toml:"ff_custom_data_dir"`
// FFLimitBuildDir [Feature Flag] enforces a limited structure on the builds_dir by creating
// a user driven process to automatically obtain concurrent directories by requesting file locks
FFLimitBuildDir bool `toml:"ff_limit_build_dir"`
// MaxBuildDir indicates how many concurrent build directories can be left on the system. Only observed
// in conjunction with LimitBuildDir. (default: 0, only limited by cumulative runner concurrency).
MaxBuildDir int `toml:"max_build_dir"`
// UncapBuildDirCleanup by default cleanup is limited to a single builds_dir in every job. This
// is to limit a CI job becoming "stuck" during cleanup, during which we lack the ability to
// directly notify the user of any cleanup actions.
UncapBuildDirCleanup bool `toml:"uncap_build_dir_cleanup"`
}
General represents configuration options that can apply throughout the job (from authorization to execution).
type Logging ¶
type Logging struct {
// Enabled allows for system logging of key results from the jacamar-auth application.
Enabled bool `toml:"enabled"`
// Level denotes the logging level of messages saved (default: info).
// Note, I had a typo in the level, to avoid breaking changes relating to logging we will
// need to support both.
Level string `toml:"log_level"`
CorrectedLevel string `toml:"level"`
// Location identifies where logs will be saved, this can be a distinct file or syslog (default).
// In the case of syslog, we will connect to the appropriate log daemon, targeting the
// syslog server if related values are not specified.
Location string `toml:"location"`
// Network specified (e.g., tcp) used for remote log daemon connections.
Network string `toml:"network"`
// Address specified (e.g., localhost:1234) used for remote log daemon connections.
Address string `toml:"address"`
}
Logging represents configuration of how the jacamar-auth application will log relevant information.
type Options ¶
type Options struct {
General General `toml:"general"`
Batch Batch `toml:"batch"`
Auth Auth `toml:"auth"`
}
Options represent all defined aspects of a potential job that are directly obtained through the specified configuration or command line interaction.
type Podman ¶ added in v0.19.0
type Podman struct {
// ApplicationPath full path to the Podman application, used in constructing all commands.
ApplicationPath string `toml:"application_path"`
// RunnerImage helper image that are used by all jobs for all standard runner interactions.
RunnerImage string `toml:"runner_image"`
// RunnerOptions 'podman run ...' options that are used for runner defined job steps associated
// with the runner_image.
RunnerOptions []string `toml:"runner_options"`
// RunnerEntryPoint override the default ENTRYPOINT/COMMAND uses for runner defined job steps.
RunnerEntryPoint []string `toml:"runner_entry_point"`
// RunnerPullPolicy define policy (e.g., Always) observed by runner images.
// See: https://docs.gitlab.com/ee/ci/yaml/#imagepull_policy
RunnerPullPolicy string `toml:"runner_pull_policy"`
// DefaultImage used when no user defined 'Image' is present for the job. There is no default
// and jobs can fail if a user does not provide their own.
DefaultImage string `toml:"default_image"`
// CustomOptions any 'podman run ...' options that are used for user defined job steps.
CustomOptions []string `toml:"custom_options"`
// CustomEntryPoint override the default ENTRYPOINT/COMMAND used for user defined job steps.
CustomEntryPoint []string `toml:"user_entry_point"`
// StepScriptOnly limits the use of containers to the step_script/build_script. This is
// useful in cases where a deployment may limit containers to compute environment
// (i.e., Slurm/LSF executors).
StepScriptOnly bool `toml:"step_script_only"`
// DisableContainerRemoval prevents the '--rm' options from being used on generated commands.
DisableContainerRemoval bool `toml:"disable_container_removal"`
// ImageAllowlist when defined only images that match this list of regular expressions will
// be observed. Errors in the provided expressions will result in failed jobs. Please note
// the check is enforced by the jacamar-auth application.
ImageAllowlist []string `toml:"image_allowlist"`
// UserVolumeVariable defines the prefix for a CI variable users can leverage to mount
// volumes. Leaving this undefined means no user defined volumes will be mounted.
UserVolumeVariable string `toml:"user_volume_variable"`
// PrepScript allows for an admin defined script to be run during the prepare_exec stage.
// This aims to allow for standard influence over a user's configurations/files to ensure
// Podman to run correctly (i.e., set default configurations in a users home directory).
PrepScript string `toml:"prep_script"`
// ArchiveFormat defines how images will be archived after being pulled.
// If set to "none" these steps will be skipped. See the official documentation for details:
// https://docs.podman.io/en/latest/markdown/podman-save.1.html#format-format
// By default Jacamar will attempt to use the best option for your deployment, for example
// when a Batch executor is encountered the "docker-archive" will be used and for basic
// shell interactions the "none" option will be preferred.
ArchiveFormat string `toml:"archive_format"`
}
Podman configurations are observed in cases where the 'run_mechanism = "podman"' is configured.
type RunAs ¶
type RunAs struct {
// ValidationScript the path to a script where the RunAs user will be checked
// against known job context.
ValidationScript string `toml:"validation_script"`
// ValidationPlugin the path to a Go plugin where the RunAs user will be checked
// against known job context.
ValidationPlugin string `toml:"validation_plugin"`
// SHA256 sum of plugin/script, if provided will be checked.
SHA256 string `toml:"sha256"`
// RunAsVariable defines the name of the variable the RunAs user will be passed in as
// from the .gitlab-ci.yml file.
RunAsVariable string `toml:"user_variable"`
// ValidationEnv manages a list of "key=value" strings that an administrator can use
// to dictate additional context to the validation script. These will take lowest
// priority so avoid using the key for any existing RunAs or system environment
// variables.
ValidationEnv []string `toml:"validation_env"`
}
RunAs represents configuration of the RunAs (e.g. service user) to affect or validate the user context in the auth flow.
type Seccomp ¶
type Seccomp struct {
// Disabled globally turns off all rules.
Disabled bool `toml:"disabled"`
// BlockAll globally blocks all system calls from being used, this means reliance on
// a manually defined list of AllowCalls.
BlockAll bool `toml:"block_all"`
// BlockCalls list of system calls that will be prevented and result in error
// when encountered.
BlockCalls []string `toml:"block_calls"`
// AllowCalls list of system call that will be allowed, this take precedence over
// any manually or system defined blocked calls.
AllowCalls []string `toml:"allow_calls"`
// LimitSetuid limits potential setuid/setgid to target only the authorized user.
// This is only observed during "setuid" downscope.
LimitSetuid bool `toml:"limit_setuid"`
// TTYRules disables all rules relating to the shared terminals when dealing
// with interactive sessions (non-system services).
TTYRules bool `toml:"tty_rules"`
// FilterPlugin path to a Go plugin where the filter can be modified. Setting this
// value implies that plugin support should be enabled.
FilterPlugin string `toml:"validation_plugin"`
// DisableNoNewPrivs disables or prevents the application of PR_SET_NO_NEW_PRIVS based upon
// the usage of seccomp filters. This only applies when seccomp is enabled.
DisableNoNewPrivs bool `toml:"disable_no_new_privs"`
// LogAllowedActions sets the default to 'ActLog' which will log all system calls.
LogAllowedActions bool `toml:"log_allowed_actions"`
// ErrorNumBlockActions modifies the desired block actions and will return an error code rather
// than terminating the thread. This will cause less harsh shutdowns when a filter is
// encountered but may prove harder to troubleshoot.
ErrorNumBlockActions bool `toml:"error_num_block_actions"`
}
Seccomp organizes all configurations relating to the manage of seccomp(2) rules.
Source Files
Click to show internal directories.
Click to hide internal directories.