lighthouse

module
v1.1.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 7, 2021 License: Apache-2.0

README

Lighthouse

Lighthouse is a lightweight ChatOps based webhook handler which can trigger Jenkins X Pipelines, Tekton Pipelines or Jenkins Jobs based on webhooks from multiple git providers such as GitHub, GitHub Enterprise, BitBucket Server and GitLab.

Installing

Lighthouse is bundled and released as Helm Chart. You find the install instruction in the Chart's README.

Depending on the pipeline engine you want to use, you can find more detailed instructions in one of the following documents:

Background

Lighthouse derived originally from Prow and started with a copy of its essential code.

Currently, Lighthouse supports the standard Prow plugins and handles push webhooks to branches to then trigger a pipeline execution on the agent of your choice.

Lighthouse uses the same config.yaml and plugins.yaml for configuration than Prow.

Comparisons to Prow

Lighthouse reuses the Prow plugin source code and a bunch of plugins from Prow

Its got a few differences though:

  • rather than being GitHub specific Lighthouse uses jenkins-x/go-scm so it can support any Git provider
  • Lighthouse does not use a ProwJob CRD; instead, it has its own LighthouseJob CRD.
Porting Prow commands

If there are any prow commands you want which we've not yet ported over, it is relatively easy to port Prow plugins.

We've reused the prow plugin code and configuration code; so it is mostly a case of switching imports of k8s.io/test-infra/prow to github.com/jenkins-x/lighthouse/pkg/prow, then modifying the GitHub client structs from, say, github.PullRequest to scm.PullRequest.

Most of the GitHub structs map 1-1 to the jenkins-x/go-scm equivalents (e.g. Issue, Commit, PullRequest). However, the go-scm API does tend to return slices to pointers to resources by default. There are some naming differences in different parts of the API as well. For example, compare the githubClient API for Prow lgtm versus the Lighthouse lgtm.

Development

Building

To build the code, fork and clone this git repository, then type:

make build

make build will build all relevant Lighthouse binaries natively for your OS which you then can run locally. For example, to run the webhook controller, you would type:

./bin/webhooks

To see which other Make rules are available, run:

make help
Environment variables

While Prow only supports GitHub as SCM provider, Lighthouse supports several Git SCM providers. Lighthouse achieves the abstraction over the SCM provider using the go-scm library. To configure your SCM, go-scm uses the following environment variables :

Name Description
GIT_KIND the kind of git server: github, gitlab, bitbucket, gitea, stash
GIT_SERVER the URL of the server if not using the public hosted git providers: https://github.com, https://bitbucket.org or https://gitlab.com
GIT_USER the git user (bot name) to use on git operations
GIT_TOKEN the git token to perform operations on git (add comments, labels etc.)
HMAC_TOKEN the token sent from the git provider in webhooks
Testing

To run the unit tests, type:

make test

For development purposes, it is also nice to start an instance of the binary you want to work. Provided you have a connection to a cluster with Lighthouse installed, the locally started controller will join the cluster, and you can test your development changes directly in-cluster.

For example to run the webhook controller locally:

make build
GIT_TOKEN=<git-token> ./bin/webhooks -namespace <namespace> -bot-name <git-bot-user>

In the case of the webhook controller, you can also test webhook deliveries locally using a ngrok tunnel. Install ngrok and start a new tunnel:

$ ngrok http 8080
ngrok by @inconshreveable                                                                                                                                                                     (Ctrl+C to quit)

Session Status                online
Account                       ***
Version                       2.3.35
Region                        United States (us)
Web Interface                 http://127.0.0.1:4040
Forwarding                    http://e289dd1e1245.ngrok.io -> http://localhost:8080
Forwarding                    https://e289dd1e1245.ngrok.io -> http://localhost:8080

Connections                   ttl     opn     rt1     rt5     p50     p90
                              0       0       0.00    0.00    0.00    0.00

Now you can use your ngrok URL to register a webhook handler with your Git provider.

NOTE Remember to append /hook to the generated ngrok URL. In the case of the above example http://e289dd1e1245.ngrok.io/hook

Any events that happen on your Git provider are now sent to your local webhook instance.

Debugging Lighthouse

You can setup a remote debugger for Lighthouse using delve via:

dlv --listen=:2345 --headless=true --api-version=2 exec ./bin/lighthouse -- $*  

You can then debug from your Go-based IDE (e.g. GoLand / IDEA / VS Code).

Debugging webhooks

If you want to debug lighthouse locally from webhooks in your cluster there are a couple of tools that could help:

Localizer

If you install localizer (see the blog for more detail you can easily debug webhooks on your cluster.

  • first run localizer:
sudo localizer

Then run/debug lighthouse locally.

e.g. in your IDE run the cmd/webhooks/main.go (passing --namespace jx as program arguments)

Then to get the webhooks to trigger your local process:

localizer expose jx/hook --map 80:8080

when you have finished debugging, return things to normal via:

localizer expose jx/hook --stop
Telepresence

You can replace the running version in your cluster with the one running locally using telepresence.
For webhooks, just run:

telepresence --swap-deployment lighthouse-webhooks --env-file /tmp/webhooks-env

in another terminal:

export $(cat /tmp/webhooks-env | xargs)
dlv --listen=:2345 --headless=true --api-version=2 exec ./bin/webhooks -- --namespace=jx

You can do the same for any other deployment (keeper, foghorn...), just make sur to check the command args used for it an set them instead of --namespace=jx.

Using a local go-scm

If you are hacking on support for a specific Git provider, you may find yourself working on the Lighthouse code and the jenkins-x/go-scm code together. Go modules lets you easily swap out the version of a dependency with a local copy of the code; so you can edit code in Lighthouse and jenkins-x/go-scm at the same time.

Just add this line to the end of your go.mod file:

replace github.com/jenkins-x/go-scm => /workspace/go/src/github.com/jenkins-x/go-scm

Using the exact path to where you cloned jenkins-x/go-scm

Directories

Path Synopsis
cmd
gc
hack module
pkg
apis/lighthouse/v1alpha1
Package v1alpha1 contains LighthouseJob +k8s:openapi-gen=true +k8s:deepcopy-gen=package,register +k8s:conversion-gen=github.com/jenkins-x/lighthouse/pkg/apis/lighthouse +k8s:defaulter-gen=TypeMeta +groupName=lighthouse.jenkins.io
Package v1alpha1 contains LighthouseJob +k8s:openapi-gen=true +k8s:deepcopy-gen=package,register +k8s:conversion-gen=github.com/jenkins-x/lighthouse/pkg/apis/lighthouse +k8s:defaulter-gen=TypeMeta +groupName=lighthouse.jenkins.io
client/clientset/versioned
This package has the automatically generated clientset.
This package has the automatically generated clientset.
client/clientset/versioned/fake
This package has the automatically generated fake clientset.
This package has the automatically generated fake clientset.
client/clientset/versioned/scheme
This package contains the scheme of the automatically generated clientset.
This package contains the scheme of the automatically generated clientset.
client/clientset/versioned/typed/lighthouse/v1alpha1
This package has the automatically generated typed clients.
This package has the automatically generated typed clients.
client/clientset/versioned/typed/lighthouse/v1alpha1/fake
Package fake has the automatically generated clients.
Package fake has the automatically generated clients.
commentpruner
Package commentpruner facilitates efficiently deleting bot comments as a reaction to webhook events.
Package commentpruner facilitates efficiently deleting bot comments as a reaction to webhook events.
config
Package config knows how to read and parse config.yaml.
Package config knows how to read and parse config.yaml.
config/secret
Package secret implements an agent to read and reload the secrets.
Package secret implements an agent to read and reload the secrets.
engines/jenkins
Package jenkins includes a client and the operational logic for managing Jenkins masters in prow.
Package jenkins includes a client and the operational logic for managing Jenkins masters in prow.
errorutil
Package errorutil provides utilities for errors
Package errorutil provides utilities for errors
genfiles
Package genfiles understands the .generated_files config file.
Package genfiles understands the .generated_files config file.
git
git/localgit
Package localgit creates a local git repo that can be used for testing code that uses a git.Client.
Package localgit creates a local git repo that can be used for testing code that uses a git.Client.
interrupts
Package interrupts exposes helpers for graceful handling of interrupt signals
Package interrupts exposes helpers for graceful handling of interrupt signals
jobutil
Package jobutil contains helpers for working with LighthouseJobs.
Package jobutil contains helpers for working with LighthouseJobs.
keeper
Package keeper contains a controller for managing a keeper pool of PRs.
Package keeper contains a controller for managing a keeper pool of PRs.
keeper/history
Package history provides an append only, size limited log of recent actions that Keeper has taken for each subpool.
Package history provides an append only, size limited log of recent actions that Keeper has taken for each subpool.
logrusutil
Package logrusutil implements some helpers for using logrus
Package logrusutil implements some helpers for using logrus
metrics
Package metrics contains utilities for working with metrics in prow.
Package metrics contains utilities for working with metrics in prow.
pluginhelp
Package pluginhelp defines structures that represent plugin help information.
Package pluginhelp defines structures that represent plugin help information.
plugins/blockade
Package blockade defines a plugin that adds the 'do-not-merge/blocked-paths' label to PRs that modify protected file paths.
Package blockade defines a plugin that adds the 'do-not-merge/blocked-paths' label to PRs that modify protected file paths.
plugins/cat
Package cat adds cat images to an issue or PR in response to a /meow comment
Package cat adds cat images to an issue or PR in response to a /meow comment
plugins/cherrypickunapproved
Package cherrypickunapproved adds the `do-not-merge/cherry-pick-not-approved` label to PRs against a release branch which do not have the `cherry-pick-approved` label.
Package cherrypickunapproved adds the `do-not-merge/cherry-pick-not-approved` label to PRs against a release branch which do not have the `cherry-pick-approved` label.
plugins/dog
Package dog adds dog images to the issue or PR in response to a /woof comment
Package dog adds dog images to the issue or PR in response to a /woof comment
plugins/hold
Package hold contains a plugin which will allow users to label their own pull requests as not ready or ready for merge.
Package hold contains a plugin which will allow users to label their own pull requests as not ready or ready for merge.
plugins/lgtm
Package lgtm implements the lgtm plugin
Package lgtm implements the lgtm plugin
plugins/milestone
Package milestone implements the `/milestone` command which allows members of the milestone maintainers team to specify a milestone to be applied to an Issue or PR.
Package milestone implements the `/milestone` command which allows members of the milestone maintainers team to specify a milestone to be applied to an Issue or PR.
plugins/milestonestatus
Package milestonestatus implements the `/status` command which allows members of the milestone maintainers team to specify a `status/*` label to be applied to an Issue or PR.
Package milestonestatus implements the `/status` command which allows members of the milestone maintainers team to specify a `status/*` label to be applied to an Issue or PR.
plugins/override
Package override supports the /override context command.
Package override supports the /override context command.
plugins/pony
Package pony adds pony images to the issue or PR in response to a /pony comment
Package pony adds pony images to the issue or PR in response to a /pony comment
plugins/sigmention
Package sigmention recognize SIG '@' mentions and adds 'sig/*' and 'kind/*' labels as appropriate.
Package sigmention recognize SIG '@' mentions and adds 'sig/*' and 'kind/*' labels as appropriate.
plugins/size
Package size contains a Prow plugin which counts the number of lines changed in a pull request, buckets this number into a few size classes (S, L, XL, etc), and finally labels the pull request with this size.
Package size contains a Prow plugin which counts the number of lines changed in a pull request, buckets this number into a few size classes (S, L, XL, etc), and finally labels the pull request with this size.
plugins/skip
Package skip implements the `/skip` command which allows users to clean up commit statuses of non-blocking presubmits on PRs.
Package skip implements the `/skip` command which allows users to clean up commit statuses of non-blocking presubmits on PRs.
plugins/stage
Package stage defines a Prow plugin that defines the stage of the issue in the features process.
Package stage defines a Prow plugin that defines the stage of the issue in the features process.
plugins/welcome
Package welcome implements a prow plugin to welcome new contributors
Package welcome implements a prow plugin to welcome new contributors
plugins/wip
Package wip will label a PR a work-in-progress if the author provides a prefix to their pull request title to the same effect.
Package wip will label a PR a work-in-progress if the author provides a prefix to their pull request title to the same effect.
scmprovider/reporter
Package reporter contains helpers for writing comments in scm providers.
Package reporter contains helpers for writing comments in scm providers.
test
e2e

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL