visualizer

README

Jenkins X Pipelines Visualizer

This is a Web UI for Jenkins X, with a clear goal: visualize the pipelines - and their logs.

See the announcement blog post for more details and a demo.

Current Status

This project has been started in September 2020 and shared after a couple of hours of work. It's working - and is deployed and used at Dailymotion - even if the UI is very basic for the moment.

Note that it is being used on GKE with logs stored in a GCS bucket.

Roadmap

• Improve the UI: visualize the pipeline (stages/steps), group logs per stage/step, and so on
• Split the API and the UI, to make it easier to iterate on the UI

Features

• very simple
• very fast: get your logs in milliseconds, not seconds. Yes, I'm looking at you, JXUI.
• either retrieve the build logs from the persistent storage (tested with GCS), or stream them from the running pods if the pipeline is still running.
• retrieve the build logs even for the garbage-collected pipelines (the JXUI just returns 404)
• read-only. Only requires READ permissions on the JX and Tekton Pipelines CRDs
• URLs backward-compatible with JXUI - so that you can easily swap the JXUI URL for the jx-pipelines-visualizer one in the Lighthouse config, and have Lighthouse set links to jx-pipelines-visualizer in GitHub Pull Requests.

Out of scope

• Auth - use a reverse-proxy in front or anything else to handle it
  • for example Vouch and Okta
  • or dex, oauth2-proxy, ...
• Create/Update/Delete operations. It is meant to be a read-only web UI
• Anything in JX which is not related to the pipelines

Installation

With Jenkins X v2

In the Git repository for your dev environment:

• Update the env/requirements.yaml file with the following:

  - name: jx-pipelines-visualizer
    repository: https://dailymotion.github.io/jx-pipelines-visualizer/charts/
    version: 0.0.3
  

• Create a new file env/jx-pipelines-visualizer/values.tmpl.yaml with the following content:

  {{- if .Requirements.storage.logs.enabled }}
  config:
    archivedLogsURLTemplate: >-
      {{ .Requirements.storage.logs.url }}{{`/jenkins-x/logs/{{.Owner}}/{{.Repository}}/{{if hasPrefix .Branch "pr"}}{{.Branch | upper}}{{else}}{{.Branch}}{{end}}/{{.Build}}.log`}}
  {{- end }}
  
  ingress:
    enabled: true
    hosts:
      - pipelines{{.Requirements.ingress.namespaceSubDomain}}{{.Requirements.ingress.domain}}
    {{- if .Requirements.ingress.tls.enabled }}
    tls:
      enabled: true
      secrets:
        # re-use the existing tls secret managed by jx
        {{- if .Requirements.ingress.tls.production }}
        tls-{{ .Requirements.ingress.domain | replace "." "-" }}-p: {}
        {{- else }}
        tls-{{ .Requirements.ingress.domain | replace "." "-" }}-s: {}
        {{- end }}
    {{- end }}
    annotations:
      kubernetes.io/ingress.class: nginx
  

• If you want Lighthouse to add links to your jx-pipelines-visualizer instance from your Pull/Merge Request checks, update the env/lighthouse-jx/values.tmpl.yaml file and add the following:

  env:
    LIGHTHOUSE_REPORT_URL_BASE: "https://pipelines{{.Requirements.ingress.namespaceSubDomain}}{{.Requirements.ingress.domain}}"
  

This will expose the UI at pipelines.your.domain.tld - without any auth. You can add basic auth with a few annotations - re-using the Jenkins X Auth Secret:

nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/auth-secret: jx-basic-auth

With Helm v3

$ helm repo add jx-pipelines-visualizer https://dailymotion.github.io/jx-pipelines-visualizer/charts/
$ helm install jx-pipelines-visualizer jx-pipelines-visualizer/jx-pipelines-Visualizer

With Helm v2

$ helm repo add jx-pipelines-visualizer https://dailymotion.github.io/jx-pipelines-visualizer/charts/
$ helm repo update
$ helm install --name jx-pipelines-visualizer jx-pipelines-visualizer/jx-pipelines-visualizer

Configuration

See the values.yaml file for the configuration.

If you are not using the Helm Chart, the binary is using CLI flags only - no config files. You can run jx-pipelines-visualizer -h to see all the flags.

Running locally

go run cmd/server/main.go

How It Works

It uses the "informer" Kubernetes pattern to keep a local cache of the Jenkins X PipelineActivities, and index them in an in-memory Bleve index.

It uses part of jx code to retrieve the build logs - mainly the part to stream the build logs from the running pods. It is the same code used by the jx get build logs command.

Documentation

Index

• type Informer
  • func (i *Informer) Start(ctx context.Context)
• type Pipeline
  • func PipelineFromPipelineActivity(pa *jenkinsv1.PipelineActivity) Pipeline
• type Pipelines
• type Query
  • func (q Query) ToBleveQuery() query.Query
• type Store
  • func NewStore() (*Store, error)
  • func (s *Store) Add(p Pipeline) error
  • func (s *Store) All() (*Pipelines, error)
  • func (s *Store) Delete(name string) error
  • func (s *Store) Query(q Query) (*Pipelines, error)

Types

type Informer

type Informer struct {
	JXClient       *jxclientset.Clientset
	Namespace      string
	ResyncInterval time.Duration
	Store          *Store
	Logger         *logrus.Logger
}

func (*Informer) Start

func (i *Informer) Start(ctx context.Context)

type Pipeline

type Pipeline struct {
	Name            string
	Owner           string
	Repository      string
	Branch          string
	Build           string
	Context         string
	Author          string
	AuthorAvatarURL string
	Commit          string
	Status          string
	Start           time.Time
	End             time.Time
	Duration        time.Duration
}

func PipelineFromPipelineActivity

func PipelineFromPipelineActivity(pa *jenkinsv1.PipelineActivity) Pipeline

type Pipelines

type Pipelines struct {
	Pipelines []Pipeline
	Counts    struct {
		Owners       map[string]int
		Repositories map[string]int
		Branches     map[string]int
		Builds       map[string]int
	}
}

type Query

type Query struct {
	Owner      string
	Repository string
	Branch     string
}

func (Query) ToBleveQuery

func (q Query) ToBleveQuery() query.Query

type Store

type Store struct {
	Index bleve.Index
}

func NewStore

func NewStore() (*Store, error)

func (*Store) Add

func (s *Store) Add(p Pipeline) error

func (*Store) All

func (s *Store) All() (*Pipelines, error)

func (*Store) Delete

func (s *Store) Delete(name string) error

func (*Store) Query

func (s *Store) Query(q Query) (*Pipelines, error)