gcpviz

README

gcpviz - Cloud Asset Inventory visualization

gcpviz is a visualization tool that takes input from Cloud Asset Inventory, creates relationships between assets and outputs a format compatible with graphviz. You can use it to visualize all resources in the export (examples are provided for the basic use cases).

Requirements

The tool requires the following:

• Golang 1.14 (or later)
• Graphviz

Installing locally

You can install the tool locally by running:

go install github.com/GoogleCloudPlatform/professional-services/tools/gcpviz/cmd/gcpviz

Building a Docker image

Easiest way to get started is to build a container from the tool, that includes all the necessary dependencies. You can build a Docker image using the supplied Dockerfile:

docker build -t gcr.io/YOUR-PROJECT/gcpviz .
docker push gcr.io/YOUR-PROJECT/gcpviz

Running Docker image locally and generate a graph

mkdir cai
cp resource_inventory.json cai/
docker run --rm -v $PWD/cai:/gcpviz/cai gcr.io/YOUR-PROJECT/gcpviz gcpviz.sh network --query-file queries/data.js

You'll then find network.gv, network.svg and network.png under the cai/ directory.

Usage

  ██████   ██████ ██████  ██    ██ ██ ███████ 
 ██       ██      ██   ██ ██    ██ ██    ███  
 ██   ███ ██      ██████  ██    ██ ██   ███   
 ██    ██ ██      ██       ██  ██  ██  ███    
  ██████   ██████ ██        ████   ██ ███████

  -alsologtostderr
        log to standard error as well as files
  -cpuprofile file
        write cpu profile to file
  -export-file string
        location of JSON export file (default "graph.json")
  -graph-file string
        location of Graph & Asset database file (default "graph.db")
  -graph-parameter value
        override graph style parameters using SJSON (ie. "options.overlap=vpsc")
  -graph-title string
        Title for the graph
  -labels-file string
        location of node/edge labels file (default "labels.yaml")
  -log_backtrace_at value
        when logging hits line file:N, emit a stack trace
  -log_dir string
        If non-empty, write log files in this directory
  -logtostderr
        log to standard error instead of files
  -memprofile file
        write memory profile to file
  -mode string
        mode of operation (generate, visualize, export)
  -no-banner
        disables banner
  -no-color
        disables color in output
  -query-file string
        location of Gizmo query file (default "query.js")
  -query-parameter value
        additional parameter to pass to Gizmo query (param=value)
  -relations-file string
        location of relations file (default "relations.yaml")
  -resource-data data
        adds resource data to graph under data predicate
  -resource-inventory-file string
        location of resource inventory file from Cloud Asset Inventory (default "resource_inventory.json")
  -stderrthreshold value
        logs at or above this threshold go to stderr
  -style-file string
        location of graph style file (default "style.yaml")
  -v value
        log level for V logs
  -vmodule value
        comma-separated list of pattern=N settings for file-filtered logging

Exporting Cloud Asset Inventory and creating graph

You'll need a Cloud Storage bucket to store the export and roles/cloudasset.viewer permission (on organisational level).

Then run the command:

export ORGANIZATION_ID=<YOUR-ORGANIZATION-ID>
export CAI_BUCKET_NAME=<YOUR-BUCKET>
gcloud asset export \
   --content-type resource \
   --organization $ORGANIZATION_ID  \
   --output-path "gs://$CAI_BUCKET_NAME/resource_inventory.json"

Copy the resource_inventory.json to your machine and generate the graph file:

gcpviz -resource-inventory-file resource_inventory.json -mode generate 

Creating graphs

The tool has many options - feel free to play around with them until you get the look you want! A number of sample queries into the graph tree has been provided:

• Visualize basic network
• Visualize load balancing components
• Visualize Google Kubernetes Engine
• Visualize Compute instances
• Visualize storage and data sets
• Visualize security components
• Visualize VPNs

To create a graph, simply run (remember, you'll need to generate the graph file first):

gcpviz -query-file queries/gke.js -mode visualize > gke.gv
dot -Kneato -Tsvg -Gdpi=60 gke.gv -o gke.svg

Exporting the graph

You can also use the -mode export for export the enriched assets and edges.

Sample graphs

Basic networking components

GKE clusters with workloads

Instances

Storage buckets, BigQuery datasets and Cloud SQL

Customizing your graph

To customize the entities that are displayed in graph, you can create new queries or adapt the examples in the queries/ directory. The query language is based on Gizmo. The graph has two types of predicates: child for parent-child relationships and uses for when a resources is attached or consumes another resource. Subgraph types are the Cloud Asset Inventory asset_types.

You can customize your graph styling by editing the following files:

• relations.yaml: contains jsonpath mappings that build uses relationships between objects.
• style.yaml: contains graph, node and edge styles (you can override these styles using -graph-parameter or just make a new style file)
• labels.yaml: contains formatting for node labels and clickable links.

Cool tips

• You can visualize multiple organizations by combining resource inventories (and modifying the queries a little bit).
• The queries have some filtering in place (empty projects, etc.), you can remove it.
• You can also start from any other node than organization root by changing the query root (see one-project-example.js and pass the project via -query-parameters "project=//cloudresourcemanager.googleapis.com/projects/PROJECT_ID")
• A Gitlab sample pipeline has been included.
• A Cloud Asset Inventory export can contain some sensitive information. A simple tool, called redactor.py, has been included to remove some fields from the export.
• A few asset types have clickable links in a SVG! Try it out.
• If you have a huge resource inventory, only export the assets you need for your graph by specifying --asset-types when doing the gcloud asset export.
• You can now access the resource properties if you specify -resource-data flag during graph database creation. For an example how to use it, see standalone-projects.js.

Documentation

Overview

# Copyright 2022 Google LLC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. #

Index

• Variables
• func DaysLeft(s string) string
• func GetLastPart(s string) string
• func GetPartFromEnd(s string, idx int) string
• func GetRegion(s string) string
• func Join(s []interface{}) string
• func JoinNicely(s []interface{}) string
• func NotLast(x int, a interface{}) bool
• func Replace(from string, to string, input string) string
• func ToLower(s string) string
• type GcpViz
  • func NewGcpViz(relationsFile string, labelsFile string, styleFile string, ...) (*GcpViz, error)
  • func (v *GcpViz) AddAsset(tx *bolt.Tx, asset validator.Asset, resource interface{}, addResourceData bool) error
  • func (v *GcpViz) Create(dbFile string) error
  • func (v *GcpViz) EnrichAssets() error
  • func (v *GcpViz) EscapeLabel(label string) string
  • func (v *GcpViz) ExportNodes(wg *sync.WaitGroup, ctx context.Context, out io.Writer) error
  • func (v *GcpViz) GenerateNodes(wg *sync.WaitGroup, ctx context.Context, gizmoQuery string, ...) error
  • func (v *GcpViz) Load(dbFile string) error
  • func (v *GcpViz) ReadAssetsFromFile(input string, addResourceData bool) error
  • func (v *GcpViz) Save() error
  • func (v *GcpViz) UpdateAsset(tx *bolt.Tx, name string, resource interface{}) error
• type GraphStyle
• type IpAddressLink
• type NodeStyle
• type RawResourceRelations
• type ResourceRelations
• type TemplateResource
• type TemplateResourceResource

Variables

var Edges map[string]map[string]*template.Template

var HeadLabels map[string]*template.Template

var Labels map[string]*template.Template

var Links map[string]*template.Template

var Nodes map[string]*template.Template

var TailLabels map[string]*template.Template

Functions

func DaysLeft

func DaysLeft(s string) string

func GetLastPart

func GetLastPart(s string) string

func GetPartFromEnd

func GetPartFromEnd(s string, idx int) string

func GetRegion

func GetRegion(s string) string

func Join

func Join(s []interface{}) string

func JoinNicely

func JoinNicely(s []interface{}) string

func NotLast

func NotLast(x int, a interface{}) bool

func Replace

func Replace(from string, to string, input string) string

func ToLower

func ToLower(s string) string

Types

type GcpViz

type GcpViz struct {
	QS            graph.QuadStore
	QW            graph.QuadWriter
	Relations     ResourceRelations
	AssetDatabase *bolt.DB
	Assets        *bolt.Bucket
	Aliases       *bolt.Bucket
	Graph         *bolt.Bucket

	OrgRoots []string

	TotalVertexes int64
	TotalEdges    int64
	TotalAliases  int64
	TotalIps      int64
	// contains filtered or unexported fields
}

func NewGcpViz

func NewGcpViz(relationsFile string, labelsFile string, styleFile string, override map[string]string) (*GcpViz, error)

func (*GcpViz) AddAsset

func (v *GcpViz) AddAsset(tx *bolt.Tx, asset validator.Asset, resource interface{}, addResourceData bool) error

func (*GcpViz) Create

func (v *GcpViz) Create(dbFile string) error

func (*GcpViz) EnrichAssets

func (v *GcpViz) EnrichAssets() error

func (*GcpViz) EscapeLabel

func (v *GcpViz) EscapeLabel(label string) string

func (*GcpViz) ExportNodes

func (v *GcpViz) ExportNodes(wg *sync.WaitGroup, ctx context.Context, out io.Writer) error

func (*GcpViz) GenerateNodes

func (v *GcpViz) GenerateNodes(wg *sync.WaitGroup, ctx context.Context, gizmoQuery string, parameters map[string]interface{}, out io.Writer) error

func (*GcpViz) Load

func (v *GcpViz) Load(dbFile string) error

func (*GcpViz) ReadAssetsFromFile

func (v *GcpViz) ReadAssetsFromFile(input string, addResourceData bool) error

func (*GcpViz) Save

func (v *GcpViz) Save() error

func (*GcpViz) UpdateAsset

func (v *GcpViz) UpdateAsset(tx *bolt.Tx, name string, resource interface{}) error

type GraphStyle

type GraphStyle struct {
	Global  map[string]string            `yaml:"global" json:"global"`
	Options map[string]string            `yaml:"options" json:"options"`
	Edges   map[string]map[string]string `yaml:"edges" json:"edges"`
	Nodes   map[string]string            `yaml:"nodes" json:"nodes"`
}

var Style GraphStyle

type IpAddressLink

type IpAddressLink struct {
	Ip        *net.IPNet
	Resource  string
	AssetType string
}

type NodeStyle

type NodeStyle struct {
	Label     string `json:"label"`
	HeadLabel string `json:"headLabel"`
	TailLabel string `json:"tailLabel"`
	Link      string `json:"link"`
	Resource  *TemplateResourceResource
}

type RawResourceRelations

type RawResourceRelations struct {
	AssetTypes  map[string][]string                     `yaml:"asset_types"`
	Aliases     map[string][]string                     `yaml:"aliases"`
	Enrich      map[string]map[string]map[string]string `yaml:"enrich"`
	IpAddresses map[string][]string                     `yaml:"ip_addresses"`
}

type ResourceRelations

type ResourceRelations struct {
	AssetTypes  map[string][]jsonpath.FilterFunc
	Aliases     map[string][]jsonpath.FilterFunc
	Enrich      map[string]map[string]map[string]jsonpath.FilterFunc
	IpAddresses map[string][]jsonpath.FilterFunc
}

type TemplateResource

type TemplateResource struct {
	Name      string                   `json:"name"`
	AssetType string                   `json:"asset_type"`
	Resource  TemplateResourceResource `json:"resource"`
	Ancestors []string                 `json:"ancestors"`
}

type TemplateResourceResource

type TemplateResourceResource struct {
	Data                 interface{} `json:"data"`
	DiscoveryDocumentUri string      `json:"discovery_document_uri"`
	DiscoveryName        string      `json:"discovery:name"`
	Version              string      `json:"version"`
	Parent               string      `json:"parent"`
}