README
¶
monitor
Monitor the compliance of an asset to a configuration rule
1 Compile and test the microservice locally
1.1 Prepare local test
- Deploy the infrastructure prerequisites using the proposed Terraform module
- create-download a key JSON format for the created service account for this microservice
- set environment variables
export MONITOR_ENVIRONMENT="dev"
export MONITOR_PROJECT_ID="<your_project_id>"
export MONITOR_COMPLIANCE_STATUS_TOPIC_ID="ram-complianceStatus"
export MONITOR_VIOLATION_TOPIC_ID="ram-violation"
export MONITOR_START_PROFILER=true
export K_SERVICE="localgoapp"
export K_REVISION=${K_SERVICE}-$(git rev-parse HEAD)
export GOOGLE_APPLICATION_CREDENTIALS="<path_to_monitor_service_account_json_key_file>"
- Craft an HTTP POST request implementing CloudEvents format, using a tool like POSTMAN. Example data at the end of this page
Test example for POSTMAN
pm.test("Status code is 200", function () {
pm.response.to.have.status(200);
});
1.2 Run the app locally and test it
Run monitor locally:
go run cmd/main.go
1.3 Check results
- When launching
monitorlocally it should log without errors thecoldstartandinit donelog entries, like:
go run main.go
{"microservice_name":"monitor","severity":"NOTICE","message":"coldstart","init_id":"44ce298c-fb19-4099-a834-c5ff0eb99fd8"}
{"microservice_name":"monitor","severity":"NOTICE","message":"init done","init_id":"44ce298c-fb19-4099-a834-c5ff0eb99fd8"}
Serving function...
- When posting the crafted HTTP request on
localhost:8080/it should- respond with 200 HTTP status code and empty body
- log entries for one compliance status and zero to many violations depending on tested asset configuration and rule code.
- publish the compliance status, and violation(s) to their respective Pubsub topics
To see published Pubsub messages subscribe to the two topics. Example:
gcloud pubsub subscriptions create test_compliance_status --topic=ram-complianceStatus --project=${MONITOR_PROJECT_ID}
gcloud pubsub subscriptions create test_violation --topic=ram-violation --project=${MONITOR_PROJECT_ID}
gcloud pubsub subscriptions pull test_compliance_status --auto-ack --project=${MONITOR_PROJECT_ID}
gcloud pubsub subscriptions pull test_violation --auto-ack --project=${MONITOR_PROJECT_ID}
2 build the microservice container image and test it locally
2.1 Build the container image locally and prepare to test
cd <repoRoot>
pack build --builder gcr.io/buildpacks/builder:v1 --env GOOGLE_FUNCTION_SIGNATURE_TYPE=cloudevent --env GOOGLE_FUNCTION_TARGET=EntryPoint monitor
docker images
Duplicate the previous test, and use a different port number in the URL, e.g. 8081
2.2 Run the container image locally and test it
Reference doc: Test a Cloud Run service locally using docker with GCP access
Make the key file readable by docker run process
chmod 644 <path to your service account key file>
PORT=8080 && docker run \
-p 8081:${PORT} \
-e PORT=${PORT} \
-e K_SERVICE=dockerrun \
-e K_CONFIGURATION=dockerrun \
-e K_REVISION=dockerrun-$(git rev-parse HEAD) \
-e MONITOR_ENVIRONMENT="dev" \
-e MONITOR_PROJECT_ID=${MONITOR_PROJECT_ID} \
-e MONITOR_COMPLIANCE_STATUS_TOPIC_ID=${MONITOR_COMPLIANCE_STATUS_TOPIC_ID} \
-e MONITOR_VIOLATION_TOPIC_ID=${MONITOR_VIOLATION_TOPIC_ID} \
-e MONITOR_START_PROFILER=true \
-e GOOGLE_APPLICATION_CREDENTIALS=/tmp/keys/key.json \
-v $GOOGLE_APPLICATION_CREDENTIALS:/tmp/keys/key.json:ro \
monitor
Run test as previously for the local app, using the mapped port number.
2.3 Check results
Same as for the local app test.
3 Build the microservice container image remotely
3.1 Using Cloud Build CLI tool
cd ram/monitor
gcloud alpha builds submit . \
--project=<your_build_project> \
--pack=image=<your_container_iamge_repo_path>,builder=gcr.io/buildpacks/builder:v1,env=GOOGLE_FUNCTION_SIGNATURE_TYPE=cloudevent,env=GOOGLE_FUNCTION_TARGET=EntryPoint
3.2 Using GitLag CI
- Gitlab / ram group / settings / CICD / set up the following variables:
- DEV_PROJECT_ID
- DEV_SA_KEY_BUILD
- QA_PROJECT_ID
- SA_DEPLOY_QA_JSON_KEY
Sample data to craft a test request
Header
ce-source: //pubsub.googleapis.com/assetRule
ce-id: asset_rule_a
ce-specversion: 1.0
ce-type: google.cloud.pubsub.topic.v1.messagePublished
ce-time: 2021-07-27T10:47:00Z
Content-Type: application/json; charset=utf-8
Body
{
"message": {
"data": "<your_base64_encoded_data>",
"messageId": "asset_rule_a",
"publishTime": "2021-07-27T10:47:00.000000000Z"
},
"subscription": "projects/MY_PROJECT/subscriptions/MY_SUBSCRIPTION"
}
Data to be base64 encoded, example
{
"rule": {
"name": "monitor_bq_dataset_location",
"deploymentTime": "2021-03-23T09:32:53.949581856Z",
"regoModules": {
"audit.rego": "\n#\n# Copyright 2024 Google LLC\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n\npackage validator.gcp.lib\n\n# RULE: the audit rule is result if its body {} evaluates to true\n# The rule body {} can be understood intuitively as: expression-1 AND expression-2 AND ... AND expression-N\naudit[result] {\n\t# iterate over each asset\n\tasset := input.assets[_]\n\ttrace(sprintf(\"asset name: %v\", [asset.name]))\n\n\t# assign the constrains in a variable\n\tconstraints := input.constraints\n\t# iterate over each constraint\n\tconstraint := constraints[_]\n\ttrace(sprintf(\"constraint kind: %v\", [constraint.kind])) \n\n\t# use a custom function to retreive constraint.spec, if not defined returns a default value that is an empty object\n\tspec := _get_default(constraint, \"spec\", {})\n\t# use a custom function to retreive constraint.spec.match, if not defined returns a default value that is an empty objecy\n\tmatch := _get_default(spec, \"match\", {})\n\t# use a custom function to retreive constraint.spec.match.target, if not defined returns a default value that is an array with one target object targetting any organization, and childs\n\ttarget := _get_default(match, \"target\", [\"organization/*\"])\n\t# use a custom function to retreive constraint.spec.match.gcp, if not defined returns a default value that is an empty object\n\tgcp := _get_default(match, \"gcp\", {})\n\t# use a custom function to retreive constraint.spec.match.gcp.target, if not defined returns what we already got in target variable\n\tgcp_target := _get_default(gcp, \"target\", target)\n\t# iterate over each target and use builtin regex to check if the asset ancestry path matches one of them\n\t# TRUE if the asset ancestry path matches (regex) one of the target (iterate targets)\n\t# FALSE when the ancestry path do not matches at least one of the targer\n\ttrace(sprintf(\"asset.ancestry_path: %v\", [asset.ancestry_path]))\n\ttrace(sprintf(\"targets: %v\", [gcp_target]))\n\ttrace(sprintf(\"is in scope:\",[re_match(gcp_target[_], asset.ancestry_path)]))\n\tre_match(gcp_target[_], asset.ancestry_path)\n\n\t# use a custom function to retreive constraint.spec.match.exclude, if not defined returns a default value that is an empty array\n\texclude := _get_default(match, \"exclude\", [])\n\t# use a custom function to retreive constraint.spec.match.gcp.exclude, if not defined returns what we already got in exlucde variable\n\tgcp_exclude := _get_default(gcp, \"exclude\", exclude)\n\t# iterate over the exclusion list (the pattern) and use regex builtin function to check if the asset ancestry path (the value) matches one of then\n\t# assign to exclusion_match variable the virtual document generated by a rule which is a set built by using a comprehension\n\t# This set containts the asset ancestry path if it maches one of the exclusion\n\t# or is empty (set()) if the asset ancestry path does not matched any of the exclusion\n\texclusion_match := {asset.ancestry_path | re_match(gcp_exclude[_], asset.ancestry_path)}\n\ttrace(sprintf(\"exclusions: %v\", [gcp_exclude]))\n\ttrace(sprintf(\"Excluded if count exclusion_match \u003e 0: %v\", [count(exclusion_match)]))\n\t# this expression evaluate to true when count is zero, aka the ancestry path does not matches any of the exclusion, otherwise evaluates to false\n\tcount(exclusion_match) == 0\n\n\t# Use a with statement to programatically call the rego rule that is specified in the YAML constraint file\n\tviolations := data.templates.gcp[constraint.kind].deny with input.asset as asset\n\t\t with input.constraint as constraint\n\n\t# Iterates through each violation found\n\tviolation := violations[_]\n\t# if the asset is in target, and not excluded and at least one violation is founds, then returns for each violation a result object with the 4 following fields:\n\tresult := {\n\t\t\"asset\": asset.name,\n\t\t\"constraint\": constraint.metadata.name,\n\t\t\"constraint_config\": constraint,\n\t\t\"violation\": violation,\n\t}\n}\n\n# has_field returns whether an object has a field\n_has_field(object, field) {\n\tobject[field]\n}\n\n# False is a tricky special case, as false responses would create an undefined document unless\n# they are explicitly tested for\n_has_field(object, field) {\n\tobject[field] == false\n}\n\n_has_field(object, field) = false {\n\tnot object[field]\n\tnot object[field] == false\n}\n\n# get_default returns the value of an object's field or the provided default value.\n# It avoids creating an undefined state when trying to access an object attribute that does\n# not exist\n_get_default(object, field, _default) = output {\n\t_has_field(object, field)\n\toutput = object[field]\n}\n\n_get_default(object, field, _default) = output {\n\t_has_field(object, field) == false\n\toutput = _default\n}\n",
"constraints.rego": "\n#\n# Copyright 2024 Google LLC\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n\npackage validator.gcp.lib\n\n# Function to fetch the constraint spec\n# Usage:\n# get_constraint_params(constraint, params)\n\nget_constraint_params(constraint) = params {\n\tparams := constraint.spec.parameters\n}\n\n# Function to fetch constraint info\n# Usage:\n# get_constraint_info(constraint, info)\n\nget_constraint_info(constraint) = info {\n\tinfo := {\n\t\t\"name\": constraint.metadata.name,\n\t\t\"kind\": constraint.kind,\n\t}\n}\n",
"monitor_bq_dataset_location.rego": "# Copyright 2024 Google LLC\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\npackage templates.gcp.GCPBigQueryDatasetLocationConstraintV1\n\nimport data.validator.gcp.lib as lib\n\n############################################\n# Find BigQuery Dataset Location Violations\n############################################\ndeny[{\n\t\"msg\": message,\n\t\"details\": metadata,\n}] {\n\tconstraint := input.constraint\n\tlib.get_constraint_params(constraint, params)\n\n\t# Verify that resource is BigQuery dataset\n\tasset := input.asset\n\tasset.asset_type == \"bigquery.googleapis.com/Dataset\"\n\n\t# Check if resource is in exempt list\n\texempt_list := params.exemptions\n\tmatches := {asset.name} \u0026 cast_set(exempt_list)\n\tcount(matches) == 0\n\n\t# Check that location is in allowlist/denylist\n\ttarget_locations := params.locations\n\tasset_location := asset.resource.data.location\n\tlocation_matches := {asset_location} \u0026 cast_set(target_locations)\n\ttarget_location_match_count(params.mode, desired_count)\n\tcount(location_matches) == desired_count\n\n\tmessage := sprintf(\"%v is in a disallowed location.\", [asset.name])\n\tmetadata := {\"location\": asset_location}\n}\n\n#################\n# Rule Utilities\n#################\n\n# Determine the overlap between locations under test and constraint\n# By default (allowlist), we violate if there isn't overlap\ntarget_location_match_count(mode) = 0 {\n\tmode != \"denylist\"\n}\n\ntarget_location_match_count(mode) = 1 {\n\tmode == \"denylist\"\n}\n",
"util.rego": "\n#\n# Copyright 2018 Google LLC\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n\npackage validator.gcp.lib\n\n# has_field returns whether an object has a field\nhas_field(object, field) {\n\tobject[field]\n}\n\n# False is a tricky special case, as false responses would create an undefined document unless\n# they are explicitly tested for\nhas_field(object, field) {\n\tobject[field] == false\n}\n\nhas_field(object, field) = false {\n\tnot object[field]\n\tnot object[field] == false\n}\n\n# get_default returns the value of an object's field or the provided default value.\n# It avoids creating an undefined state when trying to access an object attribute that does\n# not exist\nget_default(object, field, _default) = output {\n\thas_field(object, field)\n\toutput = object[field]\n}\n\nget_default(object, field, _default) = output {\n\thas_field(object, field) == false\n\toutput = _default\n}\n"
},
"constraints": [
{
"apiVersion": "constraints.gatekeeper.sh/v1alpha1",
"kind": "GCPBigQueryDatasetLocationConstraintV1",
"metadata": {
"name": "to_be_adapted",
"annotation": {
"category": "Personal Data Compliance",
"description": "BQ Dataset must be located in \u003cto_be_adapted\u003e for projects in \u003cto_be_adapted\u003e."
}
},
"spec": {
"severity": "critical",
"match": {
"exclude": null,
"target": [
"organization/"
]
},
"parameters": {
"exemptions": [],
"locations": [
"EU",
"europe-north1",
"europe-west1",
"europe-west3",
"europe-west4"
],
"mode": "allowlist"
}
}
}
]
},
"feedMessage": {
"type": "cai",
"asset": {
"name": "//bigquery.googleapis.com/projects/<your_project_id>/datasets/brunore_ds_01",
"owner": "",
"violationResolver": "",
"ancestryPathDisplayName": "ramtests.brunore.org/noncompliant/<your_project_id>",
"ancestryPath": "organization/<org_id>/folder/<folder_number>/project/<your_project_numer>",
"ancestry_path": "organization/<org_id>/folder/<folder_number>/project/<your_project_numer>",
"ancestorsDisplayName": [
"<your_project_id>",
"noncompliant",
"ramtests.brunore.org"
],
"ancestors": [
"projects/<your_project_numer>",
"folders/<folder_number>",
"organizations/<org_id>"
],
"assetType": "bigquery.googleapis.com/Dataset",
"asset_type": "bigquery.googleapis.com/Dataset",
"iamPolicy": null,
"iam_policy": null,
"resource": {
"version": "v2",
"discovery_document_uri": "https://www.googleapis.com/discovery/v1/apis/bigquery/v2/rest",
"discovery_name": "Dataset",
"parent": "//cloudresourcemanager.googleapis.com/projects/<your_project_numer>",
"data": {
"creationTime": "1604982398240",
"datasetReference": {
"datasetId": "brunore_ds_01",
"projectId": "<your_project_id>"
},
"id": "<your_project_id>:brunore_ds_01",
"kind": "bigquery#dataset",
"lastModifiedTime": "1605594631154",
"location": "US"
},
"location": "US"
},
"projectID": "<your_project_id>"
},
"window": {
"startTime": "2021-07-21T10:09:17.141Z"
},
"deleted": false,
"origin": "batch-export"
},
"step_stack": [
{
"step_id": "every-monday-at-01am10/2662791119629040",
"step_timestamp": "2021-07-21T10:08:53.58Z"
},
{
"step_id": "brunore-cai-exports-dev-003/dumpinventory_org<org_id>_bigquery_Dataset.dump/1626862157134339",
"step_timestamp": "2021-07-21T10:09:17.141Z"
},
{
"step_id": "dumpinventory_org<org_id>_bigquery_Dataset.dump/2662784568720615",
"step_timestamp": "2021-07-21T10:09:17.302Z"
}
]
}
Documentation
¶
Overview ¶
Package monitor in the monitor go module is a function based on the function framework to check asset compliance
Triggered by ¶
A cloud event containing one asset config (the feed message) and one rule ¶
Output ¶
- Cloud Event on PubSub violation topic.
- Cloud Event on complianceStatus topic.
Cardinality ¶
- When compliant: one-one, only the compliance state, no violations.
- When not compliant: one-few, 1 compliance state + n violations.
Automatic retrying ¶
Yes.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Env ¶
type Env struct {
KConfiguration string `envconfig:"k_configuration"`
KRevision string `envconfig:"k_revision"`
KService string `envconfig:"k_service"`
}
Env list environment variables
type Global ¶
type Global struct {
CommonEv glo.CommonEntryValues
// contains filtered or unexported fields
}
Global structure for global variables to optimize the performances in serverless mode
type ServiceEnv ¶
type ServiceEnv struct {
ComplianceStatusTopicID string `envconfig:"compliance_status_topic_id" default:"ram-complianceStatus"`
Environment string `envconfig:"environment" default:"dev"`
LogOnlySeveritylevels string `envconfig:"log_only_severity_levels" default:"WARNING NOTICE CRITICAL"`
ProjectID string `envconfig:"project_id" required:"true"`
ViolationTopicID string `envconfig:"violation_topic_id" default:"ram-violation"`
StartProfiler bool `envconfig:"start_profiler" default:"false"`
}
ServiceEnv list environment variables prefixed with the name of the microservice
Source Files
¶
Directories
Click to show internal directories.
Click to hide internal directories.