README
¶
Directory metadata
DIR_METADATA files are a source-focused mechanism by which owners can provide users of their code important information. They map a directory to external entities, including:
- The Monorail component where bugs should be filed.
- The team responsible for the code.
- The OS type.
This infrastructure is overall generic. It has Chromium-specific proto fields, but their usage is not required, and overall this infra can be used by other projects as well.
DIR_METADATA files
DIR_METADATA files are structured text-protobuf files that are amenable to programmatic interaction. Example:
team_email: "team@chromium.org"
os: LINUX
monorail {
project: "chromium"
component: "Infra"
}
For file schema, see Metadata message.
Inheritance
DIR_METADATA files apply to all contents of a directory including its
subdirectories. By default, individual fields are inherited by subdirectories.
In the following, example, the value of monorail.project field in directory
a/b is "chromium".
a/DIR_METADATA
monorail {
project: "chromium"
component: "Component"
}
a/b/DIR_METADATA
monorail { component: "Component>Foo" }
dirmd tool
dirmd is a tool to work with DIR_METADATA files. Features:
- Gather all metadata in a given directory tree and export it to a single JSON file. Optionally remove all redundant metadata, or instead compute inherited metadata.
- Compute inherited metadata for a given set of directories.
- Convert from text proto to JSON which is easier to interpret by programs that don't have an easy access to the protobuf files.
- Validate a given set of files. Used in PRESUBMIT.
- Fall back to legacy
OWNERSfiles, so that metadata can migrate off of OWNERS files smoothly.
The tool is deployed via depot_tools and is available as a CIPD package.
The tool also hosts implementation of the metadata-exporter builder.
Source code: ./cmd/dirmd.
Library
Go package infra/tools/dirmd
can be used work with METADATA files programmatically.
The dirmd executable is a thin wrapper around it.
Continuous export
metadata-exporter
builder contunuously exports metadata from the src.git to
gs://chrome-metadata/metadata_reduced.json
and
gs://chrome-metadata/metadata_computed.json.
metadata_reduced.json is semantic equivalent of metadata_computed.json, but
it has all redundant information removed (see
MappingForm.REDUCED).
As of July 2020, the update latency is up to 20 min.
Links
Documentation
¶
Overview ¶
Package dirmd implements parsing and transformations of directory metadata.
Index ¶
- Constants
- func HandleMetadata(md *dirmdpb.Metadata, cm map[string]int64, dir string) (*dirmdpb.Metadata, error)
- func HandleMixins(md *dirmdpb.Metadata, cm map[string]int64, root string) (map[string]*dirmdpb.Metadata, error)
- func Merge(dst, src *dirmdpb.Metadata)
- func MigrateMetadata(dir string) error
- func MigrateMonorailMetadata(dir string, cm map[string]int64, root string) error
- func ParseFile(fileName string) (*dirmdpb.Metadata, error)
- func ParseOwners(r io.Reader) (md *dirmdpb.Metadata, ignored []string, err error)
- func ReadFile(fileName string) (*dirmdpb.Metadata, error)
- func ReadMetadata(dir string, onlyDirmd bool) (*dirmdpb.Metadata, error)
- func ReadOwners(dir string) (md *dirmdpb.Metadata, ignored []string, err error)
- func ToLocationTags(mapping *Mapping) (*sinkpb.LocationTags_Repo, error)
- func ValidateFile(fileName string) error
- type Mapping
Constants ¶
const Filename = "DIR_METADATA"
Filename is the standard name of the metadata file.
const MonorailMissingError string = "Monorail component is undefined."
const OwnersFilename = "OWNERS"
OwnersFilename is the filename for OWNERS files.
Variables ¶
This section is empty.
Functions ¶
func HandleMetadata ¶
func HandleMixins ¶
func Merge ¶
Merge merges metadata from src to dst. Does nothing if src is nil.
The current implementation is just proto.Merge, but it may change in the future.
func MigrateMetadata ¶
MigrateMetadata moves metadata from legacy OWNERS of the given directory into DIR_METADATA file.
func MigrateMonorailMetadata ¶
MigrateMonorailMetadata migrates Monorail Component to a Buganizer Component ID. TODO(crbug.com/1505875) - Remove this once migration is complete.
func ParseFile ¶
ParseFile parses the metadata file to dirmd.Metadata.
A valid file has a base filename "DIR_METADATA" or "OWNERS". The format of its contents correspond to the base name.
func ParseOwners ¶
ParseOwners extracts metadata from a content of an OWNERS file.
func ReadMetadata ¶
ReadMetadata reads metadata from a single directory. See also ReadMapping.
Returns (nil, nil) if the metadata is not defined.
func ReadOwners ¶
ReadOwners reads metadata from legacy OWNERS of the given directory and returns a dirmdpb.Metadata message along with the lines that were ignored. Returns (nil, nil, nil) if OWNERS file does not exist.
func ToLocationTags ¶
func ToLocationTags(mapping *Mapping) (*sinkpb.LocationTags_Repo, error)
ToLocationTags converts all dir metadata to test location tags.
func ValidateFile ¶
ValidateFile returns a non-nil error if the metadata file is invalid.
A valid file has a base filename "DIR_METADATA" or "OWNERS". The format of its contents correspond to the base name.
Types ¶
type Mapping ¶
Mapping is a mapping from a directory to its metadata.
It wraps the corresponding protobuf message and adds utility functions.
func ReadMapping ¶
func ReadMapping(ctx context.Context, form dirmdpb.MappingForm, onlyDirmd bool, dirs ...string) (*Mapping, error)
ReadMapping reads all metadata from files in git in the given directories.
Each directory must reside in a git checkout. One of the repos must be the root repo, while other repos must be its sub-repos. In other words, all git repos referred to by the directories must be subdirectories of one of the repos. The root dir of the root repo becomes the metadata root.
Unless the form is sparse, the returned mapping includes metadata of ancestors and descendants of the specified directories. In the sparse form, metadata of only the specified directories is returned, which is usually much faster.
If onlyDirmd is true, only metadata from DIR_METADATA files will be included; otherwise metadata from DIR_METADATA and OWNERS files will be included.
Descendants of the specified directories are discovered using "git ls-files <dir>" and not FS walk. This means files outside of the repo are ignored, as well as files matched by .gitignore files. Note that when reading ancestors of the specified directories, the .gitignore files are not respected. This inconsistency should not make a difference in the vast majority of cases because it is confusing to have git-ignored DIR_METADATA in the middle of the ancestry chain. Such a case might indicate that DIR_METADATA files are used incorrectly. This behavior can be changed, but it would come with a performance penalty.
func (*Mapping) ComputeAll ¶
ComputeAll computes full metadata for each dir.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package cli contains the top-level commands pertaining to dirmd.
|
Package cli contains the top-level commands pertaining to dirmd. |
|
updater
Package updater computes inherited metadata and uploads it to GCS.
|
Package updater computes inherited metadata and uploads it to GCS. |
|
cmd
|
|
|
Package git encapsulates git command interactions for dirmd.
|
Package git encapsulates git command interactions for dirmd. |