README ¶
OpenShift v3 Diagnostics
This is a tool to help administrators and users resolve common problems that occur with OpenShift v3 deployments. It will likely remain under continuous development as the OpenShift Origin project progresses.
The goals of the diagnostics tool are summarized in this Trello
card. Diagnostics are included as an
openshift
binary sub-command that analyzes OpenShift as it finds it,
whether from the perspective of an OpenShift client or on an OpenShift
host.
Expected environment
OpenShift can be deployed in many ways: built from source, included
in a VM image, in a Docker image, or as enterprise RPMs. Each of these
would imply different configuration and environment. In order to keep
assumptions about environment to a minimum, the diagnostics have been
added to the openshift
binary itself so that wherever there is an
OpenShift server or client, the diagnostics can run in the exact same
environment.
Diagnostics looks for config files in standard locations. If not found, related diagnostics are just skipped. Non-standard locations can be specified with flags.
Standard config file locations are:
- Client:
- as indicated by --config flag
- as indicated by $KUBECONFIG env var
- ~/.kube/config file
- Master:
- as indicated by --master-config flag
- /etc/openshift/master/master-config.yaml
- Node:
- as indicated by --node-config flag
- /etc/openshift/node/node-config.yaml
Host environment
Master/node diagnostics will be most useful in a specific target environment, which is a deployment using RPMs and ansible deployment logic. This provides two major benefits:
- master/node configuration is based on a configuration file in a standard location
- all components log to journald
Having configuration files where ansible places them means you will generally not even need to specify where to find them. Running:
openshift ex diagnostics
by itself will look for master and node configs (in addition to client config file) in the standard locations and use them if found; so this should make the ansible-installed use case as simple as possible. It's also very easy to use configuration files when they are not in the expected Enterprise locations:
openshift ex diagnostics --master-config=... --node-config=...
Having logs in journald is necessary for the current log analysis logic. Other usage may have logs going into files, output to stdout, combined node/master... it may not be too hard to extend analysis to other log sources but the priority has been to look at journald logs as created by components in systemd-based deployments (including docker, openvswitch, etc.).
Client environment
The user may only have access as an ordinary user, as a cluster-admin user, and/or may be running on a host where OpenShift master or node services are operating. The diagnostics will attempt to use as much access as the user has available.
A client with ordinary access should be able to diagnose its connection to the master and look for problems in builds and deployments for the current context.
A client with cluster-admin access should be able to diagnose the status of infrastructure.
Writing diagnostics
Developers are encouraged to add to the available diagnostics as they encounter problems that are not easily communicated in the normal operations of the program, for example components with misconfigured connections, problems that are buried in logs, etc. The sanity you save may be your own.
A diagnostic is an object that conforms to the Diagnostic interface (see pkg/diagnostics/types/diagnostic.go). The diagnostic object should be built in one of the builders in the pkg/cmd/admin/diagnostics package (based on whether it depends on client, cluster-admin, or host configuration). When executed, the diagnostic logs its findings into a result object. It should be assumed that they may run in parallel.
Diagnostics should prefer providing information over perfect accuracy, as they are the first line of (self-)support for users. On the other hand, judgment should be exercised to prevent sending users down useless paths or flooding them with non-issues that obscure real problems.
- Errors should be reserved for things that are almost certainly broken or causing problems, for example a broken URL.
- Warnings indicate issues that may be a problem but could be valid for some configurations / situations, for example a node being disabled.
Message IDs
All messages should have a unique, unchanging, otherwise-meaningless message ID to facilitate the user greping for specific errors/warnings without having to depend on text that may change. Although nothing yet depends on them being unique, the message ID scheme attempts to ensure they are. That scheme is:
Initials of package + index of file in package + index of message in file
E.g. "DClu1001" is in package diagnostics/cluster (which needed to be differentiated from diagnostics/client), the first file indexed, and the first message in the file. This format is not important; it's just a convenience to help keep IDs unique. But don't change existing IDs.