README
¶
Go Plugin System over RPC
go-plugin
is a Go (golang) plugin system over RPC. It is the plugin system
that has been in use by HashiCorp tooling for over 4 years. While initially
created for Packer, it is additionally in use by
Terraform, Nomad, and
Vault.
While the plugin system is over RPC, it is currently only designed to work over a local [reliable] network. Plugins over a real network are not supported and will lead to unexpected behavior.
This plugin system has been used on millions of machines across many different projects and has proven to be battle hardened and ready for production use.
Features
The HashiCorp plugin system supports a number of features:
Plugins are Go interface implementations. This makes writing and consuming plugins feel very natural. To a plugin author: you just implement an interface as if it were going to run in the same process. For a plugin user: you just use and call functions on an interface as if it were in the same process. This plugin system handles the communication in between.
Cross-language support. Plugins can be written (and consumed) by almost every major language. This library supports serving plugins via gRPC. gRPC-based plugins enable plugins to be written in any language.
Complex arguments and return values are supported. This library
provides APIs for handling complex arguments and return values such
as interfaces, io.Reader/Writer
, etc. We do this by giving you a library
(MuxBroker
) for creating new connections between the client/server to
serve additional interfaces or transfer raw data.
Bidirectional communication. Because the plugin system supports complex arguments, the host process can send it interface implementations and the plugin can call back into the host process.
Built-in Logging. Any plugins that use the log
standard library
will have log data automatically sent to the host process. The host
process will mirror this output prefixed with the path to the plugin
binary. This makes debugging with plugins simple. If the host system
uses hclog then the log data
will be structured. If the plugin also uses hclog, logs from the plugin
will be sent to the host hclog and be structured.
Protocol Versioning. A very basic "protocol version" is supported that can be incremented to invalidate any previous plugins. This is useful when interface signatures are changing, protocol level changes are necessary, etc. When a protocol version is incompatible, a human friendly error message is shown to the end user.
Stdout/Stderr Syncing. While plugins are subprocesses, they can continue
to use stdout/stderr as usual and the output will get mirrored back to
the host process. The host process can control what io.Writer
these
streams go to to prevent this from happening.
TTY Preservation. Plugin subprocesses are connected to the identical
stdin file descriptor as the host process, allowing software that requires
a TTY to work. For example, a plugin can execute ssh
and even though there
are multiple subprocesses and RPC happening, it will look and act perfectly
to the end user.
Host upgrade while a plugin is running. Plugins can be "reattached"
so that the host process can be upgraded while the plugin is still running.
This requires the host/plugin to know this is possible and daemonize
properly. NewClient
takes a ReattachConfig
to determine if and how to
reattach.
Cryptographically Secure Plugins. Plugins can be verified with an expected checksum and RPC communications can be configured to use TLS. The host process must be properly secured to protect this configuration.
Architecture
The HashiCorp plugin system works by launching subprocesses and communicating
over RPC (using standard net/rpc
or gRPC). A single
connection is made between any plugin and the host process. For net/rpc-based
plugins, we use a connection multiplexing
library to multiplex any other connections on top. For gRPC-based plugins,
the HTTP2 protocol handles multiplexing.
This architecture has a number of benefits:
-
Plugins can't crash your host process: A panic in a plugin doesn't panic the plugin user.
-
Plugins are very easy to write: just write a Go application and
go build
. Or use any other language to write a gRPC server with a tiny amount of boilerplate to support go-plugin. -
Plugins are very easy to install: just put the binary in a location where the host will find it (depends on the host but this library also provides helpers), and the plugin host handles the rest.
-
Plugins can be relatively secure: The plugin only has access to the interfaces and args given to it, not to the entire memory space of the process. Additionally, go-plugin can communicate with the plugin over TLS.
Usage
To use the plugin system, you must take the following steps. These are
high-level steps that must be done. Examples are available in the
examples/
directory.
-
Choose the interface(s) you want to expose for plugins.
-
For each interface, implement an implementation of that interface that communicates over a
net/rpc
connection or other a gRPC connection or both. You'll have to implement both a client and server implementation. -
Create a
Plugin
implementation that knows how to create the RPC client/server for a given plugin type. -
Plugin authors call
plugin.Serve
to serve a plugin from themain
function. -
Plugin users use
plugin.Client
to launch a subprocess and request an interface implementation over RPC.
That's it! In practice, step 2 is the most tedious and time consuming step.
Even so, it isn't very difficult and you can see examples in the examples/
directory as well as throughout our various open source projects.
For complete API documentation, see GoDoc.
Roadmap
Our plugin system is constantly evolving. As we use the plugin system for new projects or for new features in existing projects, we constantly find improvements we can make.
At this point in time, the roadmap for the plugin system is:
Semantic Versioning. Plugins will be able to implement a semantic version. This plugin system will give host processes a system for constraining versions. This is in addition to the protocol versioning already present which is more for larger underlying changes.
Plugin fetching. We will integrate with go-getter to support automatic download + install of plugins. Paired with cryptographically secure plugins (above), we can make this a safe operation for an amazing user experience.
What About Shared Libraries?
When we started using plugins (late 2012, early 2013), plugins over RPC were the only option since Go didn't support dynamic library loading. Today, Go still doesn't support dynamic library loading, but they do intend to. Since 2012, our plugin system has stabilized from millions of users using it, and has many benefits we've come to value greatly.
For example, we intend to use this plugin system in Vault, and dynamic library loading will simply never be acceptable in Vault for security reasons. That is an extreme example, but we believe our library system has more upsides than downsides over dynamic library loading and since we've had it built and tested for years, we'll likely continue to use it.
Shared libraries have one major advantage over our system which is much higher performance. In real world scenarios across our various tools, we've never required any more performance out of our plugin system and it has seen very high throughput, so this isn't a concern for us at the moment.
Documentation
¶
Overview ¶
The plugin package exposes functions and helpers for communicating to plugins which are implemented as standalone binary applications.
plugin.Client fully manages the lifecycle of executing the application, connecting to it, and returning the RPC client for dispensing plugins.
plugin.Serve fully manages listeners to expose an RPC server from a binary that plugin.Client can connect to.
Index ¶
- Constants
- Variables
- func CleanupClients()
- func DefaultGRPCServer(opts []grpc.ServerOption) *grpc.Server
- func Discover(glob, dir string) ([]string, error)
- func Serve(opts *ServeConfig)
- func ServeMux(m ServeMuxMap)
- func TestConn(t testing.T) (net.Conn, net.Conn)
- func TestPluginGRPCConn(t testing.T, ps map[string]Plugin) (*GRPCClient, *GRPCServer)
- func TestPluginRPCConn(t testing.T, ps map[string]Plugin) (*RPCClient, *RPCServer)
- func TestRPCConn(t testing.T) (*rpc.Client, *rpc.Server)
- type BasicError
- type Client
- type ClientConfig
- type ClientProtocol
- type GRPCClient
- type GRPCPlugin
- type GRPCServer
- type GRPCServerConfig
- type HandshakeConfig
- type MuxBroker
- type NetRPCUnsupportedPlugin
- type Plugin
- type Protocol
- type RPCClient
- type RPCServer
- type ReattachConfig
- type SecureConfig
- type ServeConfig
- type ServeMuxMap
- type ServerProtocol
Constants ¶
const CoreProtocolVersion = 1
CoreProtocolVersion is the ProtocolVersion of the plugin system itself. We will increment this whenever we change any protocol behavior. This will invalidate any prior plugins but will at least allow us to iterate on the core in a safe way. We will do our best to do this very infrequently.
const GRPCServiceName = "plugin"
GRPCServiceName is the name of the service that the health check should return as passing.
Variables ¶
var ( // ErrProcessNotFound is returned when a client is instantiated to // reattach to an existing process and it isn't found. ErrProcessNotFound = errors.New("Reattachment process not found") // ErrChecksumsDoNotMatch is returned when binary's checksum doesn't match // the one provided in the SecureConfig. ErrChecksumsDoNotMatch = errors.New("checksums did not match") // ErrSecureNoChecksum is returned when an empty checksum is provided to the // SecureConfig. ErrSecureConfigNoChecksum = errors.New("no checksum provided") // ErrSecureNoHash is returned when a nil Hash object is provided to the // SecureConfig. ErrSecureConfigNoHash = errors.New("no hash implementation provided") // ErrSecureConfigAndReattach is returned when both Reattach and // SecureConfig are set. ErrSecureConfigAndReattach = errors.New("only one of Reattach or SecureConfig can be set") )
Error types
var Killed uint32 = 0
If this is 1, then we've called CleanupClients. This can be used by plugin RPC implementations to change error behavior since you can expected network connection errors at this point. This should be read by using sync/atomic.
Functions ¶
func CleanupClients ¶
func CleanupClients()
This makes sure all the managed subprocesses are killed and properly logged. This should be called before the parent process running the plugins exits.
This must only be called _once_.
func DefaultGRPCServer ¶
func DefaultGRPCServer(opts []grpc.ServerOption) *grpc.Server
DefaultGRPCServer can be used with the "GRPCServer" field for Server as a default factory method to create a gRPC server with no extra options.
func Discover ¶
Discover discovers plugins that are in a given directory.
The directory doesn't need to be absolute. For example, "." will work fine.
This currently assumes any file matching the glob is a plugin. In the future this may be smarter about checking that a file is executable and so on.
TODO: test
func Serve ¶
func Serve(opts *ServeConfig)
Serve serves the plugins given by ServeConfig.
Serve doesn't return until the plugin is done being executed. Any errors will be outputted to os.Stderr.
This is the method that plugins should call in their main() functions.
func ServeMux ¶
func ServeMux(m ServeMuxMap)
ServeMux is like Serve, but serves multiple types of plugins determined by the argument given on the command-line.
This command doesn't return until the plugin is done being executed. Any errors are logged or output to stderr.
func TestConn ¶
TestConn is a helper function for returning a client and server net.Conn connected to each other.
func TestPluginGRPCConn ¶
func TestPluginGRPCConn(t testing.T, ps map[string]Plugin) (*GRPCClient, *GRPCServer)
TestPluginGRPCConn returns a plugin gRPC client and server that are connected together and configured. This is used to test gRPC connections.
func TestPluginRPCConn ¶
TestPluginRPCConn returns a plugin RPC client and server that are connected together and configured.
Types ¶
type BasicError ¶
type BasicError struct {
Message string
}
This is a type that wraps error types so that they can be messaged across RPC channels. Since "error" is an interface, we can't always gob-encode the underlying structure. This is a valid error interface implementer that we will push across.
func NewBasicError ¶
func NewBasicError(err error) *BasicError
NewBasicError is used to create a BasicError.
err is allowed to be nil.
func (*BasicError) Error ¶
func (e *BasicError) Error() string
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client handles the lifecycle of a plugin application. It launches plugins, connects to them, dispenses interface implementations, and handles killing the process.
Plugin hosts should use one Client for each plugin executable. To dispense a plugin type, use the `Client.Client` function, and then cal `Dispense`. This awkward API is mostly historical but is used to split the client that deals with subprocess management and the client that does RPC management.
See NewClient and ClientConfig for using a Client.
func NewClient ¶
func NewClient(config *ClientConfig) (c *Client)
Creates a new plugin client which manages the lifecycle of an external plugin and gets the address for the RPC connection.
The client must be cleaned up at some point by calling Kill(). If the client is a managed client (created with NewManagedClient) you can just call CleanupClients at the end of your program and they will be properly cleaned.
func (*Client) Client ¶
func (c *Client) Client() (ClientProtocol, error)
Client returns the protocol client for this connection.
Subsequent calls to this will return the same client.
func (*Client) Kill ¶
func (c *Client) Kill()
End the executing subprocess (if it is running) and perform any cleanup tasks necessary such as capturing any remaining logs and so on.
This method blocks until the process successfully exits.
This method can safely be called multiple times.
func (*Client) Protocol ¶
Protocol returns the protocol of server on the remote end. This will start the plugin process if it isn't already started. Errors from starting the plugin are surpressed and ProtocolInvalid is returned. It is recommended you call Start explicitly before calling Protocol to ensure no errors occur.
func (*Client) ReattachConfig ¶
func (c *Client) ReattachConfig() *ReattachConfig
ReattachConfig returns the information that must be provided to NewClient to reattach to the plugin process that this client started. This is useful for plugins that detach from their parent process.
If this returns nil then the process hasn't been started yet. Please call Start or Client before calling this.
func (*Client) Start ¶
Starts the underlying subprocess, communicating with it to negotiate a port for RPC connections, and returning the address to connect via RPC.
This method is safe to call multiple times. Subsequent calls have no effect. Once a client has been started once, it cannot be started again, even if it was killed.
type ClientConfig ¶
type ClientConfig struct { // HandshakeConfig is the configuration that must match servers. HandshakeConfig // Plugins are the plugins that can be consumed. Plugins map[string]Plugin // One of the following must be set, but not both. // // Cmd is the unstarted subprocess for starting the plugin. If this is // set, then the Client starts the plugin process on its own and connects // to it. // // Reattach is configuration for reattaching to an existing plugin process // that is already running. This isn't common. Cmd *exec.Cmd Reattach *ReattachConfig // SecureConfig is configuration for verifying the integrity of the // executable. It can not be used with Reattach. SecureConfig *SecureConfig // TLSConfig is used to enable TLS on the RPC client. TLSConfig *tls.Config // Managed represents if the client should be managed by the // plugin package or not. If true, then by calling CleanupClients, // it will automatically be cleaned up. Otherwise, the client // user is fully responsible for making sure to Kill all plugin // clients. By default the client is _not_ managed. Managed bool // The minimum and maximum port to use for communicating with // the subprocess. If not set, this defaults to 10,000 and 25,000 // respectively. MinPort, MaxPort uint // StartTimeout is the timeout to wait for the plugin to say it // has started successfully. StartTimeout time.Duration // If non-nil, then the stderr of the client will be written to here // (as well as the log). This is the original os.Stderr of the subprocess. // This isn't the output of synced stderr. Stderr io.Writer // SyncStdout, SyncStderr can be set to override the // respective os.Std* values in the plugin. Care should be taken to // avoid races here. If these are nil, then this will automatically be // hooked up to os.Stdin, Stdout, and Stderr, respectively. // // If the default values (nil) are used, then this package will not // sync any of these streams. SyncStdout io.Writer SyncStderr io.Writer // AllowedProtocols is a list of allowed protocols. If this isn't set, // then only netrpc is allowed. This is so that older go-plugin systems // can show friendly errors if they see a plugin with an unknown // protocol. // // By setting this, you can cause an error immediately on plugin start // if an unsupported protocol is used with a good error message. // // If this isn't set at all (nil value), then only net/rpc is accepted. // This is done for legacy reasons. You must explicitly opt-in to // new protocols. AllowedProtocols []Protocol // Logger is the logger that the client will used. If none is provided, // it will default to hclog's default logger. Logger hclog.Logger }
ClientConfig is the configuration used to initialize a new plugin client. After being used to initialize a plugin client, that configuration must not be modified again.
type ClientProtocol ¶
type ClientProtocol interface { io.Closer // Dispense dispenses a new instance of the plugin with the given name. Dispense(string) (interface{}, error) // Ping checks that the client connection is still healthy. Ping() error }
ClientProtocol is an interface that must be implemented for new plugin protocols to be clients.
type GRPCClient ¶
type GRPCClient struct { Conn *grpc.ClientConn Plugins map[string]Plugin }
GRPCClient connects to a GRPCServer over gRPC to dispense plugin types.
func (*GRPCClient) Dispense ¶
func (c *GRPCClient) Dispense(name string) (interface{}, error)
ClientProtocol impl.
type GRPCPlugin ¶
type GRPCPlugin interface { // GRPCServer should register this plugin for serving with the // given GRPCServer. Unlike Plugin.Server, this is only called once // since gRPC plugins serve singletons. GRPCServer(*grpc.Server) error // GRPCClient should return the interface implementation for the plugin // you're serving via gRPC. GRPCClient(*grpc.ClientConn) (interface{}, error) }
GRPCPlugin is the interface that is implemented to serve/connect to a plugin over gRPC.
type GRPCServer ¶
type GRPCServer struct { // Plugins are the list of plugins to serve. Plugins map[string]Plugin // Server is the actual server that will accept connections. This // will be used for plugin registration as well. Server func([]grpc.ServerOption) *grpc.Server // TLS should be the TLS configuration if available. If this is nil, // the connection will not have transport security. TLS *tls.Config // DoneCh is the channel that is closed when this server has exited. DoneCh chan struct{} // Stdout/StderrLis are the readers for stdout/stderr that will be copied // to the stdout/stderr connection that is output. Stdout io.Reader Stderr io.Reader // contains filtered or unexported fields }
GRPCServer is a ServerType implementation that serves plugins over gRPC. This allows plugins to easily be written for other languages.
The GRPCServer outputs a custom configuration as a base64-encoded JSON structure represented by the GRPCServerConfig config structure.
func (*GRPCServer) Config ¶
func (s *GRPCServer) Config() string
Config is the GRPCServerConfig encoded as JSON then base64.
func (*GRPCServer) Serve ¶
func (s *GRPCServer) Serve(lis net.Listener)
type GRPCServerConfig ¶
type GRPCServerConfig struct { StdoutAddr string `json:"stdout_addr"` StderrAddr string `json:"stderr_addr"` }
GRPCServerConfig is the extra configuration passed along for consumers to facilitate using GRPC plugins.
type HandshakeConfig ¶
type HandshakeConfig struct { // ProtocolVersion is the version that clients must match on to // agree they can communicate. This should match the ProtocolVersion // set on ClientConfig when using a plugin. ProtocolVersion uint // MagicCookieKey and value are used as a very basic verification // that a plugin is intended to be launched. This is not a security // measure, just a UX feature. If the magic cookie doesn't match, // we show human-friendly output. MagicCookieKey string MagicCookieValue string }
HandshakeConfig is the configuration used by client and servers to handshake before starting a plugin connection. This is embedded by both ServeConfig and ClientConfig.
In practice, the plugin host creates a HandshakeConfig that is exported and plugins then can easily consume it.
type MuxBroker ¶
MuxBroker is responsible for brokering multiplexed connections by unique ID.
It is used by plugins to multiplex multiple RPC connections and data streams on top of a single connection between the plugin process and the host process.
This allows a plugin to request a channel with a specific ID to connect to or accept a connection from, and the broker handles the details of holding these channels open while they're being negotiated.
The Plugin interface has access to these for both Server and Client. The broker can be used by either (optionally) to reserve and connect to new multiplexed streams. This is useful for complex args and return values, or anything else you might need a data stream for.
func (*MuxBroker) Accept ¶
Accept accepts a connection by ID.
This should not be called multiple times with the same ID at one time.
func (*MuxBroker) AcceptAndServe ¶
AcceptAndServe is used to accept a specific stream ID and immediately serve an RPC server on that stream ID. This is used to easily serve complex arguments.
The served interface is always registered to the "Plugin" name.
type NetRPCUnsupportedPlugin ¶
type NetRPCUnsupportedPlugin struct{}
NetRPCUnsupportedPlugin implements Plugin but returns errors for the Server and Client functions. This will effectively disable support for net/rpc based plugins.
This struct can be embedded in your struct.
func (NetRPCUnsupportedPlugin) Client ¶
func (p NetRPCUnsupportedPlugin) Client(*MuxBroker, *rpc.Client) (interface{}, error)
func (NetRPCUnsupportedPlugin) Server ¶
func (p NetRPCUnsupportedPlugin) Server(*MuxBroker) (interface{}, error)
type Plugin ¶
type Plugin interface { // Server should return the RPC server compatible struct to serve // the methods that the Client calls over net/rpc. Server(*MuxBroker) (interface{}, error) // Client returns an interface implementation for the plugin you're // serving that communicates to the server end of the plugin. Client(*MuxBroker, *rpc.Client) (interface{}, error) }
Plugin is the interface that is implemented to serve/connect to an inteface implementation.
type RPCClient ¶
type RPCClient struct {
// contains filtered or unexported fields
}
RPCClient connects to an RPCServer over net/rpc to dispense plugin types.
func NewRPCClient ¶
NewRPCClient creates a client from an already-open connection-like value. Dial is typically used instead.
func (*RPCClient) Close ¶
Close closes the connection. The client is no longer usable after this is called.
func (*RPCClient) Ping ¶
Ping pings the connection to ensure it is still alive.
The error from the RPC call is returned exactly if you want to inspect it for further error analysis. Any error returned from here would indicate that the connection to the plugin is not healthy.
func (*RPCClient) SyncStreams ¶
SyncStreams should be called to enable syncing of stdout, stderr with the plugin.
This will return immediately and the syncing will continue to happen in the background. You do not need to launch this in a goroutine itself.
This should never be called multiple times.
type RPCServer ¶
type RPCServer struct { Plugins map[string]Plugin // Stdout, Stderr are what this server will use instead of the // normal stdin/out/err. This is because due to the multi-process nature // of our plugin system, we can't use the normal process values so we // make our own custom one we pipe across. Stdout io.Reader Stderr io.Reader // DoneCh should be set to a non-nil channel that will be closed // when the control requests the RPC server to end. DoneCh chan<- struct{} // contains filtered or unexported fields }
RPCServer listens for network connections and then dispenses interface implementations over net/rpc.
After setting the fields below, they shouldn't be read again directly from the structure which may be reading/writing them concurrently.
func (*RPCServer) ServeConn ¶
func (s *RPCServer) ServeConn(conn io.ReadWriteCloser)
ServeConn runs a single connection.
ServeConn blocks, serving the connection until the client hangs up.
type ReattachConfig ¶
ReattachConfig is used to configure a client to reattach to an already-running plugin process. You can retrieve this information by calling ReattachConfig on Client.
type SecureConfig ¶
SecureConfig is used to configure a client to verify the integrity of an executable before running. It does this by verifying the checksum is expected. Hash is used to specify the hashing method to use when checksumming the file. The configuration is verified by the client by calling the SecureConfig.Check() function.
The host process should ensure the checksum was provided by a trusted and authoritative source. The binary should be installed in such a way that it can not be modified by an unauthorized user between the time of this check and the time of execution.
type ServeConfig ¶
type ServeConfig struct { // HandshakeConfig is the configuration that must match clients. HandshakeConfig // TLSProvider is a function that returns a configured tls.Config. TLSProvider func() (*tls.Config, error) // Plugins are the plugins that are served. Plugins map[string]Plugin // GRPCServer should be non-nil to enable serving the plugins over // gRPC. This is a function to create the server when needed with the // given server options. The server options populated by go-plugin will // be for TLS if set. You may modify the input slice. // // Note that the grpc.Server will automatically be registered with // the gRPC health checking service. This is not optional since go-plugin // relies on this to implement Ping(). GRPCServer func([]grpc.ServerOption) *grpc.Server }
ServeConfig configures what sorts of plugins are served.
func (*ServeConfig) Protocol ¶
func (c *ServeConfig) Protocol() Protocol
Protocol returns the protocol that this server should speak.
type ServeMuxMap ¶
type ServeMuxMap map[string]*ServeConfig
ServeMuxMap is the type that is used to configure ServeMux
type ServerProtocol ¶
type ServerProtocol interface { // Init is called once to configure and initialize the protocol, but // not start listening. This is the point at which all validation should // be done and errors returned. Init() error // Config is extra configuration to be outputted to stdout. This will // be automatically base64 encoded to ensure it can be parsed properly. // This can be an empty string if additional configuration is not needed. Config() string // Serve is called to serve connections on the given listener. This should // continue until the listener is closed. Serve(net.Listener) }
ServerProtocol is an interface that must be implemented for new plugin protocols to be servers.
Source Files
¶
Directories
¶
Path | Synopsis |
---|---|
examples
|
|
grpc/proto
Package proto is a generated protocol buffer package.
|
Package proto is a generated protocol buffer package. |
grpc/shared
Package shared contains shared data between the host and plugins.
|
Package shared contains shared data between the host and plugins. |
test
|
|
grpc
Package grpctest is a generated protocol buffer package.
|
Package grpctest is a generated protocol buffer package. |