uds

README

UDS - Packages for using Unix Domain Sockets

Usage Note

This package is considered beta. Until this note is removed, changes to this that cause an incompatibility will be noted with a Minor version change, not a Major version change in the overall repo.

TLDR

• Provides convience over standard net.Listen("unix") and net.Dial("unix")
• Provides high level packages to do streams and RPCs(protocol buffers/JSON)
• Provides authentication on the server
• Benchmarked against gRPC, mostly faster(except at 1kiB or lower) and WAY LESS allocations
• Linux/OSX only

Introduction

Go provides unix sockets via its net.Dial() and net.Listen() calls. But how to use these to achive IPC (interprocess communication) isn't straightforward.

This package and its sub-packages provide authenticated streaming and RPCs utilizing:

• Data chunks (whatever you want to send in []byte)
• JSON messages
• Protocol buffer(proto) messages

If you simply want to use UDS with an io.ReadWriteCloser interface without dealing with all the Dial()/Listen() noise (plus you get authentication), the main uds/ package is for you.

Works on Linux and OSX only. Haven't investigated all the BSD syscall options needed and don't want to tackle Windows (but I do take PRs).

Use cases

You need to do IPC for something cross platform (or at least Linux and OSX) and want that communication authenticated.

UDS is on most platforms (though not all options are). However, it is not the fastest IPC.

Shared memory seems to be the fasted (but its a pain). Various message queuing destroys UDS in performance, but they aren't very cross platform (so you'd have to use a different one per platform).

I've used UDS to communicate to sub-processes when I want to mount and unmount "plugins". Sometimes between languages.

Alternatives

There are some other packages out there, but I haven't seen them supporting auth.

gRPC works great for IPC on UDS, at least with protos. While I love gRPC, I wanted more options, authentication and much lower heap allocations.

Everyone loves some benchmarks

So I certainly haven't benched everything. But I did bench the proto RPC, as it uses the chunking rpc package, which uses chunked streams.

I compared this to gRPC as it provides a managed RPC mechansim over unix sockets. Interestingly, gRPC seem to handle the smallest data requests (1.0 kB) better than this package, but this seems to be do better for all others. I think the small data size test is due to my use of memory pools for reuse and slightly more overhead in my packets.

No matter what my internal settings, I would beat gRPC at 10kB and double their performance in the 102 kB size. To get better performance in large sizes, I had to add some kernel buffer space over the defaults, which lead to close to double performance.

But the real killer here is allocations. This decimates gRPC in heap allocation reduction. Keep this in mind for high performance applications. If you delve deep into making Go fast, everything points at one thing: keeping your allocations down. Once you leave the micro-benchmark world (like these benchmarks), your app starts crawling if the GC has to deal with lots of objects. The key to that is buffer reuse and gRPC's design for ease of use hurts that ability.

Platform was OSX running on an 8-core Macbook Pro, Circa 2019. You can guess that your Threadripper Linux box will do much better.

Test Results(uds):
==========================================================================
[Speed]

[16 Users][10000 Requests][1.0 kB Bytes] - min 65.286µs/sec, max 5.830322ms/sec, avg 373.437µs/sec, rps 42534.71

[16 Users][10000 Requests][10 kB Bytes] - min 108.214µs/sec, max 7.875121ms/sec, avg 576.661µs/sec, rps 27596.17

[16 Users][10000 Requests][102 kB Bytes] - min 760.896µs/sec, max 13.829294ms/sec, avg 1.188321ms/sec, rps 13419.44

[16 Users][10000 Requests][1.0 MB Bytes] - min 7.26139ms/sec, max 80.21521ms/sec, avg 14.509374ms/sec, rps 1101.91


[Allocs]

[10000 Requests][1.0 kB Bytes] - allocs 290,365

[10000 Requests][10 kB Bytes] - allocs 300,598

[10000 Requests][102 kB Bytes] - allocs 302,142

[10000 Requests][1.0 MB Bytes] - allocs 309,110

Test Results(grpc):
==========================================================================
[Speed]

[16 Users][10000 Requests][1.0 kB Bytes] - min 59.624µs/sec, max 3.571806ms/sec, avg 305.171µs/sec, rps 51137.15

[16 Users][10000 Requests][10 kB Bytes] - min 93.19µs/sec, max 2.397846ms/sec, avg 875.864µs/sec, rps 18216.72

[16 Users][10000 Requests][102 kB Bytes] - min 1.221068ms/sec, max 8.495421ms/sec, avg 4.434272ms/sec, rps 3606.63

[16 Users][10000 Requests][1.0 MB Bytes] - min 21.448849ms/sec, max 54.920306ms/sec, avg 34.307985ms/sec, rps 466.28


[Allocs]

[10000 Requests][1.0 kB Bytes] - allocs 1,505,165

[10000 Requests][10 kB Bytes] - allocs 1,681,061

[10000 Requests][102 kB Bytes] - allocs 1,947,529

[10000 Requests][1.0 MB Bytes] - allocs 3,250,309

Benchmark Guide:

[# Users] = number of simultaneous clients
[# Requests] = the total number of requests we sent
[# Bytes] = the size of our input and output requests
min = the minimum seen RTT
max = the maximum seen RTT
av = the average seen RTT
rps = requests per second

Documentation

Overview

Package uds provides a server and client for Unix Domain Sockets. This provides a lot of convenience around the "net" package for handling all the file setup and detecting closed connections. It also provides the ability to authenticate connections.

The package currently only works for Linux/Darwin, as those are the systems I use.

This package takes the stance that Read() and Write() calls by default should infinitely block unless the socket is closed. This eases development.

We also handle write only connections where Write() calls may not detect a closed connection. This can be done setting the WriteOnly option.

This package is fairly straight forward in that you can uses server.Conn objects and Client objects as io.ReadWriteClose objects. We also provide higher level clients that handle chunk data, a chunk data RPC client/server, json streams, json RPC client/server, ..

Unix/Linux Note:

Socket paths may have a length limit that is different than the normal
filesystem. On OSX, you can receive "bind: invalid argument" when the name is too long.

On Linux there seems to be an 108 character limit for socket path names.
On OSX it is 104 character limit.

https://github.com/golang/go/issues/6895 .
I have set this as the default limit for all clients so I don't have to figure out the limit on
every type of system and interpret non-sensical errors.

Index

• type Client
  • func NewClient(socketAddr string, uid, gid int, fileModes []os.FileMode) (*Client, error)
  • func (c *Client) Close() error
  • func (c *Client) Read(b []byte) (int, error)
  • func (c *Client) ReadByte() (byte, error)
  • func (c *Client) ReadDeadline(t time.Time)
  • func (c *Client) ReadTimeout(timeout time.Duration)
  • func (c *Client) UnixConn() *net.UnixConn
  • func (c *Client) Write(b []byte) (int, error)
  • func (c *Client) WriteDeadline(t time.Time)
  • func (c *Client) WriteOnly()
  • func (c *Client) WriteTimeout(timeout time.Duration)
• type Conn
  • func (c *Conn) Close() error
  • func (c *Conn) Read(b []byte) (int, error)
  • func (c *Conn) ReadByte() (byte, error)
  • func (c *Conn) ReadDeadline(t time.Time)
  • func (c *Conn) ReadTimeout(timeout time.Duration)
  • func (c *Conn) UnixConn() *net.UnixConn
  • func (c *Conn) Write(b []byte) (int, error)
  • func (c *Conn) WriteDeadline(t time.Time)
  • func (c *Conn) WriteOnly()
  • func (c *Conn) WriteTimeout(timeout time.Duration)
• type Cred
  • func Current() (Cred, *user.User, error)
• type ID
  • func (i ID) Int() int
  • func (i ID) Int32() int32
  • func (i ID) String() string
• type Server
  • func NewServer(socketAddr string, uid, gid int, fileMode os.FileMode) (*Server, error)
  • func (c *Server) Close() error
  • func (c *Server) Closed() chan error
  • func (c *Server) Conn() chan *Conn

Types

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client provides a UDS client for connecting to a UDS server.

func NewClient

func NewClient(socketAddr string, uid, gid int, fileModes []os.FileMode) (*Client, error)

NewClient creates a new UDS client to the socket at socketAddr that must have the uid and gid specified. fileModes provides a list of acceptable file modes that the socket can be in (suggest 0770, 1770). If not specified, 0770 and 1770 are used.

func (*Client) Close

func (c *Client) Close() error

Close closes the connection to the server.

func (*Client) Read

func (c *Client) Read(b []byte) (int, error)

Read implements io.Reader.Read(). This will block until it has read into the buffer. This differs from native behavior which times out. If you want to have read timeouts, use ReadDeadline()/ReadTimeout() to change. This must be done before every Read() call.

func (*Client) ReadByte

func (c *Client) ReadByte() (byte, error)

ReadByte implements io.ByteReader.

func (*Client) ReadDeadline

func (c *Client) ReadDeadline(t time.Time)

ReadDeadline caused the next Read() call to timeout at t. Must be used before every Read() call that you want to have a timeout.

func (*Client) ReadTimeout

func (c *Client) ReadTimeout(timeout time.Duration)

ReadTimeout caused the next Read() call to timeout at time.Now().Add(timeout). Must be used before every Read() call that you want to have a timeout.

func (*Client) UnixConn

func (c *Client) UnixConn() *net.UnixConn

UnixConn will return the underlying UnixConn object. You can use this to use its underlying methods or change buffering. Note that SetReadDeadline/SetWriteDeadline/SetDeadline do not work, use one of Client's methods instead.

func (*Client) Write

func (c *Client) Write(b []byte) (int, error)

Write implements io.Reader.Writer(). This will block until it has written the buffer. This differs from native behavior which times out. If you want to have write timeouts, use WriteDeadeline()/WriteTimeout() to change. This must be done before every Write() call.

func (*Client) WriteDeadline

func (c *Client) WriteDeadline(t time.Time)

WriteDeadline caused the next Write() call to timeout at t. Must be used before every Write() call that you want to have a timeout.

func (*Client) WriteOnly

func (c *Client) WriteOnly()

WriteOnly let's the Client know that it will only be used for writing. This will cause a *special* read to happen on all writes to make sure the connection isn't closed, as writes are not guaranteed to error on a UDS. If doing reads, this isn't required. If this is set and you read fro the Conn the read will panic.

func (*Client) WriteTimeout

func (c *Client) WriteTimeout(timeout time.Duration)

WriteTimeout caused the next Write() call to timeout at time.Now().Add(timeout). Must be used before every Write() call that you want to have a timeout.

type Conn

type Conn struct {
	Cred Cred
	// contains filtered or unexported fields
}

Conn represents a UDS connection from a client. Must take a pointer if this will be copied after being received.

func (*Conn) Close

func (c *Conn) Close() error

Close implements io.Closer.Close().

func (*Conn) Read

func (c *Conn) Read(b []byte) (int, error)

Read implements io.Reader.Read(). This has an inifite read timeout. If you want to have a timeout, call ReadDeadline() or ReadTimeout() before calling. You must do this for every Read() call.

func (*Conn) ReadByte

func (c *Conn) ReadByte() (byte, error)

ReadByte implements io.ByteReader.

func (*Conn) ReadDeadline

func (c *Conn) ReadDeadline(t time.Time)

ReadDeadline caused the next Read() call to timeout at t. Must be used before every Read() call that you want to have a timeout.

func (*Conn) ReadTimeout

func (c *Conn) ReadTimeout(timeout time.Duration)

ReadTimeout caused the next Read() call to timeout at time.Now().Add(timeout). Must be used before every Read() call that you want to have a timeout.

func (*Conn) UnixConn

func (c *Conn) UnixConn() *net.UnixConn

UnixConn will return the underlying UnixConn object. You can use this to use its underlying methods or change buffering. Note that .SetReadDeadline()/.SetWriteDeadline()/.SetDeadline() will not do anything. Use ReadTimeout()/ReadDeadline()/WriteTimeout()/WriteDeadline() defined on Conn.

func (*Conn) Write

func (c *Conn) Write(b []byte) (int, error)

Write implements io.Writer.Write(). This has an inifite write timeout. If you want to have a timeout, call WriteDeadline() or WriteTimeout() before calling. You must do this for every Write() call.

func (*Conn) WriteDeadline

func (c *Conn) WriteDeadline(t time.Time)

WriteDeadline caused the next Write() call to timeout at t. Must be used before every Write() call that you want to have a timeout.

func (*Conn) WriteOnly

func (c *Conn) WriteOnly()

WriteOnly let's the Conn know that this Conn will only be used for writing. This will cause a *special* read to happen on all writes to make sure the connection isn't closed, as writes are not guaranteed to error on a UDS. If doing reads, this isn't required. If this is set and you read from the Conn, Read() will panic.

func (*Conn) WriteTimeout

func (c *Conn) WriteTimeout(timeout time.Duration)

WriteTimeout caused the next Write() call to timeout at time.Now().Add(timeout). Must be used before every Write() call that you want to have a timeout.

type Cred

type Cred struct {
	// PID is the process id of the process.
	PID ID
	// UID is the user id of the process.
	UID ID
	// GID is the group id of the process.
	GID ID
}

Cred provides the credentials of the local process contacting the server.

func Current

func Current() (Cred, *user.User, error)

Current provides information about the current process and user.

type ID

type ID int

ID represents a numeric ID. Go in various libraries stores IDs such as Uid or Gid as strings. However in other more OS specific libraries, it might be int or int32. This simply unifies that so it is easier to translate for whatever need you have. The string representation is probably a compatibility feature to allow an OS that doesn't use numeric IDs to work. However I am not interested in supporting those in this package.

func (ID) Int

func (i ID) Int() int

Int returns the ID as an int.

func (ID) Int32

func (i ID) Int32() int32

Int32 returns the ID as an int32.

func (ID) String

func (i ID) String() string

String returns the ID as a string.

type Server

type Server struct {
	// contains filtered or unexported fields
}

Server provides a Unix Domain Socket server that clients can connect on.

func NewServer

func NewServer(socketAddr string, uid, gid int, fileMode os.FileMode) (*Server, error)

NewServer creates a new UDS server that creates and listens to the file at socketPath. uid and gid are the uid and gid that file will be set to and fileMode is the file mode it will inherit. If sockerAddr exists this will attempt to delete it. Suggest fileMode of 0770.

func (*Server) Close

func (c *Server) Close() error

Close stops listening for connections on the socket.

func (*Server) Closed

func (c *Server) Closed() chan error

Closed returns a channel that returns an error when the connection to the server is closed. This can be because you have called Close(), the socket had a read error or the socket file was removed. An io.EOF error will not be returned (as this is normal operation). Normally this is used to block and return the final status of the server.

func (*Server) Conn

func (c *Server) Conn() chan *Conn

Conn returns a channel that is populated with connection to the server. The channel is closed when the server's is no longer serving.