httd

README

Design decisions

Response Header

Rate Limit Bucket - Reset timestamp

Discord provices a bunch of header fields to state when a bucket is reset. I have no clue why there are so many, so the function CorrectDiscordHeader makes sure that the header field X-RateLimit-Reset is populated and displays milliseconds not seconds; when Retry-After or X-RateLimit-Reset-After or a json body with the field retry_after is set and holds content.

Visually the following fields wrapped in ~~ should be ignored. The X-RateLimit-Reset is the one of interest.

> GET /api/v6/some-endpoint
> X-RateLimit-Precision: millisecond

< HTTP/1.1 429 TOO MANY REQUESTS
< Content-Type: application/json
~~< Retry-After: 6457~~
< X-RateLimit-Limit: 10
< X-RateLimit-Remaining: 0
< X-RateLimit-Reset: 1470173023000
~~< X-RateLimit-Reset-After: 7~~
< X-RateLimit-Bucket: abcd1234
{
  "message": "You are being rate limited.",
  ~~"retry_after": 6457,~~
  "global": false
}

Rate Limit and endpoint relationships

Linking a given endpoint to a bucket is a serious hassle as Discord does not return the bucket hash on each response. This is really unfortunate as we can't establish relationships between buckets and endpoints before Discord decides to randomly send them (...). This also regards the HTTP methods, even tho the documentation states otherwise. The docs are useless for insight on this matter.

To tackle this, every hashed endpoint is assumed to have its own bucket. To hash an endpoint before it can be linked to a bucket, the snowflakes, except major snowflakes, must be replaced with the string {id}. Major snowflakes are the first snowflakes in any endpoint with the prefixes ["/channels", "/guilds", "/webhooks"].

// examples of before and after hashed endpoints (without http methods)
/test => /test
/test/4234 => /test/{id}
/channels => /channels 
/channels?limit=12 => /channels 
/channels/895349573 => /channels/895349573
/guilds/35347862384/roles/23489723 => /guilds/35347862384/roles/{id}

This will of course cause some hashed endpoints to point to the same bucket, once Discord gives us insight on a per request basis. While endpoint A and B might use the same bucket we won't know unless a request for both A and B returns the hash.

The list of buckets can then be consolidated, memory wise, if two or more hashed endpoints uses the same bucket key/hash. The most recent bucket should overwrite the older bucket and all related endpoints should then point to the same bucket.

Concurrent requests

Due to the above, we can only send one request (per bucket) at the time until Discord returns rate limit information for the given bucket. Once the bucket is reset after Discord Disgord must return to sending request through the same bucket sequentially to reduce potential rate limits.

For a normal header, without any rate limit information, we can only send sequential requests for the bucket.

< HTTP/1.1 200 OK

When Discord gives us some more bucket info we can use this to send concurrent requests. In this example we can send up to 7 concurrent requests until the time hits the Reset unix. Then we're back to sequential bucket requests.

< HTTP/1.1 200 OK
< X-RateLimit-Limit: 10
< X-RateLimit-Remaining: 7
< X-RateLimit-Reset: 1470173023000
< X-RateLimit-Bucket: abcd1234

Discord is messed up and each bucket runs requests in a sequential fashion. For now this is the best that can be done with Discord.

Documentation

Index

• Constants
• Variables
• func HeaderToTime(header http.Header) (t time.Time, err error)
• func NormalizeDiscordHeader(statusCode int, header http.Header, body []byte) (h http.Header, err error)
• func SupportsDiscordAPIVersion(version int) bool
• type Client
  • func NewClient(conf *Config) (*Client, error)
  • func (c *Client) BucketGrouping() (group map[string][]string)
  • func (c *Client) Do(ctx context.Context, r *Request) (resp *http.Response, body []byte, err error)
• type Config
• type Details
• type ErrREST
  • func (e *ErrREST) Error() string
• type Error
  • func (e *Error) Error() string
• type HttpClientDoer
• type Manager
  • func NewManager(defaultRelations map[string]string) *Manager
  • func (r *Manager) Bucket(id string, cb func(bucket RESTBucket))
  • func (r *Manager) BucketGrouping() (group map[string][]string)
  • func (r *Manager) Consolidate()
  • func (r *Manager) ProxyID(id string) (pID string)
  • func (r *Manager) UpdateProxyID(id, pID, bucketHash string)
• type RESTBucket
• type RESTBucketManager
• type RateLimitResponseStructure
• type Request
  • func (r *Request) HashEndpoint() string
  • func (r *Request) PopulateMissing()
• type Requester
• type Snowflake

Constants

const (
	BaseURL = "https://discord.com/api"

	RegexpSnowflakes     = `([0-9]+)`
	RegexpURLSnowflakes  = `\/` + RegexpSnowflakes + `\/?`
	RegexpEmoji          = `([.^/]+)\s?`
	RegexpReactionPrefix = `\/channels\/([0-9]+)\/messages\/\{id\}\/reactions\/`

	// Header
	AuthorizationFormat = "%s"
	UserAgentFormat     = "DiscordBot (%s, %s) %s"

	ContentEncoding = "Content-Encoding"
	ContentType     = "Content-Type"
	ContentTypeJSON = "application/json"
	GZIPCompression = "gzip"
)

defaults and string format's for Discord interaction

const (
	XAuditLogReason         = "X-Audit-Log-Reason"
	XRateLimitPrecision     = "X-RateLimit-Precision"
	XRateLimitBucket        = "X-RateLimit-Bucket"
	XRateLimitLimit         = "X-RateLimit-Limit"
	XRateLimitRemaining     = "X-RateLimit-Remaining"
	XRateLimitReset         = "X-RateLimit-Reset"
	XRateLimitResetAfter    = "X-RateLimit-Reset-After"
	XRateLimitGlobal        = "X-RateLimit-Global"
	RateLimitRetryAfter     = "Retry-After"
	DisgordNormalizedHeader = "X-Disgord-Normalized-Kufdsfksduhf-S47yf"
	XDisgordNow             = "X-Disgord-Now-fsagkhf"
)

http rate limit identifiers

const (
	MethodGet    httpMethod = http.MethodGet
	MethodDelete httpMethod = http.MethodDelete
	MethodPost   httpMethod = http.MethodPost
	MethodPatch  httpMethod = http.MethodPatch
	MethodPut    httpMethod = http.MethodPut
)

const GlobalHash = "global"

Variables

var (
	ErrRateLimited error = &Error{"rate limited", time.Unix(0, 0)}
)

Functions

func HeaderToTime

func HeaderToTime(header http.Header) (t time.Time, err error)

HeaderToTime takes the response header from Discord and extracts the timestamp. Useful for detecting time desync between discord and client

func NormalizeDiscordHeader

func NormalizeDiscordHeader(statusCode int, header http.Header, body []byte) (h http.Header, err error)

NormalizeDiscordHeader overrides header fields with body content and make sure every header field uses milliseconds and not seconds. Regards rate limits only.

func SupportsDiscordAPIVersion

func SupportsDiscordAPIVersion(version int) bool

SupportsDiscordAPIVersion check if a given discord api version is supported by this package.

Types

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client for handling Discord REST requests

func NewClient

func NewClient(conf *Config) (*Client, error)

NewClient ...

func (*Client) BucketGrouping

func (c *Client) BucketGrouping() (group map[string][]string)

func (*Client) Do

func (c *Client) Do(ctx context.Context, r *Request) (resp *http.Response, body []byte, err error)

type Config

type Config struct {
	APIVersion int
	BotToken   string

	HttpClient HttpClientDoer

	CancelRequestWhenRateLimited bool

	// RESTBucketManager stores all rate limit buckets and dictates the behaviour of how rate limiting is respected
	RESTBucketManager RESTBucketManager

	// Header field: `User-Agent: DiscordBot ({Source}, {Version}) {Extra}`
	UserAgentVersion   string
	UserAgentSourceURL string
	UserAgentExtra     string
}

Config is the configuration options for the httd.Client structure. Essentially the behaviour of all requests sent to Discord.

type Details

type Details struct {
	Ratelimiter     string
	Endpoint        string // always as a suffix to Ratelimiter(!)
	ResponseStruct  interface{}
	SuccessHTTPCode int
}

Details ...

type ErrREST

type ErrREST struct {
	Code           int      `json:"code"`
	Msg            string   `json:"message"`
	Suggestion     string   `json:"-"`
	HTTPCode       int      `json:"-"`
	Bucket         []string `json:"-"`
	HashedEndpoint string   `json:"-"`
}

func (*ErrREST) Error

func (e *ErrREST) Error() string

type Error

type Error struct {
	// contains filtered or unexported fields
}

func (*Error) Error

func (e *Error) Error() string

type HttpClientDoer

type HttpClientDoer interface {
	Do(req *http.Request) (*http.Response, error)
}

type Manager

type Manager struct {
	// contains filtered or unexported fields
}

func NewManager

func NewManager(defaultRelations map[string]string) *Manager

func (*Manager) Bucket

func (r *Manager) Bucket(id string, cb func(bucket RESTBucket))

func (*Manager) BucketGrouping

func (r *Manager) BucketGrouping() (group map[string][]string)

func (*Manager) Consolidate

func (r *Manager) Consolidate()

func (*Manager) ProxyID

func (r *Manager) ProxyID(id string) (pID string)

func (*Manager) UpdateProxyID

func (r *Manager) UpdateProxyID(id, pID, bucketHash string)

type RESTBucket

type RESTBucket interface {
	// Transaction allows a selective atomic transaction. For distributed systems, the buckets can be
	// eventual consistent until a rate limit is hit then they must be strongly consistent. The global bucket
	// must always be strongly consistent. Tip: it might be easier/best to keep everything strongly consistent,
	// and only care about eventual consistency to get better performance as a "bug"/"accident".
	Transaction(context.Context, func() (*http.Response, []byte, error)) (*http.Response, []byte, error)
}

RESTBucket is a REST bucket for one endpoint or several endpoints. This includes the global bucket.

type RESTBucketManager

type RESTBucketManager interface {
	// Bucket returns the bucket for a given local hash. Note that a local hash simply means
	// a hashed endpoint. This is because Discord does not specify bucket hashed ahead of time.
	// Note you should map localHashes to Discord bucket hashes once that insight have been gained.
	// Discord Bucket hashes are found in the response header, field name `X-RateLimit-Bucket`.
	Bucket(localHash string, cb func(bucket RESTBucket))

	// BucketGrouping shows which hashed endpoints falls under which bucket hash
	// here a bucket hash is defined by discord, otherwise the bucket hash
	// is the same as the hashed endpoint.
	//
	// Hashed endpoints are generated by the Request struct.
	BucketGrouping() (group map[string][]string)
}

RESTBucketManager manages the buckets and the global bucket.

type RateLimitResponseStructure

type RateLimitResponseStructure struct {
	Message    string  `json:"message"`     // A message saying you are being rate limited.
	RetryAfter float64 `json:"retry_after"` // The number of seconds to wait before submitting another request.
	Global     bool    `json:"global"`      // A value indicating if you are being globally rate limited or not
}

type Request

type Request struct {
	Ctx context.Context

	Method      httpMethod
	Endpoint    string
	Body        interface{} // will automatically marshal to JSON if the ContentType is httd.ContentTypeJSON
	ContentType string

	// Reason is a X-Audit-Log-Reason header field that will show up on the audit log for this action.
	Reason string
	// contains filtered or unexported fields
}

Request is populated before executing a Discord request to correctly generate a http request

func (*Request) HashEndpoint

func (r *Request) HashEndpoint() string

func (*Request) PopulateMissing

func (r *Request) PopulateMissing()

type Requester

type Requester interface {
	Do(ctx context.Context, req *Request) (resp *http.Response, body []byte, err error)
}

Requester holds all the sub-request interface for Discord interaction

type Snowflake

type Snowflake = snowflake.Snowflake