README
¶
Design decisions
Response Header
Rate Limit Bucket - Reset timestamp
Discord provices a bunch of header fields to state when a bucket is reset. I have no clue why there are so many, so the function CorrectDiscordHeader
makes sure that the header field X-RateLimit-Reset
is populated and displays milliseconds not seconds; when Retry-After
or X-RateLimit-Reset-After
or a json body with the field retry_after
is set and holds content.
Visually the following fields wrapped in ~~ should be ignored. The X-RateLimit-Reset
is the one of interest.
> GET /api/v6/some-endpoint
> X-RateLimit-Precision: millisecond
< HTTP/1.1 429 TOO MANY REQUESTS
< Content-Type: application/json
~~< Retry-After: 6457~~
< X-RateLimit-Limit: 10
< X-RateLimit-Remaining: 0
< X-RateLimit-Reset: 1470173023000
~~< X-RateLimit-Reset-After: 7~~
< X-RateLimit-Bucket: abcd1234
{
"message": "You are being rate limited.",
~~"retry_after": 6457,~~
"global": false
}
Rate Limit and endpoint relationships
Linking a given endpoint to a bucket is a serious hassle as Discord does not return the bucket hash on each response. This is really unfortunate as we can't establish relationships between buckets and endpoints before Discord decides to randomly send them (...). This also regards the HTTP methods, even tho the documentation states otherwise. The docs are useless for insight on this matter.
To tackle this, every hashed endpoint is assumed to have its own bucket. To hash an endpoint before it can be linked to a bucket, the snowflakes, except major snowflakes, must be replaced with the string {id}
. Major snowflakes are the first snowflakes in any endpoint with the prefixes ["/channels", "/guilds", "/webhooks"].
// examples of before and after hashed endpoints (without http methods)
/test => /test
/test/4234 => /test/{id}
/channels => /channels
/channels?limit=12 => /channels
/channels/895349573 => /channels/895349573
/guilds/35347862384/roles/23489723 => /guilds/35347862384/roles/{id}
This will of course cause some hashed endpoints to point to the same bucket, once Discord gives us insight on a per request basis. While endpoint A and B might use the same bucket we won't know unless a request for both A and B returns the hash.
The list of buckets can then be consolidated, memory wise, if two or more hashed endpoints uses the same bucket key/hash. The most recent bucket should overwrite the older bucket and all related endpoints should then point to the same bucket.
Concurrent requests
Due to the above, we can only send one request (per bucket) at the time until Discord returns rate limit information for the given bucket. Once the bucket is reset after Discord Disgord must return to sending request through the same bucket sequentially to reduce potential rate limits.
For a normal header, without any rate limit information, we can only send sequential requests for the bucket.
< HTTP/1.1 200 OK
When Discord gives us some more bucket info we can use this to send concurrent requests. In this example we can send up to 7 concurrent requests until the time hits the Reset unix. Then we're back to sequential bucket requests.
< HTTP/1.1 200 OK
< X-RateLimit-Limit: 10
< X-RateLimit-Remaining: 7
< X-RateLimit-Reset: 1470173023000
< X-RateLimit-Bucket: abcd1234
Discord is messed up and each bucket runs requests in a sequential fashion. For now this is the best that can be done with Discord.
Documentation
¶
Index ¶
- Constants
- Variables
- func HeaderToTime(header http.Header) (t time.Time, err error)
- func NormalizeDiscordHeader(statusCode int, header http.Header, body []byte) (h http.Header, err error)
- func SupportsDiscordAPIVersion(version int) bool
- type Client
- type Config
- type Details
- type ErrREST
- type Error
- type HttpClientDoer
- type Manager
- type RESTBucket
- type RESTBucketManager
- type RateLimitResponseStructure
- type Request
- type Requester
- type Snowflake
Constants ¶
const ( BaseURL = "https://discord.com/api" RegexpSnowflakes = `([0-9]+)` RegexpURLSnowflakes = `\/` + RegexpSnowflakes + `\/?` RegexpEmoji = `([.^/]+)\s?` RegexpReactionPrefix = `\/channels\/([0-9]+)\/messages\/\{id\}\/reactions\/` // Header AuthorizationFormat = "Bot %s" UserAgentFormat = "DiscordBot (%s, %s) %s" ContentEncoding = "Content-Encoding" ContentType = "Content-Type" ContentTypeJSON = "application/json" GZIPCompression = "gzip" )
defaults and string format's for Discord interaction
const ( XAuditLogReason = "X-Audit-Log-Reason" XRateLimitPrecision = "X-RateLimit-Precision" XRateLimitBucket = "X-RateLimit-Bucket" XRateLimitLimit = "X-RateLimit-Limit" XRateLimitRemaining = "X-RateLimit-Remaining" XRateLimitReset = "X-RateLimit-Reset" XRateLimitResetAfter = "X-RateLimit-Reset-After" XRateLimitGlobal = "X-RateLimit-Global" RateLimitRetryAfter = "Retry-After" DisgordNormalizedHeader = "X-Disgord-Normalized-Kufdsfksduhf-S47yf" XDisgordNow = "X-Disgord-Now-fsagkhf" )
http rate limit identifiers
const GlobalHash = "global"
Variables ¶
var (
ErrRateLimited error = &Error{"rate limited", time.Unix(0, 0)}
)
Functions ¶
func HeaderToTime ¶
HeaderToTime takes the response header from Discord and extracts the timestamp. Useful for detecting time desync between discord and client
func NormalizeDiscordHeader ¶
func NormalizeDiscordHeader(statusCode int, header http.Header, body []byte) (h http.Header, err error)
NormalizeDiscordHeader overrides header fields with body content and make sure every header field uses milliseconds and not seconds. Regards rate limits only.
func SupportsDiscordAPIVersion ¶
SupportsDiscordAPIVersion check if a given discord api version is supported by this package.
Types ¶
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client for handling Discord REST requests
func (*Client) BucketGrouping ¶
type Config ¶
type Config struct { APIVersion int BotToken string HttpClient HttpClientDoer CancelRequestWhenRateLimited bool // RESTBucketManager stores all rate limit buckets and dictates the behaviour of how rate limiting is respected RESTBucketManager RESTBucketManager // Header field: `User-Agent: DiscordBot ({Source}, {Version}) {Extra}` UserAgentVersion string UserAgentSourceURL string UserAgentExtra string }
Config is the configuration options for the httd.Client structure. Essentially the behaviour of all requests sent to Discord.
type Details ¶
type Details struct { Ratelimiter string Endpoint string // always as a suffix to Ratelimiter(!) ResponseStruct interface{} SuccessHTTPCode int }
Details ...
type ErrREST ¶
type HttpClientDoer ¶ added in v0.27.0
type Manager ¶
type Manager struct {
// contains filtered or unexported fields
}
func NewManager ¶
func (*Manager) Bucket ¶
func (r *Manager) Bucket(id string, cb func(bucket RESTBucket))
func (*Manager) BucketGrouping ¶
func (*Manager) Consolidate ¶
func (r *Manager) Consolidate()
func (*Manager) UpdateProxyID ¶
type RESTBucket ¶
type RESTBucket interface { // Transaction allows a selective atomic transaction. For distributed systems, the buckets can be // eventual consistent until a rate limit is hit then they must be strongly consistent. The global bucket // must always be strongly consistent. Tip: it might be easier/best to keep everything strongly consistent, // and only care about eventual consistency to get better performance as a "bug"/"accident". Transaction(context.Context, func() (*http.Response, []byte, error)) (*http.Response, []byte, error) }
RESTBucket is a REST bucket for one endpoint or several endpoints. This includes the global bucket.
type RESTBucketManager ¶
type RESTBucketManager interface { // Bucket returns the bucket for a given local hash. Note that a local hash simply means // a hashed endpoint. This is because Discord does not specify bucket hashed ahead of time. // Note you should map localHashes to Discord bucket hashes once that insight have been gained. // Discord Bucket hashes are found in the response header, field name `X-RateLimit-Bucket`. Bucket(localHash string, cb func(bucket RESTBucket)) // BucketGrouping shows which hashed endpoints falls under which bucket hash // here a bucket hash is defined by discord, otherwise the bucket hash // is the same as the hashed endpoint. // // Hashed endpoints are generated by the Request struct. BucketGrouping() (group map[string][]string) }
RESTBucketManager manages the buckets and the global bucket.
type RateLimitResponseStructure ¶
type RateLimitResponseStructure struct { Message string `json:"message"` // A message saying you are being rate limited. RetryAfter float64 `json:"retry_after"` // The number of seconds to wait before submitting another request. Global bool `json:"global"` // A value indicating if you are being globally rate limited or not }
type Request ¶
type Request struct { Ctx context.Context Method string Endpoint string Body interface{} // will automatically marshal to JSON if the ContentType is httd.ContentTypeJSON ContentType string // Reason is a X-Audit-Log-Reason header field that will show up on the audit log for this action. Reason string // contains filtered or unexported fields }
Request is populated before executing a Discord request to correctly generate a http request
func (*Request) HashEndpoint ¶
func (*Request) PopulateMissing ¶
func (r *Request) PopulateMissing()