README
¶
Go Redis IP Ranges (goripr)
goripr is an eficient way to store IP ranges in a redis database and mapping those ranges to specific strings.
This package wraps the widely used redis Go client and extends its feature set with a storage efficient mapping of IPv4 ranges to specific strings called reasons.
I intend to use this package in my VPN Detection, that's why the term "reason" is used. The term refers to a ban reason that is given when a player using a VPN (they usually do that with malicious intent) gets banned. The string can be used in any other way needed, especially containing JSON formatted data.
Idea
The general approach is to save the beginning and the end of a range into the database.
The beginning boundary has the property called LowerBound set to true and the last IP in a given range is called an upper boundary with the property UpperBound set to true.
Based on these properties it is possible to determine, how to cut existing boundaries, when new IP ranges are inserted into the database.
Problem it solves
The VPN detection and especially the ban server used to save all IPs from the given ranges with their corresponding reasons into the database. That is the trivial approach, but proved to be inefficient when having more than 100 million individual IPs stored in the Redis database. At it's peak the database needed ~7GB of RAM, which is not a feasible solution, especially when the source files containing the actual ranges in their respective masked shorthand notation (x.x.x.x/24) needed less than one MB of storage space.
Gains over the trivial approach
On the other hand, iterating over ~50k such range strings was also not a feasible solution, especially when the ban server should react within ~1 second. The compromise should be a slower reaction time compared to the triavial approach, but way less of a RAM overhead. I guess that the reduction of RAM usage by a factor of about 240x should also improve the response time significantly, as the ~7GB approach was burdening even high performance servers rather heavily. The current RAM that is being used is about 30MB, which is acceptable.
Input format of the package
# custom IP range
84.141.32.1 - 84.141.32.255
# single IP
84.141.32.1
# subnet mask
84.141.32.1/24
Example
package main
import (
"bufio"
"context"
"errors"
"flag"
"fmt"
"os"
"regexp"
"github.com/jxsl13/goripr/v2"
)
var (
splitRegex = regexp.MustCompile(`([0-9.\-\s/]+)#?\s*(.*)\s*$`)
defaultReason = "VPN - https://website.com"
addFile = ""
findIP = ""
)
func init() {
flag.StringVar(&addFile, "add", "", "-add filename.txt")
flag.StringVar(&findIP, "find", "", "-find 123.0.0.1")
flag.Parse()
if addFile == "" && findIP == "" {
flag.PrintDefaults()
os.Exit(1)
}
}
func parseLine(line string) (ip, reason string, err error) {
if matches := splitRegex.FindStringSubmatch(line); len(matches) > 0 {
return matches[1], matches[2], nil
}
return "", "", errors.New("empty")
}
func addIPsToDatabase(rdb *goripr.Client, ctx context.Context, filename string) error {
file, err := os.Open(filename)
if err != nil {
return err
}
scanner := bufio.NewScanner(file)
for scanner.Scan() {
ip, reason, err := parseLine(scanner.Text())
if err != nil {
continue
}
if reason == "" {
reason = defaultReason
}
err = rdb.Insert(ctx, ip, reason)
if err != nil {
if !errors.Is(err, goripr.ErrInvalidRange) {
fmt.Println(err, "Input:", ip)
}
continue
}
}
return nil
}
func main() {
ctx := context.Background()
rdb, err := goripr.NewClient(ctx, goripr.Options{
Addr: "localhost:6379",
DB: 0,
})
if err != nil {
fmt.Println("error:", err)
os.Exit(1)
}
defer rdb.Close()
if addFile != "" {
err := addIPsToDatabase(rdb, ctx, addFile)
if err != nil {
fmt.Println("error:", err)
os.Exit(1)
}
} else if findIP != "" {
reason, err := rdb.Find(ctx, findIP)
if err != nil {
fmt.Println("IP:", findIP, "error:", err)
os.Exit(1)
}
fmt.Println("IP:", findIP, "Reason:", reason)
return
}
}
// Output: IP: 84.141.32.1 Reason: any range where the first IP is smaller than the second
// Output: IP: 84.141.32.0 error: the given IP was not found in any database ranges
Example text file
84.141.32.1 - 84.141.32.255 # any range where the first IP is smaller than the second
2.56.92.0/22 # VPN subnet masking
# without a reason (uses default reason)
2.56.140.0/24
TODO
- Optional Cache of requested IPs for like 24 hours in order to improve response time for recurring requests (rejoining players)
Documentation
¶
Index ¶
- Constants
- Variables
- type Client
- func (c *Client) Close() error
- func (c *Client) Find(ctx context.Context, ip string) (reason string, err error)
- func (c *Client) Flush(ctx context.Context) error
- func (c *Client) Insert(ctx context.Context, ipRange, reason string) error
- func (c *Client) Remove(ctx context.Context, ipRange string) error
- func (c *Client) Reset(ctx context.Context) error
- func (c *Client) UpdateReasonOf(ctx context.Context, ip string, fn UpdateFunc) (err error)
- type Error
- type Options
- type UpdateFunc
Constants ¶
const ( // ErrConnectionFailed is returned when the connection to the redis database fails. ErrConnectionFailed = Error("failed to establish a connection to the redis database") // ErrDatabaseInit is returned when the initialization of the database boundaries fails. ErrDatabaseInit = Error("failed to initialize database ±inf boundaries") // ErrDatabaseInconsistent is returned when the initialization of the database boundaries fails. ErrDatabaseInconsistent = Error("the databe is in an inconsistent state") // ErrInvalidRange is returned when a passed string is not a valid range ErrInvalidRange = Error("invalid range passed, use either of these: <IP>, <IP>/<1-32>, <IP> - <IP>") // ErrIPv6NotSupported is returned if an IPv6 range or IP input is detected. ErrIPv6NotSupported = Error("IPv6 ranges are not supported") // ErrInvalidIP is returned when the passed argument is an invalid IP ErrInvalidIP = Error("invalid IP passed") // ErrNoResult is returned when a result slic is empty or some connection error occurs during retrieval of values. ErrNoResult = Error("could not retrieve any results from the database") // ErrIPNotFound is returned if the passed IP is not contained in any ranges ErrIPNotFound = Error("the given IP was not found in any database ranges") )
Variables ¶
var ( // IPRangesKey contains the key name of the sorted set that contains the IPs (integers) IPRangesKey = "________________IP_RANGES________________" // DeleteReason is given to a specific deltion range // on a second attept (not atomic) the range is then finally deleted. DeleteReason = "_________________DELETE_________________" )
Functions ¶
This section is empty.
Types ¶
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client is an extended version of the redis.Client
func (*Client) Find ¶
Find searches for the requested IP in the database. If the IP is found within any previously inserted range, the associated reason is returned. If it is not found, an error is returned instead. returns a reason or either ErrIPNotFound if no IP was found ErrDatabaseInconsistent if the database has become inconsistent.
func (*Client) Insert ¶
Insert inserts a new IP range or IP into the database with an associated reason string
func (*Client) UpdateReasonOf ¶
UpdateReasonOf updates the reason of the range that contains the passed ip.
type Error ¶
type Error string
Error is a wrapper for constant errors that are not supposed to be changed.
type Options ¶
type Options struct {
// The network type, either tcp or unix.
// Default is tcp.
Network string
// host:port address.
Addr string
// ClientName will execute the `CLIENT SETNAME ClientName` command for each conn.
ClientName string
// Dialer creates new network connection and has priority over
// Network and Addr options.
Dialer func(ctx context.Context, network, addr string) (net.Conn, error)
// Hook that is called when new connection is established.
OnConnect func(ctx context.Context, cn *redis.Conn) error
// Protocol 2 or 3. Use the version to negotiate RESP version with redis-server.
// Default is 3.
Protocol int
// Use the specified Username to authenticate the current connection
// with one of the connections defined in the ACL list when connecting
// to a Redis 6.0 instance, or greater, that is using the Redis ACL system.
Username string
// Optional password. Must match the password specified in the
// requirepass server configuration option (if connecting to a Redis 5.0 instance, or lower),
// or the User Password when connecting to a Redis 6.0 instance, or greater,
// that is using the Redis ACL system.
Password string
// CredentialsProvider allows the username and password to be updated
// before reconnecting. It should return the current username and password.
CredentialsProvider func() (username string, password string)
// Database to be selected after connecting to the server.
DB int
// Maximum number of retries before giving up.
// Default is 3 retries; -1 (not 0) disables retries.
MaxRetries int
// Minimum backoff between each retry.
// Default is 8 milliseconds; -1 disables backoff.
MinRetryBackoff time.Duration
// Maximum backoff between each retry.
// Default is 512 milliseconds; -1 disables backoff.
MaxRetryBackoff time.Duration
// Dial timeout for establishing new connections.
// Default is 5 seconds.
DialTimeout time.Duration
// Timeout for socket reads. If reached, commands will fail
// with a timeout instead of blocking. Supported values:
// - `0` - default timeout (3 seconds).
// - `-1` - no timeout (block indefinitely).
// - `-2` - disables SetReadDeadline calls completely.
ReadTimeout time.Duration
// Timeout for socket writes. If reached, commands will fail
// with a timeout instead of blocking. Supported values:
// - `0` - default timeout (3 seconds).
// - `-1` - no timeout (block indefinitely).
// - `-2` - disables SetWriteDeadline calls completely.
WriteTimeout time.Duration
// ContextTimeoutEnabled controls whether the client respects context timeouts and deadlines.
// See https://redis.uptrace.dev/guide/go-redis-debugging.html#timeouts
ContextTimeoutEnabled bool
// Type of connection pool.
// true for FIFO pool, false for LIFO pool.
// Note that FIFO has slightly higher overhead compared to LIFO,
// but it helps closing idle connections faster reducing the pool size.
PoolFIFO bool
// Maximum number of socket connections.
// Default is 10 connections per every available CPU as reported by runtime.GOMAXPROCS.
PoolSize int
// Amount of time client waits for connection if all connections
// are busy before returning an error.
// Default is ReadTimeout + 1 second.
PoolTimeout time.Duration
// Minimum number of idle connections which is useful when establishing
// new connection is slow.
// Default is 0. the idle connections are not closed by default.
MinIdleConns int
// Maximum number of idle connections.
// Default is 0. the idle connections are not closed by default.
MaxIdleConns int
// ConnMaxIdleTime is the maximum amount of time a connection may be idle.
// Should be less than server's timeout.
//
// Expired connections may be closed lazily before reuse.
// If d <= 0, connections are not closed due to a connection's idle time.
//
// Default is 30 minutes. -1 disables idle timeout check.
ConnMaxIdleTime time.Duration
// ConnMaxLifetime is the maximum amount of time a connection may be reused.
//
// Expired connections may be closed lazily before reuse.
// If <= 0, connections are not closed due to a connection's age.
//
// Default is to not close idle connections.
ConnMaxLifetime time.Duration
// TLS Config to use. When set, TLS will be negotiated.
TLSConfig *tls.Config
// Limiter interface used to implement circuit breaker or rate limiter.
Limiter redis.Limiter
}
Options keeps the settings to set up redis connection.
type UpdateFunc ¶
UpdateFunc updates the previous reason to a new reason.
Source Files