convoy

README

• Website: https://getconvoy.io
• Forum: Github Discussions
• Documentation: getconvoy.io/docs
• Deploy: Install Convoy
• Slack: Join the Community

Convoy is an open source high-performance webhooks gateway used to securely ingest, persist, debug, deliver and manage millions of events reliably with rich features such as retries, rate limiting, static ips, circuit breaking, rolling secrets and more. To get started import the openapi spec into Postman or Insomnia.

Convoy provides several key features:

• Webhooks Gateway: As a webhooks gateway, Convoy lives at the edge of your network to stream webhooks from your microservices, and send them out to your users as well as receive webhooks from your providers and route them to the required services. With this your internal systems are never exposed to the public internet.

• Scalability: Convoy acts as a dedicated message queue for webhooks, and was designed to be horizontally scalable. It includes several components like the api server, workers, scheduler, and socket server which can be scaled independently to fit the need.

• Security: Convoy ships with several security features for webhooks, such as payload signing to ensure message integrity, bearer token authentication for authenticated webhook endpoints, and static ips for network environments with strict firewall rules.

• Fan Out: Convoy is able to route an event to multiple endpoints based on the event type or payload structure.

• Rate Limiting: While Convoy is able to ingest events at a massive rate, it throttles the delivery of these events to the endpoints at a configurable rate per endpoint.

• Retries & Batch Retries: Convoy supports two retry algorithms; constant time and exponential backoff with jitter. Where automatic retries are not sufficient, convoy provides batch retries for endpoints are consecutively failed to process retried events.

• Customer-Facing Dashboards: Convoy allows you to generate customer facing webhooks dashboard to embed into your applications using an iframe. On this dashboard, users can debug webhooks, retry events, add endpoints, and configure each endpoint's subscription.

• Endpoint Failure Notifications: When endpoints consecutively fails to process events, convoy disables the endpoint and sends out a notification. Two types of notifications are supported: Email and Slack Notifications.

Installation, Getting Started

To do this make sure you have Docker 20+ installed.

# Get the code
$ git clone https://github.com/frain-dev/convoy.git

# Go to the Convoy folder
$ cd convoy

# Start Services
$ docker compose -f configs/local/docker-compose.yml up

Convoy Cloud

Get started with our free Convoy Cloud version, request access: sales@getconvoy.io

Contributing

Thank you for your interest in contributing! Please refer to CONTRIBUTING.md for guidance. For contributions to the Convoy dashboard, please refer to the web/ui directory.

License

Mozilla Public License v2.0

Documentation

Index

• Constants
• Variables
• func GetVersion() string
• func GetVersionFromFS(fs embed.FS) string
• type CacheKey
  • func (c CacheKey) Get(suffix string) CacheKey
  • func (c CacheKey) String() string
• type HttpMethod
• type Plugin
• type QueueName
• type TaskName
  • func (t TaskName) SetPrefix(prefix string) TaskName

Constants

const (
	RATE_LIMIT          = 5000
	RATE_LIMIT_DURATION = "1m"
	HTTP_TIMEOUT        = "30s"
)

const (
	EventProcessor              TaskName = "EventProcessor"
	DeadLetterProcessor         TaskName = "DeadLetterProcessor"
	CreateEventProcessor        TaskName = "CreateEventProcessor"
	CreateDynamicEventProcessor TaskName = "CreateDynamicEventProcessor"
	MetaEventProcessor          TaskName = "MetaEventProcessor"
	NotificationProcessor       TaskName = "NotificationProcessor"
	// todo(raymond): remove this and it's references
	IndexDocument                TaskName = "index document"
	TokenizeSearch               TaskName = "tokenize search"
	TokenizeSearchForProject     TaskName = "tokenize search for project"
	DailyAnalytics               TaskName = "daily analytics"
	StreamCliEventsProcessor     TaskName = "StreamCliEventsProcessor"
	MonitorTwitterSources        TaskName = "monitor twitter sources"
	RetentionPolicies            TaskName = "retention_policies"
	EmailProcessor               TaskName = "EmailProcessor"
	ExpireSecretsProcessor       TaskName = "ExpireSecretsProcessor"
	DeleteArchivedTasksProcessor TaskName = "DeleteArchivedTasksProcessor"

	EndpointsCacheKey          CacheKey = "endpoints"
	OrganisationsCacheKey      CacheKey = "organisations"
	OrganisationMemberCacheKey CacheKey = "organisation_members"
	ProjectsCacheKey           CacheKey = "projects"
	TokenCacheKey              CacheKey = "tokens"
	SourceCacheKey             CacheKey = "sources"
	IdempotencyCacheKey        CacheKey = "dedup"
)

const (
	DefaultOnPremDir = "/var/convoy/export"
	TmpExportDir     = "/tmp/convoy/export"
)

Exports dir

const (
	Concurrency = 100
)

const (
	EventDeliveryIDLength = 12
)

Variables

var F embed.FS

var MigrationFiles embed.FS

Functions

func GetVersion

func GetVersion() string

TODO(subomi): This needs to be refactored for everywhere we depend on this code.

func GetVersionFromFS

func GetVersionFromFS(fs embed.FS) string

Types

type CacheKey

type CacheKey string

func (CacheKey) Get

func (c CacheKey) Get(suffix string) CacheKey

func (CacheKey) String

func (c CacheKey) String() string

type HttpMethod

type HttpMethod string

const (
	HttpPost HttpMethod = "POST"
	HttpGet  HttpMethod = "GET"
)

type Plugin

type Plugin interface {
	Apply(http.ResponseWriter, *http.Request) error
	Name() string
	IsEnabled() bool
}

type QueueName

type QueueName string

const (
	EventQueue       QueueName = "EventQueue"
	CreateEventQueue QueueName = "CreateEventQueue"
	MetaEventQueue   QueueName = "MetaEventQueue"
	SearchIndexQueue QueueName = "SearchIndexQueue"
	StreamQueue      QueueName = "StreamQueue"
	ScheduleQueue    QueueName = "ScheduleQueue"
	DefaultQueue     QueueName = "DefaultQueue"
)

queues

type TaskName

type TaskName string

func (TaskName) SetPrefix

func (t TaskName) SetPrefix(prefix string) TaskName