bmsclient

package module
v0.0.0-...-26046d6 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 23, 2023 License: MIT Imports: 16 Imported by: 1

README

BMS gRPC Client GoDoc Build Status

This repository is part of the Botnet Monitoring System (BMS), you can read more about BMS here.

This repository contains an ergonomic client implementation of the BMS gRPC API for Go. Crawler and sensor implementations can use this Go package to send their measurement data to a BMS server.

The client is centered around the concept of sessions. To send data to BMS, a client must request a session from the server. The client starts a session by providing the server with its monitor ID, an auth token and the botnet ID it is configured to monitor (and optionally more configuration). The server checks if the requested session is valid (e.g. if the botnet exists in the database) and returns a session token.

While a session is in progress, all settings are fixed. If the client wants to change its configuration (e.g. switching the botnet it monitors), it has to end the current session and start a new one.

Remarks
  • The client currently uses an unencrypted gRPC connection. For now, you can use a VPN if concerned.
  • The client currently does not validate any messages it gets from the server (which is alright, since we trust the server to adhere to the specification).
  • The client should be fully thread-safe, but we haven't tested it thoroughly.

Installation

$ go get github.com/botnet-monitoring/grpc-client

Quick Start

// Creates a simple client (which is a client that automatically starts a session)
client, err := bmsclient.NewSimpleClient(
    "your-bms-server.local:8083",
    "some-monitor-id",
    "AAAA/some+example+token/AAAAAAAAAAAAAAAAAAA=",
    "some-botnet-id",
)
if err != nil {
    panic(err)
}

// You can send data synchronously in batches
batch := bmsclient.BotReplyBatch{
    &bmsclient.BotReply{
        Timestamp: time.Now(),
        IP:        net.ParseIP("192.0.2.1"),
        Port:      10001,
    },
    &bmsclient.BotReply{
        Timestamp: time.Now(),
        IP:        net.ParseIP("192.0.2.2"),
        Port:      10002,
    },
}
err = client.SendBotReplyBatch(&batch) // this blocks
if err != nil {
    panic(err)
}

// Or add it to the client as you gather it (concurrency-safe) and flush it at once
client.AddBotReply(time.Now(), net.ParseIP("192.0.2.3"), 10003, "", nil)
client.AddBotReply(time.Now(), net.ParseIP("192.0.2.4"), 10004, "", nil)
err = client.Flush() // this blocks
if err != nil {
    panic(err)
}

// If you're done, it'd be nice if you end the session
err = client.End()
if err != nil {
    panic(err)
}

You can find a more complex example in example_advanced_test.go.

Documentation

Overview

Package bmsclient implements an easy-to-use client for the BMS gRPC API. With it, you can send measurement data of botnets to a BMS server. See for example the BMS basecrawler which implements a stateful P2P botnet crawler using this package.

If you don't know what BMS is, you can read more here.

Concepts

The client is centered around the concept of sessions. To send data to BMS, a client must request a session from the server. The client starts a session by providing the server with its monitor ID, an auth token and the botnet ID it is configured to monitor (and optionally more configuration). The server checks if the requested session is valid (e.g. if the botnet exists in the database) and returns a session token.

While a session is in progress, all settings are fixed. If the client wants to change its configuration (e.g. switching the botnet it monitors), it has to end the current session and start a new one.

For more on sessions (e.g. why they exist) and other concepts around BMS (e.g. what campaigns are, why they exist and how to use them), you can consult the explanations here.

Modes of Usage

This package provides two modes of usage: SimpleClient is a client that internally opens a session and implements basic methods for sending data to BMS. For most use cases this should be alright and it's definitely easier to use.

However, if you have a more advanced use case, you can create also a Client instance and open multiple Session instances on it. In addition, the Session struct implements more advanced methods to send data (e.g. sending with context.Context).

Add vs. Send

This client offers two different ways to send data to BMS. The methods prefixed with Add take a single bot reply, edge or failed try. You can add the measurement data as it occurs (also concurrently) and flush it after a fixed time or a fixed number of added measurements.

The methods prefixed with Send take a batch that will be sent synchronously. After the batch is sent, the client will by default wait for the server to acknowledge it. This means that if a send method returned without error, it is guaranteed that the server received the batch successfully. The flush methods internally use the send methods.

Example (Advanced)

An advanced example that creates the session separately from the client. 

package main

import (
	"net"
	"time"

	bmsclient "github.com/botnet-monitoring/grpc-client"
)

func main() {
	// First, create a client
	client, err := bmsclient.NewClient(
		"your-bms-server.local:8083",
		"some-monitor-id",
		"AAAA/some+example+token/AAAAAAAAAAAAAAAAAAA=",
	)
	if err != nil {
		panic(err)
	}

	// Then, add a new session to the client
	session, err := client.NewSession("some-botnet-id")
	if err != nil {
		panic(nil)
	}

	// Now you can send data on the created session
	// For example synchronously in batches
	batch := bmsclient.BotReplyBatch{
		&bmsclient.BotReply{
			Timestamp: time.Now(),
			IP:        net.ParseIP("192.0.2.1"),
			Port:      10001,
		},
		&bmsclient.BotReply{
			Timestamp: time.Now(),
			IP:        net.ParseIP("192.0.2.2"),
			Port:      10002,
		},
	}
	_, err = session.SendBotReplyBatch(&batch) // this blocks
	if err != nil {
		panic(err)
	}

	// Or add it to the client as you gather it and flush it at once
	session.AddBotReply(time.Now(), net.ParseIP("192.0.2.3"), 10003, "", nil)
	session.AddBotReply(time.Now(), net.ParseIP("192.0.2.4"), 10004, "", nil)
	err = session.Flush() // this blocks
	if err != nil {
		panic(err)
	}

	// If you're done, you can end the session
	err = session.End()
	if err != nil {
		panic(err)
	}
}

Output:
Example (Simple)

A simple example that uses the simple client. The simple client internally creates a session and thereby combines a client with exactly one session for easier usage. 

package main

import (
	"net"
	"time"

	bmsclient "github.com/botnet-monitoring/grpc-client"
)

func main() {
	// Create a simple client (which is a client that automatically starts a session)
	client, err := bmsclient.NewSimpleClient(
		"your-bms-server.local:8083",
		"some-monitor-id",
		"AAAA/some+example+token/AAAAAAAAAAAAAAAAAAA=",
		"some-botnet-id",
	)
	if err != nil {
		panic(err)
	}

	// You can send data in batches and send it synchronously
	batch := bmsclient.BotReplyBatch{
		&bmsclient.BotReply{
			Timestamp: time.Now(),
			IP:        net.ParseIP("192.0.2.1"),
			Port:      10001,
		},
		&bmsclient.BotReply{
			Timestamp: time.Now(),
			IP:        net.ParseIP("192.0.2.2"),
			Port:      10002,
		},
	}
	_, err = client.SendBotReplyBatch(&batch) // this blocks
	if err != nil {
		panic(err)
	}

	// Or add it to the client as you gather it and flush it at once
	client.AddBotReply(time.Now(), net.ParseIP("192.0.2.3"), 10003, "", nil)
	client.AddBotReply(time.Now(), net.ParseIP("192.0.2.4"), 10004, "", nil)
	err = client.Flush() // this blocks
	if err != nil {
		panic(err)
	}

	// If you're done, you can end the session
	err = client.End()
	if err != nil {
		panic(err)
	}
}

Output:

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type BotReply 

type BotReply struct {
	// The timestamp the bot was observed
	Timestamp time.Time

	// An ID which the bot can be identified with (optional, as not available for all botnets)
	// Set an empty string when not available
	BotID string

	// The IP address of the bot
	IP net.IP

	// The port the bot responded on
	Port uint16

	// Potential other data that was observed
	// Set an empty struct or nil when not available
	//
	// Please also note that this goes through [encoding/json.Marshal], so fields that should go into the database should be exported
	// This also means you can use field tags like `json:"some_field_name"`
	OtherData interface{}
}

A BotReply struct represents the data collected when observing a bot.

type BotReplyBatch 

type BotReplyBatch []*BotReply

BotReplyBatch can contain multiple/many instances of BotReply.

Since it's a slice of pointers, it should be alright to make this quite big (a few hundred still works fine).

type Client 

type Client struct {
	// contains filtered or unexported fields
}

The Client struct represents a BMS client. A Client can have multiple sessions that can send data to BMS.

To create a Client, use NewClient.

func NewClient 

func NewClient(bmsServer string, monitorID string, authToken string, options ...ClientOption) (*Client, error)

NewClient creates a new BMS client based on the given parameters.

It will dial the gRPC connection (to check if a compatible server is there), but not start to authenticate until you start a session with Client.NewSession. Dialing the gRPC connection may block up to 20 seconds (the default connect timeout).

In theory you could pass additional configuration via options, but there are currently no implementations of ClientOption.

The passed auth token must be base64-encoded and exactly 32 bytes long.

Example 
package main

import (
	bmsclient "github.com/botnet-monitoring/grpc-client"
)

func main() {
	client, err := bmsclient.NewClient("your-bms-server.local:8083", "some-monitor-id", "AAAA/some+example+token/AAAAAAAAAAAAAAAAAAA=")
	if err != nil {
		panic(err)
	}

	// Do something with the client
	_ = client
}

Output:

func (*Client) NewSession 

func (c *Client) NewSession(botnetID string, options ...SessionOption) (*Session, error)

NewSession creates a new session for the BMS client it was called on.

Calling NewSession will register the client's session with the BMS server. The client tells the server its configuration, the server checks whether the configuration is valid (and conforms with the campaign if the client has set one), then the server will respond with a session token and a session timeout.

While a session is in progress, all settings are fixed. If the client wants to change its configuration (e.g. switching the botnet it monitors), it has to end the current session and start a new one. You can end a session with Session.End (and similar methods).

While you don't have to do anything with the session token (will be attached internally to all batches), you have to ensure that the session stays active by not letting the session timeout elapse. You can get the session timeout via Session.GetSessionTimeout.

Example 
package main

import (
	bmsclient "github.com/botnet-monitoring/grpc-client"
)

func main() {
	client, err := bmsclient.NewClient("your-bms-server.local:8083", "some-monitor-id", "AAAA/some+example+token/AAAAAAAAAAAAAAAAAAA=")
	if err != nil {
		panic(err)
	}

	session, err := client.NewSession("some-botnet-id")
	if err != nil {
		panic(err)
	}

	// Do something with the session
	_ = session
}

Output:
Example (WithOptions) 
package main

import (
	"net"

	bmsclient "github.com/botnet-monitoring/grpc-client"
)

func main() {
	client, err := bmsclient.NewClient(
		"your-bms-server.local:8083",
		"some-monitor-id",
		"AAAA/some+example+token/AAAAAAAAAAAAAAAAAAA=",
	)
	if err != nil {
		panic(err)
	}

	otherData := struct {
		Some string `json:"some_field"`
	}{
		Some: "value",
	}

	session, err := client.NewSession(
		"some-botnet-id",
		bmsclient.WithCampaign("some-campaign"),
		bmsclient.WithFixedFrequency(30),
		bmsclient.WithAdditionalConfig(otherData),
		bmsclient.WithIP(net.ParseIP("198.51.100.1")),
		bmsclient.WithPort(10101),
	)
	if err != nil {
		panic(err)
	}

	// Do something with the session
	_ = session
}

Output:

type ClientOption 

type ClientOption func(*Client)

Functions which implement ClientOption can be passed to NewClient as additional options.

Currently there are no implementations of it.

type DisconnectReason 

type DisconnectReason bmsapi.DisconnectReason

Possible reasons a client wants to end the session with. 

const (
	// The client wants to specify no reason to end the session.
	DisconnectReasonUnspecified DisconnectReason = 0
	// The client has done its purpose and thereby wants to end the session.
	DisconnectReasonFinished DisconnectReason = 1
	// The clients wants to end the session in order to reconnect soon.
	DisconnectReasonBeRightBack DisconnectReason = 2
	// The clients wants to end the session in order to reconnect soon with a new configuration.
	DisconnectReasonBeRightBackWithNewConfig DisconnectReason = 3
	// The client had some error and thereby wants to the the session.
	DisconnectReasonClientError DisconnectReason = 4
	// The client wants to end the session because of some other reason.
	DisconnectReasonOther DisconnectReason = 5
)

type Edge 

type Edge struct {
	// The timestamp the edge was observed
	Timestamp time.Time

	// An ID which the source bot can be identified with (optional, as not available for all botnets)
	// Set an empty string when not available
	SrcBotID string

	// The IP address of the source bot
	SrcIP net.IP

	// The port the source bot responded on
	SrcPort uint16

	// An ID which the destination bot can be identified with (optional, as not available for all botnets)
	// Set an empty string when not available
	DstBotID string

	// The IP address of the destination bot
	DstIP net.IP

	// The port the destination bot is expected to be reached
	DstPort uint16
}

An Edge struct represents a connection from one bot to another. It is mostly relevant for P2P botnets where bots maintain peer lists of other bots.

Most of the time observing a bot reply also coincides with observing one (or multiple) edges. For example when requesting peers from a bot in a P2P botnet, the bot responds with multiple other bots. These bots will be the destination bots of the edge, while the bots which responded will be the source bot. Please also note that the returned bots are not necessarily online and actually don't even need to be meaningful.

type EdgeBatch 

type EdgeBatch []*Edge

EdgeBatch can contain multiple/many instances of Edge.

Since it's a slice of pointers, it should be alright to make this quite big (a few hundred still works fine).

type FailedTry 

type FailedTry struct {
	// The timestamp the bot observation failed
	Timestamp time.Time

	// An ID which the bot can be identified with (optional, as not available for all botnets)
	// Set an empty string when not available
	BotID string

	// The IP address that was tried to reach
	IP net.IP

	// The port that was tried to reach
	Port uint16

	// The reason the observation failed (e.g. "timeout", "connection refused")
	Reason string

	// Potential other data that was observed
	// Set an empty struct or nil when not available
	//
	// Please also note that this goes through [encoding/json.Marshal], so fields that should go into the database should be exported
	// This also means you can use field tags like `json:"some_field_name"`
	OtherData interface{}
}

A FailedTry struct represents the data collected when not observing a bot (but it should have worked). E.g. a crawler tried to reach a bot, but the bot did not respond before the timeout was reached.

Note that not all monitors can measure this. For example a crawler that uses a stateless loop for requesting peers and a stateless loop for receiving responses (e.g. by listening on a fixed port) cannot know with certainty whether a response relates to the last request or whether it came late and was meant for an earlier request.

type FailedTryBatch 

type FailedTryBatch []*FailedTry

FailedTryBatch can contain multiple/many instances of FailedTry.

Since it's a slice of pointers, it should be alright to make this quite big (a few hundred still works fine).

type GRPCDataIngestion 

type GRPCDataIngestion interface {
	SendBotReplyBatch(botReplyBatch *BotReplyBatch) (uint32, error)
	SendBotReplyBatchWithContext(ctx context.Context, botReplyBatch *BotReplyBatch) (uint32, error)

	SendFailedTryBatch(failedTryBatch *FailedTryBatch) (uint32, error)
	SendFailedTryBatchWithContext(ctx context.Context, failedTryBatch *FailedTryBatch) (uint32, error)

	SendEdgeBatch(edgeBatch *EdgeBatch) (uint32, error)
	SendEdgeBatchWithContext(ctx context.Context, edgeBatch *EdgeBatch) (uint32, error)

	AddBotReply(timestamp time.Time, ip net.IP, port uint16, botID string, otherData interface{})
	AddFailedTry(timestamp time.Time, ip net.IP, port uint16, botID string, reason string, otherData interface{})
	AddEdge(timestamp time.Time, srcIP net.IP, srcPort uint16, srcBotID string, dstIP net.IP, dstPort uint16, dstBotID string)
	Flush() error
	FlushBotReplies() error
	FlushFailedTries() error
	FlushEdges() error
	FlushWithContext(ctx context.Context) error
	FlushBotRepliesWithContext(ctx context.Context) error
	FlushFailedTriesWithContext(ctx context.Context) error
	FlushEdgesWithContext(ctx context.Context) error
}

GRPCDataIngestion is an interface that Session implements. It contains a collection of methods to send data to BMS.

This interface can also be implemented by external packages, so that crawlers can use the same method calls. E.g. useful for implementing local storage in a crawler to aid when the BMS server goes down.

type Session 

type Session struct {
	// contains filtered or unexported fields
}

The Session struct represents a session of a BMS client.

To create a Session, use Client.NewSession.

func (*Session) AddBotReply 

func (s *Session) AddBotReply(timestamp time.Time, ip net.IP, port uint16, botID string, otherData interface{})

AddBotReply adds a bot reply to the session's internal storage that can be flushed with Session.Flush or Session.FlushBotReplies.

An empty string passed as bot ID will be interpreted as a non-existing ID. An empty struct or nil passed as additional data will be interpreted as no additional data.

Please also note that the additional data go through encoding/json.Marshal, so fields that should go into the database should be exported. This also means you can use field tags like `json:"some_field_name"`.

func (*Session) AddBotReplyStruct 

func (s *Session) AddBotReplyStruct(botReply *BotReply)

AddBotReplyStruct is like Session.AddBotReply but takes the bot reply as a struct.

func (*Session) AddEdge 

func (s *Session) AddEdge(timestamp time.Time, srcIP net.IP, srcPort uint16, srcBotID string, dstIP net.IP, dstPort uint16, dstBotID string)

AddEdge adds an edge to the session's internal storage that can be flushed with Session.Flush or Session.FlushEdges.

An empty string passed as bot ID will be interpreted as a non-existing ID.

func (*Session) AddEdgeStruct 

func (s *Session) AddEdgeStruct(edge *Edge)

AddEdgeStruct is like Session.AddEdge but takes the edge as a struct.

func (*Session) AddFailedTry 

func (s *Session) AddFailedTry(timestamp time.Time, ip net.IP, port uint16, botID string, reason string, otherData interface{})

AddFailedTry adds a failed try to the session's internal storage that can be flushed with Session.Flush or Session.FlushFailedTries.

An empty string passed as bot ID / reason will be interpreted as a non-existing ID / reason. An empty struct or nil passed as additional data will be interpreted as no additional data.

Please also note that the additional data go through encoding/json.Marshal, so fields that should go into the database should be exported. This also means you can use field tags like `json:"some_field_name"`.

func (*Session) AddFailedTryStruct 

func (s *Session) AddFailedTryStruct(failedTry *FailedTry)

AddFailedTryStruct is like Session.AddFailedTry but takes the failed try as a struct.

func (*Session) End 

func (s *Session) End() error

End ends a session.

It will flush all unset batches (e.g. bot replies added via Session.AddBotReply) and then communicate to the server that we're not intending to send any further data. Additionally, it will wait for all sent batches to be acknowledged. While flushing and waiting, this method blocks (use Session.EndWithContext if you want to set a timeout).

func (*Session) EndWithContext 

func (s *Session) EndWithContext(ctx context.Context) error

EndWithContext is like Session.End but takes a context which for example can be used to cancel it after some timeout.

func (*Session) EndWithContextAndReason 

func (s *Session) EndWithContextAndReason(ctx context.Context, reason DisconnectReason) error

EndWithContextAndReason is a mix out of Session.EndWithContext and Session.EndWithReason.

func (*Session) EndWithReason 

func (s *Session) EndWithReason(reason DisconnectReason) error

EndWithReason is like Session.End but takes a reason why the session should end.

func (*Session) Flush 

func (s *Session) Flush() error

Flush flushes the bot replies, edges and failed tries added to the internal storage via the Add* methods.

It uses the also exported Send* methods for this, so it will block while sending. If you don't want it to wait for the server ack, you can use WithIgnoreServerAcks. If you want to cancel it while it waits for the ack (which might take long), you can use Session.FlushWithContext.

Flush first flushes bot replies, then edges, then failed tries. It will stop on the first error it encounters, which might result in unflushed failed tries, although it maybe would have been possible to flush them. As sending errors very likely are due to the underlying connections having problems, it's also likely that sending anything further would have failed either way.

func (*Session) FlushBotReplies 

func (s *Session) FlushBotReplies() error

FlushBotReplies is like Session.Flush but only flushes bot replies.

func (*Session) FlushBotRepliesWithContext 

func (s *Session) FlushBotRepliesWithContext(ctx context.Context) error

FlushBotRepliesWithContext is like Session.FlushBotReplies but takes a context which for example can be used to cancel it after some timeout.

func (*Session) FlushEdges 

func (s *Session) FlushEdges() error

FlushEdges is like Session.Flush but only flushes edges.

func (*Session) FlushEdgesWithContext 

func (s *Session) FlushEdgesWithContext(ctx context.Context) error

FlushEdgesWithContext is like Session.FlushEdges but takes a context which for example can be used to cancel it after some timeout.

func (*Session) FlushFailedTries 

func (s *Session) FlushFailedTries() error

FlushFailedTries is like Session.Flush but only flushes failed tries.

func (*Session) FlushFailedTriesWithContext 

func (s *Session) FlushFailedTriesWithContext(ctx context.Context) error

FlushFailedTriesWithContext is like Session.FlushFailedTries but takes a context which for example can be used to cancel it after some timeout.

func (*Session) FlushWithContext 

func (s *Session) FlushWithContext(ctx context.Context) error

FlushWithContext is like Session.Flush but takes a context which for example can be used to cancel it after some timeout.

func (*Session) GetLastBotReplyTime 

func (s *Session) GetLastBotReplyTime() time.Time

GetLastBotReplyTime returns the time the last bot reply batch was successfully received by the server (if any).

This method can be used to implement local caching when the server goes offline temporarily.

func (*Session) GetLastEdgeTime 

func (s *Session) GetLastEdgeTime() time.Time

GetLastEdgeTime returns the time the last edge batch was successfully received by the server (if any).

This method can be used to implement local caching when the server goes offline temporarily.

func (*Session) GetLastFailedTryTime 

func (s *Session) GetLastFailedTryTime() time.Time

GetLastFailedTryTime returns the time the last failed try batch was successfully received by the server (if any).

This method can be used to implement local caching when the server goes offline temporarily.

func (*Session) GetNumberOfUnsentBotReplies 

func (s *Session) GetNumberOfUnsentBotReplies() int

GetNumberOfUnsentBotReplies returns the number of unsent bot replies added via Session.AddBotReply and Session.AddBotReplyStruct.

func (*Session) GetNumberOfUnsentEdges 

func (s *Session) GetNumberOfUnsentEdges() int

GetNumberOfUnsentEdges returns the number of unsent edges added via Session.AddEdge and Session.AddEdgeStruct.

func (*Session) GetNumberOfUnsentFailedTries 

func (s *Session) GetNumberOfUnsentFailedTries() int

GetNumberOfUnsentFailedTries returns the number of unsent failed tries added via Session.AddFailedTry and Session.AddFailedTryStruct.

func (*Session) GetSessionTimeout 

func (s *Session) GetSessionTimeout() time.Duration

GetSessionTimeout returns the timeout that the server set for the session.

After inactivity longer than the timeout the server may close the session. This doesn't mean that the server closes the session right after the timeout elapses, but it's guaranteed to not be closed before that. Sending a batch (even an empty one) every time right before the timeout elapses is enough to renew the session.

func (*Session) GetSessionToken 

func (s *Session) GetSessionToken() [4]byte

GetSessionToken returns the session token that the server set for the session.

It should not be needed to do anything with the session token, as the client internally attaches it to its messages to the server.

func (*Session) SendBotReplyBatch 

func (s *Session) SendBotReplyBatch(botReplyBatch *BotReplyBatch) (uint32, error)

SendBotReplyBatch sends the given bot reply batch synchronously.

By default (see WithIgnoreServerAcks) it will wait for the server to acknowledge sent batches. While it waits, the method will block. In case of a server error, it might even wait forever. Therefore you likely want to use Session.SendBotReplyBatchWithContext to set a timeout.

Example 
package main

import (
	"time"

	bmsclient "github.com/botnet-monitoring/grpc-client"
)

func main() {
	// Create a client
	client, err := bmsclient.NewClient("your-bms-server.local:8083", "some-monitor-id", "AAAA/some+example+token/AAAAAAAAAAAAAAAAAAA=")
	if err != nil {
		panic(err)
	}

	// Create a session
	session, err := client.NewSession("some-botnet-id")
	if err != nil {
		panic(err)
	}

	// Create a batch
	batch := &bmsclient.BotReplyBatch{
		&bmsclient.BotReply{
			Timestamp: time.Now(),
		},
	}

	// Send the batch
	_, err = session.SendBotReplyBatch(batch)
	if err != nil {
		panic(err)
	}

	// End the session because we're done
	err = session.End()
	if err != nil {
		panic(err)
	}
}

Output:

func (*Session) SendBotReplyBatchWithContext 

func (s *Session) SendBotReplyBatchWithContext(ctx context.Context, botReplyBatch *BotReplyBatch) (uint32, error)

SendBotReplyBatchWithContext is like Session.SendBotReplyBatch but takes an additional context (e.g. to be able to cancel it when executing as a Go routine).

Example 
package main

import (
	"context"
	"net"
	"sync"
	"time"

	bmsclient "github.com/botnet-monitoring/grpc-client"
)

func main() {
	// Create a client
	client, err := bmsclient.NewClient("your-bms-server.local:8083", "some-monitor-id", "AAAA/some+example+token/AAAAAAAAAAAAAAAAAAA=")
	if err != nil {
		panic(err)
	}

	// Create a session
	session, err := client.NewSession("some-botnet-id")
	if err != nil {
		panic(err)
	}

	// Create a context that times out after 500ms
	ctx, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond)
	defer cancel()

	var wg sync.WaitGroup

	for i := 0; i < 5; i++ {
		// Add the Go routine below to the wait group
		wg.Add(1)

		// Create a batch to send
		batch := &bmsclient.BotReplyBatch{
			&bmsclient.BotReply{
				Timestamp: time.Now(),
				IP:        net.ParseIP("192.0.2.1"),
				Port:      uint16(i), // Use the index variable here to make sure the bot reply is in fact different
			},
		}

		// Send the batch in the background
		go func() {
			// Mark this Go routine as finished when it returns
			defer wg.Done()

			// Send the batch
			_, err = session.SendBotReplyBatchWithContext(ctx, batch)
			if err != nil {
				panic(err)
			}
		}()
	}

	// Wait for all Go routines to finish
	wg.Wait()

	// End the session because we're done
	err = session.End()
	if err != nil {
		panic(err)
	}

	// Note that we use a wait group here, although End by default is configured to wait for all sent batches to be acknowledged.
	// This is necessary because (depending on how the Go routine is scheduled) it can happen that End locks the sending stream before Send began sending.
	// If using this example without the wait group, it can happen that not all batches will be sent (and it depends on your use case if that's ok for you).
}

Output:

func (*Session) SendEdgeBatch 

func (s *Session) SendEdgeBatch(edgeBatch *EdgeBatch) (uint32, error)

SendEdgeBatch sends the given edge batch synchronously.

By default (see WithIgnoreServerAcks) it will wait for the server to acknowledge sent batches. While it waits, the method will block. In case of a server error, it might even wait forever. Therefore you likely want to use Session.SendEdgeBatchWithContext to set a timeout.

func (*Session) SendEdgeBatchWithContext 

func (s *Session) SendEdgeBatchWithContext(ctx context.Context, edgeBatch *EdgeBatch) (uint32, error)

SendEdgeBatchWithContext is like Session.SendEdgeBatch but takes an additional context (e.g. to be able to cancel it when executing as a Go routine).

func (*Session) SendFailedTryBatch 

func (s *Session) SendFailedTryBatch(failedTryBatch *FailedTryBatch) (uint32, error)

SendFailedTryBatch sends the given failed try batch synchronously.

By default (see WithIgnoreServerAcks) it will wait for the server to acknowledge sent batches. While it waits, the method will block. In case of a server error, it might even wait forever. Therefore you likely want to use Session.SendFailedTryBatchWithContext to set a timeout.

func (*Session) SendFailedTryBatchWithContext 

func (s *Session) SendFailedTryBatchWithContext(ctx context.Context, failedTryBatch *FailedTryBatch) (uint32, error)

SendFailedTryBatchWithContext is like Session.SendFailedTryBatch but takes an additional context (e.g. to be able to cancel it when executing as a Go routine).

type SessionInfo 

type SessionInfo interface {
	GetSessionToken() [4]byte
	GetSessionTimeout() time.Duration
	GetLastBotReplyTime() time.Time
	GetLastFailedTryTime() time.Time
	GetLastEdgeTime() time.Time
}

SessionInfo is an interface that is implemented by Session and SimpleClient.

It makes it easier for you to switch from SimpleClient to Client and Client.NewSession.

type SessionOption 

type SessionOption func(*Session)

Functions which implement SessionOption can be passed to Client.NewSession as additional options.

func WithAdditionalConfig 

func WithAdditionalConfig(config interface{}) SessionOption

WithAdditionalConfig can be used to set arbitrary additional config for the session.

Please also note that this goes through encoding/json.Marshal, so fields that should go into the database should be exported. This also means you can use field tags like `json:"some_field_name"`.

func WithBotReplyBatchResponseFailureHook 

func WithBotReplyBatchResponseFailureHook(hook hooks.BotReplyBatchResponseFailureHook) SessionOption

WithBotReplyBatchResponseFailureHook can be used to pass a function that will be executed when a server error on the bot reply batch stream is received.

Please note that the stream will be closed after the first error. So the hook will be executed only once.

func WithBotReplyBatchResponseSuccessHook 

func WithBotReplyBatchResponseSuccessHook(hook hooks.BotReplyBatchResponseSuccessHook) SessionOption

WithBotReplyBatchResponseSuccessHook can be used to pass a function that will be executed when a server ack for a bot reply batch is received.

func WithCampaign 

func WithCampaign(campaignID string) SessionOption

WithCampaign can be used to set a campaign for the session.

Note that when being part of a campaign, the server will check whether the session configuration conforms with the campaign configuration (e.g. monitoring the correct botnet, using the correct frequency).

func WithEdgeBatchResponseFailureHook 

func WithEdgeBatchResponseFailureHook(hook hooks.EdgeBatchResponseFailureHook) SessionOption

WithEdgeBatchResponseFailureHook can be used to pass a function that will be executed when a server error on the edge batch stream is received.

Please note that the stream will be closed after the first error. So the hook will be executed only once.

func WithEdgeBatchResponseSuccessHook 

func WithEdgeBatchResponseSuccessHook(hook hooks.EdgeBatchResponseSuccessHook) SessionOption

WithEdgeBatchResponseSuccessHook can be used to pass a function that will be executed when a server ack for an edge batch is received.

func WithFailedTryBatchResponseFailureHook 

func WithFailedTryBatchResponseFailureHook(hook hooks.FailedTryBatchResponseFailureHook) SessionOption

WithFailedTryBatchResponseFailureHook can be used to pass a function that will be executed when a server error on the failed try batch stream is received.

Please note that the stream will be closed after the first error. So the hook will be executed only once.

func WithFailedTryBatchResponseSuccessHook 

func WithFailedTryBatchResponseSuccessHook(hook hooks.FailedTryBatchResponseSuccessHook) SessionOption

WithFailedTryBatchResponseSuccessHook can be used to pass a function that will be executed when a server ack for failed try batch is received.

func WithFixedFrequency 

func WithFixedFrequency(frequency uint32) SessionOption

WithFixedFrequency can be used to set the frequency for the session.

This is mostly useful for crawlers (as sensors don't actively contact bots). The frequency is passed as seconds that the crawler waits between contacting (or trying to contact) the same bot again.

func WithIP 

func WithIP(publicIP net.IP) SessionOption

WithIP can be used to set the public IP address of the monitor for the session.

This is for example useful for sensors that just listen for incoming traffic (as the IP address might influence the measurements or can be used to identify the sensor in the peerlists of other bots).

func WithIgnoreServerAcks 

func WithIgnoreServerAcks(ignoreServerAcks bool) SessionOption

WithIgnoreServerAcks can be used to ignore the acknowledgements of sent batches that the server received.

By default, sending batches will wait for the server to acknowledge the reception of a batch. While the send method waits for the server to acknowledge, it will block. Please note that this might take long (or in case of an error even forever), so it is recommended to pass a context with timeout (e.g. by using Session.SendBotReplyBatchWithContext).

If you ignore server acks, it might happen that a batch isn't sent before the program exits. This is because the underlying google.golang.org/grpc.ClientStream.SendMsg returns as soon as the message is scheduled to sent (but not actually sent yet). See their documentation for more information.

func WithPort 

func WithPort(monitorPort uint16) SessionOption

WithPort can be used to set the port that a monitor listens on for incoming bot traffic.

This is for example useful for monitors that listen for incoming traffic on a fixed port. It can also be used to distinguish between several monitors that run on the same host (and therefore share the same IP address) but use different fixed ports.

type SimpleClient 

type SimpleClient struct {
	// contains filtered or unexported fields
}

SimpleClient is a simple to use BMS client.

It creates a client internally and initiates a session. If you need multiple sessions in the same client, use the Client.

To create a SimpleClient, use NewSimpleClient.

func NewSimpleClient 

func NewSimpleClient(bmsServer string, monitorID string, authToken string, botnetID string, options ...SimpleClientOption) (*SimpleClient, error)

NewSimpleClient creates a simple to use BMS client based on the given parameters.

The simple client does not distinguish between client and sessions and just creates a session for you internally. If you're done sending, you can call SimpleClient.End (and similar methods) to make sure all data is sent.

While you don't have to do anything with the session token (will be attached internally to all batches), you have to ensure that the session stays active by not letting the session timeout elapse. You can get the session timeout via SimpleClient.GetSessionTimeout.

The passed auth token must be base64-encoded and exactly 32 bytes long.

Example 
package main

import (
	bmsclient "github.com/botnet-monitoring/grpc-client"
)

func main() {
	client, err := bmsclient.NewSimpleClient("your-bms-server.local:8083", "some-monitor-id", "AAAA/some+example+token/AAAAAAAAAAAAAAAAAAA=", "some-botnet-id")
	if err != nil {
		panic(err)
	}

	// Do something with the client
	_ = client
}

Output:

func (*SimpleClient) AddBotReply 

func (s *SimpleClient) AddBotReply(timestamp time.Time, ip net.IP, port uint16, botID string, otherData interface{})

AddBotReply is a wrapper around Session.AddBotReply.

func (*SimpleClient) AddEdge 

func (s *SimpleClient) AddEdge(timestamp time.Time, srcIP net.IP, srcPort uint16, srcBotID string, dstIP net.IP, dstPort uint16, dstBotID string)

AddEdge is a wrapper around Session.AddEdge.

func (*SimpleClient) AddFailedTry 

func (s *SimpleClient) AddFailedTry(timestamp time.Time, ip net.IP, port uint16, botID string, reason string, otherData interface{})

AddFailedTry is a wrapper around Session.AddFailedTry.

func (*SimpleClient) End 

func (s *SimpleClient) End() error

End is a wrapper around Session.End.

func (*SimpleClient) EndWithReason 

func (s *SimpleClient) EndWithReason(reason DisconnectReason) error

EndWithReason is a wrapper around Session.EndWithReason.

func (*SimpleClient) Flush 

func (s *SimpleClient) Flush() error

Flush is a wrapper around Session.Flush.

func (*SimpleClient) FlushBotReplies 

func (s *SimpleClient) FlushBotReplies() error

FlushBotReplies is a wrapper around Session.FlushBotReplies.

func (*SimpleClient) FlushEdges 

func (s *SimpleClient) FlushEdges() error

FlushEdges is a wrapper around Session.FlushEdges.

func (*SimpleClient) FlushFailedTries 

func (s *SimpleClient) FlushFailedTries() error

FlushFailedTries is a wrapper around Session.FlushFailedTries.

func (*SimpleClient) GetLastBotReplyTime 

func (s *SimpleClient) GetLastBotReplyTime() time.Time

GetLastBotReplyTime is a wrapper around Session.GetLastBotReplyTime.

func (*SimpleClient) GetLastEdgeTime 

func (s *SimpleClient) GetLastEdgeTime() time.Time

GetLastEdgeTime is a wrapper around Session.GetLastEdgeTime.

func (*SimpleClient) GetLastFailedTryTime 

func (s *SimpleClient) GetLastFailedTryTime() time.Time

GetLastFailedTryTime is a wrapper around Session.GetLastFailedTryTime.

func (*SimpleClient) GetSessionTimeout 

func (s *SimpleClient) GetSessionTimeout() time.Duration

GetSessionTimeout is a wrapper around Session.GetSessionTimeout.

func (*SimpleClient) GetSessionToken 

func (s *SimpleClient) GetSessionToken() [4]byte

GetSessionToken is a wrapper around Session.GetSessionToken.

func (*SimpleClient) SendBotReplyBatch 

func (s *SimpleClient) SendBotReplyBatch(botReplyBatch *BotReplyBatch) (uint32, error)

SendBotReplyBatch is a wrapper around Session.SendBotReplyBatch.

func (*SimpleClient) SendEdgeBatch 

func (s *SimpleClient) SendEdgeBatch(edgeBatch *EdgeBatch) (uint32, error)

SendEdgeBatch is a wrapper around Session.SendEdgeBatch.

func (*SimpleClient) SendFailedTryBatch 

func (s *SimpleClient) SendFailedTryBatch(failedTryBatch *FailedTryBatch) (uint32, error)

SendFailedTryBatch is a wrapper around Session.SendFailedTryBatch.

type SimpleClientOption 

type SimpleClientOption func(*SimpleClient)

Functions which implement SimpleClientOption can be passed to NewSimpleClient as additional options.

func WithClientOptions 

func WithClientOptions(options ...ClientOption) SimpleClientOption

WithClientOptions can be used to pass one or multiple instances of ClientOption to NewSimpleClient to configure its internal client.

As there currently are no implementations for ClientOption, this option is rather useless.

func WithSessionOptions 

func WithSessionOptions(options ...SessionOption) SimpleClientOption

WithSessionOptions can be used to pass one or multiple instances of SessionOption to NewSimpleClient to configure its internal session.

Example 
package main

import (
	bmsclient "github.com/botnet-monitoring/grpc-client"
)

func main() {
	client, err := bmsclient.NewSimpleClient(
		"your-bms-server.local:8083",
		"some-monitor-id",
		"AAAA/some+example+token/AAAAAAAAAAAAAAAAAAA=",
		"some-botnet-id",

		// Pass it to NewSimpleClient on creation and wrap one or multiple session options in it
		bmsclient.WithSessionOptions(
			bmsclient.WithCampaign("some-campaign"),
		),
	)
	if err != nil {
		panic(err)
	}

	// Do something with the client
	_ = client
}

Output:

type SimpleGRPCDataIngestion 

type SimpleGRPCDataIngestion interface {
	SendBotReplyBatch(botReplyBatch *BotReplyBatch) (uint32, error)
	SendFailedTryBatch(failedTryBatch *FailedTryBatch) (uint32, error)
	SendEdgeBatch(edgeBatch *EdgeBatch) (uint32, error)

	AddBotReply(timestamp time.Time, ip net.IP, port uint16, botID string, otherData interface{})
	AddFailedTry(timestamp time.Time, ip net.IP, port uint16, botID string, reason string, otherData interface{})
	AddEdge(timestamp time.Time, srcIP net.IP, srcPort uint16, srcBotID string, dstIP net.IP, dstPort uint16, dstBotID string)
	Flush() error
	FlushBotReplies() error
	FlushFailedTries() error
	FlushEdges() error
}

SimpleGRPCDataIngestion is an interface that the SimpleClient implements. It contains a collection of basic methods to send data to BMS.

SimpleGRPCDataIngestion is a (guaranteed) subset of GRPCDataIngestion which enables you to easily switch from SimpleClient to Client and Client.NewSession.

Source Files

View all Source files

Directories

Path Synopsis
Package hooks contains the definitions of hooks that can be passed to NewSession via WithBotReplyBatchResponseSuccessHook (and similar).
 Package hooks contains the definitions of hooks that can be passed to NewSession via WithBotReplyBatchResponseSuccessHook (and similar).
internal
encoding

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.