sillybot

README

Silly Bot

A simple Discord and Slack bot written in Go that natively serves LLM and Stable Diffusion to chat, generate images, and memes!

⚠️Warning⚠️ : This is a work in progress. There is no privacy control. Please file an issue for feature requests or bugs found.

I made a talk about it: youtu.be/uGkrjsMl-ok!

Usage

Type in chat:

/meme_auto sillybot meme generator awesome

Receive:

Talk with it and use its commands as described at:

• For Discord, see cmd/discord-bot/README.md.
• For Slack, see cmd/slack-bot/README.md.

Features

• Generates memes in full automatic mode; it generates both labels and image description.
• Generates memes in manual mode for more precision.
• Chat interface with resettable system prompt.
• Uses WebSocket so no need to setup a web server! 🎉
• Works on Ubuntu (linux), macOS and Windows! 🪟
• Supported backends:
  • Uses llama.cpp.
  • Optionally uses pytorch python backend for experiments.
• Runs as a chat bot LLM instructs. Tested to work with:
  • Gemma-2-9B instruct in quantized form with 8K context window.
  • Gemma-2-27B instruct in quantized form with 8K context window.
  • Meta-Llama-3.1-8B-instruct in quantized form with 128K context window.
  • Meta-Llama-3-70B-instruct in quantized form with 8K context window.
  • Mistral-7B-Instruct-v0.3 in quantized form with 32K (!) context window.
  • Mistral-Nemo-Instruct-2407 in quantized form with 128K (!) context window.
  • Phi-3-mini-4k-instruct "Phi-3.1" in quantized form with 4K context window.
  • Phi-3-mini-128k-instruct "Phi-3.1" in quantized form with 128K (!!) context window. Requires a ton of RAM.
  • Phi-3-medium-4k-instruct in quantized form with 4K context window.
  • Phi-3-medium-128k-instruct in quantized form with 128K (!!) context window. Requires a ton of RAM.
  • Qwen2-0.5B-Instruct in quantized form with 32K (!) context window. Extremely small model, super fast. Perfect to run in unit tests.
  • Qwen2-1.5B-Instruct in quantized form with 32K (!) context window. Super small model, super fast.
  • Qwen2-7B-Instruct in quantized form with 32K (!) context window.
• Runs as an image generator. Tested to work with:
  • Segmind SSD-1B coupled with LCM Lora is quite fast. Segmind SSD-1B renders under 10s on a MacBook Pro M3 Max, under 100s on an Intel i7-13700 on Ubuntu and under 160s on a i9-13900H on Windows 11. (Depends on the number of steps selected)
  • Stable Diffusion 3 Medium which is too slow on a M3 Max IMO.

Hardware requirement

• LLM: any <4 years old computer, really. A GPU is not required. If you are unsure, start with Qwen2 0.5B by using model: "qwen2-0_5b-instruct-q5_k_m" in the llm: section of config.yml. This requires 1GiB of RAM. Go up with larger models from there.
  • I recommend to chat with the bot, use the /metrics command and iterate to use increasingly larger models as long as you can keep >=200tok/s for prompt parsing and >=20tok/s for generation.
  • When using models with large context window (>8K) like Llama 3.1 and Mistral Nemo, you may want to explicitly limit the context window to a smaller size like 16384, 32768 or 65536 so it reduces memory requirements. Counter-intuitively, reducing the context window too much will slow down generation.
• Image Generation: a nVidia GPU with 6GiB of video memory (VRAM) (12GiB is better) or a MacBook Pro. While it works on CPU, expect a minute or two to generate each image.

Pro-tip: You can use 2 computers: one running the LLM and one the image generation! Start the llm/imagegen server manually then use the remote: option in config.yml.

Installation

Both function essentially the same but the Application configuration on the server (Discord or Slack) is different:

• For Discord, see cmd/discord-bot/README.md.
• For Slack, see cmd/slack-bot/README.md.

Dev

Acknowledgements

This project greatly benefit from llama.cpp by Georgi Gerganov, previous versions leveraged llamafile by Justine Tunney, all open source contributors and all the companies providing open-weights models.

Author

sillybot was created with ❤️️ and passion by Marc-Antoine Ruel.

Documentation

Overview

Package sillybot implements the common code used by both discord-bot and slack-bot.

Index

• Variables
• func LoadModels(ctx context.Context, cache string, cfg *Config) (*llm.Session, *imagegen.Session, error)
• type Config
  • func (c *Config) LoadOrDefault(config string) error
  • func (c *Config) Validate() error
• type Settings

Variables

var DefaultConfig []byte

Default configuration with well known models and sane presets.

Functions

func LoadModels

func LoadModels(ctx context.Context, cache string, cfg *Config) (*llm.Session, *imagegen.Session, error)

LoadModels loads the LLM and ImageGen models.

Both take a while to start, so load them in parallel for faster initialization.

Types

type Config

type Config struct {
	Bot struct {
		LLM      llm.Options
		ImageGen imagegen.Options `yaml:"image_gen"`
		Settings Settings
	}
	KnownLLMs []llm.KnownLLM
}

Config defines the configuration format.

func (*Config) LoadOrDefault

func (c *Config) LoadOrDefault(config string) error

LoadOrDefault loads a config or write the default to disk.

func (*Config) Validate

func (c *Config) Validate() error

Validate checks for obvious errors in the fields.

type Settings

type Settings struct {
	// PromptSystem is the default system prompt to use. Is a Go template as
	// documented at https://pkg.go.dev/text/template. Values provided by LLM are:
	// - Now: current time in ISO-8601, including the server's time zone.
	// - Model: the model name.
	PromptSystem string `yaml:"prompt_system"`
	// PromptLabels is the prompt used to generate meme labels via a short
	// description.
	PromptLabels string `yaml:"prompt_labels"`
	// PromptImage is the prompt used to generate an image via a short
	// description.
	PromptImage string `yaml:"prompt_image"`
}

Settings is the bot settings.