slackdump

README

============
Slack Dumper
============

Purpose:  archive your private and public Slack messages, users, channels,
files and emojis.  Generate Slack Export without admin privileges.

|screenshot|

**Quick links**:

- Join the discussion in Telegram_.
- `Buy me a cup of tea`_, or use **Github Sponsors** button on the top of the
  page.
- Reference documentation: |go ref|
- How to's:

  - `Mattermost migration`_ steps
  - `SlackLogViewerとSlackdumpを一緒に使用する`_
  - `Step by Step guide by Viviana Marquez`_ (requires Medium subscription)
  - `Overview on Medium.com`_  (outdated)

.. contents::
   :depth: 2

Description
===========

Typical use scenarios:

* archive your private conversations from Slack when the administrator
  does not allow you to install applications OR you don't want to use
  potentially privacy-violating third-party tools,
* archive channels from Slack when you're on a free "no archive" subscription,
  so you don't lose valuable knowledge in those channels,
* create a Slack Export archive without admin access, or
* save your favourite emojis.

There are four modes of operation (more on this in `User Guide`_) :

#. List users/channels
#. Dumping messages and threads
#. Creating a Slack Export in Mattermost or Standard modes.
#. Emoji download mode.

Slackdump accepts two types of input (see `Dumping Conversations`_ section):

#. the URL/link of the channel or thread, OR
#. the ID of the channel.


Quick Start
===========

On macOS::

  brew install slackdump

On other OSes:

#. Download the latest release for your operating system from the releases_
   page.
#. Unpack the archive to any directory.
#. Run the ``./slackdump`` or ``slackdump.exe`` executable (see note below).
#. You know the drill:  use arrow keys to select the menu item, and Enter (or
   Return) to confirm.

By default, Slackdump uses the EZ-Login 3000 automatic login, and interactive
mode.

.. NOTE::
  On Windows and macOS you may be presented with "Unknown developer" window,
  this is fine.  Reason for this is that the executable hasn't been signed by
  the developer certificate.

  To work around this:

  - **on Windows**: click "more information", and press "Run
    Anyway" button.
  - **on macOS**: open the folder in Finder, hold Option and double click the
    executable, choose Run.


Slackord2: Migrating to Discord
===============================

If you're migrating to Discord, the recommended way is to use Slackord2_ - a
great tool with a nice GUI, that is compatible with the export files generated
by Slackdump.

User Guide
==========

For more advanced features and instructions, please see the `User Guide`_.

Previewing Results
==================

Once the data is dumped, you can use one of the following tools to preview the
results:

- `SlackLogViewer`_ - a fast and powerful Slack Export viewer written in C++.
- `Slackdump2Html`_ - a great Python application that converts Slack Dump to a
  static browsable HTML, works on Dump mode files.
- `slack export viewer`_ - Slack Export Viewer is a well known viewer for
  slack export files.

Using as a library
==================

Download:

.. code:: go

  go get github.com/rusq/slackdump/v2


Example
-------
.. code:: go

  package main

  import (
    "context"
    "log"

    "github.com/rusq/slackdump/v2"
    "github.com/rusq/slackdump/v2/auth"
  )

  func main() {
    provider, err := auth.NewValueAuth("xoxc-...", "xoxd-...")
    if err != nil {
        log.Print(err)
        return
    }
    sd, err := slackdump.New(context.Background(), provider)
    if err != nil {
        log.Print(err)
        return
    }
    _ = sd
  }

See |go ref|

Using Custom Logger
-------------------
Slackdump uses a simple `rusq/dlog`_ as a default logger (it is a wrapper around
the standard logger that adds `Debug*` functions).

If you want to use the same default logger that Slackdump uses in your
application, it is available as ``logger.Default``.

No doubts that everyone has their own favourite logger that is better than other
miserable loggers.  Please read below for instructions on plugging your
favourite logger.


Logrus
~~~~~~
Good news is logrus_ can be plugged in straight away, as it implements the
``logger.Interface`` out of the box.

.. code:: go

  lg := logrus.New()
  sd, err := slackdump.New(context.Background(), provider, WithLogger(lg))
    if err != nil {
        log.Print(err)
        return
    }
  }


Glog and others
~~~~~~~~~~~~~~~
If you need to use some other logger, such as glog_, it is a matter of wrapping
the calls to satisfy the ``logger.Interface`` (defined in the `logger`_
package), and then setting the ``Logger`` variable in `slackdump.Options` (see
`options.go`_), or using `WithLogger` option.


FAQ
===

:Q: **Do I need to create a Slack application?**

:A: No, you don't.  Just run the application and EZ-Login 3000 will take
    care of the authentication or, alternatively, grab that token and
    cookie from the browser Slack session.  See `User Guide`_.

:Q: **I'm getting "invalid_auth" error**

:A: Go get the new Cookie from the browser and Token as well.

:Q: **Slackdump takes a very long time to cache users**

:A: Disable the user cache with ``-no-user-cache`` flag.

:Q: **How to read the export file?**

:A: For Slack Workspace Export, use SlackLogViewer_ which is extremely fast
    with an advanced search function, or `slack export viewer`_ which is a
    Python application and runs in a browser.  For the generic dump files, see
    `examples`_ directory for some python and shell examples.

:Q: **My Slack Workspace is on the Free plan.  Can I get data older than
    90-days?**

:A: No, unfortunately you can't.  Slack doesn't allow to export data older
    than 90 days for free workspaces, the API does not return any data before 90
    days for workspaces on the Free plan.

Thank you
=========
Big thanks to all contributors, who submitted a pull request, reported a bug,
suggested a feature, helped to reproduce, or spent time chatting with me on
the Telegram or Slack to help to understand the issue and tested the proposed
solution.

Also, I'd like to thank all those who made a donation to support the project:

- Vivek R.
- Fabian I.
- Ori P.
- Shir B. L.
- Emin G.
- Robert Z.
- Sudhanshu J.

Bulletin Board
--------------

Messages that were conveyed with the donations:

- 25/01/2022: Stay away from `TheSignChef.com`_, ya hear, they don't pay what
  they owe to their employees.


.. _`Buy me a cup of tea`: https://ko-fi.com/rusq_
.. _Telegram: https://t.me/slackdump
.. _`Overview on Medium.com`: https://medium.com/@gilyazov/downloading-your-private-slack-conversations-52e50428b3c2
.. _User Guide: doc/README.rst
.. _Dumping Conversations: doc/usage-channels.rst
.. _Mattermost migration: doc/usage-export.rst
.. _rusq/dlog: https://github.com/rusq/dlog
.. _logrus: https://github.com/sirupsen/logrus
.. _glog: https://github.com/golang/glog
.. _logger: logger/logger.go
.. _options.go: options.go
.. _examples: examples
.. _slack export viewer: https://github.com/hfaran/slack-export-viewer
.. _releases: https://github.com/rusq/slackdump/releases/
.. _Slackord2: https://github.com/thomasloupe/Slackord2
.. _SlackLogViewer: https://github.com/thayakawa-gh/SlackLogViewer/releases
.. _Slackdump2Html: https://github.com/kununu/slackdump2html
.. _`Step by Step guide by Viviana Marquez`: https://vivianamarquez.medium.com/a-step-by-step-guide-to-downloading-slack-messages-without-admin-rights-954f20397e83
.. _`SlackLogViewerとSlackdumpを一緒に使用する`: https://kenkyu-note.hatenablog.com/entry/2022/09/02/090949

..
  bulletin board links

.. _`TheSignChef.com`: https://www.glassdoor.com.au/Reviews/TheSignChef-com-Reviews-E793259.htm

.. |go ref| image:: https://pkg.go.dev/badge/github.com/rusq/slackdump/v2.svg
              :alt: Go Reference
           :target: https://pkg.go.dev/github.com/rusq/slackdump/v2/

.. |screenshot| image:: doc/slackdump.webp
               :target: https://github.com/rusq/slackdump/releases/

Documentation

Index

• Variables
• func TestAuth(ctx context.Context, provider auth.Provider) error
• type Option
  • func CacheDir(dir string) Option
  • func DownloadFiles(b bool) Option
  • func MaxUserCacheAge(d time.Duration) Option
  • func NumWorkers(n int) Option
  • func RetryDownloads(attempts int) Option
  • func RetryThreads(attempts int) Option
  • func Tier2Boost(eventsPerMin uint) Option
  • func Tier2Burst(eventsPerSec uint) Option
  • func Tier3Boost(eventsPerMin uint) Option
  • func Tier3Burst(eventsPerSec uint) Option
  • func UserCacheFilename(s string) Option
  • func WithLogger(l logger.Interface) Option
• type Options
• type ProcessFunc
• type ProcessResult
  • func (pr ProcessResult) String() string
• type ProcessResults
  • func (prs ProcessResults) String() string
• type Session
  • func New(ctx context.Context, creds auth.Provider, opts ...Option) (*Session, error)
  • func NewWithOptions(ctx context.Context, authProvider auth.Provider, opts Options) (*Session, error)
  • func (sd *Session) Client() *slack.Client
  • func (sd *Session) CurrentUserID() string
  • func (sd *Session) Dump(ctx context.Context, link string, oldest, latest time.Time, ...) (*types.Conversation, error)
  • func (sd *Session) DumpAll(ctx context.Context, link string) (*types.Conversation, error)
  • func (sd *Session) DumpAllMessages(ctx context.Context, channelID string) (*types.Conversation, error)
  • func (sd *Session) DumpAllURL(ctx context.Context, slackURL string) (*types.Conversation, error)
  • func (s *Session) DumpEmojis(ctx context.Context) (map[string]string, error)
  • func (sd *Session) DumpMessages(ctx context.Context, channelID string, oldest, latest time.Time, ...) (*types.Conversation, error)
  • func (sd *Session) DumpMessagesRaw(ctx context.Context, channelID string, oldest, latest time.Time, ...) (*types.Conversation, error)
  • func (sd *Session) DumpRaw(ctx context.Context, link string, oldest, latest time.Time, ...) (*types.Conversation, error)
  • func (sd *Session) DumpThread(ctx context.Context, channelID, threadTS string, oldest, latest time.Time, ...) (*types.Conversation, error)
  • func (sd *Session) DumpURL(ctx context.Context, slackURL string, oldest, latest time.Time, ...) (*types.Conversation, error)
  • func (sd *Session) GetChannelMembers(ctx context.Context, channelID string) ([]string, error)
  • func (sd *Session) GetChannels(ctx context.Context, chanTypes ...string) (types.Channels, error)
  • func (sd *Session) GetUsers(ctx context.Context) (types.Users, error)
  • func (sd *Session) Me() (slack.User, error)
  • func (sd *Session) SetFS(fs fsadapter.FS)
  • func (sd *Session) StreamChannels(ctx context.Context, chanTypes []string, cb func(ch slack.Channel) error) error

Examples

• New (BrowserAuth)
• New (CookieFile)
• New (TokenAndCookie)

Variables

var AllChanTypes = []string{"mpim", "im", "public_channel", "private_channel"}

AllChanTypes enumerates all API-supported channel types as of 03/2023.

var DefOptions = Options{
	DumpFiles:           false,
	Workers:             defNumWorkers,
	DownloadRetries:     3,
	Tier2Boost:          20,
	Tier2Burst:          1,
	Tier2Retries:        20,
	Tier3Boost:          120,
	Tier3Burst:          1,
	Tier3Retries:        3,
	Tier4Boost:          1,
	Tier4Burst:          1,
	Tier4Retries:        3,
	ConversationsPerReq: 200,
	ChannelsPerReq:      100,
	RepliesPerReq:       200,
	UserCacheFilename:   "users.cache",
	MaxUserCacheAge:     4 * time.Hour,
	CacheDir:            ".",
	Logger:              logger.Default,
}

DefOptions is the default options used when initialising slackdump instance.

var (
	// ErrNoUserCache is returned when the user cache is not available.
	ErrNoUserCache = errors.New("user cache unavailable")
)

Functions

func TestAuth

func TestAuth(ctx context.Context, provider auth.Provider) error

TestAuth attempts to authenticate with the given provider. It will return AuthError if faled.

Types

type Option

type Option func(*Options)

Option is the signature of the option-setting function.

func CacheDir

func CacheDir(dir string) Option

func DownloadFiles

func DownloadFiles(b bool) Option

DownloadFiles enables or disables the conversation/thread file downloads.

func MaxUserCacheAge

func MaxUserCacheAge(d time.Duration) Option

MaxUserCacheAge allows to set the maximum user cache age. If set to 0 - it will always use the API output, and never load cache.

func NumWorkers

func NumWorkers(n int) Option

NumWorkers allows to set the number of file download workers. n should be in range [1, NumCPU]. If not in range, will be reset to a defNumWorkers number, which seems reasonable.

func RetryDownloads

func RetryDownloads(attempts int) Option

RetryDownloads sets the number of attempts to download a file when getting rate limited.

func RetryThreads

func RetryThreads(attempts int) Option

RetryThreads sets the number of attempts when dumping conversations and threads, and getting rate limited.

func Tier2Boost

func Tier2Boost(eventsPerMin uint) Option

Tier2Boost allows to deliver a magic kick to the limiter, to override the base slack Tier limits. The resulting events per minute will be calculated like this:

events_per_sec =  (<slack_tier_epm> + <eventsPerMin>) / 60.0

func Tier2Burst

func Tier2Burst(eventsPerSec uint) Option

Tier2Burst allows to set the limiter burst value.

func Tier3Boost

func Tier3Boost(eventsPerMin uint) Option

Tier3Boost allows to deliver a magic kick to the limiter, to override the base slack Tier limits. The resulting events per minute will be calculated like this:

events_per_sec =  (<slack_tier_epm> + <eventsPerMin>) / 60.0

func Tier3Burst

func Tier3Burst(eventsPerSec uint) Option

Tier3Burst allows to set the limiter burst value.

func UserCacheFilename

func UserCacheFilename(s string) Option

UserCacheFilename allows to set the user cache filename.

func WithLogger

func WithLogger(l logger.Interface) Option

WithLogger allows to set the custom logger.

type Options

type Options struct {
	DumpFiles           bool          // will we save the conversation files?
	Workers             int           // number of file-saving workers
	DownloadRetries     int           // if we get rate limited on file downloads, this is how many times we're going to retry
	Tier2Boost          uint          // Tier-2 limiter boost
	Tier2Burst          uint          // Tier-2 limiter burst
	Tier2Retries        int           // Tier-2 retries when getting 429 on channels fetch
	Tier3Boost          uint          // Tier-3 limiter boost allows to increase or decrease the slack Tier req/min rate.  Affects all tiers.
	Tier3Burst          uint          // Tier-3 limiter burst allows to set the limiter burst in req/sec.  Default of 1 is safe.
	Tier3Retries        int           // number of retries to do when getting 429 on conversation fetch
	Tier4Boost          uint          // Tier-4 limiter boost allows to increase or decrease the slack Tier req/min rate.  Affects all tiers.
	Tier4Burst          uint          // Tier-4 limiter burst allows to set the limiter burst in req/sec.  Default of 1 is safe.
	Tier4Retries        int           // number of retries to do when getting 429 on conversation fetch
	ConversationsPerReq int           // number of messages we get per 1 API request. bigger the number, less requests, but they become more beefy.
	ChannelsPerReq      int           // number of channels to fetch per 1 API request.
	RepliesPerReq       int           // number of thread replies per request (slack default: 1000)
	UserCacheFilename   string        // user cache filename
	MaxUserCacheAge     time.Duration // how long the user cache is valid for.
	NoUserCache         bool          // disable fetching users from the API.
	CacheDir            string        // cache directory
	Logger              logger.Interface
}

Options is the option set for the Session.

type ProcessFunc

type ProcessFunc func(msg []types.Message, channelID string) (ProcessResult, error)

ProcessFunc is the signature of the function Dump* functions accept and call for each API call result. It can be used to modify in-place the slice of messages, returned from API, before they are appended to the slice that will be returned by Dump*. Messages passed to this function are unsorted.

type ProcessResult

type ProcessResult struct {
	// Entity is the type of entity that this result is related to.
	Entity string
	// Count is the number of items processed during processing.
	Count int
}

ProcessResult contains the result of processing.

func (ProcessResult) String

func (pr ProcessResult) String() string

type ProcessResults

type ProcessResults []ProcessResult

ProcessResults is the slice of ProcessResult

func (ProcessResults) String

func (prs ProcessResults) String() string

type Session

type Session struct {

	// Users contains the list of users and populated on NewSession
	Users     types.Users          `json:"users"`
	UserIndex structures.UserIndex `json:"-"`
	// contains filtered or unexported fields
}

Session stores basic session parameters.

func New

func New(ctx context.Context, creds auth.Provider, opts ...Option) (*Session, error)

New creates new session with the default options and populates the internal cache of users and channels for lookups.

Example (BrowserAuth)

provider, err := auth.NewBrowserAuth(context.Background())
if err != nil {
	log.Print(err)
	return
}
sd, err := New(context.Background(), provider)
if err != nil {
	log.Print(err)
	return
}
_ = sd

Example (CookieFile)

provider, err := auth.NewCookieFileAuth("xoxc-...", "cookies.txt")
if err != nil {
	log.Print(err)
	return
}
sd, err := New(context.Background(), provider)
if err != nil {
	log.Print(err)
	return
}
_ = sd

Example (TokenAndCookie)

provider, err := auth.NewValueAuth("xoxc-...", "xoxd-...")
if err != nil {
	log.Print(err)
	return
}
sd, err := New(context.Background(), provider)
if err != nil {
	log.Print(err)
	return
}
_ = sd

func NewWithOptions

func NewWithOptions(ctx context.Context, authProvider auth.Provider, opts Options) (*Session, error)

New creates new Session with provided options, and populates the internal cache of users and channels for lookups. If it fails to authenticate, AuthError is returned.

func (*Session) Client

func (sd *Session) Client() *slack.Client

Client returns the underlying slack.Client.

func (*Session) CurrentUserID

func (sd *Session) CurrentUserID() string

func (*Session) Dump

func (sd *Session) Dump(ctx context.Context, link string, oldest, latest time.Time, processFn ...ProcessFunc) (*types.Conversation, error)

Dump dumps messages or threads specified by link. link can be one of the following:

• Channel URL - i.e. https://ora600.slack.com/archives/CHM82GF99
• Thread URL - i.e. https://ora600.slack.com/archives/CHM82GF99/p1577694990000400
• ChannelID - i.e. CHM82GF99
• ChannelID:ThreadTS - i.e. CHM82GF99:1577694990.000400

oldest and latest timestamps set a timeframe within which the messages should be retrieved, also one can provide process functions.

func (*Session) DumpAll

func (sd *Session) DumpAll(ctx context.Context, link string) (*types.Conversation, error)

DumpAll dumps all messages. See description of Dump for what can be provided in link.

func (*Session) DumpAllMessages

func (sd *Session) DumpAllMessages(ctx context.Context, channelID string) (*types.Conversation, error)

DumpAllMessages fetches messages from the conversation identified by channelID.

Deprecated: Use DumpAll, this function will be removed in v3.

func (*Session) DumpAllURL

func (sd *Session) DumpAllURL(ctx context.Context, slackURL string) (*types.Conversation, error)

DumpAllURL dumps messages from the slackURL.

Deprecated: Use DumpAll, this function will be removed in v3.

func (*Session) DumpEmojis

func (s *Session) DumpEmojis(ctx context.Context) (map[string]string, error)

func (*Session) DumpMessages

func (sd *Session) DumpMessages(ctx context.Context, channelID string, oldest, latest time.Time, processFn ...ProcessFunc) (*types.Conversation, error)

DumpMessages dumps messages in the given timeframe between oldest and latest. If oldest or latest are zero time, they will not be accounted for. Having both oldest and latest as Zero-time, will make this function behave similar to DumpMessages. ProcessFn is a slice of post-processing functions that will be called for each message chunk downloaded from the Slack API.

Deprecated: Use Dump, this function will be unexported in v3.

func (*Session) DumpMessagesRaw

func (sd *Session) DumpMessagesRaw(ctx context.Context, channelID string, oldest, latest time.Time, processFn ...ProcessFunc) (*types.Conversation, error)

DumpMessagesRaw dumps all messages, but does not account for any options defined, such as DumpFiles, instead, the caller must hassle about any processFns they want to apply.

Deprecated: Use DumpRaw, this function will be unexported in v3.

func (*Session) DumpRaw

func (sd *Session) DumpRaw(ctx context.Context, link string, oldest, latest time.Time, processFn ...ProcessFunc) (*types.Conversation, error)

DumpRaw dumps all messages, but does not account for any options defined, such as DumpFiles, instead, the caller must hassle about any processFns they want to apply.

func (*Session) DumpThread

func (sd *Session) DumpThread(
	ctx context.Context,
	channelID,
	threadTS string,
	oldest, latest time.Time,
	processFn ...ProcessFunc,
) (*types.Conversation, error)

DumpThread dumps a single thread identified by (channelID, threadTS). Optionally one can provide a number of processFn that will be applied to each chunk of messages returned by a one API call.

Deprecated: Use Dump, this function will be unexported in v3.

func (*Session) DumpURL

func (sd *Session) DumpURL(ctx context.Context, slackURL string, oldest, latest time.Time, processFn ...ProcessFunc) (*types.Conversation, error)

DumpURL acts like DumpAllURL but allows to specify oldest and latest timestamps to define a window within which the messages should be retrieved, also one can provide process functions.

Deprecated: Use Dump, this function will be removed in v3.

func (*Session) GetChannelMembers

func (sd *Session) GetChannelMembers(ctx context.Context, channelID string) ([]string, error)

GetChannelMembers returns a list of all members in a channel.

func (*Session) GetChannels

func (sd *Session) GetChannels(ctx context.Context, chanTypes ...string) (types.Channels, error)

GetChannels list all conversations for a user. `chanTypes` specifies the type of messages to fetch. See github.com/rusq/slack docs for possible values. If large number of channels is to be returned, consider using StreamChannels.

func (*Session) GetUsers

func (sd *Session) GetUsers(ctx context.Context) (types.Users, error)

GetUsers retrieves all users either from cache or from the API.

func (*Session) Me

func (sd *Session) Me() (slack.User, error)

Me returns the current authenticated user in a rather dirty manner. If the user cache is unitnitialised, it returns ErrNoUserCache.

func (*Session) SetFS

func (sd *Session) SetFS(fs fsadapter.FS)

SetFS sets the filesystem to save attachments to (slackdump defaults to the current directory otherwise).

func (*Session) StreamChannels

func (sd *Session) StreamChannels(ctx context.Context, chanTypes []string, cb func(ch slack.Channel) error) error

StreamChannels requests the channels from the API and calls the callback function cb for each.