slackdump

README

============
Slack Dumper
============

Purpose:  archive your private and public Slack messages, users, channels,
files and emojis.  Generate Slack Export without admin privileges.

|screenshot|

**Quick links**:

- Join the discussion in Telegram_.
- `Buy me a cup of tea`_, or use **Github Sponsors** button on the top of the
  page.
- Reference documentation: |go ref|
- How to's:

  - `Mattermost migration`_ steps
  - `SlackLogViewerとSlackdumpを一緒に使用する`_
  - `Step by Step guide by Viviana Marquez`_ (requires Medium subscription)
  - `Overview on Medium.com`_  (outdated)

.. contents::
   :depth: 2

Description
===========

Typical use scenarios:

* archive your private conversations from Slack when the administrator
  does not allow you to install applications OR you don't want to use
  potentially privacy-violating third-party tools,
* archive channels from Slack when you're on a free "no archive" subscription,
  so you don't lose valuable knowledge in those channels,
* create a Slack Export archive without admin access, or
* save your favourite emojis.

There are four modes of operation (more on this in `User Guide`_) :

#. List users/channels
#. Dumping messages and threads
#. Creating a Slack Export in Mattermost or Standard modes.
#. Emoji download mode.

Slackdump accepts two types of input (see `Dumping Conversations`_ section):

#. the URL/link of the channel or thread, OR
#. the ID of the channel.


Quick Start
===========

#. Download the latest release for your operating system from the releases_
   page.
#. Unpack the archive to any directory.
#. Run the ``./slackdump`` or ``slackdump.exe`` executable (see note below).
#. You know the drill:  use arrow keys to select the menu item, and Enter (or
   Return) to confirm.

By default, Slackdump uses the EZ-Login 3000 automatic login, and interactive
mode.

.. NOTE::
  On Windows and macOS you may be presented with "Unknown developer" window,
  this is fine.  Reason for this is that the executable hasn't been signed by
  the developer certificate.

  To work around this:

  - **on Windows**: click "more information", and press "Run
    Anyway" button.
  - **on macOS**: open the folder in Finder, hold Option and double click the
    executable, choose Run.


Slackord2: Migrating to Discord
===============================

If you're migrating to Discord, the recommended way is to use Slackord2_ - a
great tool with a nice GUI, that is compatible with the export files generated
by Slackdump.

User Guide
==========

For more advanced features and instructions, please see the `User Guide`_.

Previewing Results
==================

Once the data is dumped, you can use one of the following tools to preview the
results:

- `SlackLogViewer`_ - a fast and powerful Slack Export viewer written in C++.
- `Slackdump2Html`_ - a great Python application that converts Slack Dump to a
  static browsable HTML, works on Dump mode files.
- `slack export viewer`_ - Slack Export Viewer is a well known viewer for
  slack export files.

Using as a library
==================

Download:

.. code:: go

  go get github.com/rusq/slackdump/v2


Example
-------
.. code:: go

  package main

  import (
    "context"
    "log"

    "github.com/rusq/slackdump/v2"
    "github.com/rusq/slackdump/v2/auth"
  )

  func main() {
    provider, err := auth.NewValueAuth("xoxc-...", "xoxd-...")
    if err != nil {
        log.Print(err)
        return
    }
    sd, err := slackdump.New(context.Background(), provider)
    if err != nil {
        log.Print(err)
        return
    }
    _ = sd
  }

See |go ref|

Using Custom Logger
-------------------
Slackdump uses a simple `rusq/dlog`_ as a default logger (it is a wrapper around
the standard logger that adds `Debug*` functions).

If you want to use the same default logger that Slackdump uses in your
application, it is available as ``logger.Default``.

No doubts that everyone has their own favourite logger that is better than other
miserable loggers.  Please read below for instructions on plugging your
favourite logger.


Logrus
~~~~~~
Good news is logrus_ can be plugged in straight away, as it implements the
``logger.Interface`` out of the box.

.. code:: go

  lg := logrus.New()
  sd, err := slackdump.New(context.Background(), provider, WithLogger(lg))
    if err != nil {
        log.Print(err)
        return
    }
  }


Glog and others
~~~~~~~~~~~~~~~
If you need to use some other logger, such as glog_, it is a matter of wrapping
the calls to satisfy the ``logger.Interface`` (defined in the `logger`_
package), and then setting the ``Logger`` variable in `slackdump.Options` (see
`options.go`_), or using `WithLogger` option.


FAQ
===

:Q: **Do I need to create a Slack application?**

:A: No, you don't.  Just run the application and EZ-Login 3000 will take
    care of the authentication or, alternatively, grab that token and
    cookie from the browser Slack session.  See `User Guide`_.

:Q: **I'm getting "invalid_auth" error**

:A: Go get the new Cookie from the browser and Token as well.

:Q: **Slackdump takes a very long time to cache users**

:A: Disable the user cache with ``-no-user-cache`` flag.

:Q: **How to read the export file?**

:A: For Slack Workspace Export, use SlackLogViewer_ which is extremely fast
    with an advanced search function, or `slack export viewer`_ which is a
    Python application and runs in a browser.  For the generic dump files, see
    `examples`_ directory for some python and shell examples.

:Q: **My Slack Workspace is on the Free plan.  Can I get data older than
    90-days?**

:A: No, unfortunately you can't.  Slack doesn't allow to export data older
    than 90 days for free workspaces, the API does not return any data before 90
    days for workspaces on the Free plan.

Thank you
=========
Big thanks to all contributors, who submitted a pull request, reported a bug,
suggested a feature, helped to reproduce, or spent time chatting with me on
the Telegram or Slack to help to understand the issue and tested the proposed
solution.

Also, I'd like to thank all those who made a donation to support the project:

- Vivek R.
- Fabian I.
- Ori P.
- Shir B. L.
- Emin G.
- Robert Z.
- Sudhanshu J.

Bulletin Board
--------------

Messages that were conveyed with the donations:

- 25/01/2022: Stay away from `TheSignChef.com`_, ya hear, they don't pay what
  they owe to their employees.


.. _`Buy me a cup of tea`: https://ko-fi.com/rusq_
.. _Telegram: https://t.me/slackdump
.. _`Overview on Medium.com`: https://medium.com/@gilyazov/downloading-your-private-slack-conversations-52e50428b3c2
.. _User Guide: doc/README.rst
.. _Dumping Conversations: doc/usage-channels.rst
.. _Mattermost migration: doc/usage-export.rst
.. _rusq/dlog: https://github.com/rusq/dlog
.. _logrus: https://github.com/sirupsen/logrus
.. _glog: https://github.com/golang/glog
.. _logger: logger/logger.go
.. _options.go: options.go
.. _examples: examples
.. _slack export viewer: https://github.com/hfaran/slack-export-viewer
.. _releases: https://github.com/rusq/slackdump/releases/
.. _Slackord2: https://github.com/thomasloupe/Slackord2
.. _SlackLogViewer: https://github.com/thayakawa-gh/SlackLogViewer/releases
.. _Slackdump2Html: https://github.com/kununu/slackdump2html
.. _`Step by Step guide by Viviana Marquez`: https://vivianamarquez.medium.com/a-step-by-step-guide-to-downloading-slack-messages-without-admin-rights-954f20397e83
.. _`SlackLogViewerとSlackdumpを一緒に使用する`: https://kenkyu-note.hatenablog.com/entry/2022/09/02/090949

..
  bulletin board links

.. _`TheSignChef.com`: https://www.glassdoor.com.au/Reviews/TheSignChef-com-Reviews-E793259.htm

.. |go ref| image:: https://pkg.go.dev/badge/github.com/rusq/slackdump/v2.svg
              :alt: Go Reference
           :target: https://pkg.go.dev/github.com/rusq/slackdump/v2/

.. |screenshot| image:: doc/slackdump.webp
               :target: https://github.com/rusq/slackdump/releases/

Documentation

Index

• Variables
• type Option
  • func WithFilesystem(fs fsadapter.FS) Option
  • func WithForceEnterprise(b bool) Option
  • func WithLimits(l network.Limits) Option
  • func WithLogger(l logger.Interface) Option
  • func WithSlackClient(cl clienter) Option
  • func WithUserCacheRetention(d time.Duration) Option
• type ProcessFunc
• type ProcessResult
  • func (pr ProcessResult) String() string
• type ProcessResults
  • func (prs ProcessResults) String() string
• type Session
  • func New(ctx context.Context, prov auth.Provider, opts ...Option) (*Session, error)
  • func (s *Session) Client() *slack.Client
  • func (s *Session) CurrentUserID() string
  • func (s *Session) Dump(ctx context.Context, link string, oldest, latest time.Time, ...) (*types.Conversation, error)
  • func (s *Session) DumpAll(ctx context.Context, link string) (*types.Conversation, error)
  • func (s *Session) DumpEmojis(ctx context.Context) (map[string]string, error)
  • func (s *Session) DumpRaw(ctx context.Context, link string, oldest, latest time.Time, ...) (*types.Conversation, error)
  • func (s *Session) GetChannels(ctx context.Context, chanTypes ...string) (types.Channels, error)
  • func (s *Session) GetUsers(ctx context.Context) (types.Users, error)
  • func (s *Session) Info() *WorkspaceInfo
  • func (s *Session) Stream(opts ...stream.Option) *stream.Stream
  • func (s *Session) StreamChannels(ctx context.Context, chanTypes []string, cb func(ch slack.Channel) error) error
• type Slacker
• type WorkspaceInfo

Examples

• New (BrowserAuth)
• New (CookieFile)
• New (TokenAndCookie)

Variables

var AllChanTypes = []string{"mpim", "im", "public_channel", "private_channel"}

AllChanTypes enumerates all API-supported channel types as of 03/2023.

var ErrNoUserCache = errors.New("user cache unavailable")

ErrNoUserCache is returned when the user cache is not initialised.

Types

type Option

type Option func(*Session)

Option is the signature of the option-setting function.

func WithFilesystem

func WithFilesystem(fs fsadapter.FS) Option

WithFilesystem sets the filesystem adapter to use for the session. If this option is not given, the default filesystem adapter is initialised with the base location specified in the Config.

func WithForceEnterprise

func WithForceEnterprise(b bool) Option

func WithLimits

func WithLimits(l network.Limits) Option

WithLimits sets the API limits to use for the session. If this option is not given, the default limits are initialised with the values specified in DefLimits.

func WithLogger

func WithLogger(l logger.Interface) Option

WithLogger sets the logger to use for the session. If this option is not given, the default logger is initialised with the filename specified in Config.Logfile. If the Config.Logfile is empty, the default logger writes to STDERR.

func WithSlackClient

func WithSlackClient(cl clienter) Option

WithSlackClient sets the Slack client to use for the session. If this

func WithUserCacheRetention

func WithUserCacheRetention(d time.Duration) Option

WithUserCacheRetention sets the retention period for the user cache. If this option is not given, the default value is 60 minutes.

type ProcessFunc

type ProcessFunc func(msg []types.Message, channelID string) (ProcessResult, error)

ProcessFunc is the signature of the processor function that Dump* family functions accept and call for each API call result. It can be used to modify in-place the slice of messages, returned from API, before they are appended to the slice that will be returned by Dump*. Messages passed to this function are unsorted.

type ProcessResult

type ProcessResult struct {
	// Entity is the type of entity that this result is related to.
	Entity string
	// Count is the number of items processed during processing.
	Count int
}

ProcessResult contains the result of processing.

func (ProcessResult) String

func (pr ProcessResult) String() string

type ProcessResults

type ProcessResults []ProcessResult

ProcessResults is the slice of ProcessResult

func (ProcessResults) String

func (prs ProcessResults) String() string

type Session

type Session struct {
	// contains filtered or unexported fields
}

Session stores basic session parameters. Zero value is not usable, must be initialised with New.

func New

func New(ctx context.Context, prov auth.Provider, opts ...Option) (*Session, error)

New creates new Slackdump session with provided options, and populates the internal cache of users and channels for lookups. If it fails to authenticate, AuthError is returned.

Example (BrowserAuth)

provider, err := auth.NewBrowserAuth(context.Background())
if err != nil {
	log.Print(err)
	return
}
fsa := openTempFS()
defer fsa.Close()
sd, err := New(context.Background(), provider)
if err != nil {
	log.Print(err)
	return
}
_ = sd

Example (CookieFile)

provider, err := auth.NewCookieFileAuth("xoxc-...", "cookies.txt")
if err != nil {
	log.Print(err)
	return
}
fsa := openTempFS()
defer fsa.Close()

sd, err := New(context.Background(), provider)
if err != nil {
	log.Print(err)
	return
}
_ = sd

Example (TokenAndCookie)

provider, err := auth.NewValueAuth("xoxc-...", "xoxd-...")
if err != nil {
	log.Print(err)
	return
}
fsa := openTempFS()
defer fsa.Close()

sd, err := New(context.Background(), provider)
if err != nil {
	log.Print(err)
	return
}
_ = sd

func (*Session) Client

func (s *Session) Client() *slack.Client

Client returns the underlying slack.Client.

func (*Session) CurrentUserID

func (s *Session) CurrentUserID() string

CurrentUserID returns the user ID of the authenticated user.

func (*Session) Dump

func (s *Session) Dump(ctx context.Context, link string, oldest, latest time.Time, processFn ...ProcessFunc) (*types.Conversation, error)

Dump dumps messages or threads specified by link. link can be one of the following:

• Channel URL - i.e. https://ora600.slack.com/archives/CHM82GF99
• Thread URL - i.e. https://ora600.slack.com/archives/CHM82GF99/p1577694990000400
• ChannelID - i.e. CHM82GF99
• ChannelID:ThreadTS - i.e. CHM82GF99:1577694990.000400

oldest and latest timestamps set a timeframe within which the messages should be retrieved, also one can provide process functions.

func (*Session) DumpAll

func (s *Session) DumpAll(ctx context.Context, link string) (*types.Conversation, error)

DumpAll dumps all messages. See description of Dump for what can be provided in link.

func (*Session) DumpEmojis

func (s *Session) DumpEmojis(ctx context.Context) (map[string]string, error)

func (*Session) DumpRaw

func (s *Session) DumpRaw(ctx context.Context, link string, oldest, latest time.Time, processFn ...ProcessFunc) (*types.Conversation, error)

DumpRaw dumps all messages, but does not account for any options defined, such as DumpFiles, instead, the caller must hassle about any processFns they want to apply.

func (*Session) GetChannels

func (s *Session) GetChannels(ctx context.Context, chanTypes ...string) (types.Channels, error)

GetChannels list all conversations for a user. `chanTypes` specifies the type of messages to fetch. See github.com/rusq/slack docs for possible values. If large number of channels is to be returned, consider using StreamChannels.

func (*Session) GetUsers

func (s *Session) GetUsers(ctx context.Context) (types.Users, error)

GetUsers retrieves all users either from cache or from the API. If Session.usercache is not empty, it will return the cached users. Otherwise, it will try fetching them from the API and cache them.

func (*Session) Info

func (s *Session) Info() *WorkspaceInfo

Info returns a workspace information. Slackdump retrieves workspace information during the initialisation when performing authentication test, so no API call is involved at this point.

func (*Session) Stream

func (s *Session) Stream(opts ...stream.Option) *stream.Stream

Stream streams the channel, calling proc functions for each chunk.

func (*Session) StreamChannels

func (s *Session) StreamChannels(ctx context.Context, chanTypes []string, cb func(ch slack.Channel) error) error

StreamChannels requests the channels from the API and calls the callback function cb for each.

type Slacker

type Slacker interface {
	AuthTestContext(context.Context) (response *slack.AuthTestResponse, err error)
	GetConversationHistoryContext(ctx context.Context, params *slack.GetConversationHistoryParameters) (*slack.GetConversationHistoryResponse, error)
	GetConversationRepliesContext(ctx context.Context, params *slack.GetConversationRepliesParameters) (msgs []slack.Message, hasMore bool, nextCursor string, err error)
	GetUsersPaginated(options ...slack.GetUsersOption) slack.UserPagination

	GetStarredContext(ctx context.Context, params slack.StarsParameters) ([]slack.StarredItem, *slack.Paging, error)
	ListBookmarks(channelID string) ([]slack.Bookmark, error)

	GetConversationsContext(ctx context.Context, params *slack.GetConversationsParameters) (channels []slack.Channel, nextCursor string, err error)
	GetConversationInfoContext(ctx context.Context, input *slack.GetConversationInfoInput) (*slack.Channel, error)
	GetUsersInConversationContext(ctx context.Context, params *slack.GetUsersInConversationParameters) ([]string, string, error)

	SearchMessagesContext(ctx context.Context, query string, params slack.SearchParameters) (*slack.SearchMessages, error)
	SearchFilesContext(ctx context.Context, query string, params slack.SearchParameters) (*slack.SearchFiles, error)
}

Slacker is the interface with some functions of slack.Client.

type WorkspaceInfo

type WorkspaceInfo = slack.AuthTestResponse

WorkspaceInfo is an type alias for slack.AuthTestResponse.