discord-downloader-go

command module
v1.4.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Feb 3, 2021 License: MIT Imports: 41 Imported by: 0

README

Discord Downloader



Join the Discord

DOWNLOAD LATEST RELEASE BUILDS

This project is a fork of Seklfreak's discord-image-downloader-go

For list of differences and why I made an independent project, see below

This is a Discord bot program to download files posted in specified Discord channels to local folders. It can fetch highest possible quality files from various sources (listed below), aside from downloading any file directly linked or attached. See Features below for full list.

Need help? Have suggestions? Join the Discord server!

Sections

WARNING: Discord does not allow Automated User Accounts (Self-Bots/User-Bots)

Read more in Discord Trust & Safety Team's Official Statement...

While this project works for user logins, I do not reccomend it as you risk account termination. If you can, use a proper Discord Bot user for this program.

NOTE: This only applies to real User Accounts, not Bot users. This program currently works for either.

Getting Running

Depending on your purpose for this program, there are various ways you can run it.

Getting Started

You can either create a settings.json following the examples & variables listed below, or have the program create a default file (if it is missing when you run the program, it will make one, and ask you if you want to enter in basic info for the new file).

Credentials...

  • If using a Bot User, enter the token into the "token" setting. Remove the lines for "username" and "password" or leave blank (""). To create a Bot User, go to discord.com/developers/applications and create a New Application. Once created, go to Bot and create. The token can be found on the Bot page. To invite to your server(s), go to OAuth2 and check "bot", copy the url, paste into browser and follow prompts for adding to server(s).
  • If using a Real User (Self-Bot), fill out the "username" and "password" settings. Remove the line for "token" or leave blank ("").
  • If using a Real User (Self-Bot) with 2FA (Two-Factor Authentication), enter the token into the "token" setting. Remove the lines for "username" and "password" or leave blank (""). Token can be found from Developer Tools in browser under localStorage.token or in the Discord client Ctrl+Shift+I (Windows)/Cmd+Option+I (Mac) under Application → Local Storage → https://discordapp.com → "token". You must also set userBot within the credentials section of the settings.json to true.

Bot Permissions in Discord...

  • In order to perform basic downloading functions, the bot will need Read Message permissions in the server(s) of your designated channel(s).
  • In order to respond to commands, the bot will need Send Message permissions in the server(s) of your designated channel(s). If executing commands via an Admin Channel, the bot will only need Send Message permissions for that channel, and that permission will not be required for the source channel.
  • In order to process history commands, the bot will need Read Message History permissions in the server(s) of your designated channel(s).

How to Find Discord IDs...

  • Discord Developer Mode: Enable Developer Mode in Discord settings under Appearance.
  • Finding Channel ID: Enable Discord Developer Mode (see above), right click on the channel and Copy ID.
  • Finding User ID: Enable Discord Developer Mode (see above), right click on the user and Copy ID.
  • Finding Emoji ID: Enable Discord Developer Mode (see above), right click on the emoji and Copy ID.
  • Finding DM/PM ID: Inspect Element on the DM icon for the desired user. Look for href="/channels/@me/CHANNEL_ID_HERE". Using this ID in place of a normal channel ID should work perfectly fine.

Features

  • Supported File Downloading:
    • Discord File Attachments
    • Direct Links to Files
    • Twitter (requires API key, see config section)
    • Instagram
    • Reddit
    • Imgur (Single Posts & Albums)
    • Flickr (requires API key, see config section)
    • Google Drive (requires API Credentials, see config section)
    • Tistory
    • Streamable
    • Gfycat
  • Commands:
    • Help (help - Alias: commands)
    • Ping (ping - Alias: test)
    • Status: Get an output of the current status of the bot (status - Alias: info)
    • Stats: Have the bot dump stats (stats)
    • [Must be Bot or Server Admin] History: Process all old messages in channel (history - Aliases: catalog, cache)
    • [Must be Bot Admin] Exit (nice for process managers like pm2 for instant reload) (exit - Aliases: reload, kill)

Differences from Seklfreak's discord-image-downloader-go & Why I made this

  • Go 1.15 rather than 1.13 I updated Sekl's project to match this.
  • discordgo 0.22.0 rather than 0.16.1 I updated Sekl's project to match this.
  • Implements dgrouter for commands
  • Configuration is JSON-based rather than ini to allow more elaborate settings and better organization. With this came many features such as channel-specific settings.
  • Channel-specific control of downloaded filetypes / content types (considers things like .mov as videos as well, rather than ignore them), Optional dividing of content types into separate folders.
  • Download Support for Reddit
  • (Optional) Reactions upon download success.
  • (Optional) Discord messages upon encountered errors.
  • Extensive bot status/presence customization.
  • Consistent Log Formatting, Color-Coded Logging
  • Somewhat different organization than original project; initially created from scratch then components ported over.
  • Fixed Compatability Issue with xurls that required people to edit the project, regarding xurls.Strict.FindAllString. The issue was due to some people having xurls v2 installed while the projects go.mod required v1.1; changing go.mod to require v2 specifically seems to be the correct fix. I updated Sekl's project to match this.

I've been a user of Seklfreak's project since ~2018 and it's been great for my uses, but there were certain aspects I wanted to expand upon, one of those being customization of channel configuration, and other features like message reactions upon success, differently formatted statuses, etc. If some aspects are rudimentary or messy, please make a pull request, as this is my first project using Go and I've learned everything from observation & Stack Overflow.

Guide: Saving History (Cataloging Old Messages)

This guide is to show you how to make the bot go through all old messages in a channel and catalog them as though they were being sent right now, in order to download them all.

You will need the Channel ID (see bottom of Setup) if attempting to catalog history from a specific channel or group of channels, within an admin channel.

  • ddg history to catalog the current channel the command is sent in (must be registered in channels in settings).
  • ddg history cancel to stop cataloging the current channel the command is sent in (must be registered in channels in settings).
  • ddg history <Channel ID(s)> to catalog specified channels from within a designated Admin Channel (must be registered in adminChannels in settings). You can do multiple channels per command if desired, separated by commas.
  • ddg history cancel <Channel ID(s)> to stop cataloging specified channels from within a designated Admin Channel (must be registered in adminChannels in settings). You can do multiple channels per command if desired, separated by commas.

Guide: Settings / Configuration

I tried to make the configuration as user friendly as possible, though you still need to follow proper JSON syntax (watch those commas). All settings specified below labeled [DEFAULTS] will use default values if missing from the settings file, and those labeled [OPTIONAL] will not be used if missing from the settings file.

When initially launching the bot it will create a default settings file if you do not create your own settings.json manually. All JSON settings follow camelCase format.

If you have a config.ini from Seklfreak's discord-image-downloader-go, it will import settings if it's in the same folder as the program.

Settings Examples

The following example is for a Bot Application (using a token), bound to 1 channel.

This setup exempts many options so they will use default values (see below). It shows the bare minimum required settings for the bot to function. Example - Barebones settings.json:

{
    "credentials": {
        "token": "YOUR_TOKEN"
    },
    "channels": [
        {
            "channel": "DISCORD_CHANNEL_ID_TO_DOWNLOAD_FROM",
            "destination": "FOLDER_LOCATION_TO_DOWNLOAD_TO"
        }
    ]
}

Example - Selfbot settings.json:

{
    "credentials": {
        "email": "REPLACE_WITH_YOUR_EMAIL",
        "password": "REPLACE_WITH_YOUR_PASSWORD"
    },
    "scanOwnMessages": true,
    "presenceEnabled": false,
    "channels": [
        {
            "channel": "DISCORD_CHANNEL_ID_TO_DOWNLOAD_FROM",
            "destination": "FOLDER_LOCATION_TO_DOWNLOAD_TO",
            "allowCommands": false,
            "errorMessages": false,
            "reactWhenDownloaded": false
        }
    ]
}

Example - Advanced settings.json:

{
    "credentials": {
        "token": "YOUR_TOKEN",
        "twitterAccessToken": "",
        "twitterAccessTokenSecret": "",
        "twitterConsumerKey": "",
        "twitterConsumerSecret": ""
    },
    "admins": [ "YOUR_DISCORD_USER_ID", "YOUR_FRIENDS_DISCORD_USER_ID" ],
    "adminChannels": [
        {
            "channel": "CHANNEL_ID_FOR_ADMIN_CONTROL"
        }
    ],
    "debugOutput": true,
    "commandPrefix": "downloader_",
    "allowSkipping": true,
    "filterDuplicates": true,
    "filterDuplicatesThreshold": 75,
    "downloadRetryMax": 5,
    "downloadTimeout": 120,
    "githubUpdateChecking": true,
    "presenceStatus": "dnd",
    "presenceType": 3,
    "presenceOverwrite": "{{count}} files",
    "filenameDateFormat": "2006.01.02-15.04.05_",
    "embedColor": "#29BEB0",
    "inflateCount": 123456,
    "channels": [
        {
            "channel": "THIS_CHANNEL_DOWNLOADS_EVERYTHING",
            "destination": "EVERYTHING",
            "overwriteEmbedColor": "#FF0000",
            "userBlacklist": [ "USER_ID_FOR_PERSON_I_DONT_LIKE" ],
            "divideFoldersByType": false,
            "saveImages": true,
            "saveVideos": true,
            "saveAudioFiles": true,
            "saveTextFiles": true,
            "saveOtherFiles": true,
            "savePossibleDuplicates": true,
            "extensionBlacklist": [
                ".htm",
                ".html",
                ".php"
            ]
        },
        {
            "channel": "THIS_CHANNEL_ONLY_DOWNLOADS_MEDIA",
            "destination": "media",
            "overwriteAllowSkipping": false,
            "saveImages": true,
            "saveVideos": true,
            "saveAudioFiles": true,
            "saveTextFiles": false,
            "saveOtherFiles": false
        },
        {
            "channel": "THIS_CHANNEL_ONLY_DOWNLOADS_MEDIA_FROM_ME_EXCEPT_DISCORD_FILES",
            "destination": "media2",
            "usersAllWhitelisted": false,
            "userWhitelist": [ "MY_USER_ID" ],
            "saveImages": true,
            "saveVideos": true,
            "saveAudioFiles": true,
            "saveTextFiles": false,
            "saveOtherFiles": false,
            "domainBlacklist": [ "cdn.discordapp.com" ]
        },
        {
            "channel": "THIS_CHANNEL_IS_STEALTHY",
            "destination": "stealthy files",
            "allowCommands": false,
            "errorMessages": false,
            "updatePresence": false,
            "reactWhenDownloaded": false
        },
        {
            "channels": [ "CHANNEL_1", "CHANNEL_2", "CHANNEL_3", "CHANNEL_4", "CHANNEL_5" ],
            "destination": "lots of stuff",
            "allowCommands": false,
            "errorMessages": false,
            "updatePresence": false
        }
    ]
}

List of Settings

  • credentials [key/value object]
    • token [string]
      • Required for Bot Login or User Login with 2FA, don't include if using User Login without 2FA.
    • email [string]
      • Required for User Login without 2FA, don't include if using Bot Login.
    • password [string]
      • Required for User Login without 2FA, don't include if using Bot Login.
    • [DEFAULTS] userBot [bool]
      • Default: false
      • Set to true for a User Login with 2FA, keep as false if using a normal Bot.
    • [OPTIONAL] twitterAccessToken [string]
      • Won't use Twitter API for fetching media from tweets if credentials are missing.
    • [OPTIONAL] twitterAccessTokenSecret [string]
      • Won't use Twitter API for fetching media from tweets if credentials are missing.
    • [OPTIONAL] twitterConsumerKey [string]
      • Won't use Twitter API for fetching media from tweets if credentials are missing.
    • [OPTIONAL] twitterConsumerSecret [string]
      • Won't use Twitter API for fetching media from tweets if credentials are missing.
    • [OPTIONAL] flickrApiKey [string]
      • Won't use Flickr API for fetching media from posts/albums if credentials are missing.
    • [OPTIONAL] googleDriveCredentialsJSON [string]
      • Path for Google Drive API credentials JSON file.
      • Won't use Google Drive API for fetching files if credentials are missing.
  • [OPTIONAL] admins [array of strings]
    • Array of User ID strings for users allowed to use admin commands
  • [OPTIONAL] adminChannels [array of key/value objects]
    • channel [string]
  • [DEFAULTS] debugOutput [bool]
    • Default: false
    • Output debugging information.
  • [DEFAULTS] commandPrefix [string]
    • Default: "ddg "
  • [DEFAULTS] allowSkipping [bool]
    • Default: true
    • Allow scanning for keywords to skip content downloading.
    • "skip", "ignore", "don't save", "no save"
  • [DEFAULTS] scanOwnMessages [bool]
    • Default: false
    • Scans the bots own messages for content to download, only useful if using as a selfbot.
  • [DEFAULTS] filterDuplicateImages [bool]
    • Default: false
    • Experimental feature to filter out images that are too similar to other cached images.
    • Caching of image data is stored via a database file; it will not read all pre-existing images.
  • [DEFAULTS] filterDuplicateImagesThreshold [float64]
    • Default: 0
    • Threshold for what the bot considers too similar of an image comparison score. Lower = more similar (lowest is around -109.7), Higher = less similar (does not really have a maximum, would require your own testing).
  • [OPTIONAL] autorunHistory [bool]
    • Autorun history for all registered channels in background upon launch.
    • This can take anywhere between 2 minutes and 2 hours. It depends on how many channels your bot monitors and how many messages it has to go through. It can help to disable it by-channel for channels that don't require it (see overwriteAutorunHistory in channel options).
  • [DEFAULTS] downloadRetryMax [int]
    • Default: 3
  • [DEFAULTS] downloadTimeout [int]
    • Default: 60
  • [DEFAULTS] githubUpdateChecking [bool]
    • Default: true
    • Check for updates from this repo.
  • [DEFAULTS] presenceEnabled [bool]
    • Default: true
  • [DEFAULTS] presenceStatus [string]
    • Default: "idle"
    • Presence status type.
    • "online", "idle", "dnd", "invisible", "offline"
  • [DEFAULTS] presenceType [int]
    • Default: 0
    • Presence label type. ("Playing <activity>", "Listening to <activity>", etc)
    • Game = 0, Streaming = 1, Listening = 2, Watching = 3, Custom = 4
      • If Bot User, Streaming & Custom won't work properly.
  • [OPTIONAL] presenceOverwrite [string]
  • [OPTIONAL] presenceOverwriteDetails [string]
  • [OPTIONAL] presenceOverwriteState [string]
  • [DEFAULTS] filenameDateFormat [string]
  • [OPTIONAL] embedColor [string]
    • Unused by Default
    • Supports random/rand, role/user, or RGB in hex or int format (ex: #FF0000 or 16711680).
  • [OPTIONAL] inflateCount [int]
    • Unused by Default
    • Inflates the count of total files downloaded by the bot. I only added this for my own personal use to represent an accurate total amount of files downloaded by previous bots I used.
  • [DEFAULTS] numberFormatEuropean [bool]
    • Default: false
    • Formats numbers as 123.456,78/123.46k rather than 123,456.78/123,46k.
  • allChannels [key/value objects]
    • **Follow channels below for variables, except channel ID(s) are not necessary.
    • If a pre-existing config for the channel is not found, it will download from any and every channel it has access to, following these settings.
  • allChannelsBlacklist [array of strings]
    • Unused by Default
    • Blacklists channels (by ID) from allChannels.
  • channels [array of key/value objects]
    • [THIS OR BELOW] channel [string]
      • Channel ID to monitor.
    • [THIS OR ABOVE] channels [array of strings]
      • Channel IDs to monitor, for if you want the same configuration for multiple channels.
    • destination [string]
      • Folder path for saving files, can be full path or local subfolder.
    • [DEFAULTS] enabled [bool]
      • Default: true
      • Toggles bot functionality for channel.
    • [DEFAULTS] allowCommands [bool]
      • Default: true
      • Allow use of commands like ping, help, etc.
    • [DEFAULTS] errorMessages [bool]
      • Default: true
      • Send response messages when downloads fail or other download-related errors are encountered.
    • [DEFAULTS] scanEdits [bool]
      • Default: true
      • Check edits for un-downloaded media.
    • [OPTIONAL] overwriteAutorunHistory [bool]
      • Overwrite global setting for autorunning history for all registered channels in background upon launch.
    • [DEFAULTS] updatePresence [bool]
      • Default: true
      • Update Discord Presence when download succeeds within this channel.
    • [DEFAULTS] reactWhenDownloaded [bool]
      • Default: true
      • Confirmation reaction that file(s) successfully downloaded.
    • [OPTIONAL] reactWhenDownloadedEmoji [string]
      • Unused by Default
      • Uses specified emoji rather than random server emojis. Simply pasting a standard emoji will work, for custom Discord emojis use "name:ID" format.
    • [DEFAULTS] blacklistReactEmojis [array of strings]
      • Unused by Default
      • Block specific emojis from being used for reacts. Simply pasting a standard emoji will work, for custom Discord emojis use "name:ID" format.
    • [DEFAULTS] typeWhileProcessing [bool]
      • Default: true
      • Shows " is typing..." while processing things that aren't processed instantly, like history cataloging.
    • [OPTIONAL] overwriteFilenameDateFormat [string]
    • [OPTIONAL] overwriteAllowSkipping [bool]
      • Unused by Default
      • Allow scanning for keywords to skip content downloading.
      • "skip", "ignore", "don't save", "no save"
    • [OPTIONAL] overwriteEmbedColor [string]
      • Unused by Default
      • Supports random/rand, role/user, or RGB in hex or int format (ex: #FF0000 or 16711680).
    • [DEFAULTS] usersAllWhitelisted [bool]
      • Default: true
      • Allow messages from all users to be handled. Set to false if you wish to use userWhitelist to only permit specific users messages to be handled.
    • [OPTIONAL] userWhitelist [array of strings]
      • Use with usersAllWhitelisted as false to only permit specific users to have their messages handled by the bot. Only accepts User ID's in the array.
    • [OPTIONAL] userBlacklist [array of strings]
      • Use with usersAllWhitelisted as the default true to block certain users messages from being handled by the bot. Only accepts User ID's in the array.
    • [DEFAULTS] divideFoldersByServer [bool]
      • Default: false
      • Separate files into subfolders by server of origin (e.g. "My Server", "My Friends Server")
    • [DEFAULTS] divideFoldersByChannel [bool]
      • Default: false
      • Separate files into subfolders by channel of origin (e.g. "my-channel", "my-other-channel")
    • [DEFAULTS] divideFoldersByUser [bool]
      • Default: false
      • Separate files into subfolders by user who sent (e.g. "Me#1234", "My Friend#0000")
    • [DEFAULTS] divideFoldersByType [bool]
      • Default: true
      • Separate files into subfolders by type (e.g. "images", "video", "audio", "text", "other")
    • [DEFAULTS] saveImages [bool]
      • Default: true
    • [DEFAULTS] saveVideos [bool]
      • Default: true
    • [DEFAULTS] saveAudioFiles [bool]
      • Default: false
    • [DEFAULTS] saveTextFiles [bool]
      • Default: false
    • [DEFAULTS] saveOtherFiles [bool]
      • Default: false
    • [DEFAULTS] savePossibleDuplicates [bool]
      • Default: false
    • [DEFAULTS] extensionBlacklist [array of strings]
      • Default: [ ".htm", ".html", ".php", ".exe", ".dll", ".bin", ".cmd", ".sh", ".py", ".jar" ]
      • Ignores files containing specified extensions. Ensure you use proper formatting.
    • [OPTIONAL] domainBlacklist [array of strings]
      • Ignores files from specified domains. Ensure you use proper formatting.
    • [OPTIONAL] saveAllLinksToFile [string]
      • Saves all sent links to file, does not account for any filetypes or duplicates, it just simply appends every raw link sent in the channel to the specified file.

Presence Placeholders

For presenceOverwrite, presenceOverwriteDetails, presenceOverwriteState

Key Description
{{dgVersion}} discord-go version
{{ddgVersion}} Project version
{{apiVersion}} Discord API version
{{countNoCommas}} Raw total count of downloads (without comma formatting)
{{count}} Raw total count of downloads
{{countShort}} Shortened total count of downloads
{{numGuilds}} Number of guilds bot is a member of
{{numChannels}} Number of bound channels
{{numAdminChannels}} Number of admin channels
{{numAdmins}} Number of designated admins
{{timeSavedShort}} Last save time formatted as 3:04pm
{{timeSavedShortTZ}} Last save time formatted as 3:04pm MST
{{timeSavedMid}} Last save time formatted as 3:04pm MST 1/2/2006
{{timeSavedLong}} Last save time formatted as 3:04:05pm MST - January 2, 2006
{{timeSavedShort24}} Last save time formatted as 15:04
{{timeSavedShortTZ24}} Last save time formatted as 15:04 MST
{{timeSavedMid24}} Last save time formatted as 15:04 MST 2/1/2006
{{timeSavedLong24}} Last save time formatted as 15:04:05 MST - 2 January, 2006
{{timeNowShort}} Current time formatted as 3:04pm
{{timeNowShortTZ}} Current time formatted as 3:04pm MST
{{timeNowMid}} Current time formatted as 3:04pm MST 1/2/2006
{{timeNowLong}} Current time formatted as 3:04:05pm MST - January 2, 2006
{{timeNowShort24}} Current time formatted as 15:04
{{timeNowShortTZ24}} Current time formatted as 15:04 MST
{{timeNowMid24}} Current time formatted as 15:04 MST 2/1/2006
{{timeNowLong24}} Current time formatted as 15:04:05 MST - 2 January, 2006
{{uptime}} Shortened duration of bot uptime

Info for Developers

  • I'm a complete amateur with Golang. If anything's bad please make a pull request.
  • Versioning is [MAJOR].[MINOR].[PATCH]
  • I try to be consistent with annotation but it's not perfect.
  • Logging generally follows certain standards and patterns with formatting and color-coding.

Documentation

The Go Gopher

There is no documentation for this package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL