README
¶
Wallabag downloader

This tool is designed to automatically download Wallabag articles into your local computer or Kobo ebook reader.
Features:
- fast: downloads only files that have changed, in parallel
- unattended: runs in the background, when the wifi is turned on, only requires you to tap the fake USB connection screen for the Kobo to rescan its database
- status synchronization: read books are marked as read in the Wallabag instance
- easy install: just drop the magic file in your kobo reader like any other book, edit one configuration file and you're done
The instructions here are mostly for the Kobo E-readers but may work for other platforms. I have tested this on a Debian GNU/Linux 9 ("stretch") system, a Kobo Glo HD and a Kobo Touch.
Table of Contents
- Wallabag downloader
- Download and install
- Upgrade
- Configuration
- Usage
- Support
- Troubleshooting
- Known issues
- Credits
- Contributing
- Related projects

Download and install
Quick start for Kobo devices:
- connect your reader to your computer with a USB cable
- download the latest
KoboRoot.tgz
- save the file in the
.kobo
directory of your e-reader - create the configuration file as explained in the configuration section
- disconnect the reader
When you disconnect the reader, it will perform what looks like an
upgrade, but it's just the content of the KoboRoot.tgz
being
automatically deployed. If you connect the reader again, the
KoboRoot.tgz
file should have disappeared.
When you connect your reader to a Wifi access point, the wallabako
program should run, which should create a wallabako.log.txt
file at
the top directory of the reader which you can use to diagnose
problems, see also the troubleshooting section.
Another more advanced installation method, for power users, can done with launchers like KFmon or NickelMenu. See the design document, section "Launchers" for more information for this experimental configuration.
Upgrade
To upgrade wallabako, simply follow install instruction.
Configuration
The next step is to configure Wallabako by creating a .wallabako.js
file in the top directory of the reader, with the following content:
{
"WallabagURL": "https://app.wallabag.it",
"ClientId": "14_2vun20ernfy880wgkk88gsoosk4csocs4ccw4sgwk84gc84o4k",
"ClientSecret": "69k0alx9bdcsc0c44o84wk04wkgw0c0g4wkww8c0wwok0sk4ok",
"UserName": "joelle",
"UserPassword": "your super password goes here"
}
Make sure you use a plain text editor like Gedit
or Notepad
, as
LibreOffice can cause trouble because it doesn't save the file as
plain text by default.
Let's take this one step at a time. First, the weird curly braces
syntax is because this is a JSON
configuration file. Make sure you keep all the curly braces, quotes,
colons and commas ({
, }
, "
, :
, ,
).
-
The first item is the
WallabagURL
. This is the address of the service you are using. In the above example, we use the official Wallabag.it service, but this will change depending on your provider. Make sure there is no trailing slash (/
). -
The second and third items are the "client tokens". Those are tokens that you need to create in the Wallabag web interface, in the
Developer
section. Simply copy paste those values in place. -
The fourth and fifth items are your username and passwords. We would prefer to not ask you your password, but unfortunately, that is still required by the Wallabag API
Also note that some commandline flags are hardcoded in the
wallabag-run
script. To modify those, you will
need to modify the file in the KoboRoot.tgz
file or hack the kobo to
get commandline access. There are also more settings you can set in
the configuration file, see the troubleshooting
section for more information.
Note that there are more configuration settings that can be set, see the configuration file details section for more information.

Usage
Kobo devices
If everything was deployed correctly, Wallabako should run the next
time you activate the wireless connection on your device. You will
notice it is running because after a while, the dialog that comes up
when you connect your device with a cable will come up, even though
the device is not connected! Simply tap the Connect
button to
continue the synchronisation and the library will find the new entries.
When an article downloaded with Wallabako is finished on your reader, it will be marked as read in Wallabag.
Wallabako also downloads a limited numbers of articles from Wallabag, and it will remove extra articles (for example if they are too old or were marked as read in Wallabag).
By default, Wallabako will not delete old files from your reader - you will need to remove those files through the reader interface yourself. This is to avoid unnecessary synchronisations which are distracting to the user.
Commandline
Wallabako can also be compiled installed on a regular computer, provided that you have the go suite installed. Simply do the usual:
go get gitlab.com/anarcat/wallabako
If you are unfamiliar with go, you may want to read up on the
getting started instructions. If you do not wish to install golang
at all, you can also download the standalone binaries for
64 bits (aka amd64
or x86_64
) or ARM
(e.g. Raspberry PI).
You also need to create a configuration file as detailed above.
The program looks for the file in the following locations:
$HOME/.config/wallabako.js
$HOME/.wallabako.js
/mnt/onboard/.wallabako.js
/etc/wallabako.js
You will probably want to choose the first option unless you are configuring this as a system-level daemon.
Then you can just run wallabako on the commandline. For example, this
will download your articles in the epubs
directory in your home:
wallabako -output ~/epubs
Use the -h
flag for more information about the various flags you can
use on the commandline.
The program is pretty verbose, here's an example run:
$ wallabako -output /tmp
2017/01/31 22:16:41 logging in to https://example.net/wallabag
2017/01/31 22:16:41 CSRF token found: 200 OK
2017/01/31 22:16:41 logged in successful: 302 Found
2017/01/31 22:16:42 found 65 unread entries
2017/01/31 22:16:42 URL https://example.net/wallabag/export/23152.epub older than local file /tmp/23152.epub, skipped
2017/01/31 22:16:42 URL https://example.net/wallabag/export/23179.epub older than local file /tmp/23179.epub, skipped
2017/01/31 22:16:42 URL https://example.net/wallabag/export/23170.epub older than local file /tmp/23170.epub, skipped
2017/01/31 22:16:42 URL https://example.net/wallabag/export/23180.epub older than local file /tmp/23180.epub, skipped
2017/01/31 22:16:42 URL https://example.net/wallabag/export/23160.epub older than local file /tmp/23160.epub, skipped
2017/01/31 22:16:42 processed: 5, downloaded: 0
2017/01/31 22:16:42 completed in 1.44s
You can also run the program straight from source with:
go run *.go
Building for the Kobo requires more work, see the contribution guide for details.
On-screen display
Out of the box, wallabako does not know how to give user feedback on ebook readers like the Kobo. It can (and will), however, use the fbink program if it's found. If you have [koreader][] or kfmon installed, it should be able to find it there.
This feature will show wallabako's log in a crude overlay. By default, it piles them up on two lines at the bottom of the screen. This is less intrusive but can be hard to read.
To get the full experience, try to enable FbinkInteractive
in the
configuration file, which will scroll all the messages, one per line,
from the top of the display, completely clobbering the current
display. This can be extremely distracting which is why it's off by
default.
Even without that setting, fbink
will be used to display messages,
if available. The difference is fbink
will be called repeatedly,
clobbering previous messages but limiting the amount of text flooding
the display.
And, of course, if you haven't modified your reader far enough to have
fbink
actually installed, you'll never see that good stuff. If you
want to only install fbink
(without koreader of kfmon), it seems
the canonical installation instructions are in this forum post.
Support
I will provide only limited free support for this tool. I wrote it, after all, for my own uses. People are welcome to file issues and send patches, of course, but I cannot cover for every possible use cases. There is also a discussion on MobileRead.com if you prefer that format.
Troubleshooting
To troubleshoot issues with the script, you may need to get commandline access into it, which is beyond the scope of this documentation. See the following tutorial for example.
Below are issues and solutions I have found during development that you may stumble upon. Normally, if you install the package correctly, you shouldn't get those errors so please do file a bug if you can reproduce this issue.
Table of Contents
- Logging
- Configuration file details
- Configuration file is not found even if present
- Some articles are not downloaded or disappear
- Unable to open database file
- x509: failed to load system roots and no roots provided
- Command not running
Logging
Versions from 0.3 to 1.0 were writing debugging information in the
wallabako.log.txt
on the reader. This is now disabled by default
(see this discussion for why) but can be enabled again by adding a
logfile
option in the configuration file, like this:
{
"WallabagURL": "https://app.wallabag.it",
"ClientId": "14_2vun20ernfy880wgkk88gsoosk4csocs4ccw4sgwk84gc84o4k",
"ClientSecret": "69k0alx9bdcsc0c44o84wk04wkgw0c0g4wkww8c0wwok0sk4ok",
"UserName": "joelle",
"UserPassword": "your super password goes here",
"logfile": "/mnt/onboard/wallabako.log.txt"
}
This will make a wallabako.log
file show up on your reader that you
can check to see what's going on with the command.
You can increase the verbosity of those logs with the Debug
commandline flag or configuration option (set to true
, without
quotes). WARNING: this will include your password and authentication
tokens, so be careful where you send this output.
Configuration file details
Most commandline options (except -version
and -config
) can also be
set in the configuration file. Here are the configuration options and
their matching configuration file settings:
Configuration | Flag | Default | Meaning |
---|---|---|---|
Debug |
-debug |
false |
include (lots of!) additional debugging information in logs, including passwords and confidential data |
Delete |
-delete |
false |
delete EPUB files marked as read or missing from Wallabag |
Database |
-database |
/mnt/onboard/.kobo/KoboReader.sqlite |
path to the Kobo database |
Concurrency |
-concurrency |
2 | number of downloads to process in parallel |
Count |
-count |
-1 | number of articles to fetch, -1 means use Wallabag default |
Exec |
-exec |
nothing | execute the given command when files have changed |
LogFile |
-logfile |
no logging | rotated logfile to store debug information |
OutputDir |
-output |
current directory | output directory to save files into |
PidFile |
-pidfile |
wallabako.pid |
pidfile to write to avoid multiple runs |
RetryMax |
-retry |
4 | number of attempts to login the website, with exponential backoff delay |
Timeout |
-timeout |
300 | timeout for HTTP requests, in seconds |
Tags |
-tags |
no tags filtering | a comma-separated list of tags to filter for |
Plato.LibraryPath |
N/A | /mnt/onboard |
For plato document reader integration, the value of [[libraries.path]] in Settings.toml |
Fbink |
N/A | false |
use fbink to overlay logs directly on the kobo screen, can be noisy |
FbinkInteractive |
N/A | false |
use full screen interactive fbink mode |
Some more details about specific settings:
-
The
Count
option actually defaults to 30 in Wallabag, at the time of writing. You may want to bump that up if you have more than 30 unread articles, see below for details. -
The
PidFile
is actually written in one of those directories, the first one found that works:/var/run
/run
/run/user/UID
/home/USER/.
-
The
Fbink
settings only work iffbink
is installed and available in the$PATH
, wallabako doesn't ship fbink itself.
Finally, note that some of those settings are hardcoded in the
wallabako-run
wrapper script and therefore cannot be overridden in
the configuration file. Those are:
Flag | Value |
---|---|
-output |
/mnt/onboard/wallabako |
-exec |
/usr/local/bin/fake-connect-usb |
Changing those settings could be dangerous. In particular, changing
the -output
directory while enabling -delete
could delete files
unexpectedly if they match the magic pattern (N.epub
where N is an
integer).
Here's an example of a fully-populated configuration file:
{
"Concurrency": 2,
"Count": -1,
"ClientId": "14_2vun20ernfy880wgkk88gsoosk4csocs4ccw4sgwk84gc84o4k",
"ClientSecret": "69k0alx9bdcsc0c44o84wk04wkgw0c0g4wkww8c0wwok0sk4ok",
"Debug": false,
"Delete": false,
"Database": "/mnt/onboard/.kobo/KoboReader.sqlite",
"Exec": "/usr/local/bin/fake-connect-usb",
"Fbink": false,
"FbinkInteractive": false,
"LogFile": "/mnt/onboard/wallabako.log.txt",
"OutputDir": "/mnt/onboard/wallabako",
"PidFile": "wallabako.pid",
"Plato": {
"LibraryPath": "/mnt/onboard"
},
"RetryMax": 4,
"Tags": "",
"UserName": "joelle",
"UserPassword": "your super password goes here",
"WallabagURL": "https://app.wallabag.it"
}
Configuration file is not found even if present
This can happen if you have some sort of a syntax error in the configuration file. For example, this can happen if you have a double-quote in your password and you didn't properly escape it.
The configuration file is a JSON file, parsed by the Unmarshal function of the Golang json package. Wikipedia says that:
JSON's basic data types are: [...] String: a sequence of zero or more Unicode characters. Strings are delimited with double-quotation marks and support a backslash escaping syntax.
This means that if you have a password that is, say foo"bar
, you can
escape the special character with a backslash, like this:
"UserPassword": "foo\"bar",
According to the JSON specification, strings should be delimited
by double-quote ("
) and unless you have weird control characters in
your passwords (e.g. tab or formfeed), double-quote and backslash are
the only characters you should need to escape.
Another common error is to add an extra comma (,
) on the final
entry, or omitting the brackets ({
or }
). Files with
BOM markers used to cause issues as well, but that has been fixed
in the Wallabago library since 0.7.
Some articles are not downloaded or disappear
If you can't seem to synchronize all your articles and you have a
large number of unread articles, you may want to change the Count
field in the configuration file. By default, Wallabako only downloads
a part of the database: it is limited by the number of articles
returned by the Wallabag listing (30
at the time of writing).
Also, if the Delete
option is set, older articles will be deleted
from the Kobo reader as well.
Note that it should be fairly safe to use a larger number here, as
only Concurrency
(e.g. 6) articles will be downloaded in parallel at
a time. It could make the first listing request slower, however, if
you have a huge number of articles. We have reports of operation with
60 articles without significant performance issues.
Unable to open database file
If you see this warning message repeated:
2017/03/01 21:06:49 unable to open database file
It is because the database cannot be found. By default, the database
path is hardcoded to /mnt/onboard/.kobo/KoboReader.sqlite
, which is
likely to work only on Kobo readers. If you are running this on your
desktop or another reader, you should disable the database by using
the -database
flag:
wallabako -database ""
... or configuration option in .wallabako.js
:
"Database": "",
Such configuration should silence those warnings as Wallabako will not attempt to open a database file.
Note that the warnings can also safely be ignored.
x509: failed to load system roots and no roots provided
You may see this error when running on weird environments;
2017/01/30 14:45:46 logging in to https://example.net/wallabag
2017/01/30 14:45:51 <nil> Get https://example.net/wallabag/login: x509: failed to load system roots and no roots provided
2017/01/30 14:45:51 completed in 5.12s
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x11280]
goroutine 1 [running]:
panic(0x2558a0, 0x1061e008)
/usr/lib/go-1.7/src/runtime/panic.go:500 +0x33c
main.login(0x1060ee20, 0x19, 0x1067d958, 0x7, 0x1060ee60, 0x14, 0x0)
/home/anarcat/go/src/gitlab.com/anarcat/wallabako/main.go:59 +0x280
main.main()
/home/anarcat/go/src/gitlab.com/anarcat/wallabako/main.go:147 +0x280
This is because your operating system doesn't ship standard X509
certificates in the location the program expects them to be. A
workaround I have found is to copy the
/etc/ssl/certs/ca-certificates.crt
provided by the ca-certificates
package in Debian in the machine.
By default, the tarball creation script adds that magick file to the
KoboRoot.tgz
archive, which should work around this problem. But
this was never tested from scratch.
Note: it may be possible to fix the program to ignore the SystemRootsError but I would advise against it, if only for obvious security reasons...
Command not running
If you notice that udev is not running your command, for some reason,
you can restart it with --debug
which is very helpful. Example:
[root@(none) ~]# ps ax | grep udev
621 root 0:00 /sbin/udevd -d
1242 root 0:00 grep udev
[root@(none) ~]# kill 621
[root@(none) ~]# /sbin/udevd --debug
[1256] parse_file: reading '/lib/udev/rules.d/50-firmware.rules' as rules file
[1256] parse_file: reading '/lib/udev/rules.d/50-udev-default.rules' as rules file
[1256] parse_file: reading '/lib/udev/rules.d/60-persistent-input.rules' as rules file
[1256] parse_file: reading '/lib/udev/rules.d/75-cd-aliases-generator.rules' as rules file
[1256] parse_file: reading '/etc/udev/rules.d/90-wallabako.rules' as rules file
[1256] parse_file: reading '/lib/udev/rules.d/95-udev-late.rules' as rules file
[1256] parse_file: reading '/lib/udev/rules.d/kobo.rules' as rules file
[...]
[1276] util_run_program: '/usr/local/bin/wallabako-run' (stdout) '2017/01/31 00:03:50 logging in to https://example.net/wallabag'
[1256] event_queue_insert: seq 859 queued, 'remove' 'module'
[1256] event_fork: seq 859 forked, pid [1289], 'remove' 'module', 0 seconds old
[1276] util_run_program: '/usr/local/bin/wallabako-run' (stdout) '2017/01/31 00:03:50 failed to get login page:Get https://example.net/wallabag/login: dial tcp: lookup example.net on 192.168.0.1:53: dial udp 192.168.0.1:53: connect: network is unreachable'
In the above case, network is down, probably because the command ran
too fast. You can adjust the delay in wallabako-run
, but really this
should be automated in the script (which should retry a few times
before giving up).
Uninstalling
Unfortunately, there is no easy way to uninstall Wallabako (bug 36). While Kobo
provides a way to automatically install files on the readers, there
is no (at least obvious) way of removing those files once they are
installed. To cleanup the install, you need to manually remove the
files provided in the KoboRoot.tgz
file, which are, at the time of
writing:
etc/udev/rules.d/90-wallabako.rules
etc/wallabako.js
etc/ssl/certs/ca-certificates.crt
usr/local/bin/wallabako
usr/local/bin/fake-connect-usb
usr/local/bin/wallabako-run
Of those, the one likely to cause problems is possibly only
/etc/udev/rules.d/90-wallabako.rules
, and only because it calls
/usr/local/bin/wallabako-run
which in turns calls
/usr/local/bin/fake-connect-usb
. So removing the .rules
file
should be enough to keep Wallabako from starting at all.
If you cannot access those files directly (which is likely, unless you
have SSH access or have KSM installed), a workaround disable
the configuration file. Remove the .wallabako.js
configuration file
at the root of your Kobo filesystem: this will "unconfigure" Wallabako
which will still fire automatically when wifi comes up, but it will do
nothing, which shouldn't cause problems anymore.
Known issues
Like any program, Wallabako is imperfect. There is a list of known issues on the main site, but some issues do not depend on Wallabako, as they are problems with Wallabag itself. Most of the issues we have found in Wallabag are documented in issue #2821.
Credits
Wallabako was written by The Anarcat and reviewed by friendly Debian
developers juliank
and stapelberg
. smurf
also helped in
reviewing the code and answering my million newbie questions about go.
Also thanks to Norbert Preining for pulishing the Kobo firmware images that got me started into this path and allowed me to easily root my reader. This inspired me to start the related kobo-ssh project to build smaller, ssh-only images.
This program and documentation is distributed under the AGPLv3 license, see the LICENSE file for more information.
Contributing
See the contribution guide for more information. In short: this is a free software project and you are welcome to join us in improving it, both by fixing things, reporting issues or documentation.
Design notes
Moved a separate document.
Remaining issues
There are XXX
markers in the source code that show other issues that
need to be checked. The other known issues previously stored in this
file have been moved to the Gitlab issue queue to allow for better
visibility and public collaboration.
Related projects
Other Kobo-related software has support for Wallabako, and may be easier to use than this program.
- KOReader includes a native plug-in that fully integrates with Wallabag servers. The KOReader wiki has installation instructions and details about Wallabag integration. That implementation is discussed in the design document as well.
- Plato also includes an article fetcher with support for Wallabag
Note that Wallabako will also automatically fetch read/unread statuses Plato and Koreader metadata, on top of the built-in Nickel interface shipped with the Kobo.
Documentation
¶
Overview ¶
Parse read status from the Nickel UI database.
Nickel is Kobo's builtin and proprietary UI which stores book details in a SQLite database.
We refer to it as "Nickel" here because that's the internal name used by Kobo. This is to distinguish this UI from the Kobo *device* itself, which also happens to run other UIs and programs like koreader, Plato, etc.
The code simply reads the book status from the sqlite database.
Note that this file relies on an external variable named wallabakoSqliteBackend which is defined in one of nickel_*.go files, depending on compile-time tags. Those files are also responsible for importing the correct modules.