skicka

README

skicka

Utility for working with files and folders stored on Google Drive.

Note: skicka is not an official Google product!

Intro

skicka makes it easy to copy files to and from Google Drive and to work with files stored on Google Drive.

For example, skicka upload ~/Pictures /Pictures copies the entire contents of the local ~/Pictures directory to a folder Pictures in Google Drive. If you then run skicka download /Pictures ~/Pictures2, then the contents of your ~/Pictures2 directory will match the contents of ~/Pictures.

More generally, skicka makes it easy to list the files in Google Drive folders, compute the space used by Drive folders, and copy files between your computer to Google Drive. If you'd like to encrypt your files before uploading them, skicka supports AES-256 encryption.

What skicka is not

skicka is not a general solution for automatically synchronizing files stored in Google Drive across multiple machines. In particular, it doesn't have logic to reconcile concurrent changes to the same file on different machines.

Furthermore, although skicka has been robust in usage so far and has no known data corruption bugs, it should for now be treated as "alpha" software. Bug reports are welcome.

Getting Started

1. You must have a Go compiler installed.
2. Download and build skicka: go get github.com/google/skicka
3. Either copy the skicka executable in $GOPATH/bin to a directory in your PATH, or add $GOPATH/bin to your PATH.
4. Run skicka init to create a skeleton ~/.skicka.config file. Various options can be set in this file to control skicka's operation; see comments in the file for more information.
5. Authorize skicka to access your Google Drive files: run skicka ls, and skicka will attempt to open an authorization page in your web browser. Click the "Accept" button to authorize skicka. You only need to perform this step once.
   • Alternatively, you can authorize skicka by running skicka -no-browser-auth ls and skicka will ask you to visit a URL: go to the URL in a browser, log into your Google account (if needed), and give permission for the application to access your Google Drive files. After you click 'accept', copy the code from your browser window to the terminal with the "Enter verification code" prompt from skicka.
6. After skicka is authorized, it will download and locally cache information about the files you have stored in Google Drive. This may take a while if you have many files on Drive; a progress indicator will try to keep you posted about how this is going. Once this data is stored locally, future runs of skicka will get going much more quickly.
7. If you're going store encrypted files in Google Drive, create an encryption key: set the environment variable SKICKA_PASSPHRASE to hold your passphrase and run skicka genkey. Copy the lines of text that are printed to the [encryption] section of your ~/.skicka.config file.
8. Try it out: run skicka ls -l /. A list of the files and folders in the root directory of your Google Drive should be printed.
9. Join the mailing list!

Usage

For a general overview of skicka's commands, run skicka help.

To copy a local directory hierarchy to Google Drive, use the upload command. As the upload progresses, skicka periodically reports how much data has been uploaded and how much time has elapsed.

% skicka upload ~/Pictures /Pictures
11 / 11 [=====================================================] 100.00 % 
skicka: preparation time 1s, sync time 10s
skicka: updated 0 Drive files, 8 local files
skicka: 4.52 MiB read from disk, 0 B written to disk
skicka: 4.52 MiB uploaded (482 kiB/s), 0 B downloaded (0 B/s)
skicka: 32.76 MiB peak memory used
%

If you'd like to encrypt the files, create an encryption key as described above in "Getting Started" and then use the -encrypt command-line option. skicka looks for your encryption passphrase in the SKICKA_PASSPHRASE environment variable; thus, you might use:

% env SKICKA_PASSPHRASE=mySecretPassphrase skicka upload -encrypt ~/Pictures /EncryptedPictures

To download a directory hierarchy from Google Drive to your local disk, use download. As the download progresses, status is periodically reported:

% skicka download /Pictures/2014 ~/Pictures.copy/2014
10 / 10 [=====================================================] 100.00 % 
skicka: preparation time 1s, sync time 6s
skicka: updated 0 Drive files, 10 local files
skicka: 0 B read from disk, 16.18 MiB written to disk
skicka: 0 B uploaded (0 B/s), 16.18 MiB downloaded (2.33 MiB/s)
skicka: 50.23 MiB peak memory used
%

The ls command can be used to list files and directories in Google Drive. For example, after uploading your ~/Pictures directory, you might run:

% skicka ls /Pictures
2012
2013
2014

To see more detail, ls -l can be used:

% skicka ls -l /Pictures/2013
-rw-r--r--  2.62 MiB  Sun Mar 10 14:41:08 2013  IMG_1127.JPG
-rw-r--r--  2.63 MiB  Sun Mar 10 14:41:09 2013  IMG_1128.JPG
-rw-r--r--  2.32 MiB  Sun Mar 10 14:41:16 2013  IMG_1129.JPG
-rw-r--r--  2.47 MiB  Sun Mar 10 14:43:16 2013  IMG_1130.JPG
[...]

In addition to the filename, we see the file size, the file permissions, and the local modification time that the file on Google Drive is synced to. The file permissions are based on the permissions of the file when it was uploaded to Google Drive.

(For even more detail, ls -ll can be used, which also prints the MD5 checksums of the files on Google Drive.)

The contents of a single file can be downloaded using cat:

% skicka cat /Pictures/2013/IMG_1129.JPG > img.jpg

Google Drive folders can be created with mkdir. (As with the Unix command, the -p option can be specified to indicate that the intermediate directories in the path should be created).

% skicka mkdir /Pictures/2015

The amount of space used by the children of a given Google Drive folder can be reported by du:

% skicka du /Pictures
 81.56 MiB  /Pictures/2012
 92.14 MiB  /Pictures/2013
121.02 MiB  /Pictures/2014
294.72 MiB  /Pictures

You can remove a file with rm; by default, files are moved to the Trash in Google Drive:

% skicka rm /Pictures/2014/embarassing.jpg

If you'd like to delete the file immediately with no chance to recover it, use the -s command-line option to rm. To remove a folder and everything inside it, use -r.

Finally, there is a fsck command that checks the file system on Google Drive for problems and verifies that the local cache of file metadata is in-sync with the files stored on Drive.

FAQs

Can skicka work with Google Drive files that it didn't create itself?

Yes. The only limitations are that regular Google Drive files don't store the Unix permissions or the local modification of the original file when it was uploaded. Therefore, in this case skicka download uses 644 permissions for files it creates and 755 permissions for directories.

skicka ls -l indicates that a file has world-readable permissions on Google Drive; does this mean anyone can access it?

No. Those permissions are only used to set the local file permissions when the file is downloaded with skicka download. The access permissions for the files stored on Drive are handled with Drive's regular mechanisms.

How can I speed up uploads?

There's a fixed per-file overhead for each file uploaded to Google Drive that limits skicka to creating roughly five files a second; if files are relatively small, this overhead will be more of a limit than the time spent transferring the contents of the files.

If the uploaded files don't need to be accessed individually, creating a tar or zip archive of them before uploading may help in this case.

I occasionally see "operation timed out" or "broken pipe" errors when uploading; what's going on?

A variety of transient errors can happen when using RESTful APIs like the Google Drive API. When these errors are encountered, skicka makes a number of attempts to retry the operation before giving up.

It may be that skicka should make more attempts before giving up or that there are better error handling strategies; one trade-off is that if there is a serious error (like the internet connection is lost), then it's useful for the user to know this sooner rather than later.

If you do see these errors, re-run the operation you were performing; any files that weren't transferred the first time should be taken care of with a second run.

Does skicka support rate-limited uploads and downloads?

Yes. By default, skicka doesn't try to limit its bandwidth usage. However, if you add a line bytes-per-second-limit=... to the [upload] or [download] section of your .skicka.config file, you can specify a maximum number of bytes per second to transfer for uploads or downloads, respectively.

If this line isn't present (or has a value of zero), then bandwidth won't be limited.

Can an http proxy be used with skicka?

Yes--just set the HTTP_PROXY environment variable appropriately.

"skicka"?!?

Swedish, "to send".

Implementation notes

How files are stored in Google Drive

When a directory hierarchy is uploaded, Google Drive file is created for each local file and a Google Drive folder is created for each local directory. skicka stores the time the local copy file was last modified in the "modifiedDate" Google Drive File resource. The Unix file permissions of the file or directory are stored in using a custom "Permissions" file property, stored as a string with the octal file permissions.

See the discussion of encryption below for details about how encrypted files are represented.

Synchronization algorithm

When deciding if a local file needs to be uploaded to Google Drive, skicka performs the following checks.

1. If there is no corresponding file on Drive, the local file will be uploaded.
2. Otherwise, if there is a corresponding file on Drive and the sizes of the files are different, the local file will be uploaded.
3. Otherwise, if the local file's modification time is after the modification time of the file the last time it was synced to Drive, then an MD5 checksum of the file contents is computed. If this checksum differs from the checksum of the file stored on Google Drive, the file will be uploaded. (Thus, if touch is run on a file to update its modification time but the file's contents aren't modified, skicka won't unnecessarily re-upload the file.)

Note that skicka trusts that file modification times are meaningful: if a file's contents are modified leaving the file size is unchanged and if the modification time is set to be in the past, then skicka won't compute an MD5 checksum and won't know that the file should be uploaded. To override this behavior, run skicka upload with the -ignore-times flag; if this flag is provided, then the MD5 checksum check in the third step will be applied regardless of the file modification time.

Note also that this algorithm is an algorithm to efficiently mirror the contents of a set of local files on Google Drive; it's not a general bidirectional synchronization algorithm. For example, if a file is modified both on Drive and on the local filesystem, a skicka upload run will clobber the file contents on Drive. In other words, the assumption is that the source directory hierarchy is by definition the canonical one and the destination directory's role is to perfectly reflect the source.

When downloading from Google Drive, skicka follows the same general approach: only files that don't yet exist, have different sizes, or different MD5 checksums from the corresponding local file will be downloaded. The -ignore-times option can also be used to bypass the file modification time check and to force a comparison of file contents to decide whether to download.

Encryption

If the -encrypt flag is provided to the upload command, skicka will encrypt the contents of each file before uploading it to Google Drive. Conversely, the download and cat commands transparently decrypt encrypted files when downloading them (if the encryption key and passphrase are available!)

Encryption requires both an encryption key and a passphrase; your passphrase should be stored in the SKICKA_PASSPHRASE environment variable. To generate an encryption key, run skicka genkey. For example:

% env SKICKA_PASSPHRASE=mySecretPassphrase skicka genkey
[...]

skicka will generate a few lines of output that are your encrypted encryption key; to save your key, edit your ~/.skicka.config file and add the printed text to the [encryption] section of the file. You can now upload and encrypt files:

% env SKICKA_PASSPHRASE=mySecretPassphrase skicka upload -encrypt ~/backups /backups

If you lose your encryption key or forget your passphrase, your encrypted data is lost forever. If you use encryption, please be very careful with this information.

skicka only encrypts file contents: it doesn't encrypt filenames or hide file sizes, for example. If this is problematic for your usage, you should probably use tar or zip to put the files you want to encrypt into a single file before uploading it.

Key generation and storage

When skicka genkey is executed, an encryption key is generated as follows:

1. skicka generates a random 32-byte salt using Go's rand.Reader, which returns cryptographically secure pseudo-random numbers. The hex-encoded salt is printed out, and should be recorded in the salt field of the [encryption] section of the config file.
2. The user's passphrase, read from the SKICKA_PASSPHRASE environment variable, is run through the PBKDF2 key derivation function, using 65536 iterations and the SHA-256 hash function to derive a 64-byte hash.
3. The first 32 bytes of the hash are hex encoded and printed out; they should be copied to the passphrase-hash field of the [encryption] section of the config file. These bytes are later used only to validate that the user has provided the correct passphrase on subsequent runs of skicka.
4. A random 32-byte encryption key is generated (again with rand.Reader). This is the key that will actually be used for encryption and decryption of file contents.
5. A random 16-byte initialization vector is generated with rand.Reader. It is hex encoded and printed out, and should be copied to the encrypted-key-iv configuration file field.
6. The encryption key from #4 is encrypted using the initialization vector from #5, using the second 32 bytes of the hash computed in #2 as the encryption key. The result should be copied to the encrypted-key field of the config file.

Upon subsequent runs of skicka, the salt is loaded from the config file so that PBKDF2 can be used as in #2 above to hash the user's passphrase. If the first 32 bytes of the passphrase hash match the stored bytes, then the second 32 bytes of the hash and the stored IV are used to decrypt the encrypted encryption key.

Given the encryption key, when a file is to be encrypted before being uploaded to Google Drive, skicka uses the key along with a fresh 16-byte initialization vector for each file to encrypt the file using AES-256. The initialization vector is prepended to the file contents before upload. (Thus, encrypted files are 16 bytes larger on Google Drive than they are locally.)

The initialization vector is also stored hex-encoded as a Google Drive file Property with the name "IV". We store the initialization vector redundantly so that if one downloads the encrypted file contents, it's possible to decrypt the file using the file contents (and the key!) alone. Conversely, also having the IV available in a property makes it possible to encrypt the contents of a local version of a file without needing to download any of the contents of the corresponding file from Google Drive.