proton_api_bridge

package module
v0.0.0-...-45a3c93 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 22, 2024 License: MIT Imports: 28 Imported by: 0

README

Proton API Bridge

Thanks to Proton open sourcing proton-go-api and the web, iOS, and Android client codebases, we don't need to completely reverse engineer the APIs by observing the web client traffic!

proton-go-api provides the basic building blocks of API calls and error handling, such as 429 exponential back-off, but it is pretty much just a barebone interface to the Proton API. For example, the encryption and decryption of the Proton Drive file are not provided in this library.

This codebase, Proton API Bridge, bridges the gap, so software like rclone can be built on top of this quickly. This codebase handles the intricate tasks before and after calling Proton APIs, particularly the complex encryption scheme, allowing developers to implement features for other software on top of this codebase.

Currently, only Proton Drive APIs are bridged, as we are aiming to implement a backend for rclone.

Sidenotes

We are using a fork of the proton-go-api, as we are adding quite some new code to it. We are actively rebasing on top of the master branch of the upstream, as we will try to commit back to the upstream once we feel like the code changes are stable.

Unit testing and linting

golangci-lint run && go test -race -failfast -v ./...

Drive APIs

In collaboration with Azimjon Pulatov, in memory of our good old days at Meta, London, in the summer of 2022.

Thanks to Anson Chen for the motivation and some initial help on various matters!

Currently, the development are split into 2 versions. V1 supports the features required by rclone, such as file listing. As the unit and integration tests from rclone have all been passed, we would stabilize this and then move onto developing V2. V2 will bring in optimizations and enhancements, esp. supporting thumbnails. Please see the list below.

V1

Features
  • Log in to an account without 2FA using username and password
  • Obtain keyring
  • Cache access token, etc. to be able to reuse the session
    • Bug: 403: Access token does not have sufficient scope - used the wrong newClient function
  • Volume actions
    • List all volumes
  • Share actions
    • Get all shares
    • Get default share
  • Fix context with proper propagation instead of using ctx everywhere
  • Folder actions
    • List all folders and files within the root folder
      • BUG: listing directory - missing signature when there are more than 1 share -> we need to check for the "active" folder type first
    • List all folders and files recursively within the root folder
    • Delete
    • Create
    • (Feature) Update
    • (Feature) Move
  • File actions
    • Download
      • Download empty file
      • Improve large file download handling
      • Properly handle large files and empty files (check iOS codebase)
        • esp. large files, where buffering in-memory will screw up the runtime
      • Check signature and hash
    • Delete
    • Upload
      • Handle empty file
      • Parse mime type
      • Add revision
      • Modified time
      • Handle failed / interrupted upload
    • List file metadata
  • Duplicated file name handling: 422: A file or folder with that name already exists (Code=2500, Status=422)
  • Init ProtonDrive with config passed in as Map
  • Remove all log.Fatalln and use proper error propagation (basically remove HandleError and we go from there)
  • Integration tests
    • Remove drive demo code
    • Create a Drive struct to encapsulate all the functions (maybe?)
    • Move comments to proper places
    • Modify shouldRejectDestructiveActions()
    • Refactor
  • Reduce config options on caching access token
  • Remove integration test safeguarding
TODO
  • address go dependencies
    • Fixed by doing the following in the go-proton-api repo to bump to use the latest commit
      • go get github.com/ProtonMail/go-proton-api@ea8de5f674b7f9b0cca8e3a5076ffe3c5a867e01
      • go get github.com/ProtonMail/gluon@fb7689b15ae39c3efec3ff3c615c3d2dac41cec8
  • Remove mail-related apis (to reduce dependencies)
  • Make a "super class" and expose all necessary methods for the outside to call
  • Add 2FA login
  • Fix the function argument passing (using pointers)
  • Handle account with
    • multiple addresses
    • multiple keys per addresses
  • Update RClone's contribution.md file
  • Remove delete all's hardcoded string
  • Point to the right proton-go-api branch
    • Run go get github.com/henrybear327/go-proton-api@dev to update go mod
  • Pass in AppVersion as a config option
  • Proper error handling by looking at the return code instead of the error string
    • Duplicated folder name handling: 422: A file or folder with that name already exists (Code=2500, Status=422)
    • Not found: ERROR RESTY 422: File or folder was not found. (Code=2501, Status=422), Attempt 1
    • Failed upload: Draft already exists on this revision (Code=2500, Status=409)
  • Fix file upload progress -> If the upload failed, please Replace file. If the upload is still in progress, replacing it will cancel the ongoing upload
  • Concurrency control on file encryption, decryption, and block upload
Known limitations
  • No thumbnails, respecting accepted MIME types, max upload size, can't init Proton Drive, etc.
  • Assumptions
    • only one main share per account
    • only operate on active links

V2

  • Support thumbnail
  • Potential bugs
    • Confirm the HMAC algorithm -> if you create a draft using integration test, and then use the web frontend to finish the upload (you will see overwrite pop-up), and then use the web frontend to upload again the same file, but this time you will have 2 files with duplicated names
    • Might have missing signature issues on some old accounts, e.g. GetHashKey on rootLink might fail -> currently have a quick patch, but might need to double check the behavior
    • Double check the attrs field parsing, esp. for size
    • Double check the attrs field, esp. for size
  • Crypto-related operations, e.g. signature verification, still needs to cross check with iOS or web open source codebase
  • Mimetype detection by using the file content itself, or Google content sniffer
  • Remove e.g. proton.link related exposures in the function signature (this library should abstract them all)
  • Improve documentation
  • Go through Drive iOS source code and check the logic control flow
  • File
  • Commit back to proton-go-api and switch to using upstream (make sure the tag is at the tip though)
  • Support legacy 2-password mode
  • Proton Drive init (no prior Proton Drive login before -> probably will have no key, volume, etc. to start with at all)
  • linkID caching -> would need to listen to the event api though
  • Integration tests
    • Check file metadata
    • Try to check if all functions are used at least once so we know if it's functioning or not
  • Handle accounts with multiple shares
  • Use CI to run integration tests
  • Some error handling from here MAX_NAME_LENGTH, TIMEOUT
  • Mimetype restrictions
  • Address TODO and FIXME

Questions

  • rclone's folder / file rename detection? -> just implement the interface and rclone will deal with the rest!

Notes

  • Due to caching, functions using ...ByID needs to perform protonDrive.removeLinkIDFromCache(linkID, false) in order to get the latest data!

Documentation

Index

Constants

This section is empty.

Variables

View Source
var (
	LIB_VERSION = "1.0.0"

	UPLOAD_BLOCK_SIZE       = 4 * 1024 * 1024 // 4 MB
	UPLOAD_BATCH_BLOCK_SIZE = 8
	/*
		https://github.com/rclone/rclone/pull/7093#issuecomment-1637024885

		The idea is that rclone performs buffering / pre-fetching on it's own so
		we don't need to be doing this on our end.

		If you are not using rclone and instead is directly basing your work on this
		library, then maybe you can increase this value to let the library does
		the buffering work for you!
	*/
	DOWNLOAD_BATCH_BLOCK_SIZE = 1
)
View Source
var (
	ErrMainSharePreconditionsFailed          = errors.New("the main share assumption has failed")
	ErrDataFolderNameIsEmpty                 = errors.New("please supply a DataFolderName to enabling file downloading")
	ErrLinkTypeMustToBeFolderType            = errors.New("the link type must be of folder type")
	ErrLinkTypeMustToBeFileType              = errors.New("the link type must be of file type")
	ErrFolderIsNotEmpty                      = errors.New("folder can't be deleted because it is not empty")
	ErrCantLocateRevision                    = errors.New("can't create a new file upload request and can't find an active/draft revision")
	ErrInternalErrorOnFileUpload             = errors.New("either link or file creation request should be not nil")
	ErrMissingInputUploadAndCollectBlockData = errors.New("missing either session key or key ring")
	ErrLinkMustNotBeNil                      = errors.New("missing input proton link")
	ErrLinkMustBeActive                      = errors.New("can not operate on link state other than active")
	ErrDownloadedBlockHashVerificationFailed = errors.New("the hash of the downloaded block doesn't match the original hash")
	ErrDraftExists                           = errors.New("a draft exist - usually this means a file is being uploaded at another client, or, there was a failed upload attempt. Can use --protondrive-replace-existing-draft=true to temporarily override the existing draft")
	ErrCantFindActiveRevision                = errors.New("can't find an active revision")
	ErrCantFindDraftRevision                 = errors.New("can't find a draft revision")
	ErrWrongUsageOfGetLinkKR                 = errors.New("internal error for GetLinkKR - nil passed in for link")
	ErrWrongUsageOfGetLink                   = errors.New("internal error for getLink - empty linkID passed in")
	ErrSeekOffsetAfterSkippingBlocks         = errors.New("internal error for download seek - the offset after skipping blocks is wrong")
	ErrNoKeyringForSignatureVerification     = errors.New(("internal error for signature verification - no keyring is generated"))
)

Functions

func NewDefaultConfig

func NewDefaultConfig() *common.Config

func RandomString

func RandomString(n int) string

String create a random string for test purposes.

Do not use these for passwords.

func StringFn

func StringFn(n int, randIntn func(n int) int) string

Taken from: https://github.com/rclone/rclone/blob/e43b5ce5e59b5717a9819ff81805dd431f710c10/lib/random/random.go

StringFn create a random string for test purposes using the random number generator function passed in.

Do not use these for passwords.

Types

type FileDownloadReader

type FileDownloadReader struct {
	// contains filtered or unexported fields
}

func (*FileDownloadReader) Close

func (r *FileDownloadReader) Close() error

func (*FileDownloadReader) Read

func (r *FileDownloadReader) Read(p []byte) (int, error)

type FileSystemAttrs

type FileSystemAttrs struct {
	ModificationTime time.Time
	Size             int64
	BlockSizes       []int64
	Digests          string // sha1 string
}

type MailSendingParameters

type MailSendingParameters struct {
	TemplateFile          string
	EmailSubject          string
	RecipientEmailAddress string
	EmailAttachments      []string
	EmailContentIDs       []string
}

type ProtonDirectoryData

type ProtonDirectoryData struct {
	Link     *proton.Link
	Name     string
	IsFolder bool
}

type ProtonDrive

type ProtonDrive struct {
	MainShare *proton.Share
	RootLink  *proton.Link

	MainShareKR   *crypto.KeyRing
	DefaultAddrKR *crypto.KeyRing

	Config *common.Config
	// contains filtered or unexported fields
}

func NewProtonDrive

func NewProtonDrive(ctx context.Context, config *common.Config, authHandler proton.AuthHandler, deAuthHandler proton.Handler) (*ProtonDrive, *common.ProtonDriveCredential, error)

func (*ProtonDrive) About

func (protonDrive *ProtonDrive) About(ctx context.Context) (*proton.User, error)

func (*ProtonDrive) ClearCache

func (protonDrive *ProtonDrive) ClearCache()

func (*ProtonDrive) CreateNewFolder

func (protonDrive *ProtonDrive) CreateNewFolder(ctx context.Context, parentLink *proton.Link, folderName string) (string, error)

func (*ProtonDrive) CreateNewFolderByID

func (protonDrive *ProtonDrive) CreateNewFolderByID(ctx context.Context, parentLinkID string, folderName string) (string, error)

func (*ProtonDrive) DownloadFile

func (protonDrive *ProtonDrive) DownloadFile(ctx context.Context, link *proton.Link, offset int64) (io.ReadCloser, int64, *FileSystemAttrs, error)

func (*ProtonDrive) DownloadFileByID

func (protonDrive *ProtonDrive) DownloadFileByID(ctx context.Context, linkID string, offset int64) (io.ReadCloser, int64, *FileSystemAttrs, error)

func (*ProtonDrive) EmptyRootFolder

func (protonDrive *ProtonDrive) EmptyRootFolder(ctx context.Context) error

WARNING!!!! Everything in the root folder will be moved to trash Most likely only used for debugging when the key is messed up

func (*ProtonDrive) EmptyTrash

func (protonDrive *ProtonDrive) EmptyTrash(ctx context.Context) error

Empty the trash

func (*ProtonDrive) GetActiveRevisionAttrs

func (protonDrive *ProtonDrive) GetActiveRevisionAttrs(ctx context.Context, link *proton.Link) (*FileSystemAttrs, error)

Might return nil when xattr is missing

func (*ProtonDrive) GetActiveRevisionAttrsByID

func (protonDrive *ProtonDrive) GetActiveRevisionAttrsByID(ctx context.Context, linkID string) (*FileSystemAttrs, error)

func (*ProtonDrive) GetActiveRevisionWithAttrs

func (protonDrive *ProtonDrive) GetActiveRevisionWithAttrs(ctx context.Context, link *proton.Link) (*proton.Revision, *FileSystemAttrs, error)
func (protonDrive *ProtonDrive) GetLink(ctx context.Context, linkID string) (*proton.Link, error)

func (*ProtonDrive) GetRevisions

func (protonDrive *ProtonDrive) GetRevisions(ctx context.Context, link *proton.Link, revisionType proton.RevisionState) ([]*proton.RevisionMetadata, error)

func (*ProtonDrive) ListDirectory

func (protonDrive *ProtonDrive) ListDirectory(
	ctx context.Context,
	folderLinkID string) ([]*ProtonDirectoryData, error)

func (*ProtonDrive) Logout

func (protonDrive *ProtonDrive) Logout(ctx context.Context) error

func (*ProtonDrive) MoveFile

func (protonDrive *ProtonDrive) MoveFile(ctx context.Context, srcLink *proton.Link, dstParentLink *proton.Link, dstName string) error

func (*ProtonDrive) MoveFileByID

func (protonDrive *ProtonDrive) MoveFileByID(ctx context.Context, srcLinkID, dstParentLinkID string, dstName string) error

func (*ProtonDrive) MoveFileToTrashByID

func (protonDrive *ProtonDrive) MoveFileToTrashByID(ctx context.Context, linkID string) error

func (*ProtonDrive) MoveFolder

func (protonDrive *ProtonDrive) MoveFolder(ctx context.Context, srcLink *proton.Link, dstParentLink *proton.Link, dstName string) error

func (*ProtonDrive) MoveFolderByID

func (protonDrive *ProtonDrive) MoveFolderByID(ctx context.Context, srcLinkID, dstParentLinkID, dstName string) error

func (*ProtonDrive) MoveFolderToTrashByID

func (protonDrive *ProtonDrive) MoveFolderToTrashByID(ctx context.Context, linkID string, onlyOnEmpty bool) error

func (*ProtonDrive) SearchByNameInActiveFolder

func (protonDrive *ProtonDrive) SearchByNameInActiveFolder(
	ctx context.Context,
	folderLink *proton.Link,
	targetName string,
	searchForFile, searchForFolder bool,
	targetState proton.LinkState) (*proton.Link, error)

func (*ProtonDrive) SearchByNameInActiveFolderByID

func (protonDrive *ProtonDrive) SearchByNameInActiveFolderByID(ctx context.Context,
	folderLinkID string,
	targetName string,
	searchForFile, searchForFolder bool,
	targetState proton.LinkState) (*proton.Link, error)

if the target isn't found, nil will be returned for both return values

func (*ProtonDrive) SearchByNameRecursively

func (protonDrive *ProtonDrive) SearchByNameRecursively(ctx context.Context, folderLink *proton.Link, targetName string, isFolder bool, listAllActiveOrDraftFiles bool) (*proton.Link, error)

func (*ProtonDrive) SendEmail

func (protonDrive *ProtonDrive) SendEmail(ctx context.Context, i int, errChan chan error, config *MailSendingParameters)

func (*ProtonDrive) UploadFileByPath

func (protonDrive *ProtonDrive) UploadFileByPath(ctx context.Context, parentLink *proton.Link, filename string, filePath string, testParam int) (string, *proton.RevisionXAttrCommon, error)

func (*ProtonDrive) UploadFileByReader

func (protonDrive *ProtonDrive) UploadFileByReader(ctx context.Context, parentLinkID string, filename string, modTime time.Time, file io.Reader, testParam int) (string, *proton.RevisionXAttrCommon, error)

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL