gof3

README

gof3

As a CLI or as a library, GoF3 provides a single operation: mirroring. The origin and destination are designated by the URL of a forge and a path to the resource. For instance, mirror --from-type forgejo --from https://code.forgejo.org/forgejo/lxc-helpers --to-type F3 --to /some/directory will mirror a project in a local directory using the F3 format.

Building

• Install go >= v1.21
• make f3-cli
• ./f3-cli mirror -h

Example

To F3

Login to https://code.forgejo.org and obtain an application token with read permissions at https://code.forgejo.org/user/settings/applications.

f3-cli mirror \
  --from-type forgejo --from-forgejo-url https://code.forgejo.org \
  --from-forgejo-token $codetoken \
  --from-path /forge/organizations/actions/projects/cascading-pr \
  --to-type filesystem --to-filesystem-directory /tmp/cascading-pr

From F3

Run a local Forgejo instance with serials=1 tests/setup-forgejo.sh and obtain an application token with:

docker exec --user 1000 forgejo1 forgejo admin user generate-access-token -u root --raw --scopes 'all,sudo'

Mirror issues

f3-cli mirror \
  --from-type filesystem --from-filesystem-directory /tmp/cascading-pr \
  --from-path /forge/organizations/actions/projects/cascading-pr/issues \
  --to-type forgejo --to-forgejo-url http://0.0.0.0:3001 \
  --to-forgejo-token $localtoken

Visit them at http://0.0.0.0:3001/actions/cascading-pr/issues

Testing

Requirements

The tests require a live GitLab instance as well as a live Forgejo instance and will use up to 16GB of RAM.

• Install docker
• make coverage

License

This project is MIT licensed.

Architecture

F3 is a hierarchy designed to be stored in a file system. It is represented in memory with the tree/generic abstract data structure that can be saved and loaded from disk by the forges/filesystem driver. Each forge (e.g. forges/forgejo) is supported by a driver that is responsible for the interactions of each resource (e.g issues, asset, etc.).

Tree

tree/f3 implements a F3 hierarchy based on the tree/generic data structure. The tree has a logger for messages, options defining which forge it relates to and how and a pointer to the root node of the hierarchy (i.e. the forge F3 resource).

The node (tree/generic/node.go) has:

• a unique id (e.g. the numerical id of an issue)
• a parent
• chidren (e.g. issues children are issues, issue children are comments and reactions)
• a kind that maps to a F3 resource (e.g. issue, etc.)
• a driver for its concrete implementation for a given forge

It relies on a forge driver for the concrete implemenation of a F3 resource (issue, reaction, repository, etc.). For instance the issues driver for Forgejo is responsible for listing the existing issues and the issue driver is responsible for creating, updating or deleting a Forgejo issue.

F3 archive

The F3 JSON schemas are copied in f3/schemas. Their internal representation and validation is found in a source file named after the resource (e.g. an issue represented by f3/schemas/issue.json is implemented by f3/issue.go).

When a F3 resource includes data external to the JSON file (i.e. a Git repository or an asset file), the internal representation has a function to copy the data to the destination given in argument. For instance:

• f3/repository.go FetchFunc(destination) will git fetch --mirror the repository to the destination directory.
• f3/releaseasset.go DownloadFunc() returns a io.ReadCloser that will be used by the caller to copy the asset to its destination.

Options

The Forge options at options/interface.go define the parameters given when a forge is created:

Each forge driver is responsible for registering the options (e.g. Forgejo options) and for registering a factory that will create these options (e.g. Forgejo options registration). In addition to the options that are shared by all forges such as the logger, it may define additional options.

Driver interface

For each F3 resource, the driver is responsible for:

• copying the f3 argument to FromFormat to the forge
• ToFormat reads from the forge and convert the data into an f3/resources.go

A driver must have a unique name (e.g. forgejo) and register:

• an options factory
• a forge factory

Tree driver

The tree driver functions (e.g. forges/forgejo/tree.go) specialize NullTreeDriver.

• Factory(ctx context.Context, kind generic.Kind) generic.NodeDriverInterface creates a new node driver for a given Kind.
• GetPageSize() int returns the default page size.

Node driver

The node driver functions for each Kind (e.g. issues, issue, etc.) specialize NullNodeDriver. The examples are given for the Forgejo issue and issues drivers, matching the REST API endpoint to the driver function.

• ListPage(context.Context, page int) ChildrenSlice returns children of the node paginated GET /repos/{owner}/{repo}/issues
• Get(context.Context) get the content of the resource (e.g. GET /repos/{owner}/{repo}/issues/{index})
• Put(context.Context) NodeID create a new resource and return the identifier (e.g. POST /repos/{owner}/{repo}/issues)
• Patch(context.Context) modify an existing resource (e.g. PATCH /repos/{owner}/{repo}/issues/{index})
• Delete(context.Context) delete an existing resource (e.g. DELETE /repos/{owner}/{repo}/issues/{index})
• NewFormat() f3.Interface create a new issue F3 object
• FromFormat(f3.Interface) set the internal representation from the given F3 resource
• ToFormat() f3.Interface convert the internal representation into the corresponding F3 resource. For instance the internal representation of an issue for the Forgejo driver is the Issue struct of the Forgejo SDK.

Options

The options created by the factory are expected to provide the options interfaces:

• Required
  • LoggerInterface
  • URLInterface
• Optional
  • CLIInterface if additional CLI arguments specific to the forge are supported

For instance forges/forgejo/options/options.go is created by forges/forgejo/options.go.

Driver implementation

A driver for a forge must be self contained in a directory (e.g. forges/forgejo). Functions shared by multiple forges are grouped in the forges/helpers directory and split into one directory per Kind (e.g. forges/helpers/pullrequest).

• options.go defines the name of the forge in the Name variable (e.g. Name = "forgejo")
• options/options.go defines the options specific to the forge and the corresponding CLI flags
• main.go calls f3_tree.RegisterForgeFactory to create the forge given its name
• tree.go has the Factory() function that maps a node kind (issue, reaction, etc.) into an object that is capable of interacting with it (CRUD).
• one file per Kind (e.g. forges/forgejo/issues.go).

Idempotency

Mirroring is idempotent: it will produce the same result if repeated multiple times. The drivers functions are not required to be idempotent.

• The Put function will only be called if the resource does not already exist.
• The Patch and Delete functions will only be called if the resource exists.

Identifiers mapping

When a forge (e.g. Forgejo) is mirrored on the filesystem, the identifiers are preserved verbatim (e.g. the issue identifier). When the filesystem is mirrored to a forge, the identifiers cannot always be preserved. For instance if an issue with the identifier 1234 is downloaded from Forgejo and created on another Forgejo instance, it will be allocated an identifier by the Forgejo instance. It cannot request to be given a specific identifier.

References

A F3 resource may reference another F3 resource by a path. For instance the user that authored an issue is represented by /forge/users/1234 where 1234 is the unique identifier of the user. The reference is relative to the forge. The mirroring of a forge to another is responsible for converting the references using the identifier mapping stored in the origin forge. For instance if /forge/users/1234 stored in the filesystem is created in Forgejo as /forge/users/58, the issue stored in the filesystem with its authored as /forge/users/1234 will be created in Forgejo to be authored by /forge/users/58 instead.

Logger

The tree/generic has a pointer to a logger implementing logger.Interface which is made available to the nodes and the drivers.

Context

All functions except for setters and getters have a context.Context argument which is checked (using util/terminate.go) to not be Done before performing a long lasting operation (e.g. a REST API call or a call to the Git CLI). It is not used otherwise.

Error model

When an error that cannot be recovered from happens, panic is called, otherwise an Error is logged.

CLI

The CLI is in cmd and relies on options to figure out which options are to be implemented for each supported forge.

Hacking

Local tests

The forge instance is deleted before each run and left running for forensic analysis when the run completes.

./tests/run.sh test_forgejo # http://0.0.0.0:3001 user root, password admin1234
./tests/run.sh test_gitlab # http://0.0.0.0:8181 user root, password Wrobyak4
./tests/run.sh test_gitea # http://0.0.0.0:3001 user root, password admin1234

Restart a new forge with:

./tests/run.sh run_forgejo # http://0.0.0.0:3001 user root, password admin1234
./tests/run.sh run_gitlab # http://0.0.0.0:8181 user root, password Wrobyak4
./tests/run.sh run_gitea # http://0.0.0.0:3001 user root, password admin1234

The compliance test resources are deleted, except if the environment variable GOF3_TEST_COMPLIANCE_CLEANUP=false.

GOF3_TEST_COMPLIANCE_CLEANUP=false GOF3_FORGEJO_HOST_PORT=0.0.0.0:3001 go test -run=TestF3Forge/forgejo -v code.forgejo.org/f3/gof3/...

F3 schemas

The JSON schemas come from the f3-schemas repository and should be updated as follows:

cd f3 ; rm -fr schemas ; git --work-tree schemas clone https://code.forgejo.org/f3/f3-schemas ; rm -fr f3-schemas schemas/.gitignore schemas/.forgejo

Funding

See the page dedicated to funding in the F3 documentation