fuelserver

README

This project has been moved to [gazebo-web/fuel-server](https://github.com/gazebo-web/fuel-server) on Github

Ignition Robotics

Ignition Web Fuel Server hosts simulation resources for consumption through a REST API.

Install

1. Go version 1.14 or above.

   Make sure to set the following environment variables:

   1. $PATH: Set environment variable to point to the bin directory of where you installed Go so your system can find the go executable.

   2. $GOPATH: This is the working directory of your go project. Project dependencies and binaries will be installed to this path. If the environment variable is unset, by default Go will point to the $HOME/go directory.

2. Download the server code

   git clone https://gitlab.com/ignitionrobotics/web/fuelserver
   

3. Build and install fuelserver

   cd fuelserver
   go install
   

   go install does the following:

   1. Download dependencies if not found
   2. Builds the Project.
   3. Copies the project binary to the $GOPATH/bin directory

   A fuelserver executable should now be installed to $GOPATH/bin

4. Make sure your git config is set, i.e.

    git config --global user.name "User Name"
    git config --global user.email "user@email.com"
   

5. (Optional) Generate a self-signed certificate:

   # navigate to your fuelserver directory
   cd fuelserver
   

   wget https://raw.githubusercontent.com/golang/go/release-branch.go1.8/src/crypto/tls/generate_cert.go
   

   go run generate_cert.go --host localhost --ca true
   

   mv cert.pem key.pem ssl
   

   export IGN_SSL_CERT=`pwd`/ssl/cert.pem
   

   export IGN_SSL_KEY=`pwd`/ssl/key.pem
   

   Note: to allow self certificates for localhost in Chrome, you need to put this in the chrome address bar : chrome://flags/#allow-insecure-localhost

6. Install mysql:

   NOTE: Install a version greater than v5.6.4. In the servers, we are currently using MySQL v5.7.21

   sudo apt-get install mysql-server
   

   The installer will ask you to create a root password for mysql.

7. Create the database and a user in mysql. Replace 'newuser' with your username and 'password' with your new password:

    # Xenial
    mysql -u root -p
   
    # Bionic requires sudo
    sudo mysql -u root -p
   

   CREATE DATABASE fuel;
   

   Also create a separate database to use with tests:

   CREATE DATABASE fuel_test;
   

   CREATE USER 'newuser'@'localhost' IDENTIFIED BY 'password';
   

   GRANT ALL PRIVILEGES ON fuel.* TO 'newuser'@'localhost';
   

   FLUSH PRIVILEGES;
   

   exit
   

8. Create a Test JWT token (this is needed for tests to pass OK -- go test)

   TL;DR: Just copy and paste the following env vars in your system (.env)

    # Test RSA256 Private key WITHOUT the -----BEGIN RSA PRIVATE KEY----- and -----END RSA PRIVATE KEY-----
    # It is used by token-generator to generate the Test JWT Token
    export TOKEN_GENERATOR_PRIVATE_RSA256_KEY=MIICWwIBAAKBgQDdlatRjRjogo3WojgGHFHYLugdUWAY9iR3fy4arWNA1KoS8kVw33cJibXr8bvwUAUparCwlvdbH6dvEOfou0/gCFQsHUfQrSDv+MuSUMAe8jzKE4qW+jK+xQU9a03GUnKHkkle+Q0pX/g6jXZ7r1/xAK5Do2kQ+X5xK9cipRgEKwIDAQABAoGAD+onAtVye4ic7VR7V50DF9bOnwRwNXrARcDhq9LWNRrRGElESYYTQ6EbatXS3MCyjjX2eMhu/aF5YhXBwkppwxg+EOmXeh+MzL7Zh284OuPbkglAaGhV9bb6/5CpuGb1esyPbYW+Ty2PC0GSZfIXkXs76jXAu9TOBvD0ybc2YlkCQQDywg2R/7t3Q2OE2+yo382CLJdrlSLVROWKwb4tb2PjhY4XAwV8d1vy0RenxTB+K5Mu57uVSTHtrMK0GAtFr833AkEA6avx20OHo61Yela/4k5kQDtjEf1N0LfI+BcWZtxsS3jDM3i1Hp0KSu5rsCPb8acJo5RO26gGVrfAsDcIXKC+bQJAZZ2XIpsitLyPpuiMOvBbzPavd4gY6Z8KWrfYzJoI/Q9FuBo6rKwl4BFoToD7WIUS+hpkagwWiz+6zLoX1dbOZwJACmH5fSSjAkLRi54PKJ8TFUeOP15h9sQzydI8zJU+upvDEKZsZc/UhT/SySDOxQ4G/523Y0sz/OZtSWcol/UMgQJALesy++GdvoIDLfJX5GBQpuFgFenRiRDabxrE9MNUZ2aPFaFp+DyAe+b4nDwuJaW2LURbr8AEZga7oQj0uYxcYw==
   
    # JWT Token generated by the token-generator program using the above Test RSA keys
    # This token does not expire.
    export IGN_TEST_JWT=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ0ZXN0LXVzZXItaWRlbnRpdHkifQ.iV59-kBkZ86XKKsph8fxEeyxDiswY1zvPGi4977cHbbDEkMA3Y3t_zzmwU4JEmjbTeToQZ_qFNJGGNufK2guLy0SAicwjDmv-3dHDfJUH5x1vfi1fZFnmX_b8215BNbCBZU0T2a9DEFypxAQCQyiAQDE9gS8anFLHHlbcWdJdGw
   
    # A Test RSA256 Public key, without the -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----.
    # It is used to override the AUTH0_RSA256_PUBLIC_KEY when tests are run.
    export TEST_RSA256_PUBLIC_KEY=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDdlatRjRjogo3WojgGHFHYLugdUWAY9iR3fy4arWNA1KoS8kVw33cJibXr8bvwUAUparCwlvdbH6dvEOfou0/gCFQsHUfQrSDv+MuSUMAe8jzKE4qW+jK+xQU9a03GUnKHkkle+Q0pX/g6jXZ7r1/xAK5Do2kQ+X5xK9cipRgEKwIDAQAB
   

   Long version:

   Ign Fuel assumes JWT Tokens are RSA256. In order to generate the token, follow these steps:

   1. You need to set the TOKEN_GENERATOR_PRIVATE_RSA256_KEY env var first. This var is a RSA256 Private key without the -----BEGIN RSA PRIVATE KEY----- and -----END RSA PRIVATE KEY----- prefix and suffix. It is used to generate Test JWT Tokens.

   2. Also please write down the corresponding TEST_RSA256_PUBLIC_KEY env var. This var is the RSA256 Public key (without the -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----) that will be used by the backend to validate received Test JWT tokens. NOTE: This one must be the pair of the private key TOKEN_GENERATOR_PRIVATE_RSA256_KEY.

   3. Generate the token:

      Build all ign-fuelserver packages and programs:

       go install gitlab.com/ignitionrobotics/web/fuelserver/...
      

      Note: the token generator should be installed to your go path's bin directory. By default if no GO environment variables are set, it should be ~/go

       $GOPATH/bin/token-generator
      

   4. The token-generator program will output a signed token. You will need to set the IGN_TEST_JWT env var with that generated value.

   In summary, in order to make go test work with JWT you will need to set the following env vars:

   • TOKEN_GENERATOR_PRIVATE_RSA256_KEY
   • TEST_RSA256_PUBLIC_KEY
   • IGN_TEST_JWT

9. Run the test suite

   First, make sure you have all the required env variables set:

   export IGN_DB_USERNAME=<DB username>
   

   export IGN_DB_PASSWORD=<DB password>
   

   export IGN_DB_ADDRESS=<DB IP and port. Eg: localhost:3306>
   

   export IGN_DB_NAME=fuel
   

   export IGN_DB_MAX_OPEN_CONNS=66
   

   export IGN_FUEL_RESOURCE_DIR=/tmp/fuel
   

   export IGN_TEST_JWT=<JWT token>
   

   export TEST_RSA256_PUBLIC_KEY=< RSA256 public key without the '-----BEGIN CERTIFICATE-----' and '-----END CERTIFICATE-----'>
   Note: TEST_RSA256_PUBLIC_KEY must be able to decode and validate the IGN_TEST_JWT test token.
   

   Then, run all tests:

   go test
   

10. Run the backend server

    First, make sure you have all the required env variables set (Please see the Environment Variables section). At minimum, you need to have the required variables for local development set and also the AWS_BUCKET_PREFIX variable.

    By default if no GO environment variables are set, fuelserver should be installed to ~/go/bin

    $GOPATH/bin/fuelserver
    

    Note: You may see errors about parsing IGN_FUEL_MIGRATE_* variables. They are for migration purposes and can be ignored.

11. Test in the browser

    1. If using SSL

       https://localhost:4430/1.0/models
       

    2. If not using SSL

       http://localhost:8000/1.0/models
       

Permissions

Permission handling is done using the authorization library casbin

To set up the system administrator (root user), export the following environment variable:

export IGN_FUEL_SYSTEM_ADMIN=sys_admin@email.org

Note: when running the tests, this environment variable will be overriden with a predefined string: root

Environment Variables

You may want to create an .env.bash file to define environment vars. Remember to add it to .gitignore. Then load the env vars using source .env.bash from the bash terminal where you will run go commands.

Minimum required variables for local development.

1. IGN_DB_USERNAME : Mysql user name.
2. IGN_DB_PASSWORD : Mysql user password.
3. IGN_DB_ADDRESS : Mysql address, of the form host:port.
4. IGN_DB_NAME : Mysql database name, which should be fuel.
5. IGN_DB_MAX_OPEN_CONNS : Max number of open connections in connections pool. Eg. 66.
6. IGN_FUEL_RESOURCE_DIR : the file system path where models will be stored.
7. IGN_FUEL_VERBOSITY : controls the level of output, with a default value of 2. 0 = critical messages, 1 = critical & error messages, 2 = critical & error & warning messages, 3 = critical & error & warning & informational messages, 4 = critical & error & warning & informational & debug messages
8. AUTH0_RSA256_PUBLIC_KEY : Auth0 RSA256 public key without the '-----BEGIN CERTIFICATE-----' and '-----END CERTIFICATE-----' Note: This env var will be used by the backend to decode and validate any received Auth0 JWT tokens.This env var will be used by the backend to decode and validate any received Auth0 JWT tokens. You can get this key from: https://osrfoundation.auth0.com/.well-known/jwks.json (or from your auth0 user). It is the "x5c" field.

Using AWS S3 buckets

1. AWS_BUCKET_PREFIX: set it to a prefix that will be shared by all buckets created by this server (eg. export AWS_BUCKET_PREFIX=myFuelServer-)
2. AWS_BUCKET_USE_IN_TESTS: set it to true if you want to us AWS S3 during tests. Set to false to use a mock (default: false). (eg. export AWS_BUCKET_USE_IN_TESTS=true)
3. AWS_BUCKET_PREFIX_TEST: if AWS is enabled during tests, set this env var to a prefix that will be shared by all buckets in tests. (eg. export AWS_BUCKET_PREFIX_TEST=myFuelServer-tests-)

Flagging content and sending emails

To enable flagging of content you need to set the following env vars:

1. AWS_REGION, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY

2. IGN_FLAGS_EMAIL_FROM: The email address used as from when sending email notifications.

3. IGN_FLAGS_EMAIL_TO: The email address used as to when sending email notifications.

Database

There are three databases in use:

1. ign-fuel: a production database for use with the production Elastic Beanstalk environment,

2. ign-fuel-staging: a staging database for use with the staging Elastic Beanstalk environment, and

3. ign-fuel-integration: an integration database for use with the integration Elastic Beanstalk environment.

The production database should never be manually altered. The staging database should match the production environment, and the purpose is to catch migration errors on the staging Elastic beanstalk instance. The integration database is used during testing and development. This database is frequently wiped and altered.

Leaderboards

There may be some cases where scores for specific organizations or circuits should not be displayed in competition leaderboards. There are environment variables available to control which organizations and circuits should not be displayed in leaderboards. These environment variables do not stop scores from being produced, they only filter /subt/leaderboard results.

1. IGN_FUEL_TEST_ORGANIZATIONS List of organizations to filter from leaderboard scores.
2. IGN_FUEL_HIDE_CIRCUIT_SCORES List of circuits to filter from leaderboard scores.

All of these environment variables can contain multiple comma-separated values.

Testing and Development

1. IGN_POPULATE_PATH : Path to a set of Gazebo models. Setting this variable will populate the database with these models.

2. IGN_TEST_JWT : A JWT token used from unit tests. Note: the TEST_RSA256_PUBLIC_KEY must be able to decode and validate this test token.

3. TEST_RSA256_PUBLIC_KEY : if present, override the Auth0 public RSA key with this test key. This one must match with the Test JWT token. It is used by unit tests (go test).

4. TOKEN_GENERATOR_PRIVATE_RSA256_KEY : RSA 256 private key Used by the Token Generator utility program to generate the Test JWT token. It must pair with the TEST_RSA256_PUBLIC_KEY public key.

Migration environment variables

These variables are used for migration. You may see errors about parsing these variables when running fuelserver locally but they are safe to ignore when doing development locally.

• IGN_FUEL_MIGRATE_UNIQUEOWNERS_TABLE
• IGN_FUEL_MIGRATE_RESET_LIKE_AND_DOWNLOADS
• IGN_FUEL_MIGRATE_RESET_ZIP_FILESIZE
• IGN_FUEL_MIGRATE_CASBIN
• IGN_FUEL_MIGRATE_MODEL_REPOSITORIES

Naming conventions

In general we will try to follow Go naming conventions. In addition, these are own conventions:

1. For constructors / factory-methods we will use the form: New<object>. See: http://www.golangpatterns.info/object-oriented/constructors

2. For HTTP POST handlers we use: <object>Create. eg: ModelCreate.

3. For HTTP DELETE handlers we use: <object>Remove. eg: ModelRemove.

4. An HTTP handler that returns a collection will have the form: <object>List<format>. eg: ModelListJSON. Note: format is optional. JSON will be assumed by default.

5. An HTTP handler that returns a single element will be: <object>Index<format>. eg: ModelIndexZip. Note: format is optional. JSON will be assumed by default.

6. For HTTP OPTIONS handler we use: <object>API.

7. We use the func suffix Impl when we delegate implementation of a handler to an internal function. Eg: UserCreate handler can delegate to userCreateImpl.

Linter

1. Get the linter

   go get -u golang.org/x/lint/golint
   

2. Run the linter

   $GOPATH/bin/golint $(go list gitlab.com/ignitionrobotics/web/fuelserver/...) | grep -v .pb.go
   

Note you can create this bash script:

#!/bin/bash
go get -u golang.org/x/lint/golint
$GOPATH/bin/golint $(go list gitlab.com/ignitionrobotics/web/fuelserver/...) | grep -v .pb.go

Proto

1. If you need to modify the proto files then you will need to run (from the proto folder):

   protoc --go_out=. *.proto
   

   Then update the generated import proto from code.google... to "github.com/golang/protobuf/proto"

Coverage

1. Run test suite with coverage enabled

   go test -cover gitlab.com/ignitionrobotics/web/fuelserver
   

2. Tip. Add this function to your ~/.bashrc

cover () {
  t="/tmp/go-cover.$$.tmp"
  go test -covermode=count -coverprofile=$t $@ && go tool cover -html=$t && unl$
}

Then run the function from your project folder. Tests will be run and a browser window will open with coverage results.

Integration deployment

If it's the first time that you deploy on integration:

1. Install eb CLI tool (if needed).

2. Configure eb:

   eb init
   

   And choose the following options:

   1. Select us-east-1 as region.

   2. Select ign-fuel-server as the application name.

   3. Select ign-fuel-server-integration as the environment name.

Otherwise, just type:

eb deploy

Staging/Production deployment

The staging and production branches push code through bitbucket pipelines to AWS Elastic Beanstalk (EBS).

1. Push code to staging. Make sure pipelines completes successfully.

2. Test staging.api.ignitionfuel.org

3. When ready, Swap Environment URLs with the production EBS environment.

4. Merge the staging branch into the production branch, and push.

5. Test staging.api.ignitionfuel.org again

6. Swap the EBS environment URLs back.

AWS Configuration

• Elastic Beanstalk runs go in a docker container

• AWS Relation Database (RDS) runs an instance of mysql.

• Each EC2 instance started by EBS mounts an NFS filesystem, via Elastic Filesystem, on /fuel. This filesystem store all the mercurial repositories.

AWS RDS

There are two mysql databases hosted on Amazon.

1. ign-fuel: The production database.

   • Endpoint: ign-fuel.cpznmiopbczj.us-east-1.rds.amazonaws.com:3306

2. ign-fuel-dev: The development and testing database. You can write tests that will run on bitbucket pipelines against this database. Make sure to clean the database up after each test.

   • Endpoint: ign-fuel-dev.cpznmiopbczj.us-east-1.rds.amazonaws.com:3306

Development

See the Database section above for information on the different database options.

Transactions

Try to create and commit transactions within main Handlers, and not in the helper functions.

REST Documentation

Swagger is used to document the REST API. This includes both model and route information.

Do not manually edit swagger.json

Process

1. Document a route or model following this documentation.

2. Install swagger inside your GOPATH

   • go get -u github.com/go-swagger/go-swagger/cmd/swagger
   • go install github.com/go-swagger/go-swagger/cmd/swagger

3. Generate the swagger.json file. This file will be used by a webserver to display the API documentation.

   ./bin/swagger generate spec -o ./src/gitlab.com/ignitionrobotics/web/fuelserver/swagger.json -b ./src/gitlab.com/ignitionrobotics/web/fuelserver/ -m
   

4. Commit and push your changes to the repository.

5. View the results at http://doc.ignitionfuel.org. Enter a new swagger.json in the Explore box to see a different version of the API.

Useful links

1. Swagger json documentation

• This page documents how to write swagger documentation that will be parsed to generate the swagger.json file.

1. Our S3 swagger website

• This is an instance of swagger-ui, where the index.html file was edited to point to our swagger file.

Log Files

Two web services are used for log management.

1. rollbar.com

Rollbar aggregates log messages from the application. Application log messages are sent to rollbar when a REST error is returned to a client or by using one of ign.Debug, ign.Info, ign.Warning, ign.Error, or ign.Critical.

Messages are sent to rollbar asynchronously. This could result in messages appearing out of order on rollbar's UI. In addition to the timestamp generated by rollbar upon message arrival, rollbar keeps a metadata.customer_timestamp which should be the application's timestamp. You can use the RQL console with metadata.custom_timestamp to reorder items. Steps:

1. Click on `RQL` in rollbar's top toolbar.
2. Enter an SQL query like the following:

    ```
    SELECT *
    FROM item_occurrence
    WHERE item.counter = 932
    ORDER BY metadata.customer_timestamp DESC
    LIMIT 0, 20
    ```
3. Click 'Run'

1. papertrail.com

Papertrail aggregates system log messages. Log file upload to Papertrail happens automatically. Configuration, including specification of system log files to monitor, is handled in .ebextensions/remote_syslog.ebextensions.config.

Debugging inside docker container

If you ever need to debug the application as if it were running in AWS or the pipelines, you need to do it from inside its docker containter. To do that:

Most ideas taken from here: Mysql and Docker https://docs.docker.com/samples/library/mysql/#-via-docker-stack-deploy-or-docker-compose

1. First create the docker image for the ign-fuelserver. docker build ign-fuelserver . Write down its image ID.

2. Then run a dockerized mysql database. docker run --name my-mysql -e MYSQL_ROOT_PASSWORD=<desired-root-pwd> -d mysql:5.7.21 This will create a mysql docker container with an empty mysql in it.

3. Then you need to connect to that mysql container instance to run commands: docker exec -it my-mysql bash. From inside the container, connect to mysql using the client (eg. mysql -u root -p) and create databases fuel and fuel_test. eg: create database fuel_test;.

4. Run the ign-fuelserver docker and link it to mysql database. docker run --name ign-fuelserver --rm --link my-mysql:mysql -ti <fuelserver-image-id> /bin/bash

5. Then from inside the server container you need to set the Env Var that points to the linked docker mysql. eg. export IGN_DB_ADDRESS="172.17.0.2:3306"

After that you can source your env vars and run commands such as go test.

Tips for local development using multiple dependent projects

This is useful when you need to test uncommitted changes from a project depedency. Eg. when you need to test changes in ign-go using ign-fuelserver.

Install vg (virtualgo) . This tool is used on top of go dep: https://github.com/GetStream/vg

First, initialize vg support in your system. In a terminal, run:

• export PATH=$PATH:$GOPATH/bin && eval "$(vg eval --shell bash)"

Tip: I have pushed a .vgenable script that can be sourced later to enable vg support in current terminal. Eg. source .vgenable

How to use vg:

1. vg init (only the first time, to initialize the project with vg)

2. vg ensure (this command delegates to dep ensure and then removes the vendor folder)

3. To add or update dependencies (into Gopck.toml) use: vg ensure -- -update <dependency> (or just use normal dep ensure -update <dependency> style, and later run vg ensure to move dependencies into vg's workspace)

4. How to switch to a local version of a dependency: eg. vg localInstall gitlab.com/ignitionrobotics/web/ign-go

Documentation

Overview

Package main Ignition Fuel Server REST API

This package provides a REST API to the Ignition Fuel server.

Schemes: https Host: fuel.ignitionrobotics.org BasePath: /1.0 Version: 0.1.0 License: Apache 2.0 Contact: info@openrobotics.org

swagger:meta go:generate swagger generate spec -m