note-vook

README

🧑🏽‍💻 Project: NoteVook

This project aims to be an exercise to discuss about software engineering technical topics like software development, pair programming, testing, deployment, etcetera. More specifically, to discuss the development of an API (Application Programming Interface) to manage annotations for videos implemented written in go programming language.

📂 Table of content

• 📹 Overview
  • ☑️ Requirements
  • 🤔 Assumptions
• 📐 Design
  • 📊 Data model
    • 🎞️ Video
    • ✍🏽 Annotation
    • 👤 User
  • 🔀 Workflows
    • 🔀 User sign up
    • 🔀 User login
    • 🔀 Authorised requests
  • 🔚 End-points
• 🏗️ Implementation details
  • 📦 Dependencies
  • 🗄️ Storage
• ⏯️ Running
  • 🍏 Development Mode
  • 🍎 Production Mode
• ✅ Testing
  • 🧪 Manual
  • ♻️ Automated
  • 💯 Coverage
• 📚 References

📹 Overview

This simple API aims to manage a video annotations database, this means we will have a collection of videos and we will be able to add text notes for some given interval of time of the videos. Additionally we would like to manage some basic security layer based-on JWT (JSON Web Token). The application should be ready to deploy as a Docker container, so we need to generate an image for it available to download in Docker Hub.

☑️ Requirements

The API should be able to manage a database for the videos and each video may have many annotations related to it. An annotation allow s to capture time related information about the video. For example, we may want to create an annotation that references to a part of the video from 00:04:00 to 00:05:00.

Allowing the client to perform following operations:

• List all the videos. It should return the list of all videos in the system.
• Create a video. It should insert a new record for video provided by the client that includes some metadata.
• Update a video. It should allow the client to update the information of a given video.
• Delete a video. The API should provide an end-point to delete videos from the system.
• View video details. It should show the details of a single video provided by the client.
• Annotate a video. It should create a new annotation for a video with start and end time.
• Update annotations. The API should allow the client to update annotation details.
• Delete annotations. The API should provide a mechanism to delete annotations.
• Security layer. It should implement a security layer based-on JWT.

🤔 Assumptions

This is a small example and it's not taking care about some corner case scenarios like following:

• The environment variables and secrets (e. g. SECRET_TOKEN_KEY to encode sign the authorisation token) for API configuration are stored in .env files (see Running section below for more information).
• In the real world the secrets should be stored and provisioned by an external system (e. g. AWS Secret Manager). In order to test and play around with the API you can leave them as blank string in the .env files.
• The videos can only be annotated by the user creator.
• In order to keep things simple, there is no ACID transactions implemented for the database. We will remove the annotations in cascade though, so if we delete a vide from database we will remove its annotations too.
• The users won't be able to edit the video ID for an annotation. If the users want to do so, it's better to remove the annotation from the video, then add a new one in the other video.
• A video with the same link can be added multiple times by different users.
• It's been assumed that the annotation type it's some sort of category and each annotation can only be of one type.
• Users were not part of the original requirements, but I added them as makes simpler the way to explain the authorisation layer.
• The user names and passwords aren't validated properly, so the client can provide any input except an empty string.
• Users can anonymously be created in the system.
• If we would like access to the API end-point programmatically (e.g. via some automation), we would need to create a new user and their correspondent password for that client.
• Even if we added the security layer with the authorisation process, this is not secure enough, there are several flaws (e. g. non-secure cookie, non-password charset checking, lack of HTTPS certificates, etcetera), but it's implemented in this way just for didactical purposes.

📐 Design

The architecture will be a HTTP API for a microservice that will consume some configuration and use ORM to represent the records in the database tables and also a Model-Controller (MC) pattern design, so the controllers will contain the handlers for the API requests, while the models will represent the data. The service will be stateless, so we won't hold any state (e. g. session management) on the server side, instead we will use authorisation tokens.

📊 Data model

In order to store and manipulate the data needed the API will rely on the entities shown in following diagram:

erDiagram
  Video {
    id integer PK
    user_id integer FK
    title string
    description string
    link integer
    duration integer
    created_at datetime
    updated_at datetime
  }

  Annotation {
    id integer PK
    video_id integer FK
    type enum
    start integer
    end integer
    title string
    notes string
    created_at datetime
    updated_at datetime
  }

  User {
    id integer PK
    nickname string
    password string
    created_at datetime
    updated_at datetime
  }

  User ||--o{ Video : "may own"
  Annotation }o--|| Video: "may have"

As we can see in the diagram, a User may own several Videos, which is made possible with the user of the foreign key user_id within the Video entity. Then, a Video may have many Annotations thanks to the foreign key video_id.

The API manages the persistency of the data with a 🪶 SQLite database, which is a simple local storage database. So, the records for entities shown in the diagram will be stored as rows in tables. SQLite manages a reduced set of data types so, we will use the actual data type (affinity) used in following subsections.

🎞️ Video

This entity will represent the videos in the system and each record will be stored in the table videos which has following fields:

⏹️	Name	Type	Description
🗝️	id	INTEGER	Auto-numeric identifier for the video
✳️	user_id	INTEGER	Foreign key for the user owner of the video
🔤	title	TEXT	Title of the video
📄	description	BLOB	Description for the video
🔤	link	TEXT	URL for a link of the video. Unique along user domain
🔢	duration	INTEGER	Duration of the video in seconds
🗓️	created_at	NUMERIC	Timestamp representing the creation time
🗓️	updated_at	NUMERIC	Timestamp representing the last update time

✍🏽 Annotation

This entity will represent the annotations for the videos in the system and each record will be stored in the table annotations which has following fields:

⏹️	Name	Type	Description
🗝️	id	INTEGER	Auto-numeric identifier for the annotation
✳️	video_id	INTEGER	Foreign key for the video
🔢	type	INTEGER	Annotation type or category
🔢	start	INTEGER	Start point in seconds within the video timeline
🔢	end	INTEGER	End point in seconds within the video timeline
🔤	title	TEXT	Title or headline of the annotation
📄	notes	BLOB	Optional. Additional notes
🗓️	created_at	NUMERIC	Timestamp representing the creation time
🗓️	updated_at	NUMERIC	Timestamp representing the last update time

👤 User

The records for this entity will represent the users in the system and each record will be stored in the table users which has following fields:

⏹️	Name	Type	Description
🗝️	id	INTEGER	Auto-numeric identifier for the user
✳️	nickname	TEXT	Nickname of the user. Unique along this table
🔢	password	TEXT	Hash for the password of the user
🗓️	created_at	NUMERIC	Timestamp representing the creation time
🗓️	updated_at	NUMERIC	Timestamp representing the last update time

🔀 Workflows

There are three general workflows in this API: user sign up, user login and all the other operations that require authorisation.

🔀 User sign up

Following diagram describes the happy path for a user signup operation:

sequenceDiagram

actor Unknown
participant gin.Engine
participant UsersController
participant User
participant bcrypt
participant GORM
participant Database

Unknown->>+gin.Engine: POST /signup @JSON: credentials
gin.Engine->>+UsersController: Signup(@gin.Context)
UsersController->>+gin.Engine: bind(@JSON credentials)
gin.Engine->>-UsersController: returns @credentials
UsersController->>+bcrypt: Hash(@credentials.Password)
bcrypt-->>-UsersController: returns Hashed Password
UsersController->>+User: create instance (@credentials)
User-->>-UsersController: returns @user
UsersController->>+GORM: Create(@user)
GORM->>GORM: generates SQL Statement
GORM->>+Database: query(INSERT INTO users(...) VALUES(...))
Database-->>-GORM: returns query result
GORM-->>-UsersController: populated @user
UsersController-->>-gin.Engine: HTTP 201 OK and message
gin.Engine->>-Unknown: HTTP 201 Created (JSON with message)

🔀 User login

Following diagram describes the happy path for a user login operation:

sequenceDiagram

actor Unknown
participant gin.Engine
participant UsersController
participant User
participant bcrypt
participant JWT
participant GORM
participant Database

Unknown->>+gin.Engine: POST /login @JSON: credentials
gin.Engine->>+UsersController: Login(@gin.Context)
UsersController->>+gin.Engine: bind(@JSON credentials)
gin.Engine->>-UsersController: returns @credentials
UsersController->>+User: create empty instance
User-->>-UsersController: returns empty @user
UsersController->>+GORM: First(@user, @user.Nickname)
GORM->>GORM: generates SQL Statement
GORM->>+Database: query(SELECT * FROM users WHERE nickname = @user.Nickname)
Database-->>-GORM: returns query result
GORM-->>-UsersController: returns populated @user
UsersController->>+bcrypt: CompareHashAndPassword(@credentials.Password)
bcrypt-->>-UsersController: returns comparison result
UsersController->>+JWT: generates token signed with SECRET_TOKEN_KEY
JWT-->>-UsersController: returns signed @jwt.Token(7 days for expiration)
UsersController-->>-gin.Engine: HTTP 200 OK and message
gin.Engine->>-Unknown: HTTP 200 OK JSON with message and Authorisation Cookie with @jwt.Token

🔀 Authorised requests

Following diagram describes the happy path for any other operation that needs authorisation, meaning after user has been logged in:

sequenceDiagram

actor KnownUser
participant gin.Engine
participant UsersController
participant User
participant JWT
participant GORM
participant AnotherController
participant AnotherModel
participant Database

KnownUser->>+gin.Engine: GET /another-controller/action @JSON: credentials
gin.Engine->>+UsersController: Authorise(@gin.Context)
UsersController->>+gin.Engine: get Authorisation cookie
gin.Engine->>-UsersController: returns @cookie

UsersController->>+JWT: Parse @cookie.Value
JWT-->>+UsersController: checks algorithm for consistency and ask for key
UsersController-->>-JWT: result for algorithm check and SECRET_TOKEN_KEY
JWT-->>-UsersController: decodes the token and returns Parsed and Decoded @jwt.Token

UsersController->>UsersController: Extracts claims and check expiration
UsersController->>+User: New instance with Nickname only
User-->>-UsersController: returns @user with Nickname only
UsersController->>+GORM: First(@user, @user.Nickname)
GORM->>GORM: generates SQL Statement
GORM->>+Database: query(SELECT * FROM users WHERE nickname = @user.Nickname)
Database-->>-GORM: returns query result
GORM-->>-UsersController: returns populated @user 
UsersController-->>-gin.Engine: @user within the @gin.Context

gin.Engine->>+AnotherController: ActionHandler (@gin.Context with @user)
AnotherController->>+AnotherModel: do something
AnotherModel-->>-AnotherController: returns some result
AnotherController->>+GORM: do something
GORM->>GORM: generates SQL statement
GORM->>+Database: query (generated SQL statement)
Database-->-GORM: returns query result
GORM-->-AnotherController: returns query result
AnotherController->>AnotherController: May do something else (e. g. business logic)
AnotherController-->>-gin.Engine: HTTP 200 OK and message
gin.Engine->>-KnownUser: HTTP 200 OK JSON with message

🔚 End-points

The input for all the API end-points will be always in JSON format and the Cookie Authorisation JWT token in most of the cases and the output will be in the same format. The end-points for the API are described in following table:

Method	Address	Description	Success Status	Possible Failure Status
HEAD	/health	Service health check	200 OK	* Any
POST	/signup	User sign up to create users	201 Created	400 Bad Request
POST	/login	User login and get authorisation token	200 OK	400 Bad Request, 500 Internal Server Error
GET	/videos	List of all videos owned by logged user	200 OK	401 Unauthorised
POST	/videos	Create a video record in the system	200 Created	401 Unauthorised, 400 Bad Request
GET	/videos/:id	Get video details and its annotations	200 OK	401 Unauthorised, 404 Not Found
PATCH	/videos/:id	Edit details for a given video	200 OK	401 Unauthorised, 400 Bad Request, 404 Not Found
DELETE	/videos/:id	Delete a video from the system	200 OK	401 Unauthorised, 404 Not Found
POST	/annotations	Create a annotation record for a video	200 Created	401 Unauthorised, 400 Bad Request
PATCH	/annotations/:id	Edit details for an annotation	200 OK	401 Unauthorised, 400 Bad Request, 404 Not Found
DELETE	/annotations/:id	Delete an annotation	200 OK	401 Unauthorised, 404 Not Found

🏗️ Implementation details

We are using Golang as programming language for the implementation of the API operations. And the database is a single table in SQLite stored locally.

There is a continuous integration workflow that runs in GitHub Actions which is responsible to build the API, tun the unit tests if the tests succeed then it generates and pushes the image for the container to DockerHub.

📦 Dependencies

We are using following libraries for the implementation:

• gin-gonic. A web framework to implement a RESTful API via HTTP.
• gorm. A library for Object Relational Model (ORM) in order to represent the records in the database as relational objects.
• gorm/drivers/sqlite. Driver that manage SQLite dialect and connect to the database.
• godotenv. This CLI tool allows us to load environment configuration via .env files and run a command.
• crypto/bcrypt. This is part of the standard go library. It's to make use of hashing when sign up and login.
• golang-jwt. To generate and use the authorisation tokens.

And also, following ones for the development:

• testify. To have more readable assertions on the unit testing.
• mockery. To generate mocks used on unit testing.
• monkey. To perform monkey patching on the unit testing.

🗄️ Storage

A Docker container it's not persistent itself, so the Docker Compose file specify a volume to make the database persistent, that volume can be mapped to a host directory. The following sections will explain how to do that in order to run the API locally.

⏯️ Running

In order to run the application locally you will need to have Docker installed and internet connection. Using the command line with docker you can either go on two modes:

• Production: Directly downloading the latest built of the image and run it
• Development: Clone this Git repository and build the image for the container locally

🍎 Production Mode

You can docker CLI to download the latest built of the image from Docker Hub, and then run create and run a container as follow:

docker pull zatarain/note-vook:latest
docker run \
  -v $(pwd)/data:/api/data \
  -e ENVIRONMENT=prod \
  --name notevook \
  -p 4000:4000 \
  zatarain/note-vook:latest

Those commands will download the latest build image generated by the Continuous Integration and Deployment Pipeline in the Github Actions of the repository.

As you can see you need to specify following things:

• Volume for Database [-v $(pwd)/data:/api/data]: This will create (if it doesn't already exists) a data directory in the current directory where the API will store the database.
• Environment [ENVIRONMENT=prod]: This is optional, if you omit this, you will run it in development mode. The value of this variable determines the environment file and database tha will be used:

ENVIRONMENT	File	Database	Description
	.env	data/beta.db	Development is enabled by leaving the variable empty
test	test.env	data/test.db	This is used when running the Unit Testing
prod	prod.env	data/prod.db	Production environment

• Port binding [-p 4000:4000]: The image it's built to run the API on port 4000 withing the container, but you can choose to run it in another host port if you want (e. g. -p 8080:4000).

🍏 Development Mode

In your terminal, clone repository and build image as follow:

git clone https://github.com/zatarain/note-vook.git
cd note-vook
docker compose up --build

That will follow the configuration specified in the compose.yml file to build the image and run the unit testing on building time, and then run the API in development mode.

Then you can follow the steps to play manually with the API with the steps in next section.

✅ Testing

This project is able to be tested in manual way and with automated unit testing. This section will explain how can you play around with the API once you run it following the steps of the previous section.

🧪 Manual

In order to play around with the API, there are JSON files in the directory test/ of the repository for each of the end-points in the API. There are JSON files for input for the POST and PATCH end-points and there are examples of outputs and possible error messages returned by the API.

For instance to add a new video you can use curl command in your terminal as follow:

1. First login to get the Authorisation cookie:

curl -X POST http://localhost:4000/login --data @test/login.input.json -c -

You will get an output like following:

{"message":"Yaaay! You are logged in :)"}# Netscape HTTP Cookie File
# https://curl.haxx.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.

#HttpOnly_localhost     FALSE   /       FALSE   1685430242      Authorisation   eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHBpcmF0aW9uIjoxNjg1NDMwMjQyLCJpZGVudGlmaWVyIjoiYW5kcmVzIn0.ggF_5yEW5MQ6v7oY_c16h2sa2hxvxV12HeN4ZHPqtBc

1. Last line is the Authorisation cookie, we can use it for the subsequent calls:

curl -X POST http://localhost:4000/videos --data @test/videos/add.input.json -b "Authorisation=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHBpcmF0aW9uIjoxNjg1NDMwMjQyLCJpZGVudGlmaWVyIjoiYW5kcmVzIn0.ggF_5yEW5MQ6v7oY_c16h2sa2hxvxV12HeN4ZHPqtBc"

Then it should return a JSON output similar to following:

{"id":10,"user_id":3,"title":"synthwave radio 🌌 - beats to chill/game to","description":"🤗 Thank you for listening, I hope you will have a good time here","link":"https://www.youtube.com/watch?v=MVPTGNGiI-4","duration":"00:07:30","created_at":"2023-05-23T07:09:08.504423801Z","updated_at":"2023-05-23T07:09:08.504423801Z","annotations":null}%

Following is an screenshot of that test. As you can see, even the emojis are well supported 😊:

Another option to play with the API is use Postman:

If you are in development mode you should be able to see the logs from Database and the HTTP requests in the terminal where the API is running:

Or also within Docker desktop application:

Even more, if you have a SQLite database client (e. g. SQLite Viewer for VSCode) you may see how the records are in the Database:

♻️ Automated

Automated unit testing has been implemented and they run on each push and pull requests within the GitHub Actions Pipeline and when the Docker image is build. Following is how they are shown in GitHub website:

Some of those unit testing use DDT (Data Driven Testing) approach in order to test different inputs for similar scenarios or expected behaviours.

The mocks within the mocks directory/package of the repository were generated with mockery command line tool to simulate the interaction with the Database via gorm.DB and the HTTP requests and responses from gin.Engine during the unit testing.

💯 Coverage

You can follow the test coverage reports of this project in the CodeCov website:

IMPORTANT NOTE: Even that we can see there is a good coverage , that doesn't mean the API is flawless, as it was mentioned in the Assumptions section there are many chances to improve and for sure it may have even more.

📚 References

• SQLite Data Types
• GORM Documentation
• Gin Documentation
• Testify Documentation
• Monkey Patching Documentation
• Crypto/Bcrypt Documentation
• GoJWT Documentation
• GoDotEnv
• GitHub Actions Documentation

---