pipeline-backend
pipeline-backend
manages all pipeline resources within Versatile Data Pipeline (VDP) to streamline data from a source connector to models, and to a destination connector at the end.
Local dev
On the local machine, clone vdp
repository in your workspace, move to the repository folder, and launch all dependent microservices:
$ cd <your-workspace>
$ git clone https://github.com/instill-ai/vdp.git
$ cd vdp
$ make latest PROFILE=pipeline
Clone pipeline-backend
repository in your workspace and move to the repository folder:
$ cd <your-workspace>
$ git clone https://github.com/instill-ai/pipeline-backend.git
$ cd pipeline-backend
Build the dev image
$ make build
Run the dev container
$ make dev
Now, you have the Go project set up in the container, in which you can compile and run the binaries together with the integration test in each container shell.
Run the server
$ docker exec -it pipeline-backend /bin/bash
$ go run ./cmd/migration
$ go run ./cmd/main
Run the integration test
$ docker exec -it pipeline-backend /bin/bash
$ make integration-test
Stop the dev container
$ make stop
CI/CD
- pull_request to the
main
branch will trigger the Integration Test
workflow running the integration test using the image built on the PR head branch.
- push to the
main
branch will trigger
- the
Integration Test
workflow building and pushing the :latest
image on the main
branch, following by running the integration test, and
- the
Release Please
workflow, which will create and update a PR with respect to the up-to-date main
branch using release-please-action.
Once the release PR is merged to the main
branch, the release-please-action will tag and release a version correspondingly.
The images are pushed to Docker Hub repository.
License
See the LICENSE file for licensing information.