playground/

README

The Beam Playground is a web application to run Beam code snippets in a modern browser. This directory holds code to build, test, and deploy the frontend and backend services.

• Setup development prerequisites
• Common tasks
• Run Beam Playground locally
  • Configure frontend to use local backend
  • Run local deployment using Gradle task
  • Deploy examples
• How to add your own example
• Deployment guide
  • Manual deployment
  • Manual CloudBuild setup
• Project structure
• Contribution guide

Setup development prerequisites

Google Cloud Shell note: Google Cloud Shell already has most of the prerequisites installed. Only the following has to be installed manually:

• Flutter - run flutter precache
• Go protobuf dependencies
• Dart protobuf dependencies
• buf
• sbt

1. Install Go 1.20+

   Ubuntu 22.04 and newer:

   sudo apt install golang
   

   Other Linux variants: Follow manual at https://go.dev/doc/install

2. Install flutter

   Ubuntu 22.04 or newer:

   sudo apt install flutter
   

   Other Linux variants: Follow manual at https://flutter.dev/docs/get-started/install/linux

3. Install protoc

   Ubuntu 22.04 or newer/Debian 11 or newer:

   sudo apt install protobuf-compiler
   

   Other Linux variants: Follow manual at https://grpc.io/docs/protoc-installation/

4. Install Go protobuf dependencies: Go gRPC Quickstart

   Do not forget to update your PATH environment variable

5. Install Dart protobuf dependencies: Dart gRPC Quickstart

   Do not forget to update your PATH environment variable

6. Install npm Ubuntu 22.04 and newer/Debian 11 and newer:

   sudo apt install npm
   

   Other Linux variants: Follow manual at https://docs.npmjs.com/downloading-and-installing-node-js-and-npm

7. Install buf

   npm install -g @bufbuild/buf
   

8. Install Docker Ubuntu 22.04 and newer/Debian 11 and newer:

   sudo apt install docker.io
   

   Other Linux variants: Follow manual at https://docs.docker.com/engine/install/

   To verify your docker installation, run:

   docker ps
   

   It should finish without any errors. If you get a permission denied error, you need to add your user to the docker group:

   sudo usermod -aG docker $USER
   

   Then, log out and log back in to apply the changes.

9. Install Docker Compose Ubuntu 22.04 and newer/Debian 11 and newer:

   sudo apt install docker-compose
   

   Other Linux variants: Follow manual at https://docs.docker.com/compose/install/

10. Install gcloud CLI by following the manual for your system

11. Install Cloud Datastore Emulator

    gcloud components install cloud-datastore-emulator
    

    or, if you have install gcloud CLI using APT:

    sudo apt install google-cloud-cli-datastore-emulator
    

12. Install sbt following instruction at https://www.scala-sbt.org/1.x/docs/Installing-sbt-on-Linux.html

    Optional: Run sbt comamnd once after installation to let it cache its dependencies

13. Make sure that you have the full installation of Python 3.8 or newer. On Debian or Ubuntu it can be installed using sudo apt install python3-full.

Common tasks

To get an overview of common tasks, see TASKS.md

Run Beam Playground locally

Configure frontend to use local backend

Note: Follow this step only if you want to have a local deployment of Playground. Skip this step entirely if you want to deploy Playground to Google Cloud.

Uncommend lines after // Uncomment the following lines to use local backend. in frontend/playground_components/lib/src/constants/backend_urls.dart

Run local deployment using Gradle task

For more information read the corresponding section in TASKS.md

Run the deployment script:

./gradlew :playground:dockerComposeLocalUp

To shut down the playground, run:

./gradlew :playground:dockerComposeLocalDown

Deploy examples

1. Go to /playground/infrastucture directory

   cd playground/infrastucture
   

2. Setup python venv

   python3 -m venv venv
   

3. Activate venv

   source venv/bin/activate
   

4. Install dependencies

   pip install -r requirements.txt
   

5. Setup the environment variables

   export BEAM_ROOT_DIR=$(realpath ../../)
   export SDK_CONFIG="../../playground/sdks.yaml"
   export BEAM_EXAMPLE_CATEGORIES="../categories.yaml"
   export BEAM_USE_WEBGRPC=yes
   export BEAM_CONCURRENCY=4
   export DATASTORE_EMULATOR_HOST=localhost:8081
   

6. Run the ci_cd.py script

   export SERVER_ADDRESS=<runner_address> # see the note below
   python ci_cd.py --step CD \
                   --sdk <SDK> \ # SDK_GO, SDK_JAVA, SDK_PYTHON, SDK_SCIO
                   --namespace Playground \
                   --datastore-project test \
                   --origin PG_EXAMPLES \
                   --subdirs $BEAM_ROOT_DIR/sdks $BEAM_ROOT_DIR/examples $BEAM_ROOT_DIR/learning/katas # For SCIO examples see note below
   

   Note: The SERVER_ADDRESS variable should be set to the address of the runner server for the particular SDK. For the local deployment the default values are:

   SDK	Address
   Go	localhost:8084
   Java	localhost:8086
   Python	localhost:8088
   SCIO	localhost:8090

   Note: SCIO examples are not committed to the Beam tree. You will need to use fetch_scala_examples.py script to download them to some directory and then use --subdirs option to point to that directory. For example:

   python fetch_scala_examples.py --output-dir /tmp/scio-examples
   python ci_cd.py --step CD \
                   --sdk SDK_SCIO \
                   --namespace Playground \
                   --datastore-project test \
                   --origin PG_EXAMPLES \
                   --subdirs /tmp/scio-examples
   

   See this section for more information.

How to add your own example

Please refer to this document.

Deployment guide

Manual deployment

See deployment guide at terraform/README.md

Manual CloudBuild setup

To set up CloudBuild triggers manually please refer to this guide

Project structure

Several directories in this repository are used for the Beam Playground project. The list of the directories used can be seen below.

Directory	Purpose
/examples	Contains code of the examples for Java SDK. The examples are getting loaded into the main catalog.
/learning/beamdoc	Contains code of the examples which should not be available in the main Playground catalog.
/learning/katas	Containes small code examples, loaded into main catalog.
/playground	Root of Playground sources.
/playground/api	Protobuf definitions for the Playground gRPC API.
/playground/backend	Root of Playground backend sources. See this document for detailed description.
/playground/frontend	Root of Playground frontend sources. See this document for detailed description.
/playground/infrastructure	Scripts used in Playground CI/CD pipelines for verifying and uploading examples.
/playground/kafka-emulator	Sources of a Kafka emulator used for demonstrating KafkaIO examples.
/playground/terraform	Terraform configuration files for Playground deployment to Google Cloud Platform.
/sdks	Source of the BEAM SDKs. Used by Playground as a source of examples for Go and Python SDKs.

Contribution guide

• Backend: see backend/README.md and backend/CONTRIBUTE.md
• Frontend: see frontend/README.md and frontend/CONTRIBUTE.md