sling-cli

README

Slings from a data source to a data target.

See docs.slingdata.io for more details.

Sling is a passion project turned into a free CLI Product which offers an easy solution to create and maintain small to medium volume data pipelines using the Extract & Load (EL) approach. It focuses on data movement between:

• Database to Database
• File System to Database
• Database to File System

https://github.com/slingdata-io/sling-cli/assets/7671010/e10ee716-1de8-4d53-8eb2-95c6d9d7f9f0

Some key features:

• Single Binary deployment (built with Go). See installation page.
• Use Custom SQL as a stream: --src-stream='select * from my_table where col1 > 10'
• Manage / View / Test / Discover your connections with the sling conns sub-command
• Use Environment Variable as connections if you prefer (export MY_PG='postgres//...)'
• Provide YAML or JSON configurations (perfect for git version control).
• Powerful Replication logic, to replication many tables with a wildcard (my_schema.*).
• Reads your existing DBT connections
• Use your environment variable in your YAML / JSON config (select * from my_table where date = '{date}')
• Convenient Transformations, such as the flatten option, which auto-creates columns from your nested fields.
• Run Pre & Post SQL commands.
• many more!

---

Example Replication:

---

Available Connectors:

• Databases: bigquery bigtable clickhouse duckdb mariadb motherduck mysql oracle postgres redshift snowflake sqlite sqlserver starrocks prometheus proton
• File Systems: azure b2 dospaces gs local minio r2 s3 sftp wasabi
• File Formats: csv, parquet, xlsx, json, avro, xml, sas7bday

Here are some additional links:

• https://slingdata.io
• https://docs.slingdata.io
• https://blog.slingdata.io

---

Ever wanted to quickly pipe in a CSV or JSON file into your database? Use sling to do so:

cat my_file.csv | sling run --tgt-conn MYDB --tgt-object my_schema.my_table

Or want to copy data between two databases? Do it with sling:

sling run --src-conn PG_DB --src-stream public.transactions \
  --tgt-conn MYSQL_DB --tgt-object mysql.bank_transactions \
  --mode full-refresh

Sling can also easily manage our local connections with the sling conns command:

$ sling conns set MY_PG url='postgresql://postgres:myPassword@pghost:5432/postgres'

$ sling conns list
+--------------------------+-----------------+-------------------+
| CONN NAME                | CONN TYPE       | SOURCE            |
+--------------------------+-----------------+-------------------+
| AWS_S3                   | FileSys - S3    | sling env yaml    |
| FINANCE_BQ               | DB - BigQuery   | sling env yaml    |
| DO_SPACES                | FileSys - S3    | sling env yaml    |
| LOCALHOST_DEV            | DB - PostgreSQL | dbt profiles yaml |
| MSSQL                    | DB - SQLServer  | sling env yaml    |
| MYSQL                    | DB - MySQL      | sling env yaml    |
| ORACLE_DB                | DB - Oracle     | env variable      |
| MY_PG                    | DB - PostgreSQL | sling env yaml    |
+--------------------------+-----------------+-------------------+

$ sling conns discover LOCALHOST_DEV
9:05AM INF Found 344 streams:
 - "public"."accounts"
 - "public"."bills"
 - "public"."connections"
 ...

Installation

Brew on Mac

brew install slingdata-io/sling/sling

# You're good to go!
sling -h

Scoop on Windows

scoop bucket add sling https://github.com/slingdata-io/scoop-sling.git
scoop install sling

# You're good to go!
sling -h

Binary on Linux

curl -LO 'https://github.com/slingdata-io/sling-cli/releases/latest/download/sling_linux_amd64.tar.gz' \
  && tar xf sling_linux_amd64.tar.gz \
  && rm -f sling_linux_amd64.tar.gz \
  && chmod +x sling

# You're good to go!
sling -h

Compiling From Source

Requirements:

• Install Go 1.22+ (https://go.dev/doc/install)
• Install a C compiler (gcc, tdm-gcc, mingw, etc)

Linux or Mac

git clone https://github.com/slingdata-io/sling-cli.git
cd sling-cli
bash scripts/build.sh

./sling --help

Windows (PowerShell)

git clone https://github.com/slingdata-io/sling-cli.git
cd sling-cli

.\scripts\build.ps1

.\sling --help

Automated Dev Builds

Here are the links of the official development builds, which are the latest builds of the upcoming release.

• Linux (x64): https://f.slingdata.io/dev/latest/sling_linux_amd64.tar.gz
• Mac (arm64): https://f.slingdata.io/dev/latest/sling_darwin_arm64.tar.gz
• Windows (x64): https://f.slingdata.io/dev/latest/sling_windows_amd64.tar.gz

Installing via Python Wrapper

pip install sling

Then you should be able to run sling --help from command line.

Contributing

We welcome contributions to improve Sling! Here are some guidelines to help you get started.

Branch Naming Convention

When creating a new branch for your contribution, please use the following naming convention:

• feature/your-feature-name for new features
• bugfix/issue-description for bug fixes
• docs/update-description for documentation updates

Testing Guidelines

Sling has three main test suites: Database, File and CLI. When contributing, please ensure that your changes pass the relevant tests.

Running Tests

To run the full test suite, run below. However you'd need to define all the needed connections as shown here, so it's recommended to target specific tests instead.

./scripts/build.sh
./scripts/test.sh

Targeting Specific Tests

You can target specific tests or suites using environment variables:

1. Database Suite:

   cd cmd/sling
   go test -v -run TestSuiteDatabasePostgres             # run all Postgres tests
   TESTS="1-3" go test -v -run TestSuiteDatabasePostgres # run Postgres tests 1, 2, 3
   

2. File Suite:

   cd cmd/sling
   go test -v -run TestSuiteFileS3               # run all S3 tests
   TESTS="1,2,3" go test -v -run TestSuiteFileS3 # run S3 tests 1, 2, 3
   

3. CLI Suite:

   cd cmd/sling
   export SLING_BIN=../../sling
   go test -v -run TestCLI             # run all CLI tests
   TESTS="31+" go test -v -run TestCLI # run CLI tests 31 and all subsequent tests
   

You can specify individual test numbers, ranges, or use the '+' suffix to run all tests from a certain number:

• TESTS="1,2,3": Run tests 1, 2, and 3
• TESTS="1-5": Run tests 1 through 5
• TESTS="3+": Run test 3 and all subsequent tests

Test Suites Overview

1. Database Suite: Tests database-related functionality.

   • Located in: cmd/sling/sling_test.go
   • Configuration: cmd/sling/tests/suite.db.template.tsv

2. File Suite: Tests file system operations.

   • Located in: cmd/sling/sling_test.go
   • Configuration: cmd/sling/tests/suite.file.template.tsv

3. CLI Suite: Tests command-line interface functionality.

   • Located in: cmd/sling/sling_cli_test.go
   • Configuration: cmd/sling/tests/suite.cli.tsv

Adding New Tests

When introducing new features or addressing bugs, it's essential to incorporate relevant tests, focusing mainly on the CLI suite file located at cmd/sling/suite.cli.tsv. The database and file suites serve as templates applicable across all connectors, making them more sensitive to modifications. Therefore, any changes to these suites will be managed internally.

When adding new test entries in the CLI suite file, feel free to create a new replication file in folder cmd/sling/tests/replications, or a corresponding source file in the cmd/sling/tests/files directory. Also include the expected output or the number of expected rows/streams in the new test entry.

Pull Request Process

1. Ensure your code adheres to the existing style and passes all tests.
2. Update the README.md with details of changes to the interface, if applicable.
3. Create a Pull Request with a clear title and description.

Thank you for contributing to Sling!