README
¶
VMware-Go-KCL
A Fork? Why?
The upstream project does not allow for the use of a single DynamoDB table for N many workers. This fork has changed the schema for the DynamoDB checkpointer to allow for this.
Overview
Amazon Kinesis enables real-time processing of streaming data at massive scale. Kinesis Streams is useful for rapidly moving data off data producers and then continuously processing the data, be it to transform the data before emitting to a data store, run real-time metrics and analytics, or derive more complex data streams for further processing.
The VMware Kinesis Client Library for GO (VMware-Go-KCL) enables Go developers to easily consume and process data from [Amazon Kinesis][kinesis].
VMware-Go-KCL brings Go/Kubernetes community with Go language native implementation of KCL matching exactly the same API and functional spec of original Java KCL v2.0 without the resource overhead of installing Java based MultiLangDaemon.
Try it out
Prerequisites
Make sure hmake is version 1.3.1 or above and go is version 1.11 or above
hmake --version
1.3.1
Make sure to launch Docker daemon with specified DNS server --dns DNS-SERVER-IP
On Ubuntu, update the file /etc/default/docker to put --dns DNS-SERVER-IP in DOCKER_OPTS.
On Mac, set DNS in Docker Preferences – Daemon – Insecure registries
Build & Run
hmake
# security scan
hmake scanast
# run test
hmake check
# run integration test
# update the worker_test.go to let it point to your Kinesis stream
hmake test
Documentation
VMware-Go-KCL matches exactly the same interface and programming model from original Amazon KCL, the best place for getting reference, tutorial is from Amazon itself:
Contributing
The vmware-go-kcl project team welcomes contributions from the community. Before you start working with vmware-go-kcl, please read our Developer Certificate of Origin. All contributions to this repository must be signed as described on that page. Your signature certifies that you wrote the patch or have the right to pass it on as an open-source patch. For more detailed information, refer to CONTRIBUTING.md.
License
MIT License
Directories
¶
| Path | Synopsis |
|---|---|
|
clientlibrary
|
|
|
checkpoint
* Copyright (c) 2018 VMware, Inc.
|
* Copyright (c) 2018 VMware, Inc. |
|
common
* Copyright (c) 2018 VMware, Inc.
|
* Copyright (c) 2018 VMware, Inc. |
|
config
* Copyright (c) 2018 VMware, Inc.
|
* Copyright (c) 2018 VMware, Inc. |
|
interfaces
* Copyright (c) 2018 VMware, Inc.
|
* Copyright (c) 2018 VMware, Inc. |
|
metrics
* Copyright (c) 2018 VMware, Inc.
|
* Copyright (c) 2018 VMware, Inc. |
|
partition
* Copyright (c) 2018 VMware, Inc.
|
* Copyright (c) 2018 VMware, Inc. |
|
utils
* Copyright (c) 2018 VMware, Inc.
|
* Copyright (c) 2018 VMware, Inc. |
|
worker
* Copyright (c) 2018 VMware, Inc.
|
* Copyright (c) 2018 VMware, Inc. |