Common service structure for different types of processors
We need a single package that would take care of running and gracefully shutting down
all types of processors
Cross-session service identification
Each service instance must have its persistent ID that can be used as a consumer group label for queues.
Service discovery
We need to collect information about services (their presence, id, other properties)
Clean Code/Architecture
Decompose transport and business logic (reading events must be separated from their processing)
Task Restarts
A restart seems to be an easy concept but in the world of kantoku a few problems arise:
Should task be recreated (or sending a new task_ready event is enough)? Should output resources be recreated (if
yes, how to manage dependents of the failed task?)?
Restarting a task parallelly can cause double output resource resolution. So, it's crucial to guarantee some kind
of synchronization of the action. It can be done by delegating the restarting logic to scheduler (actually it's
not a complete solution yet, synchronization among multiple scheduler instances must be also provided) or by
setting some flag in the restarted task's info, e.g. restarted: true (this approach also has a tricky part: the
task storage should guarantee atomic writes and such stuff).
Restart context id and restart parent id: restarts represent a simple tree.
Investigate alternatives for records
Consider using raw mql requests (can be used with Postgres via FerretDB)
Consider using postgres + jsonb
Implement auto tasks parsing from source code
Use "source" column in the specifications table to identify which tasks are expected to be deleted
Either parse task from source code structures with embedded kantoku.Task
or register tasks manually in some package-wise router (in this case, code generation for task files is
preferrable)
Pipelines
Pipelines are DAGs of tasks. It's pretty simple to implement but it requires good visualization along with support for
partial restarts (a task restart without output reallocation).
Preprocessors: pure functions written in some scripting language that are used for a slight data rectification in
order to match a dependent task input's type.
Processor Nodes
Nodes that would read the events and process them with multiple handlers (status update/garbage collector/etc)
WebSocket API
Use Centrifugo for notifications about updates in a context/task/namespace
CLI (ktk)
Design a ktk cli utility similar to kubectl
API
Make it follow REST principles (stop using POST for all requests)
Consider using jsonrpc
Spawners
A spawner is a new entity. It is a program that spawns new tasks when some event happens. Might be a kafka message, a cron event or a telegram
message.
Batching
APIs and services should support batched execution. It would reduce amount of network requests to databases and
overhead of small requests
Garbage collector
Garbage collection is necessary to limit the amount of used resources
Scheduler
Dependencies
Make resource dependency resolver instant using resource initialization notifier.
What other resolvers can be useful?
Should dependencies fail or have any other statuses besides PENDING and READY?
Investigate approaches for a better dependencies engine
Make it scalable
Executor
Executor interface is pretty simple and the current implementation is sufficient for basic usage. In the future (
meaning that this task shouldn't be taken until more fundamental problems are solved) it'd be nice to have a more
complex and environment-aware version. Some points on what'd be cool to see:
Select the running machine based on some set of labels in task's info
Shard tasks by their execution context / resource usage. This would improve usage of a local resource cache
Implement Leader/Follower decomposition.
Infra
Save logs to some persistent storage (OpenSearch/Elastic/Loki)
Use Prometheus and Grafana for metrics
When environment is not set up (Docker isn't running), services start to ddos the components (Consul, Postgres,
etc). We need timeouts.
Tests
Add tests for all services
Add tests for kantoku (end2end)
Write unit tests for common stuff
Docs
Create a JS emulator of the whole system and show it using reactflow
Web UI
Task List
Fix failed and ok statuses
Add running status (a.k.a received)
Make ID a hyperlink to task (and remove the eye action)
Make one-sided filter on updated_at
Add updated_at column and make it sortable
Task Show
Show the task's status (ok, failed, running...)
Show type (aka spec) and info about it
List dependencies
Show updated_at
List parameters (aka inputs) and outputs as a form
Show all info as json editor, make its style fit the antd's theme