dsmapperlite

package
v0.0.0-...-7bf56e6 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Dec 3, 2024 License: Apache-2.0 Imports: 6 Imported by: 0

Documentation

Overview

Package dsmapperlite implements an in-process datastore mapper.

Unlike its bigger sibling dsmapper, it doesn't distribute mapping operations across machines, but in exchange has a very simple API. There's no need to install it as a server module or setup task queue tasks etc. Just use is a library.

Useful for quickly visiting up to 100K entities.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Map

func Map[E any](ctx context.Context, q *datastore.Query, shards, batchSize int, cb func(ctx context.Context, shard int, entity E) error) error

Map passes all entities matching the query to the callback, in parallel, in some random order.

Runs up to `shards` number of parallel goroutines, where each one executes a datastore query and passes the resulting entities to the callback (along with the shard index). Each query fetches entities in `batchSize` pages before handling them. The overall memory consumption is thus `O(shards * batchSize * averageEntitySize)`.

Within a shard, the callback is called sequentially, but different shards are processed in parallel. If the callback needs to parallelize entity processing more, it should manage its own goroutine pool and pass entities to it.

If the callback returns an error, Map aborts the entire operation ASAP (but it may take some time to wind down). When this happens, the context passed to the callback is canceled.

Types

This section is empty.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL