Documentation
¶
Overview ¶
Package dsmapperlite implements an in-process datastore mapper.
Unlike its bigger sibling dsmapper, it doesn't distribute mapping operations across machines, but in exchange has a very simple API. There's no need to install it as a server module or setup task queue tasks etc. Just use is a library.
Useful for quickly visiting up to 100K entities.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Map ¶
func Map[E any](ctx context.Context, q *datastore.Query, shards, batchSize int, cb func(ctx context.Context, shard int, entity E) error) error
Map passes all entities matching the query to the callback, in parallel, in some random order.
Runs up to `shards` number of parallel goroutines, where each one executes a datastore query and passes the resulting entities to the callback (along with the shard index). Each query fetches entities in `batchSize` pages before handling them. The overall memory consumption is thus `O(shards * batchSize * averageEntitySize)`.
Within a shard, the callback is called sequentially, but different shards are processed in parallel. If the callback needs to parallelize entity processing more, it should manage its own goroutine pool and pass entities to it.
If the callback returns an error, Map aborts the entire operation ASAP (but it may take some time to wind down). When this happens, the context passed to the callback is canceled.
Types ¶
This section is empty.
Source Files