obrc

README

obrc

An unoptimized, 2m9s One Billion Row Challenge solution in Go.

(I skipped the output sorting because I'm lazy.)

Usage

Generate a CSV:

go run ./cmd/gen -n 1000 > test.csv

Run the solver:

go run ./cmd/stats -input test.csv

Benchmark

MacBook Pro 14-inch, 2021, M1 Max, 64 GB:

$ go test -bench=. .
goos: darwin
goarch: arm64
pkg: github.com/clfs/obrc
BenchmarkGenerateCSV1e3-10       	    5095	    234506 ns/op
BenchmarkGenerateCSV1e4-10       	     512	   2333130 ns/op
BenchmarkGenerateCSV1e5-10       	      50	  23381521 ns/op
BenchmarkGenerateCSV1e6-10       	       5	 232976692 ns/op
BenchmarkTakeRecordings1e3-10    	    7417	    148938 ns/op
BenchmarkTakeRecordings1e4-10    	     903	   1321701 ns/op
BenchmarkTakeRecordings1e5-10    	      93	  12592483 ns/op
BenchmarkTakeRecordings1e6-10    	       9	 124397972 ns/op
PASS
ok  	github.com/clfs/obrc	13.300s

$ go run ./cmd/gen -n 1_000_000_000 > test.csv
$ time go run ./cmd/stats -input test.csv
...
... 126.41s user 3.52s system 100% cpu 2:09.61 total

Documentation

Overview

Package obrc implements the One Billion Row Challenge.

Index

• Variables
• func Generate(w *csv.Writer, n int) error
• func NewReader(r io.Reader) *csv.Reader
• func NewWriter(w io.Writer) *csv.Writer
• func TakeRecordings(r *csv.Reader) (map[string]Recording, error)
• type Recording
  • func NewRecording(measurement float64) Recording
  • func (r *Recording) Add(measurement float64)
  • func (r *Recording) Max() float64
  • func (r *Recording) Mean() float64
  • func (r *Recording) Min() float64
• type Station
  • func RandStation() Station
  • func (s *Station) Measure() float64

Variables

var AllStations = []Station{}/* 413 elements not displayed */

AllStations contains all weather stations.

Functions

func Generate

func Generate(w *csv.Writer, n int) error

Generate writes n measurements to w.

func NewReader

func NewReader(r io.Reader) *csv.Reader

NewReader returns a new csv.Reader with recommended settings.

func NewWriter

func NewWriter(w io.Writer) *csv.Writer

NewWriter returns a new csv.Writer with recommended settings.

func TakeRecordings

func TakeRecordings(r *csv.Reader) (map[string]Recording, error)

TakeRecordings takes a recording for each station.

Types

type Recording

type Recording struct {
	// contains filtered or unexported fields
}

Recording stores aggregate statistics about a weather station.

func NewRecording

func NewRecording(measurement float64) Recording

NewRecording starts a new Recording.

func (*Recording) Add

func (r *Recording) Add(measurement float64)

Add adds a measurement to the recording.

func (*Recording) Max

func (r *Recording) Max() float64

Max returns the maximum measurement.

func (*Recording) Mean

func (r *Recording) Mean() float64

Mean returns the mean measurement.

func (*Recording) Min

func (r *Recording) Min() float64

Min returns the minimum measurement.

type Station

type Station struct {
	Name     string
	MeanTemp float64
}

Station is a weather station.

func RandStation

func RandStation() Station

RandStation returns a random weather station.

func (*Station) Measure

func (s *Station) Measure() float64

Measure returns a random temperature for the station.