db

README

postgres database

We are using POSTGRESQL as the store for the raw scraped data from the various data sources.
The schemas are quite similar to the scraped data structures.

Table of Contents

• Instagram
  • Remarks
• Twitter
• Debezium

Instagram

This database is the more sophisticated one and is running in production.

Remarks

• internal_picture_url is pointing to the downloaded picture on S3

Twitter

This database is not in production yet and at the moment only dumps the tweaked scraped data.

Debezium

The debezium connector generates a change stream from all change events in postgres (read, create, update, delete) and writes them into a kafka-topic "postgres.public.<table_name>"

To read from this stream you can:

• get kafkacat
• inspect the topic list in kafka:

  $ kafkacat -L -b my-kafka | grep 'topic "postgres'
  

• consume a topic with

  $ kafkacat -b my-kafka -t <topic_name>
  

The messages are quite verbose, since they include their own schema description. The most interesting part is the value.payload:

$ kafkacat -b my-kafka -topic postgres.public.users | jq '.value | fromjson | .payload'`

Documentation

Index

• func Create(db *gorm.DB, out interface{}, update interface{}) error
• func CreateOrUpdate(db *gorm.DB, out interface{}, where interface{}, update interface{}) error

Functions

func Create

func Create(db *gorm.DB, out interface{}, update interface{}) error

Create just create a new entry in the database

func CreateOrUpdate

func CreateOrUpdate(db *gorm.DB, out interface{}, where interface{}, update interface{}) error

CreateOrUpdate checks whether a specific (gorm) database entry already exists using a model filter, creates it in case no record is found and updates the first in case of existing record(s)