crawler

Documentation

Overview

This program crawls the LOC.gov API and identifies digitized items.

It proceeds in this way.

1. It fetches all digital collections (with some filtering)
2. It fetches the items in those digital collections (again with some filtering for full text items).

Everything gets stored in a database.

The task of fetching the full item metadata is handled by itemmd.