scripts/

README

Scripts

A list of scripts which load data into elasticsearch for use in the Search API.

A list of scripts

• load postcodes
• load parent docs
• arcgis boundaries
• geojson
  • 2011 Lower Layer Super Output Areas (LSOA)
  • 2011 Middle Layer Super Output Areas (MSOA)
  • 2011 Output Areas (OA)
  • 2015 Towns and Cities (TCITY)

Load Postcode

This script loads postcode data for all postcodes across the UK as of Febraury 2020 from a csv file downloaded from the geo portal here.

Once file is downloaded (from above link), unzip file. The data layout to postcode data should look like this: - NSPL_FEB_2020_UK - Data - NSPL_FEB_2020_UK.csv

Upload postcode data to elasticsearch index with: make postcode This will take approximately 4 minutes and 20 seconds and documents will be stored in test_postcode index.

Load Parent Docs

This script loads dummy data stored in test-data/datasets.csv and upload geo location docs into Elasticsearch.

Upload data to elasticsearch index with: make parent This will take less than a few seconds as it uploads 7 documents, this may increase over time as more documents get added test_parent index.

ARCGIS Boundaries

This script calls the geoportal API in attempt to process lsoa data in the form of JSON.

Upload data to elasticsearch index with: make arcgis

Limited to 2000 of the 34000+ geographical areas. Likely to be a paging issue. Data is added to test_arcgis index.

Load data from GEOJSON files

This script loads geographical boundaries for 2011 census data. This includes lower and middle layer output areas, as well as other output areas, towns and cities across England and Wales only.

Files can be downloaded from the geoportal -> boundaries -> census boundaries -> select geography layer. This will tend to open up a search of all relevant boundaries, select the data you would like to view/import. The new screen will have a drop down list to the right of webpage titled APIs, click the drop down and copy the GEOJSON url. Paste the url into the browser and it will automatically download the data, be patient this may take some time; below is a list of urls used for the geojson scripts (these might break if geoportal decides to move the geojson file location):

• Major Towns And Cities 2015
• Middle layer super output areas december 2011 ew-bfc
• Middle layer super output areas december 2011 ew-bfe
• Middle layer super output areas december 2011 ew-bgc
• Middle layer super output areas december 2011 ew-bsc
• Lower layer super output areas december 2011 ew-bfc
• Lower layer super output areas december 2011 ew-bfe
• Lower layer super output areas december 2011 ew-bgc
• Lower layer super output areas december 2011 ew-bsc
• Output areas december 2011 ew-bfc
• Output areas december 2011 ew-bfe
• Output areas december 2011 ew-bgc

Once the above files have downloaded, move the files to root of this repository and store under geojson folder.

Upload all the data to elasticsearch index with: make geojson This will take a long time as it i populates 700,000+ records with full polygon boundaries into elasticsearch test_geo index.

There are actually four separate scripts which handle generating data for LSOA, MSOA, OA and TCITY files. These can be run separately using make lsoa, make msoa, make oa, make tcity respectively. Be aware that if you are running this for the first time you will need to create the test_geo index, you can do this by running make refreshgeojson.

The refresh script deletes the index and recreates it with 0 data.