colouring-montreal/etl

Creating a Colouring London database from scratch

Data downloading

The scripts in this directory are used to extract, transform and load (ETL) the core datasets for Colouring London:

1. Building geometries, sourced from Ordnance Survey (OS) MasterMap (Topography Layer)
2. Unique Property Reference Numbers (UPRNs), sourced from Ordnance Survey AddressBase

To get the required datasets, you'll need to complete the following steps:

1. Sign up for the Ordnance Survey Data Exploration License. You should receive an e-mail with a link to log in to the platform (this could take up to a week).
2. Navigate to https://orders.ordnancesurvey.co.uk/orders and click the button for: ✏️ Order. From here you should be able to click another button to add a product.
3. Drop a rectangle or Polygon over London and make the following selections, clicking the "Add to basket" button for each:

4. You should be then able to check out your basket and download the files

Prerequisites

You should already have set up PostgreSQL and created a database. Make sure to create environment variables to use psql if you haven't already:

export PGPASSWORD=<pgpassword>
export PGUSER=<username>
export PGHOST=localhost
export PGDATABASE=<colouringlondondb>

Create the core database tables:

cd ~/colouring-london
psql < migrations/001.core.up.sql

There is some performance benefit to creating indexes after bulk loading data. Otherwise, it's fine to run all the migrations at this point and skip the index creation steps below.

Install GNU parallel, this is used to speed up loading bulk data.

Make data available to Ubuntu

If you didn't download the OS files to the Ubuntu machine where you are setting up your Colouring London application, you will need to make them available there. If you are using Virtualbox, you could host share a the two folders containing the files (one for MasterMap, one for AddressBase) with the VM via a shared folder (e.g. see these instructions for Mac).

Process and load Ordnance Survey data

The scripts should be run in the following order:

cd ~/colouring-london/etl

Extract the addressBase dataset.

sudo ./extract_addressbase.sh ./addressbase_dir

Extract the MasterMap data (this step could take a while).

sudo ./extract_mastermap.sh ./mastermap_dir

Filter MasterMap 'building' polygons and any others referenced by addressbase.

sudo ./filter_transform_mastermap_for_loading.sh ./addressbase_dir ./mastermap_dir

Load all building outlines.

sudo ./load_geometries.sh ./mastermap_dir

Index geometries.

psql < ../migrations/002.index-geometries.up.sql

Create a building record per outline.

sudo ./create_building_records.sh

Ensure you have the colouringlondon environment activated, then add UPRNs where they match.

source colouringlondon/bin/activate
load_uprns.py ./addressbase_dir

Run the remaining migrations in ../migrations to create the rest of the database structure.

ls ~/colouring-london/migrations/*.up.sql 2>/dev/null | while read -r migration; do psql < $migration; done;

[WIP] Updating the Colouring London database with new OS data

TODO: this section should instruct how to update and existing db