colouring-montreal/etl/README.md

109 lines
4.2 KiB
Markdown
Raw Normal View History

2022-04-01 06:23:24 -04:00
# Extract, transform and load
The scripts in this directory are used to extract, transform and load (ETL) the core datasets for Colouring London. This README acts as a guide for setting up the Colouring London database with these datasets and updating it.
2022-04-01 05:32:53 -04:00
# Contents
- :arrow_down: [Downloading Ordnance Survey data](#arrow_down-downloading-ordnance-survey-data)
- :penguin: [Making data available to Ubuntu](#penguin-making-data-available-to-ubuntu)
2022-04-01 05:37:56 -04:00
- :new_moon: [Creating a Colouring London database from scratch](#new_moon-creating-a-colouring-london-database-from-scratch)
2022-04-01 05:32:53 -04:00
- :full_moon: [Updating the Colouring London database with new OS data](#full_moon-updating-the-colouring-london-database-with-new-os-data)
# :arrow_down: Downloading Ordnance Survey data
2018-09-25 15:46:16 -04:00
2022-04-01 06:23:24 -04:00
The building geometries are sourced from Ordnance Survey (OS) MasterMap (Topography Layer). To get the required datasets, you'll need to complete the following steps:
2022-03-09 08:24:27 -05:00
1. Sign up for the Ordnance Survey [Data Exploration License](https://www.ordnancesurvey.co.uk/business-government/licensing-agreements/data-exploration-sign-up). You should receive an e-mail with a link to log in to the platform (this could take up to a week).
2. Navigate to https://orders.ordnancesurvey.co.uk/orders and click the button for: ✏️ Order. From here you should be able to click another button to add a product.
3. Drop a rectangle or Polygon over London and make the following selections, clicking the "Add to basket" button for each:
![](screenshot/MasterMap.png)
<p></p>
2022-03-10 06:46:34 -05:00
4. You should be then able to check out your basket and download the files. Note: there may be multiple `.zip` files to download for MasterMap due to the size of the dataset.
2022-03-10 07:01:41 -05:00
6. Unzip the MasterMap `.zip` files and move all the `.gz` files from each to a single folder in a convenient location. We will use this folder in later steps.
2022-03-09 08:24:27 -05:00
2022-04-01 05:32:53 -04:00
# :penguin: Making data available to Ubuntu
2022-04-01 05:20:18 -04:00
Before creating or updating a Colouring London database, you'll need to make sure the downloaded OS files are available to the Ubuntu machine where the database is hosted. If you are using Virtualbox, you could host share folder(s) containing the OS files with the VM (e.g. [see these instructions for Mac](https://medium.com/macoclock/share-folder-between-macos-and-ubuntu-4ce84fb5c1ad)).
2022-04-01 05:32:53 -04:00
# :new_moon: Creating a Colouring London database from scratch
2022-04-01 05:20:18 -04:00
2018-09-25 15:46:16 -04:00
## Prerequisites
2022-04-01 05:50:34 -04:00
You should already have set up PostgreSQL and created a database in an Ubuntu environment. Make sure to create environment variables to use `psql` if you haven't already:
2022-03-09 06:48:45 -05:00
```bash
export PGPASSWORD=<pgpassword>
export PGUSER=<username>
export PGHOST=localhost
export PGDATABASE=<colouringlondondb>
```
Create the core database tables:
```bash
2022-03-09 08:33:26 -05:00
cd ~/colouring-london
psql < migrations/001.core.up.sql
```
There is some performance benefit to creating indexes after bulk loading data.
Otherwise, it's fine to run all the migrations at this point and skip the index
creation steps below.
2022-04-01 05:50:34 -04:00
You should already have installed GNU parallel, which is used to speed up loading bulk data.
2022-04-01 05:32:53 -04:00
## Processing and loading Ordnance Survey data
2018-09-25 15:46:16 -04:00
2022-03-11 08:28:23 -05:00
Move into the `etl` directory and set execute permission on all scripts.
2018-09-25 15:46:16 -04:00
2018-09-29 13:29:57 -04:00
```bash
2022-03-10 04:53:12 -05:00
cd ~/colouring-london/etl
2022-03-11 08:28:23 -05:00
chmod +x *.sh
2022-03-10 05:39:06 -05:00
```
Extract the MasterMap data (this step could take a while).
```bash
2022-03-28 08:54:00 -04:00
sudo ./extract_mastermap.sh /path/to/mastermap_dir
2022-03-10 05:39:06 -05:00
```
2022-03-18 12:49:46 -04:00
Filter MasterMap 'building' polygons.
2022-03-10 05:39:06 -05:00
```bash
2022-03-28 08:54:00 -04:00
sudo ./filter_transform_mastermap_for_loading.sh /path/to/mastermap_dir
2022-03-10 05:39:06 -05:00
```
2022-03-28 10:22:04 -04:00
Load all building outlines. Note: you should ensure that `mastermap_dir` has permissions that will allow the linux `find` command to work without using sudo.
2022-03-10 05:39:06 -05:00
```bash
2022-03-28 10:22:04 -04:00
./load_geometries.sh /path/to/mastermap_dir
2022-03-10 05:39:06 -05:00
```
Index geometries.
```bash
2022-03-10 05:44:31 -05:00
psql < ../migrations/002.index-geometries.up.sql
2022-03-10 05:39:06 -05:00
```
2022-03-28 10:23:40 -04:00
<!-- TODO: Drop outside limit. -->
2022-03-28 10:23:40 -04:00
<!-- ```bash
2022-03-28 08:54:00 -04:00
./drop_outside_limit.sh /path/to/boundary_file
2022-03-28 10:23:40 -04:00
```` -->
2022-03-10 05:39:06 -05:00
Create a building record per outline.
```bash
2022-03-11 08:28:23 -05:00
./create_building_records.sh
2022-03-10 05:39:06 -05:00
```
2022-03-10 05:44:31 -05:00
Run the remaining migrations in `../migrations` to create the rest of the database structure.
2022-03-10 05:39:06 -05:00
```bash
2022-03-10 05:44:31 -05:00
ls ~/colouring-london/migrations/*.up.sql 2>/dev/null | while read -r migration; do psql < $migration; done;
2018-09-29 13:29:57 -04:00
```
2022-04-01 05:32:53 -04:00
# :full_moon: Updating the Colouring London database with new OS data
2022-03-10 05:06:44 -05:00
TODO: this section should instruct how to update and existing db