a_adli/colouring-montreal
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4984ad9515
colouring-montreal/etl
History
Tom Russell 73aa3df290 Clip OSMM to GLA on extraction
2018-09-27 21:37:47 +01:00
..
join_building_data Rename etl scripts 2018-09-25 20:46:16 +01:00
check_ab_mm_match.py Use default pool size (CPU count) in ETL 2018-09-25 21:47:29 +01:00
create_building_records.sh Use default pool size (CPU count) in ETL 2018-09-25 21:47:29 +01:00
extract_addressbase.sh Rename etl scripts 2018-09-25 20:46:16 +01:00
extract_mastermap.sh Clip OSMM to GLA on extraction 2018-09-27 21:37:47 +01:00
filter_addressbase_csv.py Parallel extract/filter OS data 2018-09-21 11:10:39 +01:00
filter_mastermap.py Use default pool size (CPU count) in ETL 2018-09-25 21:47:29 +01:00
filter_transform_mastermap_for_loading.sh Rename etl scripts 2018-09-25 20:46:16 +01:00
get_test_polygons.py Fix get_test_polygons script 2018-09-25 22:01:09 +01:00
load_geometries.sh Rename etl scripts 2018-09-25 20:46:16 +01:00
load_uprns.py Update UPRN-load script 2018-09-25 21:47:58 +01:00
README.md Split indexing further, UPRN requires bigint 2018-09-25 21:46:22 +01:00
requirements.txt Record python packages for etl 2018-09-09 11:34:37 +01:00

README.md

Data loading

The scripts in this directory are used to extract, transform and load (ETL) the core datasets for Colouring London:

  1. Building geometries, sourced from Ordnance Survey MasterMap (Topography Layer)
  2. Unique Property Reference Numbers (UPRNs), sourced from Ordnance Survey AddressBase

Prerequisites

Before running any of these scripts, you will need the OS data for your area of interest. AddressBase and MasterMap are available directly from Ordnance Survey.

To help test the Colouring London app, get_test_polygons.py will attempt to save a small (1.5km²) extract from OpenStreetMap to a format suitable for loading to the database.

The scripts should be run in the following order:

  1. extract_addressbase.sh
  2. extract_mastermap.sh
  3. filter_transform_mastermap_for_loading.sh
  4. load_geometries.sh
  5. create_building_records.sh
  6. (SQL migration) psql < ../migrations/002.index-geometries.sql
  7. load_uprns.py
  8. (SQL migration) psql < ../migrations/002.index-buildings.sql