a_adli/colouring-montreal
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

rearrange sections

Browse Source
This commit is contained in:
Ed Chalstrey 2022-04-01 10:20:18 +01:00
parent a3d2537e22
commit b470591420
1 changed files with 7 additions and 7 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
etl/README.md
View File

@ -1,6 +1,4 @@
# Creating a Colouring London database from scratch
## Data downloading
# Downloading Ordnance Survey data
The scripts in this directory are used to extract, transform and load (ETL) the core datasets
for Colouring London:
@ -19,6 +17,12 @@ To get the required datasets, you'll need to complete the following steps:
4. You should be then able to check out your basket and download the files. Note: there may be multiple `.zip` files to download for MasterMap due to the size of the dataset.
6. Unzip the MasterMap `.zip` files and move all the `.gz` files from each to a single folder in a convenient location. We will use this folder in later steps.
# Make data available to Ubuntu
Before creating or updating a Colouring London database, you'll need to make sure the downloaded OS files are available to the Ubuntu machine where the database is hosted. If you are using Virtualbox, you could host share folder(s) containing the OS files with the VM (e.g. [see these instructions for Mac](https://medium.com/macoclock/share-folder-between-macos-and-ubuntu-4ce84fb5c1ad)).
# Creating a Colouring London database from scratch
## Prerequisites
You should already have set up PostgreSQL and created a database. Make sure to create environment variables to use `psql` if you haven't already:
@ -43,10 +47,6 @@ creation steps below.
Install GNU parallel, this is used to speed up loading bulk data.
## Make data available to Ubuntu
If you didn't download the OS files to the Ubuntu machine where you are setting up your Colouring London application, you will need to make them available there. If you are using Virtualbox, you could host share a the folder containing the MasteerMap files with the VM via a shared folder (e.g. [see these instructions for Mac](https://medium.com/macoclock/share-folder-between-macos-and-ubuntu-4ce84fb5c1ad)).
## Process and load Ordnance Survey data
Move into the `etl` directory and set execute permission on all scripts.