a_adli/colouring-montreal
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

update the doc

Browse Source
This commit is contained in:
Ed Chalstrey 2022-04-14 14:10:07 +01:00
parent 8a540256bb
commit e10213c54c
1 changed files with 3 additions and 48 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

51
etl/README.md
View File

@ -147,7 +147,7 @@ sudo ./filter_transform_mastermap_for_loading.sh /path/to/mastermap_dir
Load all new building outlines. This step will only add geometries that are not already present (based on the `TOID`). Note: you should ensure that `mastermap_dir` has permissions that will allow the linux `find` command to work without using sudo. Load all new building outlines. This step will only add geometries that are not already present (based on the `TOID`). Note: you should ensure that `mastermap_dir` has permissions that will allow the linux `find` command to work without using sudo.
```bash ```bash
./load_geometries.sh /path/to/mastermap_dir ./load_new_geometries.sh /path/to/mastermap_dir
``` ```
Drop building outlines outside London boundary. Drop building outlines outside London boundary.
@ -159,59 +159,14 @@ cd ~/colouring-london/etl/
./drop_outside_limit.sh ~/colouring-london/app/public/geometries/boundary.shp ./drop_outside_limit.sh ~/colouring-london/app/public/geometries/boundary.shp
``` ```
Create a virtual environment for python in the `etl` folder of your repository. In the following example we have name the virtual environment *colouringlondon* but it can have any name. Create building record to match each new geometry (TOID) that doesn't already have a linked building.
```bash ```bash
pyvenv colouringlondon ./create_new_building_records.sh
```
Activate the virtual environment so we can install python packages into it.
```bash
source colouringlondon/bin/activate
```
Install python pip package manager and related tools.
```bash
pip install --upgrade pip
pip install --upgrade setuptools wheel
```
Install the required python packages.
```bash
pip install -r requirements.txt
```
Converting the OS Open TOID data OSGB36 Eastings and Northings to WGS84 longitude and latitude coordinates.
<details>
<summary>
WARNING
</summary>
EC note: When testing this in my developemt setup, due to memory constraints I actually ended up running this Python step on my Mac directly, rather than in my Ubuntu Virtualbox environment.
</details><p></p>
```bash
python convert_opentoid_bng_latlon.py /path/to/opentoids_dir
```
Assign latitude and longitude to buildings with the converted OS Open TOID data.
```bash
./load_coordinates.sh /path/to/opentoids_dir
```
Create building record to match each new geometry (TOID) that doesn't already have a linked building. Ignores geometries in the same location as pre-existing ones based on lat/lon coordinates, in which case marking the old one as valid _up to_ the current date and adding the new one as valid _from_ today, keeping both linked to the existing building record.
```bash
./update_building_records.sh
``` ```
Mark TOIDs not present in the update as demolished. Mark TOIDs not present in the update as demolished.
<!-- TODO: need the open_toids table for this -->
```bash ```bash
./mark_demolitions.sh ./mark_demolitions.sh
``` ```