Merge pull request #776 from colouring-cities/docs-vagrant

Fully document setup of a working CL development environment
This commit is contained in:
Ed Chalstrey 2022-03-01 10:21:43 +00:00 committed by GitHub
commit e100dd5c7d
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
6 changed files with 14805 additions and 21123 deletions

View File

@ -26,26 +26,19 @@ for download under a liberal open data license
## Setup and run
1. Provision database (see [migrations](migrations/README.md))
1. Load buildings and geometries to database (see [etl](etl/README.md))
1. Install app dependencies: `cd app && npm i`
1. Run tests: `npm test`
1. Run app: `npm start`
#### Test the application:
In development, run with environment variables:
You can try out the Colouring London application by setting up your own development environment, which includes the option to load test data from OpenStreetMaps (OSM). See [docs/setup-dev-environment](docs/setup-dev-environment.md).
_Last updated March 2022_
```bash
APP_COOKIE_SECRET=test_secret \
PGHOST=localhost \
PGUSER=dbuser \
PGDATABASE=dbname \
PGPASSWORD=dbpassword \
PGPORT=5432 \
TILECACHE_PATH=/path/to/tilecache/directory \
npm start
```
#### Create a production version of the application:
[Provision script](provision/vm_provision.sh) can be also of interest.
We also have documentation on setting up a production environment here: [docs/setup-production-environment](docs/setup-production-environment.md).
_Last updated December 2021_
**Note:** There are additional useful documentation within the `/docs` folder.
## Acknowledgements

35480
app/package-lock.json generated

File diff suppressed because it is too large Load Diff

View File

@ -29,12 +29,12 @@
"geojson": "^0.5.0",
"leaflet": "^1.7.1",
"lodash": "^4.17.21",
"mapnik": "^4.5.8",
"mapnik": "^4.5.9",
"markdown-to-jsx": "^7.1.3",
"node-fs": "^0.1.7",
"nodemailer": "^6.4.11",
"pg-format": "^1.0.4",
"pg-promise": "^8.7.5",
"pg-promise": "^10.11.1",
"query-string": "^6.13.1",
"react": "^17.0.2",
"react-dom": "^17.0.2",
@ -63,11 +63,11 @@
"@types/webpack-env": "^1.16.0",
"@typescript-eslint/eslint-plugin": "^4.22.0",
"@typescript-eslint/parser": "^4.22.0",
"copy-webpack-plugin": "^8.1.1",
"copy-webpack-plugin": "^6.4.1",
"eslint": "^7.24.0",
"eslint-plugin-jest": "^24.3.5",
"eslint-plugin-react": "^7.23.2",
"html-webpack-plugin": "^5.2.0",
"html-webpack-plugin": "^4.5.2",
"mini-css-extract-plugin": "^0.9.0",
"postcss": "^8.2.4",
"razzle": "^4.0.4",

View File

@ -33,7 +33,12 @@ export const AuthRoute: React.FC<RouteProps> = ({ component: Component, children
{...rest}
render={props => {
if(isAuthenticated) {
const { from } = props.location.state ?? { from: '/my-account.html'};
let state = props.location.state as any;
let from = '/my-account.html';
if ('from' in state){
from = state.from;
}
return <Redirect to={{pathname: from }} />;
} else {
if(Component) {

View File

@ -1,43 +1,118 @@
# Setting up a local development environment
This document is intended to guide you through setting up a local development environment for
Colouring London. This guide assumes you already have Ubuntu 20.04 server installed, typically
installed in a virtual environment such a Virtual Box and are able to SSH into your Ubuntu
installation for convenience.
### WARNING: Setup document suitable for development environment, not production server
This document is intended to guide you through setting up a local development environment for the Colouring London application. This guide assumes you already have either already have access to an machine with Ubuntu 18.04 installed, or can use VirtualBox to set up an Ubuntu virtual machine as below.
<details>
<summary>
Configuring an Ubuntu VM in VirtualBox
</summary><p></p>
Here we explain how to use VirtualBox and SSH into your Ubuntu installation for convenience.
When setting up the VirtualBox VM, consider the size of the database you intend to load for use with the application. Consult the [:house: Loading the building data](#house-loading-the-building-data) section of this guide and decide whether you will be using a full city database or will load test data from OSM.
For "Colouring London", we have found that the size of the database means that a VM with access to 50GB of storage is appropriate. If you are using the OSM test data, the default storage settings in VirtualBox should suffice.
##### In either case, you should set the memory to at least `2048` MB.
If you a running Ubuntu in a virtual environment you will need to configure networking to forward ports from the guest to the host. For Virtual Box the following was configured under NAT port forwarding (found under `Settings -> Network -> Advanced -> Port Forwarding`).
Name | Protocol | Host Port | Guest Port
-------- | --------- | ---------- | -----------
app | TCP | 8080 | 3000
app_dev | TCP | 3001 | 3001
ssh | TCP | 4022 | 22
The `app_dev` mapping is used in development by Razzle which rebuilds and serves client side assets on the fly.
To run the commands in the rest of this setup guide, either `ssh` into the VirtualBox environment or open the terminal within the Ubuntu GUI.
If you wish to `ssh`, you will first need to open the terminal in Ubuntu and run the following.
```bash
sudo apt-get install -y openssh-server
```
You can then `ssh` into the VirtualBox VM set up with the port forwarding described above like so, where `<linuxusername>` is the name you set up during the installation of Ubuntu (you can type `whoami` in the Ubuntu terminal to remind yourself of this).
```bash
ssh <linuxusername>@localhost -p 4022
```
</details>
## Contents
- [:tulip: Installing the tools and components](#tulip-installing-the-tools-and-components)
- [:red_circle: Installing PostgreSQL](#red_circle-installing-postgresql)
- [:rainbow: Installing Colouring London](#rainbow-installing-colouring-london)
- [:arrow_down: Installing Node.js](#arrow_down-installing-nodejs)
- [:large_blue_circle: Configuring PostgreSQL](#large_blue_circle-configuring-postgresql)
- [:arrow_forward: Configuring Node.js](#arrow_forward-configuring-nodejs)
- [:house: Loading the building data](#house-loading-the-building-data)
- [:computer: Running the application](#computer-running-the-application)
- [:eyes: Viewing the application](#eyes-viewing-the-application)
## :tulip: Installing the tools and components
First upgrade the installed packages to the latest versions, to remove any security warnings.
```bash
sudo apt-get update -y
sudo apt-get upgrade -y
```
Now install some essential tools.
```bash
sudo apt-get install -y build-essential git wget curl
```
### :red_circle: Installing PostgreSQL
Set the postgres repo for apt (these instructions were taken from [postgresql.org](https://www.postgresql.org/download/linux/ubuntu/)).
```bash
sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
```
```bash
sudo wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
```
```bash
sudo apt-get update
sudo apt-get upgrade
```
## Installing the tools and components
Now we install some essential tools.
`sudo apt-get install -y build-essential git vim-nox wget curl`
Now install python and related tools.
`sudo apt-get install -y python3 python3-pip python3-dev python3-venv`
Next install postgres and postgis to enable support for geographical objects.
`sudo apt-get install -y postgresql postgresql-contrib libpq-dev postgis postgresql-12-postgis-3`
```bash
sudo apt-get install -y postgresql-12 postgresql-contrib-12 libpq-dev postgis postgresql-12-postgis-3
```
and additional geo-spatial tools
`sudo apt-get install -y gdal-bin libspatialindex-dev libgeos-dev libproj-dev`
```bash
sudo apt-get install -y gdal-bin libspatialindex-dev libgeos-dev libproj-dev
```
Now clone the colouring london codebase.
### :rainbow: Installing Colouring London
`git clone https://github.com/colouring-london/colouring-london.git`
Now clone the `colouring-london` codebase.
```bash
cd ~ && git clone https://github.com/colouring-london/colouring-london.git
```
**Note:** We assume here that you will clone the repo into the home directory of your Ubuntu installation. Watch out for later commands in this guide that assume the repo is located at `~/colouring-london` and modify the path if appropriate.
### :arrow_down: Installing Node.js
Now install Node. It is helpful to define some local variables.
```
export NODE_VERSION=v12.14.1
```bash
export NODE_VERSION=v16.13.2
export DISTRO=linux-x64
wget -nc https://nodejs.org/dist/$NODE_VERSION/node-$NODE_VERSION-$DISTRO.tar.xz
sudo mkdir /usr/local/lib/node
@ -48,138 +123,273 @@ rm node-$NODE_VERSION-$DISTRO.tar.xz
Now add the Node installation to the path and export this to your bash profile.
```
```bash
cat >> ~/.profile <<EOF
export NODEJS_HOME=/usr/local/lib/node/node-$NODE_VERSION/bin
export PATH=\$NODEJS_HOME:\$PATH
EOF
```
Then run source to make sure node and npm are on your path:
Then run source to make sure node and npm are on your path.
```
```bash
source ~/.profile
```
You can check the updated variables as follows
```
```bash
echo $PATH
echo $NODEJS_HOME
```
## Configuring Postgres
### :large_blue_circle: Configuring PostgreSQL
Now we configure postgres. First ensure postgres is running.
`service postgresql start`
```bash
sudo service postgresql start
```
Ensure the `en_US` locale exists.
`sudo locale-gen en_US.UTF-8`
```bash
sudo locale-gen en_US.UTF-8
```
Configure the database to listen on network connection.
`sudo sed -i "s/#\?listen_address.*/listen_addresses '*'/" /etc/postgresql/10/main/postgresql.conf`
```bash
sudo sed -i "s/#\?listen_address.*/listen_addresses '*'/" /etc/postgresql/12/main/postgresql.conf
```
Allow authenticated connections from any IP (so includes the host).
`echo "host all all all md5" | sudo tee --append /etc/postgresql/10/main/pg_hba.conf > /dev/null`
```bash
echo "host all all all md5" | sudo tee --append /etc/postgresql/12/main/pg_hba.conf > /dev/null
```
Restart postgres to pick up config changes.
`service postgresql restart`
```bash
sudo service postgresql restart
```
Create a superuser role for this user (`<username>`) if it does not already exist. The
password `<pgpassword>` is arbitrary and probably should not be your Ubuntu login password.
`sudo -u postgres psql -c "SELECT 1 FROM pg_user WHERE usename = '<username>';" | grep -q 1 || sudo -u postgres psql -c "CREATE ROLE <username> SUPERUSER LOGIN PASSWORD '<pgpassword>';"`
```bash
sudo -u postgres psql -c "SELECT 1 FROM pg_user WHERE usename = '<username>';" | grep -q 1 || sudo -u postgres psql -c "CREATE ROLE <username> SUPERUSER LOGIN PASSWORD '<pgpassword>';"
```
<details>
<summary>Note for "Colouring London" devs</summary><p></p>
If you intend to load the full CL database from a dump file into your dev environment, run the above `psql` command with `<username>` as "cldbadmin" and use that username in subsequent steps, but also run the above a second time with `<username>` as "clwebapp" (see section [:house: Loading the building data](#house-loading-the-building-data) for more details).
</details><p></p>
Set environment variables, which will simplify running subsequent `psql` commands.
```bash
export PGPASSWORD=<pgpassword>
export PGUSER=<username>
export PGHOST=localhost
export PGDATABASE=<colouringlondondb>
```
Create a colouring london database if none exists. The name (`<colouringlondondb>`) is arbitrary.
```
```bash
sudo -u postgres psql -c "SELECT 1 FROM pg_database WHERE datname = '<colouringlondondb>';" | grep -q 1 || sudo -u postgres createdb -E UTF8 -T template0 --locale=en_US.utf8 -O <username> <colouringlondondb>
```
To test the app user's connection to the database, you could run `psql` interactively:
```
psql -d <colouringlondondb> -U <username> -h localhost
```bash
psql -c "create extension postgis;"
psql -c "create extension pgcrypto;"
psql -c "create extension pg_trgm;"
```
Create the necessary postgres extensions.
### :arrow_forward: Configuring Node.js
```
psql -d <colouringlondondb> -c "create extension postgis;"
psql -d <colouringlondondb> -c "create extension pgcrypto;"
psql -d <colouringlondondb> -c "create extension pg_trgm;"
```
Now upgrade the npm package manager to the most recent release with global privileges. This needs to be performed as root user, so it is necessary to export the node variables to the root user profile.
Now run all 'up' migrations to create tables, data types, indexes etc. The `.sql` scripts to
do this are located in the `migrations` folder of your local repository.
`ls ./colouring-london/migrations/*.up.sql 2>/dev/null | while read -r migration; do psql -d <colouringlondondb> < $migration; done;`
## Setting up Python
Now set up a virtual environment for python. In the following example we have named the
virtual environment *colouringlondon* but it can have any name.
`pyvenv colouringlondon`
Activate the virtual environment so we can install python packages into it.
`source colouringlondon/bin/activate`
Install python pip package manager and related tools.
```
pip install --upgrade pip
pip install --upgrade setuptools wheel
```
Now install the required python packages. This relies on the `requirements.txt` file located
in the `etl` folder of your local repository.
`pip install -r ./colouring-london/etl/requirements.txt`
## Setting up Node
Now upgrade the npm package manager to the most recent release with global privileges. This
needs to be performed as root user, so it is necessary to export the node variables to the
root user profile. Don't forget to exit from root at the end.
```
sudo su root
export NODEJS_HOME=/usr/local/lib/node/node-v12.14.1/bin/
```bash
export NODEJS_HOME=/usr/local/lib/node/node-v16.13.2/bin/
export PATH=$NODEJS_HOME:$PATH
npm install -g npm@next
exit
sudo env "PATH=$PATH" npm install -g npm@latest
```
Now install the required Node packages. This needs to done from the `app` directory of your
local repository, so that it can read from the `package.json` file.
`cd ./colouring-london/app && npm install`
```bash
cd ~/colouring-london/app
npm install
```
## :house: Loading the building data
## Running the application
<details>
<summary> With a database dump </summary><p></p>
Now we are ready to run the application. The `APP_COOKIE_SECRET` is arbitrary.
If you are a developer on the Colouring London project (or another Colouring Cities project), you may have a production database (or staging etc) that you wish to duplicate in your development environment.
`PGPASSWORD=<pgpassword> PGDATABASE=<colouringlondondb> PGUSER=<username> PGHOST=localhost PGPORT=5432 APP_COOKIE_SECRET=123456 npm start`
Log into the environment where your production database is kept and create a dump file from the db.
If you a running Ubuntu in a virtual environment you will need to configure networking to
forward ports from the guest to the host. For Virtual Box the following was configured under
NAT port forwarding.
```bash
pg_dump <colouringlondondb> > <dumpfile>
```
Name | Protocol | Host Port | Guest Port
-------- | --------- | ---------- | -----------
app | TCP | 8080 | 3000
app_dev | TCP | 3001 | 3001
ssh | TCP | 4022 | 22
You should then download the file to the machine where you are setting up your development environment. If you are using Virtualbox, you could host share the dump file with the VM via a shared folder (e.g. [see these instructions for Mac](https://medium.com/macoclock/share-folder-between-macos-and-ubuntu-4ce84fb5c1ad)).
The site can then be viewed on http://localhost:8080. The `app_dev` mapping is used in
development by Razzle which rebuilds and serves client side assets on the fly.
In your Ubuntu installation where you have been running these setup steps (e.g. Virtualbox VM), you can then recrate the db like so.
```bash
psql < <dumpfile>
```
#### Run migrations
Now run all 'up' migrations to create tables, data types, indexes etc. The `.sql` scripts to
do this are located in the `migrations` folder of your local repository.
```bash
ls ~/colouring-london/migrations/*.up.sql 2>/dev/null | while read -r migration; do psql < $migration; done;
```
</details>
<details>
<summary> With test data </summary><p></p>
This section shows how to load test buildings into the application from OpenStreetMaps (OSM).
#### Set up Python
Install python and related tools.
```bash
sudo apt-get install -y python3 python3-pip python3-dev python3-venv
```
Now set up a virtual environment for python. In the following example we have named the
virtual environment *colouringlondon* but it can have any name.
```bash
pyvenv colouringlondon
```
Activate the virtual environment so we can install python packages into it.
```bash
source colouringlondon/bin/activate
```
Install python pip package manager and related tools.
```bash
pip install --upgrade pip
pip install --upgrade setuptools wheel
```
#### Load OpenStreetMap test polygons
First install prerequisites.
```bash
sudo apt-get install -y parallel
```
Install the required python packages. This relies on the `requirements.txt` file located
in the `etl` folder of your local repository.
```bash
cd ~/colouring-london/etl/
pip install -r requirements.txt
```
To help test the Colouring London application, `get_test_polygons.py` will attempt to save a small (1.5km²) extract from OpenStreetMap to a format suitable for loading to the database.
Download the test data.
```bash
python get_test_polygons.py
```
Note: the first time you run it, you will get these warnings:
```
rm: cannot remove 'test_buildings.geojson': No such file or directory
rm: cannot remove 'test_buildings.3857.csv': No such file or directory
```
#### Run migrations
Now run all 'up' migrations to create tables, data types, indexes etc. The `.sql` scripts to
do this are located in the `migrations` folder of your local repository.
```bash
ls ~/colouring-london/migrations/*.up.sql 2>/dev/null | while read -r migration; do psql < $migration; done;
```
#### Load buildings
Load all building outlines.
```bash
./load_geometries.sh ./
```
Create a building record per outline.
```bash
./create_building_records.sh
```
</details>
## :computer: Running the application
Now we are ready to run the application.
First enter the app directory.
```bash
cd ~/colouring-london/app
```
Then create a folder for the tilecache.
```bash
mkdir tilecache
```
Create some additional variables for running the application (the `APP_COOKIE_SECRET` is arbitrary).
```bash
export PGPORT=5432
export APP_COOKIE_SECRET=123456
export TILECACHE_PATH=~/colouring-london/app/tilecache
```
Finally, simply run the application with npm.
```bash
npm start
```
**Note:** You can also specify the variables for `npm start` like so:
<details>
<summary>
Specify variables
</summary>
```bash
PGPASSWORD=<pgpassword> PGDATABASE=<colouringlondondb> PGUSER=<username> PGHOST=localhost PGPORT=5432 APP_COOKIE_SECRET=123456 TILECACHE_PATH=~/colouring-london/app/tilecache npm start
```
</details><p></p>
### :eyes: Viewing the application
The site can then be viewed on http://localhost:8080 on the host computer.
Finally to quit the application type `Ctrl-C`.

View File

@ -1,7 +1,7 @@
# Python packages for etl
fiona==1.7.13
osmnx==0.8.1
osmnx==0.13
psycopg2==2.7.5
shapely==1.6.4
shapely==1.7
retrying==1.3.3
requests==2.23.0