Maciej Ziarkowski
82a50d77d6
Allow specifying JSON columns for CSV bulk import
2019-12-10 17:25:55 +00:00
Maciej Ziarkowski
26ca7f8873
Accept CSV with building_id for API data import
2019-12-10 17:16:26 +00:00
Tom Russell
f3b9be39bf
Update load_csv python script
...
- use building_id if present in CSV
- print DEBUG even if response from API is 200
- handle empty-string sust_dec (which fails because empty-string isn't a valid
enum value as defined in 011.sustainability.up.sql) by deleting from data
if present
2019-11-21 13:17:17 +00:00
Dominic H
1d372823ed
Adjusts guide notes no code change
2019-11-08 12:36:54 +00:00
Dominic H
e00d912686
Enable python upload to staging
...
- Based on load_csv.py with edit for staging
2019-11-08 12:31:40 +00:00
Dominic H
9943bdd9a5
Test load to staging
...
- Copy of main load_csv.py with edit to get round ssh error
2019-10-02 16:35:56 +01:00
Tom Russell
46ac6c7a40
Fix load_csv script - tested against localhost
2019-10-02 09:43:12 +01:00
Tom Russell
c055bea38b
Add generic CSV upload script (rename load_data to load_shapefile)
2019-09-30 10:39:43 +01:00
Maciej Ziarkowski
2c9b5ea3d8
Modify routes, refactor API structure
2019-08-14 14:05:49 +01:00
Tom Russell
482ab5060c
Set postcode zoom and class
2019-02-11 09:07:26 +00:00
Tom Russell
731f299a18
Load open postcode data
2019-02-05 13:36:43 +00:00
Tom Russell
42f72bea9c
Add script to upload conservation areas
2019-01-20 15:29:54 +00:00
Tom Russell
d72dd90351
Comment sections in etl run script
2018-10-21 20:47:31 +01:00
Tom Russell
3441bf88e2
Update load_data to use API
2018-10-20 18:37:02 +01:00
Tom Russell
4415b21af5
Ensure index exists for uprn link
2018-10-04 21:13:25 +01:00
Tom Russell
a10e4bf5c6
Update run_all etl script
2018-10-04 19:01:40 +01:00
Tom Russell
30086766db
Boundary file not needed in initial extraction
2018-10-04 19:01:17 +01:00
Tom Russell
20e4a73e73
Skip altering foreign key restrictions
2018-10-04 19:00:56 +01:00
Tom Russell
4c62da548b
Drop outside limit of boundary
2018-10-04 18:59:53 +01:00
Tom Russell
b180602a5b
Uncomment copy-uprns block
2018-10-03 20:10:47 +01:00
Tom Russell
b73fb7118e
Fix sed quoting
2018-10-03 20:10:27 +01:00
Tom Russell
f06b820d19
Remove clipsrc for speedup
2018-10-03 20:10:16 +01:00
Tom Russell
4696e3e079
Update etl to load UPRNs to table
2018-10-02 21:12:46 +01:00
Tom Russell
79724cc449
Copy from stdin (cat user-accessible file) when loading geometries
2018-09-30 21:23:19 +01:00
Tom Russell
2a1902f6ce
Update ETL docs
2018-09-29 18:29:57 +01:00
Tom Russell
73aa3df290
Clip OSMM to GLA on extraction
2018-09-27 21:37:47 +01:00
Tom Russell
322a976f7a
Fix get_test_polygons script
2018-09-25 22:01:09 +01:00
Tom Russell
342167f9c9
Update UPRN-load script
2018-09-25 21:47:58 +01:00
Tom Russell
d9797385d9
Use default pool size (CPU count) in ETL
2018-09-25 21:47:29 +01:00
Tom Russell
3f9c9f3221
Split indexing further, UPRN requires bigint
2018-09-25 21:46:22 +01:00
Tom Russell
bddd7e769f
Rename etl scripts
2018-09-25 20:46:16 +01:00
Tom Russell
c6b3d3d5ca
Extract and load addressbase/mastermap using id, gml, parallel
2018-09-25 19:20:41 +01:00
Tom Russell
181e850225
Parallel extract/filter OS data
2018-09-21 11:10:39 +01:00
Tom Russell
f6f7cc1341
Save test polygons projected
2018-09-10 10:44:09 +01:00
Tom Russell
204740e46e
Sketch matching data by best-intersection
2018-09-09 11:58:50 +01:00
Tom Russell
5695f2dc9d
Record python packages for etl
2018-09-09 11:34:37 +01:00
Tom Russell
54633d9e04
Create building-per-geometry
2018-09-09 11:32:27 +01:00
Tom Russell
3711fc5f80
Load test polygons
2018-09-09 11:32:12 +01:00
Tom Russell
981d3608a2
Mappings for Camden and Fitrovia CASA data
2018-08-01 17:06:00 +01:00
Tom Russell
69bb6c5790
Provide transform when joining data
2018-08-01 17:05:02 +01:00
Tom Russell
847eb75a70
Fix loading buildings
2018-08-01 15:48:31 +01:00
Tom Russell
990e4241cf
Sketch generic update from shapes
2018-08-01 14:17:10 +01:00
Tom Russell
1c30f4d436
Skip trying to join buildings/geometries up front
2018-08-01 13:50:08 +01:00
Tom Russell
c7b322f214
Skip duplicate geometries
2018-08-01 13:49:16 +01:00
Tom Russell
f6c8323bfa
Load buildings from CSV
2018-08-01 13:49:03 +01:00
Tom Russell
cd65a5aaab
Load from geojson to db
2018-08-01 13:12:56 +01:00
Tom Russell
39160d8a09
Join each polygon to UPRNs if possible
2018-07-27 17:47:22 +01:00
Tom Russell
92b81928ca
Rename etl scripts to reflect stages
2018-07-27 15:48:00 +01:00
Tom Russell
105271a1e8
Sketch loading Addressbase buildings to database
...
- TODO handle intersection with geometries
2018-07-17 09:10:24 +01:00
Tom Russell
9316cfb719
Load OSMM geometries to database
2018-07-17 09:09:13 +01:00
Tom Russell
62b65f44f9
Extract OS MasterMap shell
2018-05-30 20:18:37 +01:00
Tom Russell
dd72ccd108
Extract OS AddressBase to CSV
2018-05-30 18:35:15 +01:00
Tom Russell
7c97aba950
Script comments for OS data extract
2018-05-30 14:31:23 +01:00
