m_osman/city_retrofit
1
0
Fork 0
forked from s_ranjbar/city_retrofit
Code Pull Requests Activity

Merge branch 'db_persistence' into 'master'

Browse Source 
Db persistence

See merge request Guille/hub!44
This commit is contained in:
Guillermo Gutierrez Morote 2023-01-09 21:56:46 +00:00
commit 609ad59371
37 changed files with 2821 additions and 1941 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

16
.env Normal file
View File

@ -0,0 +1,16 @@
# production database credentials
PROD_DB_USER=postgres
PROD_DB_PASSWORD=
PROD_DB_HOST=localhost
PROD_DB_PORT=5432
# test database credentials
TEST_DB_USER=postgres
TEST_DB_PASSWORD=postgres
TEST_DB_HOST=localhost
TEST_DB_PORT=5432
#Gitlab token
HUB_TOKEN=9s_CJYh5TcWhyYL416MM
DEV_SECRET_NAME=dp.st.dev.Axvak1ILOlCOwUNGajv7fg5VPaacFR6OL1kdb3YGWHX

2
.gitignore vendored
View File

@ -5,4 +5,6 @@
/data/energy_systems/heat_pumps/*.csv
/data/energy_systems/heat_pumps/*.insel
.DS_Store
.env
logs
**/__pycache__/

66
DEPLOYMENT.md Normal file
View File

@ -0,0 +1,66 @@
## Installing PostgreSQL Database Server on Linux (Ubuntu) ##
Execute the *install_postgresql_linux.sh* script to install PostgreSQL database
*NB: PostgreSQL DB Server runs on a default port of 5432.*
## Installing PostgreSQL Database Server on Windows ##
1. Download a Windows installer from [this link](https://www.enterprisedb.com/downloads/postgres-postgresql-downloads).
2. Double click on the installer file and follow the prompts of the installation wizard
3. On the component selection page of the installation wizard make sure to select *PostgreSQL Server and Commandline tools*
4. You can optionally select pgAdmin 4 to install a graphical UI to access your database
5. On the password page when prompted, enter the default password (postgres) and confirm it
6. You can change the default password of 5432 on the port page. You should ensure that whatever port number you
provide is not used by another service.
7. Follow the installation wizard prompt to complete your installation. You can verify your installation by
searching for the *psql* tool from your start menu
## Installing PostgreSQL Database Server on Mac OS X ##
1. Download the Mac OS X installer from [this link](https://www.enterprisedb.com/downloads/postgres-postgresql-downloads).
2. Launch the installation wizard by double-clicking it and follow the rest of steps as described above on
Installing PostgreSQL Database Server on Windows.
NB: Hub has been tested with version 15 of PostgreSQL
## Create Database and Database User ##
1. Connect to the PostgreSQL database server via psql by executing `sudo -u postgres psql`. You will be
be prompted for a password, the default password of *postgres* user is *postgres*. The above command may not work on
a Windows system using the default command line tool. You can access the psql tool from Windows start menu and follow
the rest of the instructions from step 2 below
2. Execute `create user <username> with encrypted password '<password>';` in the psql console to create a user.
3. Execute `create database <database-name>;` in the psql console to create a database.
4. Execute `grant all privileges on database <database-name> to <username>;` to grant the all privileges on the database
to the user created in step 2 above.
5. The created database by default, has on schema named public which you can use. However, if you wish to create
another schema, you can do so by following [this link](https://www.postgresqltutorial.com/postgresql-administration/postgresql-create-schema/).
**NB: You can grant selected privileges to the user on the database using commands [on this page](https://tableplus.com/blog/2018/04/postgresql-how-to-grant-access-to-users.html).*
The user should however have read and write permission to all tables in the database. You can as well create a database and user using the PgAdmin UI tool*
## Setting Up Database Connection Parameters
1. Create a .env file that contains the configuration parameters as explained in the *Database Configuration Parameters*
section in persistence/README.md file.
2. The .env file should contain the following credentials: database user, database password, database host an,d database port
3. Provide the *absolute path* to the .env file to the persistence importers and exporters whenever using them in your code
as shown below:
```python
from exports.db_factory import DBFactory
from pathlib import Path
dotenv_path = (Path(__file__).parent / '.env').resolve()
factory = DBFactory(db_name='hub_db', app_env='PROD', dotenv_path=dotenv_path)
```
## Create Database Tables ##
Use the *DBSetup* class in the persistence package to create the required database tables as described below
```python
from persistence import DBSetup
from pathlib import Path
dotenv_path = (Path(__file__).parent / '.env').resolve()
DBSetup(db_name='hub_db', app_env='PROD', dotenv_path=dotenv_path)
```
The *DBSetUp* class also creates a default admin user with default credentials that can be changed.
with the import UserFactory class. The admin user (name, email, password and role) is logged into the console after it is created by the
*constructor of DBSetup*. Use can also manage users (create, read, update and delete) with user import and export factories.
**NB: Make sure to change the default admin user credentials**

8
city_model_structure/city.py
View File

@ -14,7 +14,6 @@ import pyproj
from typing import List, Union
from pyproj import Transformer
from pathlib import Path
from city_model_structure.building import Building
from city_model_structure.city_object import CityObject
from city_model_structure.city_objects_cluster import CityObjectsCluster
@ -96,7 +95,7 @@ class City:
@property
def country_code(self):
"""
Get city country code
Get models country code
:return: str
"""
return self._get_location().country
@ -290,11 +289,12 @@ class City:
selected_region_upper_corner = [center[0] + radius, center[1] + radius, center[2] + radius]
selected_region_city = City(selected_region_lower_corner, selected_region_upper_corner, srs_name=self.srs_name)
selected_region_city.climate_file = self.climate_file
# selected_region_city.climate_reference_city = self.climate_reference_city
for city_object in self.city_objects:
location = city_object.centroid
if location is not None:
distance = math.sqrt(math.pow(location[0]-center[0], 2) + math.pow(location[1]-center[1], 2)
+ math.pow(location[2]-center[2], 2))
distance = math.sqrt(math.pow(location[0] - center[0], 2) + math.pow(location[1] - center[1], 2)
+ math.pow(location[2] - center[2], 2))
if distance < radius:
selected_region_city.add_city_object(city_object)
return selected_region_city

3
data/energy_systems/heat_pumps/constants.yaml
View File

@ -13,13 +13,12 @@ Cp: 4190
Rhow: 1000
TESDiameter: 5
AuxHeaterEfficiency: 0.9
HPNominalCapacity: 256
# These come from the data model according to other student's work
ElecGridEF: 0.5
ElectricityPrice: 0.073
# Water to Water HP constants
HPNominalCapacity: 256
LowestPossibleLoadFlow: 4.73
HighestPossibleLoadFlow: 9.46

1671
data/energy_systems/heat_pumps/w2w_parallel.txt
View File

File diff suppressed because it is too large Load Diff

1680
data/energy_systems/heat_pumps/w2w_series.txt
View File

File diff suppressed because it is too large Load Diff

46
exports/db_factory.py Normal file
View File

@ -0,0 +1,46 @@
"""
DBFactory performs read related operations
SPDX - License - Identifier: LGPL - 3.0 - or -later
Copyright © 2022 Concordia CERC group
Project CoderPeter Yefi peteryefi@gmail.com
"""
from persistence import CityRepo
from persistence import HeatPumpSimulationRepo
class DBFactory:
"""
DBFactory class
"""
def __init__(self, db_name, app_env, dotenv_path):
self._city_repo = CityRepo(db_name=db_name, app_env=app_env, dotenv_path=dotenv_path)
self._hp_simulation_repo = HeatPumpSimulationRepo(db_name=db_name, app_env=app_env, dotenv_path=dotenv_path)
def get_city(self, city_id):
"""
Retrieve a single city from postgres
:param city_id: the id of the city to get
"""
return self._city_repo.get_by_id(city_id)
def get_city_by_name(self, city_name):
"""
Retrieve a single city from postgres
:param city_name: the name of the city to get
"""
return self._city_repo.get_by_name(city_name)
def get_hp_simulation(self, hp_sim_id: int):
"""
Retrieve a single heat pump simulation from postgres
:param hp_sim_id: the id of the heat pump to get
"""
return self._hp_simulation_repo.get_by_id(hp_sim_id)
def get_hp_simulation_by_city(self, city_id: int):
"""
Retrieve a single city from postgres
:param city_id: the id of the city
"""
return self._hp_simulation_repo.get_by_city(city_id)

12
exports/energy_systems/air_source_hp_export.py
View File

@ -6,7 +6,7 @@ Copyright © 2022 Concordia CERC group
Project Coder Peter Yefi peteryefi@gmail.com
"""
from exports.energy_systems.heat_pump_export import HeatPumpExport
from typing import List, Tuple, Union
from typing import List, Dict, Union
class AirSourceHPExport(HeatPumpExport):
@ -15,17 +15,18 @@ class AirSourceHPExport(HeatPumpExport):
after executing insel
"""
def __init__(self, base_path, city, output_path, sim_type):
def __init__(self, base_path, city, output_path, sim_type, demand_path=None):
"""
:param base_path: path to energy system files
:param city: the city object
:param output_path: the file to hold insel simulation results
:param sim_type: the simulation type to run: 0 for series, 1 for parallel
:param demand_path: path to hourly energy demand file
"""
tmp_file = 'heat_pumps/as_series.txt' if sim_type == 0 else 'heat_pumps/as_parallel.txt'
template_path = (base_path / tmp_file)
super().__init__(base_path, city, output_path, template_path)
super().__init__(base_path, city, output_path, template_path, demand_path)
def _extract_model_coff(self, hp_model: str, data_type='heat') -> Union[List, None]:
"""
@ -43,7 +44,7 @@ class AirSourceHPExport(HeatPumpExport):
return energy_system.air_source_hp.cooling_capacity_coff
return None
def execute_insel(self, user_input, hp_model, data_type):
def execute_insel(self, user_input, hp_model, data_type) -> Union[Dict, None]:
"""
Runs insel and produces output files
Runs insel and write the necessary files
@ -54,7 +55,6 @@ class AirSourceHPExport(HeatPumpExport):
:param data_type: a string that indicates whether
insel should run for heat or cooling performance
:return:
:return:
"""
capacity_coeff = self._extract_model_coff(hp_model, data_type)
super(AirSourceHPExport, self)._run_insel(user_input, capacity_coeff, 'air_source.insel')
return super(AirSourceHPExport, self)._run_insel(user_input, capacity_coeff, 'air_source.insel')

73
exports/energy_systems/heat_pump_export.py
View File

@ -5,7 +5,7 @@ Copyright © 2022 Concordia CERC group
Project Coder Peter Yefi peteryefi@gmail.com
"""
import os
from typing import List, Tuple, Union, Dict
from typing import List, Union, Dict
import yaml
from string import Template
import pandas as pd
@ -17,18 +17,18 @@ class HeatPumpExport:
of some defined function
"""
def __init__(self, base_path, city, output_path, template, water_temp=None):
def __init__(self, base_path, city, output_path, template, demand_path=None, water_temp=None):
self._template_path = template
self._water_temp = water_temp
self._constants_path = (base_path / 'heat_pumps/constants.yaml')
# needed to compute max demand.
self._demand_path = (base_path / 'heat_pumps/demand.txt')
self._demand_path = (base_path / 'heat_pumps/demand.txt') if demand_path is None else demand_path
self._city = city
self._input_data = None
self._base_path = base_path
self._output_path = output_path
def _run_insel(self, user_input: Dict, capacity_coeff: List, filename: str) -> None:
def _run_insel(self, user_input: Dict, capacity_coeff: List, filename: str) -> Union[Dict, None]:
"""
Runs insel and write the necessary files
:param user_input: a dictionary containing the user
@ -56,11 +56,11 @@ class HeatPumpExport:
insel_file_handler.write(insel_template)
# Now run insel
self._delete_existing_output_files()
os.system('insel {}'.format(insel_file))
os.system('/usr/local/bin/insel {}'.format(insel_file))
# Writer headers to csv output files generated by insel
self._write_insel_output_headers()
# User output
self._get_user_out_put()
return self._get_user_out_put()
except IOError as err:
print("I/O exception: {}".format(err))
finally:
@ -72,17 +72,33 @@ class HeatPumpExport:
Write headers to the various csv file generated by insel
:return:
"""
header = [
'Year', ' Month', ' Day', 'Hour', 'Minute', 'HP Heat Output (kW)', 'Heating Demand (kW)', 'HP output flow rate',
'Building Required Flow Rate', 'TES Charging Rate (kg/s)', 'Water Flow Rate After Splitter',
'water temperature after splitter', 'TES Discharging Rate (kg/s)', 'TES discharge temperature',
'Mixer Outlet Flow Rate (kg/s)', 'Mixer outlet temperature', 'Auxiliary heater fuel flow rate',
'Auxiliary heater energy input (kW)', 'Building Inlet Flow Rate (kg/s)', 'Building inlet temperature',
'Building return temperature', 'TES Return Flow Rate (kg/s)', 'TES return temperature',
'TES Bypass Line Flow Rate (kg/s)', 'TES bypass line temperature', 'Flow Rate from TES to mixer 2 (kg/s)',
'Temperature from Tes to mixer', 'HP Inlet Flow Rate (kg/s)', 'HP Inlet temperature', 'TES Node 1 Temperature',
'TES Node 2 Temperature', 'TES Node 3 Temperature', 'TES Node 4 Temperature', 'TES Energy Content (J)',
'HP Electricity Consumption (kW)', 'HP COP', 'Ambient Temperature', 'HP Operational Cost (CAD)',
'Auxiliary Heater Operational Cost (CAD)', 'Operational CO2 Emissions of HP (g)',
'Operational CO2 Emissions of Auxiliary Heater (g)']
if 'series' in str(self._template_path):
header = [
'Year', ' Month', ' Day', 'Hour', 'Minute', 'HP Heat Output (kW)',
'HP Electricity Consumption (kW)', 'HP COP', 'TES Charging Rate (kg/s)',
'TES Discharging Rate (kg/s)', 'TES Node 1 Temperature', 'TES Node 2 Temperature',
'TES Node 3 Temperature', 'TES Node 4 Temperature', 'TES Energy Content (J)',
'TES Energy Content (kWh)', 'TES Energy Content Variation (kWh)',
'Auxiliary Heater Fuel Flow Rate (kg/s)', 'Auxiliary Heater Energy Input (kW)',
'HP Operational Cost (CAD)', 'Auxiliary Heater Operational Cost (CAD)',
'Operational CO2 Emissions of HP (g)',
'Operational CO2 Emissions of Auxiliary Heater (g)',
'Return Temperature', 'Demand (kW)']
header_data = {
self._input_data['fileOut1']: ['Year', ' Month', ' Day', 'Hour', 'Minute', 'HP Heat Output (kW)',
'HP Electricity Consumption (kW)', 'HP COP', 'TES Charging Rate (kg/s)',
'TES Discharging Rate (kg/s)', 'TES Node 1 Temperature', 'TES Node 2 Temperature',
'TES Node 3 Temperature', 'TES Node 4 Temperature', 'TES Energy Content (J)',
'TES Energy Content (kWh)', 'TES Energy Content Variation (kWh)',
'Auxiliary Heater Fuel Flow Rate (kg/s)', 'Auxiliary Heater Energy Input (kW)',
'HP Operational Cost (CAD)', 'Auxiliary Heater Operational Cost (CAD)',
'Operational CO2 Emissions of HP (g)',
'Operational CO2 Emissions of Auxiliary Heater (g)',
'Return Temperature', 'Demand (kW)'],
self._input_data['fileOut1']: header,
self._input_data['fileOut2']: ['Day', 'Operational Daily Emissions from Heat Pumps (g)',
'Operational Daily Emissions from Auxiliary Heater (g)'],
self._input_data['fileOut3']: ['Month', 'Monthly Operational Costs of Heat Pumps (CAD)',
@ -102,7 +118,7 @@ class HeatPumpExport:
file_path = file_path.strip("'")
df = pd.read_csv(file_path, header=None, sep='\s+')
# ignore ambient temperature for air source series run
if df.shape[1] > 25:
if df.shape[1] > 25 and 'series' in str(self._template_path):
df.drop(columns=df.columns[-1],
axis=1,
inplace=True)
@ -161,6 +177,8 @@ class HeatPumpExport:
with open(self._constants_path) as file:
constants_dict = yaml.load(file, Loader=yaml.FullLoader)
for key, value in constants_dict.items():
if key in ['LowestPossibleLoadFlow', 'HighestPossibleLoadFlow'] and self._water_temp is None:
continue
self._input_data[key] = value
# compute water to water HP specific values
if 55 <= self._input_data['HPSupTemp'] <= 60:
@ -206,19 +224,30 @@ class HeatPumpExport:
self._input_data["a10"] = a_coeff[9]
self._input_data["a11"] = a_coeff[10]
def _get_user_out_put(self):
def _get_user_out_put(self) -> Union[Dict, None]:
"""
Extracts monthly electricity demand and fossil fuel consumption
from output files generated by insel
:return:
:return: Dict for json output
"""
electricity_df = pd.read_csv(self._input_data['fileOut8'].strip("'")).iloc[:, 2]
fossil_df = pd.read_csv(self._input_data['fileOut4'].strip("'")).iloc[:, 2]
monthly_electricity_df = pd.read_csv(self._input_data['fileOut8'].strip("'")).iloc[:, 2]
monthly_fossil_df = pd.read_csv(self._input_data['fileOut4'].strip("'")).iloc[:, 2]
data = [electricity_df, fossil_df]
if self._output_path is None:
return {
'hourly_electricity_demand': pd.read_csv(self._input_data['fileOut10'].strip("'")).iloc[:, 5].tolist(),
'monthly_electricity_demand': monthly_electricity_df.tolist(),
'daily_electricity_demand': pd.read_csv(self._input_data['fileOut6'].strip("'")).iloc[:, 2].tolist(),
'daily_fossil_consumption': pd.read_csv(self._input_data['fileOut9'].strip("'")).iloc[:, 2].tolist(),
'monthly_fossil_consumption': monthly_fossil_df.tolist()
}
data = [monthly_electricity_df, monthly_fossil_df]
df = pd.concat(data, axis=1)
df = pd.concat([df, df.agg(['sum'])])
s = pd.Series(["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sept", "Oct", "Nov", "Dec", "Total"])
df = df.set_index([s])
df.to_csv(self._output_path)

12
exports/energy_systems/water_to_water_hp_export.py
View File

@ -6,7 +6,7 @@ Copyright © 2022 Concordia CERC group
Project Coder Peter Yefi peteryefi@gmail.com
"""
from exports.energy_systems.heat_pump_export import HeatPumpExport
from typing import List, Tuple, Union
from typing import List, Dict, Union
class WaterToWaterHPExport(HeatPumpExport):
@ -15,17 +15,19 @@ class WaterToWaterHPExport(HeatPumpExport):
after executing insel
"""
def __init__(self, base_path, city, output_path, sim_type):
def __init__(self, base_path, city, output_path, sim_type, demand_path):
"""
:param base_path: path to energy system files
:param city: the city object
:param output_path: the file to hold insel simulation results
:param sim_type: the simulation type to run: 1 for series, 0 for parallel
:param demand_path: path to hourly energy demand file
"""
tmp_file = 'heat_pumps/w2w_series.txt' if sim_type == 0 else 'heat_pumps/w2w_parallel.txt'
template_path = (base_path / tmp_file)
water_temp = (base_path / 'heat_pumps/wt_hourly3.txt')
super().__init__(base_path, city, output_path, template_path, water_temp)
super().__init__(base_path=base_path, city=city, output_path=output_path, template=template_path,
demand_path=demand_path, water_temp=water_temp)
def _extract_model_coff(self, hp_model: str) -> Union[List, None]:
"""
@ -39,7 +41,7 @@ class WaterToWaterHPExport(HeatPumpExport):
return energy_system.water_to_water_hp.power_demand_coff
return None
def execute_insel(self, user_input, hp_model):
def execute_insel(self, user_input, hp_model) -> Union[Dict, None]:
"""
Runs insel and produces output files
Runs insel and write the necessary files
@ -50,4 +52,4 @@ class WaterToWaterHPExport(HeatPumpExport):
:return:
"""
pow_demand_coeff = self._extract_model_coff(hp_model)
super(WaterToWaterHPExport, self)._run_insel(user_input, pow_demand_coeff, 'w2w.insel')
return super(WaterToWaterHPExport, self)._run_insel(user_input, pow_demand_coeff, 'w2w.insel')

10
exports/energy_systems_factory.py
View File

@ -16,7 +16,8 @@ class EnergySystemsExportFactory:
Exports factory class for energy systems
"""
def __init__(self, city, user_input, hp_model, output_path, sim_type=0, data_type='heat', base_path=None):
def __init__(self, city, user_input, hp_model, output_path, sim_type=0, data_type='heat', base_path=None,
demand_path=None):
"""
:param city: the city object
@ -26,7 +27,9 @@ class EnergySystemsExportFactory:
:param sim_type: the simulation type, 0 for series 1 for parallel
:param data_type: indicates whether cooling or heating data is used
:param base_path: the data directory of energy systems
:param demand_path: path to hourly energy dempand file
"""
self._city = city
if base_path is None:
base_path = Path(Path(__file__).parent.parent / 'data/energy_systems')
@ -36,6 +39,7 @@ class EnergySystemsExportFactory:
self._data_type = data_type
self._output_path = output_path
self._sim_type = sim_type
self._demand_path = demand_path
def _export_heat_pump(self, source):
"""
@ -44,10 +48,10 @@ class EnergySystemsExportFactory:
:return: None
"""
if source == 'air':
AirSourceHPExport(self._base_path, self._city, self._output_path, self._sim_type)\
return AirSourceHPExport(self._base_path, self._city, self._output_path, self._sim_type, self._demand_path)\
.execute_insel(self._user_input, self._hp_model, self._data_type)
elif source == 'water':
WaterToWaterHPExport(self._base_path, self._city, self._output_path, self._sim_type)\
return WaterToWaterHPExport(self._base_path, self._city, self._output_path, self._sim_type, self._demand_path)\
.execute_insel(self._user_input, self._hp_model)
def export(self, source='air'):

31
exports/user_factory.py Normal file
View File

@ -0,0 +1,31 @@
"""
User performs user related crud operations
SPDX - License - Identifier: LGPL - 3.0 - or -later
Copyright © 2022 Concordia CERC group
Project CoderPeter Yefi peteryefi@gmail.com
"""
from persistence import UserRepo
class UserFactory:
"""
UserFactory class
"""
def __init__(self, db_name, app_env, dotenv_path):
self._user_repo = UserRepo(db_name=db_name, app_env=app_env, dotenv_path=dotenv_path)
def login_user(self, email: str, password: str):
"""
Retrieve a single city from postgres
:param email: the email of the user
:param password: the password of the user
"""
return self._user_repo.get_user_by_email_and_password(email, password)
def get_user_by_email(self, email):
"""
Retrieve a single user
:param email: the email of the user to get
"""
return self._user_repo.get_by_email(email)

43
helpers/auth.py Normal file
View File

@ -0,0 +1,43 @@
import bcrypt
import re
class Auth(object):
@staticmethod
def validate_password(password: str) -> bool:
"""
Validates a password
:param password: the password to validate
:return:
"""
pattern = "^(?=.*[a-z])(?=.*[A-Z])(?=.*\d)(?=.*[@$!%*#?&])[A-Za-z\d@$!#%*?&]{6,20}$"
pattern = re.compile(pattern)
if not re.search(pattern, password):
raise ValueError("Password must be between 6 to 20 characters and must have at least a number, an uppercase "
"letter, a lowercase letter, and a special character")
return True
@staticmethod
def hash_password(password: str) -> str:
"""
Hashes a password
:param password: the password to be hashed
:return:
"""
return bcrypt.hashpw(password.encode('utf-8'), bcrypt.gensalt(14)).decode('utf-8')
@staticmethod
def check_password(password: str, hashed_password) -> bool:
"""
Hashes a password
:param password: the password to be checked
:param hashed_password: the hashed password
:return:
"""
return bcrypt.checkpw(password.encode('utf-8'), hashed_password.encode('utf-8'))

6
hub_logger/__init__.py Normal file
View File

@ -0,0 +1,6 @@
import logging as logger
from pathlib import Path
log_file = (Path(__file__).parent.parent / 'logs/hub.log').resolve()
logger.basicConfig(filename=log_file, format="%(asctime)s:%(levelname)s:{%(pathname)s:%(funcName)s:%(lineno)d} "
"- %(message)s", level=logger.DEBUG)

0
imports/__init__.py Normal file
View File

58
imports/db_factory.py Normal file
View File

@ -0,0 +1,58 @@
"""
DBFactory performs database create, delete and update operations
SPDX - License - Identifier: LGPL - 3.0 - or -later
Copyright © 2022 Concordia CERC group
Project CoderPeter Yefi peteryefi@gmail.com
"""
from persistence import CityRepo
from persistence import HeatPumpSimulationRepo
from typing import Dict
class DBFactory:
"""
DBFactory class
"""
def __init__(self, city, db_name, dotenv_path, app_env):
self._city = city
self._city_repo = CityRepo(db_name=db_name, dotenv_path=dotenv_path, app_env=app_env)
self._hp_simulation_repo = HeatPumpSimulationRepo(db_name=db_name, dotenv_path=dotenv_path, app_env=app_env)
def persist_city(self):
"""
Persist city into postgres database
"""
return self._city_repo.insert(self._city)
def update_city(self, city_id, city):
"""
Update an existing city in postgres database
:param city_id: the id of the city to update
:param city: the updated city object
"""
return self._city_repo.update(city_id, city)
def delete_city(self, city_id):
"""
Deletes a single city from postgres
:param city_id: the id of the city to get
"""
self._city_repo.delete_city(city_id)
def persist_hp_simulation(self, hp_simulation_data: Dict, city_id: int):
"""
Persist heat pump simulation results
:param hp_simulation_data: the simulation results
:param city_id: the city object used in running the simulation
:return: HeatPumpSimulation object
"""
return self._hp_simulation_repo.insert(hp_simulation_data, city_id)
def delete_hp_simulation(self, hp_sim_id):
"""
Deletes a single heat pump simulation from postgres
:param hp_sim_id: the id of the heat pump simulation to get
"""
self._hp_simulation_repo.delete_hp_simulation(hp_sim_id)

45
imports/user_factory.py Normal file
View File

@ -0,0 +1,45 @@
"""
User performs user-related crud operations
SPDX - License - Identifier: LGPL - 3.0 - or -later
Copyright © 2022 Concordia CERC group
Project CoderPeter Yefi peteryefi@gmail.com
"""
from persistence import UserRepo
from persistence import UserRoles
class UserFactory:
"""
UserFactory class
"""
def __init__(self, db_name, app_env, dotenv_path):
self._user_repo = UserRepo(db_name=db_name, app_env=app_env, dotenv_path=dotenv_path)
def create_user(self, name: str, email: str, password: str, role: UserRoles):
"""
Creates a new user
:param name: the name of the user
:param email: the email of the user
:param password: the password of the user
:param role: the role of the user
"""
return self._user_repo.insert(name, email, password, role)
def update_user(self, user_id: int, name: str, email: str, password: str, role: UserRoles):
"""
Creates a new user
:param user_id: the id of the user
:param name: the name of the user
:param email: the email of the user
:param password: the password of the user
:param role: the role of the user
"""
return self._user_repo.update(user_id, name, email, password, role)
def delete_user(self, user_id):
"""
Retrieve a single user
:param user_id: the id of the user to delete
"""
return self._user_repo.delete_user(user_id)

6
install_postgresql_linux.sh Executable file
View File

@ -0,0 +1,6 @@
#!/bin/sh
sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo apt-get update
sudo apt-get install postgresql

50
persistence/README.md Normal file
View File

@ -0,0 +1,50 @@
## Database Persistence ##
The persistence package includes classes to store different class objects in a Postgres database.
### models ###
This defines models for all class objects that we want to persist. It is used for Object Relation Mapping (ORM)
of the class objects to database table columns
### repositories ###
This defines repository classes that contain CRUD methods for database operations. The constructor of all repositories requires
The database name to connect to and the application environment (PROD or TEST). Tests use a different database
from the production environment, which is why this is necessary. An example is shown below
```python
from persistence import CityRepo
# instantiate city repo for hub production database
city_repo = CityRepo(db_name='hub', app_env='PROD')
```
All database operations are conducted with the production database (*PROD*) named *hub* in the example above
### config_db ##
This Python file is a configuration class that contains variables that map to configuration parameters in a .env file.
It also contains a method ``def conn_string()`` which returns the connection string to a Postgres database.
### Base ##
This class has a constructor that establishes a database connection and returns a reference for database-related CRUD operations.
### Database Configuration Parameter ###
A .env file (or environment variables) with configuration parameters described below are needed to establish a database connection:
```
# production database credentials
PROD_DB_USER=postgres-database-user
PROD_DB_PASSWORD=postgres-database-password
PROD_DB_HOST=database-host
PROD_DB_PORT=database-port
# test database credentials
TEST_DB_USER=postgres-database-user
TEST_DB_PASSWORD=postgres-database-password
TEST_DB_HOST=database-host
TEST_DB_PORT=database-port
```
### Database Related Unit Test
Unit tests that involve database operations require a Postgres database to be set up.
The tests connect to the database server using the default postgres user (*postgres*).
NB: You can provide any credentials for the test to connect to postgres, just make sure
the credentials are set in your .env file as explained above in *Database Configuration Parameters* section
When the tests are run, a **test_db** database is created and then the required tables for
the test. Before the tests run, the *test_db* is deleted to ensure that each test starts
on a clean slate

6
persistence/__init__.py Normal file
View File

@ -0,0 +1,6 @@
from .base_repo import BaseRepo
from .repositories.city_repo import CityRepo
from .repositories.heat_pump_simulation_repo import HeatPumpSimulationRepo
from .db_setup import DBSetup
from .repositories.user_repo import UserRepo
from .models.user import UserRoles

27
persistence/base_repo.py Normal file
View File

@ -0,0 +1,27 @@
"""
Base repository class to establish db connection
SPDX - License - Identifier: LGPL - 3.0 - or -later
Copyright © 2022 Concordia CERC group
Project Coder Peter Yefi peteryefi@gmail.com
"""
from persistence.db_config import BaseConfiguration
from sqlalchemy import create_engine
from sqlalchemy.orm import Session
class BaseRepo:
def __init__(self, db_name, dotenv_path: str, app_env='TEST'):
try:
self.config = BaseConfiguration(db_name, dotenv_path, app_env)
self.engine = create_engine(self.config.conn_string())
self.session = Session(self.engine)
except ValueError as err:
print(f'Missing value for credentials: {err}')

49
persistence/db_config.py Normal file
View File

@ -0,0 +1,49 @@
"""
Persistence (Postgresql) configuration
SPDX - License - Identifier: LGPL - 3.0 - or -later
Copyright © 2022 Concordia CERC group
Project Coder Peter Yefi peteryefi@gmail.com
"""
import os
from dotenv import load_dotenv
from sqlalchemy.ext.declarative import declarative_base
from hub_logger import logger
Base = declarative_base()
class BaseConfiguration(object):
"""
Base configuration class to hold common persistence configuration
"""
def __init__(self, db_name: str, dotenv_path: str, app_env='TEST'):
"""
:param db_name: database name
:param app_env: application environment, test or production
:param dotenv_path: the absolute path to dotenv file
"""
try:
# load environmental variables
load_dotenv(dotenv_path=dotenv_path)
self._db_name = db_name
self._db_host = os.getenv(f'{app_env}_DB_HOST')
self._db_user = os.getenv(f'{app_env}_DB_USER')
self._db_pass = os.getenv(f'{app_env}_DB_PASSWORD')
self._db_port = os.getenv(f'{app_env}_DB_PORT')
self.hub_token = os.getenv('HUB_TOKEN')
except KeyError as err:
logger.error(f'Error with credentials: {err}')
def conn_string(self):
"""
Returns a connection string postgresql
:return: connection string
"""
if self._db_pass:
return f'postgresql://{self._db_user}:{self._db_pass}@{self._db_host}:{self._db_port}/{self._db_name}'
return f'postgresql://{self._db_user}@{self._db_host}:{self._db_port}/{self._db_name}'
def get_db_user(self):
return self._db_user

37
persistence/db_setup.py Normal file
View File

@ -0,0 +1,37 @@
from persistence.models import City
from persistence import BaseRepo
from persistence.models import HeatPumpSimulation
from persistence.models import User
from persistence.repositories import UserRepo
from persistence.models import UserRoles
from hub_logger import logger
class DBSetup:
def __init__(self, db_name, app_env, dotenv_path):
"""
Creates database tables and a default admin user
:param db_name:
:param app_env:
:param dotenv_path:
"""
repo = BaseRepo(db_name=db_name, app_env=app_env, dotenv_path=dotenv_path)
City.__table__.create(bind=repo.engine, checkfirst=True)
HeatPumpSimulation.__table__.create(bind=repo.engine, checkfirst=True)
User.__table__.create(bind=repo.engine, checkfirst=True)
self._user_repo = UserRepo(db_name=db_name, app_env=app_env, dotenv_path=dotenv_path)
self._create_admin_user(self._user_repo)
def _create_admin_user(self, user_repo):
email = 'admin@hub.com'
password = 'HubAdmin#!98'
print('Creating default admin user...')
user = user_repo.insert('Administrator', email, password, UserRoles.Admin)
if type(user) is dict:
print(user)
logger.info(user)
else:
print(f'Created Admin user with email: {email}, password: {password} and role: {UserRoles.Admin}')
logger.info(f'Created Admin user with email: {email}, password: {password} and role: {UserRoles.Admin}')
print('Remember to change the admin default password and email address with the UserFactory')

5
persistence/models/__init__.py Normal file
View File

@ -0,0 +1,5 @@
from .city import City
from .heat_pump_simulation import HeatPumpSimulation
from .heat_pump_simulation import SimulationTypes
from .heat_pump_simulation import HeatPumpTypes
from .user import User, UserRoles

42
persistence/models/city.py Normal file
View File

@ -0,0 +1,42 @@
"""
Model representation of a City
SPDX - License - Identifier: LGPL - 3.0 - or -later
Copyright © 2022 Concordia CERC group
Project Coder Peter Yefi peteryefi@gmail.com
"""
from sqlalchemy import Column, Integer, String, Sequence
from sqlalchemy import DateTime, PickleType, Float
from persistence.db_config import Base
from sqlalchemy.dialects.postgresql import JSONB
import datetime
class City(Base):
"""A model representation of a city
"""
__tablename__ = "city"
id = Column(Integer, Sequence('city_id_seq'), primary_key=True)
city = Column(PickleType, nullable=False)
name = Column(String, nullable=False)
srs_name = Column(String, nullable=False)
climate_reference_city = Column(String, nullable=True)
time_zone = Column(String, nullable=True)
country_code = Column(String, nullable=False)
latitude = Column(Float)
longitude = Column(Float)
lower_corner = Column(JSONB, nullable=False)
upper_corner = Column(JSONB, nullable=False)
hub_release = Column(String, nullable=False)
city_version = Column(Integer, nullable=False)
created = Column(DateTime, default=datetime.datetime.utcnow)
def __init__(self, city, name, srs_name, country_code, l_corner, u_corner):
self.city = city
self.name = name
self.srs_name = srs_name
self.country_code = country_code
self.lower_corner = l_corner.tolist()
self.upper_corner = u_corner.tolist()

86
persistence/models/heat_pump_simulation.py Normal file
View File

@ -0,0 +1,86 @@
"""
Model representation of the results of heat pump simulation
SPDX - License - Identifier: LGPL - 3.0 - or -later
Copyright © 2022 Concordia CERC group
Project Coder Peter Yefi peteryefi@gmail.com
"""
from sqlalchemy import Column, Integer, String, Sequence
from sqlalchemy import Enum, ForeignKey, Float, DateTime
from sqlalchemy.dialects.postgresql import JSONB
from persistence.db_config import Base
import enum
import datetime
class SimulationTypes(enum.Enum):
Parallel = 'PARALLEL'
Series = 'SERIES'
class HeatPumpTypes(enum.Enum):
Air = 'Air Source'
Water = 'Water to Water'
class HeatPumpSimulation(Base):
"""A model representation of a building
Attributes:
city_id, A reference to the city which was used to run this simulation.
hourly_electricity_demand, A JSON object that has hours and their electricity demand
daily_electricity_demand, A JSON object that has days and their electricity demand
monthly_electricity_demand, A JSON object that has months and their electricity demand
daily_fossil_fuel_consumption, A JSON object that has days and fossil fuel consumption
monthly_fossil_fuel_consumption, A JSON object that has months and fossil fuel consumption
heat_pump_type, Water or air heat pump
simulation_type, The type of heat pump simulation (parallel or series)
heat_pump_model, The model of the heat pump (either water to water or air source)
start year, HP simulation start year
end year, HP simulation end year
max_hp_energy_input, Maximum heat pump energy input
max_demand_storage_hour, Hours of storage at maximum demand
building_supply_temp, building supply temperature
temp_difference, Difference in HP and building supply temperatures
fuel_lhv, The lower heating value of fuel
fuel_price, The price of fuel
fuel_efficiency, the efficiency of fuel
fuel_density, the density of fuel
hp_supply_temp, supply temperature of heat pump
"""
__tablename__ = "heat_pump_simulation"
id = Column(Integer, Sequence('hp_simulation_id_seq'), primary_key=True)
city_id = Column(Integer, ForeignKey('city.id'), nullable=False)
daily_electricity_demand = Column(JSONB, nullable=False)
hourly_electricity_demand = Column(JSONB, nullable=False)
daily_fossil_fuel_consumption = Column(JSONB, nullable=False)
monthly_fossil_fuel_consumption = Column(JSONB, nullable=False)
monthly_electricity_demand = Column(JSONB, nullable=False)
heat_pump_type = Column(Enum(HeatPumpTypes), nullable=False)
simulation_type = Column(Enum(SimulationTypes), nullable=False)
heat_pump_model = Column(String, nullable=False)
start_year = Column(Integer, nullable=False)
end_year = Column(Integer, nullable=False)
max_hp_energy_input = Column(Float, nullable=False)
max_demand_storage_hour = Column(Float, nullable=False)
building_supply_temp = Column(Float, nullable=False)
temp_difference = Column(Float, nullable=False)
fuel_lhv = Column(Float, nullable=False)
fuel_price = Column(Float, nullable=False)
fuel_efficiency = Column(Float, nullable=False)
fuel_density = Column(Float, nullable=False)
hp_supply_temp = Column(Float, nullable=False)
created = Column(DateTime, default=datetime.datetime.utcnow)
def __init__(self, city_id, hourly_elec_demand, daily_elec_demand, monthly_elec_demand, daily_fossil, monthly_fossil):
self.city_id = city_id
self.hourly_electricity_demand = hourly_elec_demand
self.daily_electricity_demand = daily_elec_demand
self.monthly_electricity_demand = monthly_elec_demand
self.daily_fossil_fuel_consumption = daily_fossil
self.monthly_fossil_fuel_consumption = monthly_fossil

45
persistence/models/user.py Normal file
View File

@ -0,0 +1,45 @@
"""
Model representation of a User
SPDX - License - Identifier: LGPL - 3.0 - or -later
Copyright © 2022 Concordia CERC group
Project Coder Peter Yefi peteryefi@gmail.com
"""
from sqlalchemy import Column, Integer, String, Sequence
from sqlalchemy import DateTime, Enum
from persistence.db_config import Base
import datetime
from sqlalchemy.orm import validates
import re
import enum
class UserRoles(enum.Enum):
Admin = 'ADMIN'
HubReader = 'HUB_READER'
class User(Base):
"""A model representation of a city
"""
__tablename__ = "user"
id = Column(Integer, Sequence('user_id_seq'), primary_key=True)
name = Column(String, nullable=False)
email = Column(String, nullable=False, unique=True)
password = Column(String, nullable=False)
role = Column(Enum(UserRoles), nullable=False, default=UserRoles.HubReader)
created = Column(DateTime, default=datetime.datetime.utcnow)
updated = Column(DateTime, default=datetime.datetime.utcnow)
@validates("email")
def validate_email(self, key, address):
pattern = r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b'
if not re.match(pattern, address):
raise ValueError("failed simple email validation")
return address
def __init__(self, name, email, password, role):
self.name = name
self.email = email
self.password = password
self.role = role

1
persistence/repositories/__init__.py Normal file
View File

@ -0,0 +1 @@
from .user_repo import UserRepo

138
persistence/repositories/city_repo.py Normal file
View File

@ -0,0 +1,138 @@
"""
City repository with database CRUD operations
SPDX - License - Identifier: LGPL - 3.0 - or -later
Copyright © 2022 Concordia CERC group
Project Coder Peter Yefi peteryefi@gmail.com
"""
from city_model_structure.city import City
from persistence import BaseRepo
from sqlalchemy.exc import SQLAlchemyError
from sqlalchemy import select
from persistence.models import City as DBCity
import pickle
import requests
from urllib3.exceptions import HTTPError
from typing import Union, Dict
from hub_logger import logger
class CityRepo(BaseRepo):
_instance = None
def __init__(self, db_name: str, dotenv_path: str, app_env: str):
super().__init__(db_name, dotenv_path, app_env)
def __new__(cls, db_name, dotenv_path, app_env):
"""
Implemented for a singleton pattern
"""
if cls._instance is None:
cls._instance = super(CityRepo, cls).__new__(cls)
return cls._instance
def insert(self, city: City) -> Union[City, Dict]:
db_city = DBCity(pickle.dumps(city), city.name, city.srs_name, city.country_code, city.lower_corner,
city.upper_corner)
db_city.climate_reference_city = city.climate_reference_city
db_city.longitude = city.longitude
db_city.latitude = city.latitude
db_city.time_zone = city.time_zone
try:
# Retrieve hub project latest release
response = requests.get("https://rs-loy-gitlab.concordia.ca/api/v4/projects/2/repository/branches/master",
headers={"PRIVATE-TOKEN": self.config.hub_token})
recent_commit = response.json()["commit"]["id"]
logger.info(f'Current commit of hub is {recent_commit}')
exiting_city = self._get_by_hub_version(recent_commit, city.name)
# Do not persist the same city for the same version of Hub
if exiting_city is None:
db_city.hub_release = recent_commit
cities = self.get_by_name(city.name)
# update version for the same city but different hub versions
if len(cities) == 0:
db_city.city_version = 0
else:
db_city.city_version = cities[-1].city_version + 1
# Persist city
self.session.add(db_city)
self.session.flush()
self.session.commit()
return db_city
else:
return {'message': f'Same version of {city.name} exist'}
except SQLAlchemyError as err:
logger.error(f'Error while adding city: {err}')
except HTTPError as err:
logger.error(f'Error retrieving Hub latest release: {err}')
def get_by_id(self, city_id: int) -> DBCity:
"""
Fetch a City based on the id
:param city_id: the city id
:return: a city
"""
try:
return self.session.execute(select(DBCity).where(DBCity.id == city_id)).first()[0]
except SQLAlchemyError as err:
logger.error(f'Error while fetching city: {err}')
def _get_by_hub_version(self, hub_commit: str, city_name: str) -> City:
"""
Fetch a City based on the name and hub project recent commit
:param hub_commit: the latest hub commit
:param city_name: the name of the city
:return: a city
"""
try:
return self.session.execute(select(DBCity)
.where(DBCity.hub_release == hub_commit, DBCity.name == city_name)).first()
except SQLAlchemyError as err:
logger.error(f'Error while fetching city: {err}')
def update(self, city_id: int, city: City):
"""
Updates a city
:param city_id: the id of the city to be updated
:param city: the city object
:return:
"""
try:
self.session.query(DBCity).filter(DBCity.id == city_id) \
.update({
'name': city.name, 'srs_name': city.srs_name, 'country_code': city.country_code, 'longitude': city.longitude,
'latitude': city.latitude, 'time_zone': city.time_zone, 'lower_corner': city.lower_corner.tolist(),
'upper_corner': city.upper_corner.tolist(), 'climate_reference_city': city.climate_reference_city,
})
self.session.commit()
except SQLAlchemyError as err:
logger.error(f'Error while updating city: {err}')
def get_by_name(self, city_name: str) -> [DBCity]:
"""
Fetch city based on the name
:param city_name: the name of the building
:return: [ModelCity] with the provided name
"""
try:
result_set = self.session.execute(select(DBCity).where(DBCity.name == city_name))
return [building[0] for building in result_set]
except SQLAlchemyError as err:
logger.error(f'Error while fetching city by name: {err}')
def delete_city(self, city_id: int):
"""
Deletes a City with the id
:param city_id: the city id
:return: a city
"""
try:
self.session.query(DBCity).filter(DBCity.id == city_id).delete()
self.session.commit()
except SQLAlchemyError as err:
logger.error(f'Error while fetching city: {err}')

108
persistence/repositories/heat_pump_simulation_repo.py Normal file
View File

@ -0,0 +1,108 @@
"""
Heat pump simulation repository with database CRUD operations
SPDX - License - Identifier: LGPL - 3.0 - or -later
Copyright © 2022 Concordia CERC group
Project Coder Peter Yefi peteryefi@gmail.com
"""
from persistence import BaseRepo, CityRepo
from sqlalchemy.exc import SQLAlchemyError
from sqlalchemy import select
from persistence.models import HeatPumpSimulation
from typing import Union, Dict
from hub_logger import logger
class HeatPumpSimulationRepo(BaseRepo):
_instance = None
def __init__(self, db_name, dotenv_path, app_env):
super().__init__(db_name, dotenv_path, app_env)
self._city_repo = CityRepo(db_name, dotenv_path, app_env)
def __new__(cls, db_name, dotenv_path, app_env):
"""
Implemented for a singleton pattern
"""
if cls._instance is None:
cls._instance = super(HeatPumpSimulationRepo, cls).__new__(cls)
return cls._instance
def insert(self, hp_sim_data: Dict, city_id: int) -> Union[HeatPumpSimulation, Dict]:
"""
Inserts the results of heat pump simulation
:param hp_sim_data: dictionary with heatpump the simulation inputs and output
:param city_id: the city that was used in running the simulation
:return: HeatPumpSimulation
"""
city = self._city_repo.get_by_id(city_id)
if city is None:
return {'message': 'city not found in database'}
try:
hp_simulation = HeatPumpSimulation(city_id, hp_sim_data["HourlyElectricityDemand"],
hp_sim_data["DailyElectricityDemand"], hp_sim_data["MonthlyElectricityDemand"],
hp_sim_data["DailyFossilFuelConsumption"],
hp_sim_data["MonthlyFossilFuelConsumption"])
hp_simulation.city_id = city_id
hp_simulation.end_year = hp_sim_data["EndYear"]
hp_simulation.start_year = hp_sim_data["StartYear"]
hp_simulation.max_demand_storage_hour = hp_sim_data["HoursOfStorageAtMaxDemand"]
hp_simulation.max_hp_energy_input = hp_sim_data["MaximumHPEnergyInput"]
hp_simulation.building_supply_temp = hp_sim_data["BuildingSuppTemp"]
hp_simulation.temp_difference = hp_sim_data["TemperatureDifference"]
hp_simulation.fuel_lhv = hp_sim_data["FuelLHV"]
hp_simulation.fuel_price = hp_sim_data["FuelPrice"]
hp_simulation.fuel_efficiency = hp_sim_data["FuelEF"]
hp_simulation.fuel_density = hp_sim_data["FuelDensity"]
hp_simulation.hp_supply_temp = hp_sim_data["HPSupTemp"]
hp_simulation.simulation_type = hp_sim_data["SimulationType"]
hp_simulation.heat_pump_model = hp_sim_data["HeatPumpModel"]
hp_simulation.heat_pump_type = hp_sim_data["HeatPumpType"]
# Persist heat pump simulation data
self.session.add(hp_simulation)
self.session.flush()
self.session.commit()
return hp_simulation
except SQLAlchemyError as err:
logger.error(f'Error while saving heat pump simulation data: {err}')
except KeyError as err:
logger.error(f'A required field is missing in your heat pump simulation dictionary: {err}')
def get_by_id(self, hp_simulation_id: int) -> HeatPumpSimulation:
"""
Fetches heat pump simulation data
:param hp_simulation_id: the city id
:return: a HeatPumpSimulation
"""
try:
return self.session.execute(select(HeatPumpSimulation).where(HeatPumpSimulation.id == hp_simulation_id)).first()[
0]
except SQLAlchemyError as err:
logger.error(f'Error while fetching city: {err}')
def get_by_city(self, city_id: int) -> [HeatPumpSimulation]:
"""
Fetch heat pump simulation results by city
:param city_id: the name of the building
:return: [HeatPumpSimulation] with the provided name
"""
try:
result_set = self.session.execute(select(HeatPumpSimulation).where(HeatPumpSimulation.city_id == city_id))
return [sim_data[0] for sim_data in result_set]
except SQLAlchemyError as err:
logger.error(f'Error while fetching city by name: {err}')
def delete_hp_simulation(self, hp_simulation_id: int):
"""
Deletes a heat pump simulation results
:param hp_simulation_id: the heat pump simulation results id
:return:
"""
try:
self.session.query(HeatPumpSimulation).filter(HeatPumpSimulation.id == hp_simulation_id).delete()
self.session.commit()
except SQLAlchemyError as err:
logger.error(f'Error while fetching city: {err}')

102
persistence/repositories/user_repo.py Normal file
View File

@ -0,0 +1,102 @@
"""
City repository with database CRUD operations
SPDX - License - Identifier: LGPL - 3.0 - or -later
Copyright © 2022 Concordia CERC group
Project Coder Peter Yefi peteryefi@gmail.com
"""
from persistence import BaseRepo
from sqlalchemy.exc import SQLAlchemyError
from sqlalchemy import select
from persistence.models import User
from persistence.models import UserRoles
from helpers.auth import Auth
from typing import Union, Dict
from hub_logger import logger
class UserRepo(BaseRepo):
_instance = None
def __init__(self, db_name: str, dotenv_path: str, app_env: str):
super().__init__(db_name, dotenv_path, app_env)
def __new__(cls, db_name, dotenv_path, app_env):
"""
Implemented for a singleton pattern
"""
if cls._instance is None:
cls._instance = super(UserRepo, cls).__new__(cls)
return cls._instance
def insert(self, name: str, email: str, password: str, role: UserRoles) -> Union[User, Dict]:
user = self.get_by_email(email)
if user is None:
try:
if Auth.validate_password(password):
user = User(name=name, email=email, password=Auth.hash_password(password), role=role)
self.session.add(user)
self.session.flush()
self.session.commit()
return user
except SQLAlchemyError as err:
logger.error(f'An error occured while creating user: {err}')
else:
return {'message': f'user with {email} email already exists'}
def update(self, user_id: int, name: str, email: str, password: str, role: UserRoles):
"""
Updates a user
:param user_id: the id of the user to be updated
:param name: the name of the user
:param email: the email of the user
:param password: the password of the user
:param role: the role of the user
:return:
"""
try:
if Auth.validate_password(password):
self.session.query(User).filter(User.id == user_id) \
.update({'name': name, 'email': email, 'password': Auth.hash_password(password), 'role': role})
self.session.commit()
except SQLAlchemyError as err:
logger.error(f'Error while updating user: {err}')
def get_by_email(self, email: str) -> [User]:
"""
Fetch user based on the email address
:param email: the email of the user
:return: [User] with the provided email
"""
try:
return self.session.execute(select(User).where(User.email == email)).first()
except SQLAlchemyError as err:
logger.error(f'Error while fetching user by email: {err}')
def delete_user(self, user_id: int):
"""
Deletes a user with the id
:param user_id: the user id
:return: None
"""
try:
self.session.query(User).filter(User.id == user_id).delete()
self.session.commit()
except SQLAlchemyError as err:
logger.error(f'Error while fetching user: {err}')
def get_user_by_email_and_password(self, email: str, password: str) -> [User]:
"""
Fetch user based on the email and password
:param email: the email of the user
:param password: the password of the user
:return: [User] with the provided email and password
"""
try:
user = self.session.execute(select(User).where(User.email == email)).first()
if user:
if Auth.check_password(password, user[0].password):
return user
return {'message': 'user not found'}
except SQLAlchemyError as err:
logger.error(f'Error while fetching user by email: {err}')

5
requirements.txt
View File

@ -16,6 +16,9 @@ rhino3dm==7.7.0
scipy
PyYAML
pyecore==0.12.2
python-dotenv
SQLAlchemy
bcrypt==4.0.1
shapely
geopandas
triangle
triangle

90
unittests/test_db_factory.py Normal file
View File

@ -0,0 +1,90 @@
"""
Test EnergySystemsFactory and various heatpump models
SPDX - License - Identifier: LGPL - 3.0 - or -later
Copyright © 2022 Concordia CERC group
Project Coder Peter Yefi peteryefi@gmail.com
"""
from unittest import TestCase
from imports.geometry_factory import GeometryFactory
from imports.db_factory import DBFactory
from exports.db_factory import DBFactory as ExportDBFactory
from persistence.base_repo import BaseRepo
from sqlalchemy import create_engine
from persistence.models import City
from pickle import loads
from sqlalchemy.exc import ProgrammingError
class TestDBFactory(TestCase):
"""
TestDBFactory
"""
@classmethod
def setUpClass(cls) -> None:
"""
Test setup
:return: None
"""
# Create test database
repo = BaseRepo(db_name='test_db', app_env='TEST', dotenv_path='../.env')
eng = create_engine(f'postgresql://{repo.config.get_db_user()}@/{repo.config.get_db_user()}')
try:
# delete test database if it exists
conn = eng.connect()
conn.execute('commit')
conn.execute('DROP DATABASE test_db')
conn.close()
except ProgrammingError as err:
print(f'Database does not exist. Nothing to delete')
cnn = eng.connect()
cnn.execute('commit')
cnn.execute("CREATE DATABASE test_db")
cnn.close()
City.__table__.create(bind=repo.engine, checkfirst=True)
city_file = "../unittests/tests_data/C40_Final.gml"
cls.city = GeometryFactory('citygml', city_file).city
cls._db_factory = DBFactory(city=cls.city, db_name='test_db', app_env='TEST', dotenv_path='../.env')
cls._export_db_factory = ExportDBFactory(db_name='test_db', app_env='TEST', dotenv_path='../.env')
def test_save_city(self):
saved_city = self._db_factory.persist_city()
self.assertEqual(saved_city.name, 'Montréal')
pickled_city = loads(saved_city.city)
self.assertEqual(len(pickled_city.buildings), 10)
self.assertEqual(pickled_city.buildings[0].floor_area, 1990.9913970530033)
self._db_factory.delete_city(saved_city.id)
def test_save_same_city_with_same_hub_version(self):
first_city = self._db_factory.persist_city()
second_city = self._db_factory.persist_city()
self.assertEqual(second_city['message'], f'Same version of {self.city.name} exist')
self.assertEqual(first_city.name, 'Montréal')
self.assertEqual(first_city.country_code, 'ca')
self._db_factory.delete_city(first_city.id)
def test_get_city_by_name(self):
city = self._db_factory.persist_city()
retrieved_city = self._export_db_factory.get_city_by_name(city.name)
self.assertEqual(retrieved_city[0].lower_corner[0], 610610.7547462888)
self._db_factory.delete_city(city.id)
def test_get_city_by_id(self):
city = self._db_factory.persist_city()
retrieved_city = self._export_db_factory.get_city(city.id)
self.assertEqual(retrieved_city.upper_corner[0], 610818.6731258357)
self._db_factory.delete_city(city.id)
def test_get_update_city(self):
city = self._db_factory.persist_city()
self.city.longitude = 1.43589
self.city.latitude = -9.38928339
self._db_factory.update_city(city.id, self.city)
updated_city = self._export_db_factory.get_city(city.id)
self.assertEqual(updated_city.longitude, 1.43589)
self.assertEqual(updated_city.latitude, -9.38928339)
self._db_factory.delete_city(city.id)

42
unittests/test_energy_systems_air_source_hp.py
View File

@ -12,6 +12,21 @@ from city_model_structure.energy_systems.air_source_hp import AirSourceHP
from exports.energy_systems_factory import EnergySystemsExportFactory
import os
# User defined paramenters
user_input = {
'StartYear': 2020,
'EndYear': 2021,
'MaximumHPEnergyInput': 8000,
'HoursOfStorageAtMaxDemand': 1,
'BuildingSuppTemp': 40,
'TemperatureDifference': 15,
'FuelLHV': 47100,
'FuelPrice': 0.12,
'FuelEF': 1887,
'FuelDensity': 0.717,
'HPSupTemp': 60
}
class TestEnergySystemsFactory(TestCase):
"""
@ -34,27 +49,20 @@ class TestEnergySystemsFactory(TestCase):
self.assertEqual(self._city.energy_systems[0].air_source_hp.model, '012')
self.assertEqual(self._city.energy_systems[16].air_source_hp.model, '140')
def test_air_source_heat_pump_export(self):
# User defined paramenters
user_input = {
'StartYear': 2020,
'EndYear': 2021,
'MaximumHPEnergyInput': 8000,
'HoursOfStorageAtMaxDemand': 1,
'BuildingSuppTemp': 40,
'TemperatureDifference': 15,
'FuelLHV': 47100,
'FuelPrice': 0.12,
'FuelEF': 1887,
'FuelDensity': 0.717,
'HPSupTemp': 60
}
EnergySystemsExportFactory(self._city, user_input, '012', self._output_path).export()
def test_air_source_series_heat_pump_export(self):
EnergySystemsExportFactory(city=self._city, user_input=user_input, hp_model='012',
output_path=self._output_path).export()
df = pd.read_csv(self._output_path)
self.assertEqual(df.shape, (13, 3))
self.assertEqual(df.iloc[0, 1], 1867715.88)
def test_air_source_parallel_heat_pump_export(self):
output = EnergySystemsExportFactory(city=self._city, user_input=user_input, hp_model='018',
output_path=None, sim_type=1).export()
self.assertEqual(output["hourly_electricity_demand"][0], 38748.5625)
self.assertIsNotNone(output["daily_fossil_consumption"])
self.assertEqual(len(output["hourly_electricity_demand"]), 8760)
def tearDown(self) -> None:
try:
os.remove(self._output_path)

23
unittests/test_energy_systems_water_to_water_hp.py
View File

@ -13,6 +13,21 @@ import pandas as pd
import os
# User defined paramenters
user_input = {
'StartYear': 2020,
'EndYear': 2021,
'MaximumHPEnergyInput': 8000,
'HoursOfStorageAtMaxDemand': 1,
'BuildingSuppTemp': 40,
'TemperatureDifference': 15,
'FuelLHV': 47100,
'FuelPrice': 0.12,
'FuelEF': 1887,
'FuelDensity': 0.717,
'HPSupTemp': 60
}
class TestEnergySystemsFactory(TestCase):
"""
@ -62,11 +77,11 @@ class TestEnergySystemsFactory(TestCase):
'b11': 10
}
EnergySystemsExportFactory(self._city, user_input, 'ClimateMaster 156 kW', self._output_path).export('water')
EnergySystemsExportFactory(city=self._city, user_input=user_input, hp_model='ClimateMaster 256 kW',
output_path=self._output_path, sim_type=1).export('water')
df = pd.read_csv(self._output_path)
print(df.shape)
#self.assertEqual(df.shape, (13, 3))
#self.assertEqual(df.iloc[0, 1], 3045398.0)
self.assertEqual(df.shape, (13, 3))
self.assertEqual(df.iloc[0, 1], 1031544.62)
def tearDown(self) -> None:
try:

118
unittests/test_heat_pump_simulation.py Normal file
View File

@ -0,0 +1,118 @@
"""
Test EnergySystemsFactory and various heatpump models
SPDX - License - Identifier: LGPL - 3.0 - or -later
Copyright © 2022 Concordia CERC group
Project Coder Peter Yefi peteryefi@gmail.com
"""
from unittest import TestCase
from imports.geometry_factory import GeometryFactory
from imports.energy_systems_factory import EnergySystemsFactory
from exports.energy_systems_factory import EnergySystemsExportFactory
from imports.db_factory import DBFactory
from exports.db_factory import DBFactory as ExportDBFactory
from persistence.base_repo import BaseRepo
from sqlalchemy import create_engine
from persistence.models import City
from persistence.models import SimulationTypes
from persistence.models import HeatPumpTypes
from persistence.models import HeatPumpSimulation
from sqlalchemy.exc import ProgrammingError
# User defined paramenters
hp_sim_data = {
'StartYear': 2020,
'EndYear': 2021,
'MaximumHPEnergyInput': 8000,
'HoursOfStorageAtMaxDemand': 1,
'BuildingSuppTemp': 40,
'TemperatureDifference': 15,
'FuelLHV': 47100,
'FuelPrice': 0.12,
'FuelEF': 1887,
'FuelDensity': 0.717,
'HPSupTemp': 60
}
class TestHeatPumpSimulation(TestCase):
"""
Heat pump simulation test cases
"""
@classmethod
def setUpClass(cls) -> None:
"""
Test setup
:return: None
"""
repo = BaseRepo(db_name='test_db', app_env='TEST', dotenv_path='../.env')
eng = create_engine(f'postgresql://{repo.config.get_db_user()}@/{repo.config.get_db_user()}')
try:
conn = eng.connect()
conn.execute('commit')
conn.execute('DROP DATABASE test_db')
conn.close()
except ProgrammingError as err:
print(f'Database does not exist. Nothing to delete')
cnn = eng.connect()
cnn.execute('commit')
cnn.execute("CREATE DATABASE test_db")
cnn.close()
# Create test tables if they do not exit
City.__table__.create(bind=repo.engine, checkfirst=True)
HeatPumpSimulation.__table__.create(bind=repo.engine, checkfirst=True)
city_file = "../unittests/tests_data/C40_Final.gml"
cls._city = GeometryFactory('citygml', city_file).city
EnergySystemsFactory('air source hp', cls._city).enrich()
cls._db_factory = DBFactory(city=cls._city, db_name='test_db', app_env='TEST', dotenv_path='../.env')
cls._export_db_factory = ExportDBFactory(db_name='test_db', app_env='TEST', dotenv_path='../.env')
def test_heat_pump_simulation_persistence(self):
output = EnergySystemsExportFactory(city=self._city, user_input=hp_sim_data, hp_model='018',
output_path=None, sim_type=1).export()
hp_sim_data["HeatPumpModel"] = '018'
hp_sim_data["SimulationType"] = SimulationTypes.Parallel
hp_sim_data["HeatPumpType"] = HeatPumpTypes.Air
hp_sim_data["HourlyElectricityDemand"] = output["hourly_electricity_demand"]
hp_sim_data["DailyElectricityDemand"] = output["daily_electricity_demand"]
hp_sim_data["MonthlyElectricityDemand"] = output["monthly_electricity_demand"]
hp_sim_data["DailyFossilFuelConsumption"] = output["daily_fossil_consumption"]
hp_sim_data["MonthlyFossilFuelConsumption"] = output["monthly_fossil_consumption"]
saved_city = self._db_factory.persist_city()
hp_sim = self._db_factory.persist_hp_simulation(hp_sim_data, saved_city.id)
self.assertEqual(hp_sim.heat_pump_type, HeatPumpTypes.Air)
self.assertEqual(hp_sim.simulation_type, SimulationTypes.Parallel)
self.assertEqual(hp_sim.fuel_efficiency, hp_sim_data["FuelEF"])
self.assertEqual(hp_sim.monthly_electricity_demand, output["monthly_electricity_demand"])
self._db_factory.delete_hp_simulation(hp_sim.id)
self._db_factory.delete_city(saved_city.id)
def test_get_heat_pump_simulation_by_city(self):
output = EnergySystemsExportFactory(city=self._city, user_input=hp_sim_data, hp_model='012',
output_path=None, sim_type=0).export()
hp_sim_data["HeatPumpModel"] = '012'
hp_sim_data["SimulationType"] = SimulationTypes.Series
hp_sim_data["HeatPumpType"] = HeatPumpTypes.Air
hp_sim_data["HourlyElectricityDemand"] = output["hourly_electricity_demand"]
hp_sim_data["DailyElectricityDemand"] = output["daily_electricity_demand"]
hp_sim_data["MonthlyElectricityDemand"] = output["monthly_electricity_demand"]
hp_sim_data["DailyFossilFuelConsumption"] = output["daily_fossil_consumption"]
hp_sim_data["MonthlyFossilFuelConsumption"] = output["monthly_fossil_consumption"]
saved_city = self._db_factory.persist_city()
self._db_factory.persist_hp_simulation(hp_sim_data, saved_city.id)
# retrieved saved simulation by city id
hp_sim = self._export_db_factory.get_hp_simulation_by_city(saved_city.id)
self.assertEqual(hp_sim[0].heat_pump_type, HeatPumpTypes.Air)
self.assertEqual(hp_sim[0].simulation_type, SimulationTypes.Series)
self.assertEqual(hp_sim[0].fuel_price, hp_sim_data["FuelPrice"])
self.assertEqual(hp_sim[0].hourly_electricity_demand, output["hourly_electricity_demand"])
self._db_factory.delete_hp_simulation(hp_sim[0].id)
self._db_factory.delete_city(saved_city.id)