Merged Develop to resolved conflicts with OEDv4, and update quoting

.github/workflows/test.yml

@@ -2,6 +2,9 @@ name: OpenData Testing
 
 on:
   push:
+    branches:
+      - master
+      - develop
   pull_request:
   workflow_dispatch:
     inputs:

OpenExposureData/Abbreviations.csv

@@ -20,4 +20,4 @@
 "Port","Portfolio"
 "Reins","Reinsurance"
 "TIV","Total Insurable Value"
-"Vuln","Vulnerability"
+"Vuln","Vulnerability"

OpenExposureData/IndustryCodeValues.csv

@@ -8264,4 +8264,4 @@
 "UKSIC","2007","98100","Undifferentiated goods-producing activities of private households for own use"
 "UKSIC","2007","98200","Undifferentiated service-producing activities of private households for own use"
 "UKSIC","2007","99000","Activities of extraterritorial organizations and bodies"
-"UKSIC","2007","99999","Dormant Company"
+"UKSIC","2007","99999","Dormant Company"

OpenExposureData/PerilValues.csv

@@ -1,48 +1,48 @@
-DB table PerilCode,Peril Description,Input format abbreviation,Grouped PerilCode
-1,Earthquake - Shake only,QEQ,No
-2,Fire Following,QFF,No
-4,Tsunami,QTS,No
-8,Sprinkler Leakage,QSL,No
-16,Landslide,QLS,No
-32,Liquefaction,QLF,No
-64,Strong wind from Tropical Cyclone,WTC,No
-128,Strong wind from Extra Tropical Cyclone,WEC,No
-256,Storm Surge from strong wind,WSS,No
-512,River / Fluvial Flood,ORF,No
-1024,Flash / Surface / Pluvial Flood,OSF,No
-2048,Straight-line wind,XSL,No
-4096,Tornado,XTD,No
-8192,Hail,XHL,No
-16384,Snow,ZSN,No
-32768,Ice,ZIC,No
-65536,Freeze,ZFZ,No
-131072,NonCat,BFR,No
-262144,Wildfire / Bushfire,BBF,No
-524288,NBCR Terrorism,MNT,No
-1048576,Conventional Terrorism,MTR,No
-2097152,Lightning,XLT,No
-4194304,Winterstorm Wind,ZST,No
-8388608,Smoke,BSK,No
-16777216,Drought Induced Subsidence,SSD,No
-33554432,Crop Hail (from convective storm),XCH,No
-67108864,Cyber Security Data and Privacy Breach,CSB,No
-134217728,Cyber Security Property Damage,CPD,No
-268435456,Pandemic Flu,PNF,No
-536870912,Volcanic Ash Cloud,VVA,No
-1073741824,Volcanic lava flow/eruption,VVE,No
-2147483648,Volcanic landslide/mudslide,VVL,No
-4294967296,Sewage Backup,SBU,No
-63,All EQ perils,QQ1,Yes
-192,Windstorm w/o storm surge,WW2,Yes
-448,Windstorm with storm surge,WW1,Yes
-1536,Flood w/o storm surge,OO1,Yes
-1572864,Terrorism,MM1,Yes
-2111488,Convective Storm,XX1,Yes
-4308992,Winter storm,ZZ1,Yes
-6420480,Convective storm (incl winter storm),XZ1,Yes
-8650752,Wildfire with smoke,BB1,Yes
-268435456,All Pandemic perils,PP1,Yes
-33554432,All Crop perils,GG1,Yes
-201326592,All Cyber perils,CC1,Yes
-3758096384,All Volcanic perils,VV1,Yes
-8589934591,All perils,AA1,Yes
+"DB table PerilCode","Peril Description","Input format abbreviation","Grouped PerilCode"
+"1","Earthquake - Shake only","QEQ","No"
+"2","Fire Following","QFF","No"
+"4","Tsunami","QTS","No"
+"8","Sprinkler Leakage","QSL","No"
+"16","Landslide","QLS","No"
+"32","Liquefaction","QLF","No"
+"64","Strong wind from Tropical Cyclone","WTC","No"
+"128","Strong wind from Extra Tropical Cyclone","WEC","No"
+"256","Storm Surge from strong wind","WSS","No"
+"512","River / Fluvial Flood","ORF","No"
+"1024","Flash / Surface / Pluvial Flood","OSF","No"
+"2048","Straight-line wind","XSL","No"
+"4096","Tornado","XTD","No"
+"8192","Hail","XHL","No"
+"16384","Snow","ZSN","No"
+"32768","Ice","ZIC","No"
+"65536","Freeze","ZFZ","No"
+"131072","NonCat","BFR","No"
+"262144","Wildfire / Bushfire","BBF","No"
+"524288","NBCR Terrorism","MNT","No"
+"1048576","Conventional Terrorism","MTR","No"
+"2097152","Lightning","XLT","No"
+"4194304","Winterstorm Wind","ZST","No"
+"8388608","Smoke","BSK","No"
+"16777216","Drought Induced Subsidence","SSD","No"
+"33554432","Crop Hail (from convective storm)","XCH","No"
+"67108864","Cyber Security Data and Privacy Breach","CSB","No"
+"134217728","Cyber Security Property Damage","CPD","No"
+"268435456","Pandemic Flu","PNF","No"
+"536870912","Volcanic Ash Cloud","VVA","No"
+"1073741824","Volcanic lava flow/eruption","VVE","No"
+"2147483648","Volcanic landslide/mudslide","VVL","No"
+"4294967296","Sewage Backup","SBU","No"
+"63","All EQ perils","QQ1","Yes"
+"192","Windstorm w/o storm surge","WW2","Yes"
+"448","Windstorm with storm surge","WW1","Yes"
+"1536","Flood w/o storm surge","OO1","Yes"
+"1572864","Terrorism","MM1","Yes"
+"2111488","Convective Storm","XX1","Yes"
+"4308992","Winter storm","ZZ1","Yes"
+"6420480","Convective storm (incl winter storm)","XZ1","Yes"
+"8650752","Wildfire with smoke","BB1","Yes"
+"268435456","All Pandemic perils","PP1","Yes"
+"33554432","All Crop perils","GG1","Yes"
+"201326592","All Cyber perils","CC1","Yes"
+"3758096384","All Volcanic perils","VV1","Yes"
+"8589934591","All perils","AA1","Yes"

OpenExposureData/PerilsCovered.csv

@@ -1,107 +1,107 @@
-Peril,PerilsCovered
-QEQ,QEQ
-QFF,QFF
-QTS,QTS
-QSL,QSL
-QLS,QLS
-QLF,QLF
-WTC,WTC
-WEC,WEC
-WSS,WSS
-ORF,ORF
-OSF,OSF
-XSL,XSL
-XTD,XTD
-XHL,XHL
-ZSN,ZSN
-ZIC,ZIC
-ZFZ,ZFZ
-BFR,BFR
-BBF,BBF
-MNT,MNT
-MTR,MTR
-XLT,XLT
-ZST,ZST
-BSK,BSK
-SSD,SSD
-XCH,XCH
-CSB,CSB
-CPD,CPD
-PNF,PNF
-VVA,VVA
-VVE,VVE
-VVL,VVL
-SBU,SBU
-QEQ,QQ1
-QFF,QQ1
-QTS,QQ1
-QSL,QQ1
-QLS,QQ1
-QLF,QQ1
-WTC,WW2
-WEC,WW2
-WTC,WW1
-WEC,WW1
-WSS,WW1
-ORF,OO1
-OSF,OO1
-MNT,MM1
-MTR,MM1
-XSL,XX1
-XTD,XX1
-XHL,XX1
-XLT,XX1
-ZSN,ZZ1
-ZIC,ZZ1
-ZFZ,ZZ1
-ZST,ZZ1
-XSL,XZ1
-XTD,XZ1
-XHL,XZ1
-XLT,XZ1
-ZSN,XZ1
-ZIC,XZ1
-ZFZ,XZ1
-ZST,XZ1
-BBF,BB1
-BSK,BB1
-XCH,GG1
-CSB,CC1
-CPD,CC1
-PNF,PP1
-VVA,VV1
-VVE,VV1
-VVL,VV1
-BBF,AA1
-BFR,AA1
-BSK,AA1
-CPD,AA1
-CSB,AA1
-MNT,AA1
-MTR,AA1
-ORF,AA1
-OSF,AA1
-PNF,AA1
-QEQ,AA1
-QFF,AA1
-QLF,AA1
-QLS,AA1
-QSL,AA1
-QTS,AA1
-SBU,AA1
-SSD,AA1
-VVA,AA1
-VVE,AA1
-VVL,AA1
-WEC,AA1
-WSS,AA1
-WTC,AA1
-XCH,AA1
-XHL,AA1
-XLT,AA1
-XSL,AA1
-XTD,AA1
-ZFZ,AA1
-ZIC,AA1
-ZSN,AA1
-ZST,AA1
+"Peril","PerilsCovered"
+"QEQ","QEQ"
+"QFF","QFF"
+"QTS","QTS"
+"QSL","QSL"
+"QLS","QLS"
+"QLF","QLF"
+"WTC","WTC"
+"WEC","WEC"
+"WSS","WSS"
+"ORF","ORF"
+"OSF","OSF"
+"XSL","XSL"
+"XTD","XTD"
+"XHL","XHL"
+"ZSN","ZSN"
+"ZIC","ZIC"
+"ZFZ","ZFZ"
+"BFR","BFR"
+"BBF","BBF"
+"MNT","MNT"
+"MTR","MTR"
+"XLT","XLT"
+"ZST","ZST"
+"BSK","BSK"
+"SSD","SSD"
+"XCH","XCH"
+"CSB","CSB"
+"CPD","CPD"
+"PNF","PNF"
+"VVA","VVA"
+"VVE","VVE"
+"VVL","VVL"
+"SBU","SBU"
+"QEQ","QQ1"
+"QFF","QQ1"
+"QTS","QQ1"
+"QSL","QQ1"
+"QLS","QQ1"
+"QLF","QQ1"
+"WTC","WW2"
+"WEC","WW2"
+"WTC","WW1"
+"WEC","WW1"
+"WSS","WW1"
+"ORF","OO1"
+"OSF","OO1"
+"MNT","MM1"
+"MTR","MM1"
+"XSL","XX1"
+"XTD","XX1"
+"XHL","XX1"
+"XLT","XX1"
+"ZSN","ZZ1"
+"ZIC","ZZ1"
+"ZFZ","ZZ1"
+"ZST","ZZ1"
+"XSL","XZ1"
+"XTD","XZ1"
+"XHL","XZ1"
+"XLT","XZ1"
+"ZSN","XZ1"
+"ZIC","XZ1"
+"ZFZ","XZ1"
+"ZST","XZ1"
+"BBF","BB1"
+"BSK","BB1"
+"XCH","GG1"
+"CSB","CC1"
+"CPD","CC1"
+"PNF","PP1"
+"VVA","VV1"
+"VVE","VV1"
+"VVL","VV1"
+"BBF","AA1"
+"BFR","AA1"
+"BSK","AA1"
+"CPD","AA1"
+"CSB","AA1"
+"MNT","AA1"
+"MTR","AA1"
+"ORF","AA1"
+"OSF","AA1"
+"PNF","AA1"
+"QEQ","AA1"
+"QFF","AA1"
+"QLF","AA1"
+"QLS","AA1"
+"QSL","AA1"
+"QTS","AA1"
+"SBU","AA1"
+"SSD","AA1"
+"VVA","AA1"
+"VVE","AA1"
+"VVL","AA1"
+"WEC","AA1"
+"WSS","AA1"
+"WTC","AA1"
+"XCH","AA1"
+"XHL","AA1"
+"XLT","AA1"
+"XSL","AA1"
+"XTD","AA1"
+"ZFZ","AA1"
+"ZIC","AA1"
+"ZSN","AA1"
+"ZST","AA1"

README.md

@@ -94,7 +94,24 @@ The diagram below highlights the proposed, long-term structure of ODS and all th
 
 ## Open Exposure Data (OED)
 
-The aim of OED is to provide the industry with a robust, open, and transparent data format. This will improve efficiency and transparency for the cat modelling community, facilitating data transfer and analytics across models and vendors. OED is a model agnostic data format and the detailed descriptions of each data field for property and how to code financial structures can all be found here: https://oasislmf.github.io/sections/OED.html
+The aim of OED is to provide the industry with a robust, open, and transparent data format. 
+This will improve efficiency and transparency for the cat modelling community, facilitating data transfer and analytics across models and vendors. 
+OED is a model agnostic data format and the detailed descriptions of each data field for property 
+and how to code financial structures can all be found here: https://oasislmf.github.io/sections/OED.html
+
+Up until version 3.4.0, those fields are covered in the 'Open Exposure Data Spec.xlsx'
+and reference and background OED information can be found in the 'docs’ folder
+(https://github.com/OasisLMF/ODS_OpenExposureData/tree/3.4.0/OpenExposureData/Docs).
+
+from 3.4.1, for ease of change tracking, the fields are now describe in csv files where each file represent a sheet in the previous xlsx version.
+those can be found directly in the OpenExposureData folder (https://github.com/OasisLMF/ODS_OpenExposureData/tree/develop/OpenExposureData).
+
+Those csv files use the minimal quoting rule
+(only quote those fields which contain special characters such as delimiter, quotechar or any of the characters in lineterminator).
+
+Examples of how to code multiple financial structures in the input files are also covered within these documents.
+
+The web (HTML) version of the OED documentation can be viewed here [https://oasislmf.github.io/sections/OED.html](https://oasislmf.github.io/sections/OED.html)
 
  
 

utils/switch-quoting.py

@@ -0,0 +1,13 @@
+#!/usr/bin/env python
+
+import pandas as pd
+import csv
+from pathlib import Path
+
+# add quoting switch => csv.QUOTE_MINIMAL // csv.QUOTE_ALL
+oed_root_path = Path(Path(__file__).parent.parent, "OpenExposureData")
+oed_files = list(oed_root_path.glob('*.csv'))
+
+for filename in oed_files:
+    df = pd.read_csv(filename, dtype=str, na_values=[], keep_default_na=False)
+    df.to_csv(filename, index=False, quoting=csv.QUOTE_MINIMAL)