Merge pull request #2 from hasadna/importer

Importer

README.md

@@ -1,17 +1,34 @@
-# Open Pension Next Generation
+This is the xls ingester project, it reads data from excele files in a given directory into an sql database.<br>
+קבצי המקור הם דוחות "הנכס הבודד" רבעוניים מהגופים הפנסוניים
+# The database structure is as follows:
+importer_kupot - רשימת כל החברות והמסלולים <br>
+importer_reports - linked to kupot - contains the report date and file name <br>
+importer_asset_details - linked to reports - contains the details of assets and values. <br>
+![image](https://github.com/hasadna/open-pension-ng/blob/importer/importer-erd1.png)
+
+
 
 ## Setup
 
 ```
 make init
 ```
+# build the database
+```
+make makemigrations
+```
 
 ## Running
 
 ```
 make serve
 ```
 
+```
+cd djang
+../venv/bin/python3 manage.py  import_from_folder path= <path to directory where excel files are>
+```
+
 
 ## Docker Compose development
 
@@ -43,3 +60,15 @@ Start the Q Cluster:
 ```
 docker-compose up -d --build qcluster
 ```
+## Open issues
+
+Some of the xlsx files do not open, an exception is thrown.
+The list of problematic files is in the database in importer_filesnotingested table.
+The error is "Failed to read workbook
+<class 'openpyxl.styles.named_styles._NamedCellStyle'>.name should be <class 'str'> but value is <class 'NoneType'>"
+<br>
+Another exception is trown with some files, seems to be caused by formula fields.
+Error is: "תעודות התחייבות ממשלתיות-R25
++++Code 300. The number of operands is more than available in stack for function "+". Formula: C13+C15++C16+C17+C18+C19+C20+C21".
+...
+השלב הבא מבחינתי הוא פיתוח ממשק משתמש לניהול הנתונים - זה לא הממשק העקרי לשימוש של המידע אלא ממשק ניהולי לבדיקת המידע.

requirements.txt

@@ -1,6 +1,9 @@
-asgiref==3.6.0
-Django==4.2
-sqlparse==0.4.4
-gunicorn==20.1.0
-uvicorn==0.20.0
-psycopg2-binary==2.9.5
+asgiref
+Django
+sqlparse
+gunicorn
+uvicorn
+psycopg2-binary
+openpyxl
+pycel
+