Basic UI

.gitignore

@@ -4,3 +4,4 @@ __pycache__/
 *.sqlite3
 venv/
 .idea
+/djang/.vscode

README.md

@@ -1,17 +1,34 @@
-# Open Pension Next Generation
+This is the xls ingester project, it reads data from excele files in a given directory into an sql database.<br>
+קבצי המקור הם דוחות "הנכס הבודד" רבעוניים מהגופים הפנסוניים
+# The database structure is as follows:
+importer_kupot - רשימת כל החברות והמסלולים <br>
+importer_reports - linked to kupot - contains the report date and file name <br>
+importer_asset_details - linked to reports - contains the details of assets and values. <br>
+![image](https://github.com/hasadna/open-pension-ng/blob/importer/importer-erd1.png)
+
+
 
 ## Setup
 
 ```
 make init
 ```
+# build the database
+```
+make makemigrations
+```
 
 ## Running
 
 ```
 make serve
 ```
 
+```
+cd djang
+../venv/bin/python3 manage.py  import_from_folder path= <path to directory where excel files are>
+```
+
 
 ## Docker Compose development
 
@@ -43,3 +60,15 @@ Start the Q Cluster:
 ```
 docker-compose up -d --build qcluster
 ```
+## Open issues
+
+Some of the xlsx files do not open, an exception is thrown.
+The list of problematic files is in the database in importer_filesnotingested table.
+The error is "Failed to read workbook
+<class 'openpyxl.styles.named_styles._NamedCellStyle'>.name should be <class 'str'> but value is <class 'NoneType'>"
+<br>
+Another exception is trown with some files, seems to be caused by formula fields.
+Error is: "תעודות התחייבות ממשלתיות-R25
++++Code 300. The number of operands is more than available in stack for function "+". Formula: C13+C15++C16+C17+C18+C19+C20+C21".
+...
+השלב הבא מבחינתי הוא פיתוח ממשק משתמש לניהול הנתונים - זה לא הממשק העקרי לשימוש של המידע אלא ממשק ניהולי לבדיקת המידע.

djang/djang/settings.py

@@ -54,6 +54,7 @@
     "django.contrib.sessions",
     "django.contrib.messages",
     "django.contrib.staticfiles",
+    "importer.apps.ImporterConfig",
 ]
 
 MIDDLEWARE = [

djang/djang/urls.py

@@ -16,7 +16,13 @@
 """
 from django.contrib import admin
 from django.urls import path
+from importer import views
 
 urlpatterns = [
     path("admin/", admin.site.urls),
+    path("", views.companies, name="חברות"),
+    path("<str:company_name>/", views.kupot, name="קופות"),
+    path("duchot/<int:kupa_id>/<str:kupa>", views.duchot, name="דוחות"),
+    path("tabs/<int:report_id>/<str:report_date>", views.tabs, name="טאבים"),
+    path("details/<int:report_id>/<str:tab>", views.details, name="")
 ]

djang/importer/admin.py

@@ -0,0 +1,11 @@
+from django.contrib import admin
+
+# Register your models here.
+
+from .models import Kupot
+from .models import Reports
+from .models import AssetDetails
+
+admin.site.register(Kupot)
+admin.site.register(Reports)
+admin.site.register(AssetDetails)

djang/importer/apps.py

@@ -0,0 +1,6 @@
+from django.apps import AppConfig
+
+
+class ImporterConfig(AppConfig):
+    default_auto_field = "django.db.models.BigAutoField"
+    name = "importer"