Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update parser_and_mv to remove A1 reading and use schemas.yaml #222

Merged
merged 5 commits into from
Jan 15, 2025
Merged

Update parser_and_mv to remove A1 reading and use schemas.yaml #222

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f6672d3
feature/parser: optimize csv_parser and fix generateXml and tests
issam71100 Jan 8, 2025
5b8fd32
⚙️ Auto-génération des classes et des specs
issam71100 Jan 9, 2025
8ce7ee3
⚙️ Auto-génération des librairies
issam71100 Jan 10, 2025
d00e2af
merge auto/auto_model_tracker
bou3108 Jan 15, 2025
935c39a
⚙️ Auto-génération des librairies
bou3108 Jan 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions .github/workflows/generate-model.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,11 @@ jobs:
# We specifically only remove FOLDERS with the exception of a couple manually created ones
run: find . -mindepth 1 -maxdepth 1 -type d ! -name 'builders' ! -name 'config' ! -name 'custom' ! -name 'edxl' ! -name 'exception' ! -name 'report' -exec rm -r {} +

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Je passe peut-être à côté de l'enjeu : pourquoi a-t-on remonté ce bloc ?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Car le schema.yaml est généré dans la fonction output_schemas_yaml, et il doit avoir été généré avant pour être utilisé dans la fonction parser_and_mv du workflow.py

- name: Run csv_parser to generate schemas.yaml
working-directory: ./csv_parser
if: steps.filter.outputs.parsing_required == 'true'
run: python workflow.py --stage output_schemas_yaml

- name: Run csv_parser and collect OpenAPI & JSON Schemas
working-directory: ./csv_parser
if: steps.filter.outputs.parsing_required == 'true'
Expand All @@ -74,11 +79,6 @@ jobs:
rm -r ./out/test-cases || true
python workflow.py --stage test_case_parser

- name: Run csv_parser to generate schemas.yaml
working-directory: ./csv_parser
if: steps.filter.outputs.parsing_required == 'true'
run: python workflow.py --stage output_schemas_yaml

- name: Collect schemas.yaml and copy it to json_schema2xsd
working-directory: ./csv_parser
if: steps.filter.outputs.parsing_required == 'true'
Expand Down
10 changes: 5 additions & 5 deletions automatic-schema-generator/templates/ContentMessage.java.tmpl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,18 +9,18 @@ import com.fasterxml.jackson.annotation.JsonTypeInfo;

{{ range (datasource "config").schemas }}
{{- if eq .automaticGeneration "Y" -}}
import com.hubsante.model.{{ .package }}.{{ .rootElement | title }};
import com.hubsante.model.{{ .package }}.{{ .rootElement | title }}Wrapper;
{{ end }}{{ end -}}
import com.hubsante.model.{{ .package }}.{{ .rootElement | title }};{{ if ne .rootElement "distributionElement" }}
import com.hubsante.model.{{ .package }}.{{ .rootElement | title }}Wrapper;{{ end }}{{ end }}
{{ end -}}
Comment on lines +12 to +14
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK.

Je me demande si à l'avenir on ne pourrait pas avoir d'autres messages qui n'étendraient pas DistributionElement.
Pour rappel, la propriété/balise "message" contient systématiquement un message fonctionnel, qui est étendu quasi systématiquement par les champs du DistributionElement => d'où la mécanique d'héritage, avec un UseCaseWrapper qui étend DistributionElement en y ajoutant le UseCase (CreateCaseWrapper = DistributionElement + CreateCase, EmsiWrapper = DistributionElement + Emsi, etc.).

On se pose quand même la question de ne pas porter systématiquement les champs du DistributionElement, qui est assez lourd et redondant avec des infos déjà présentes au niveau de l'enveloppe, pour les messages qui n'ont pas vocation à être interforces.

Si on va dans cette direction, on aura peut-être intérêt à terme à remplacer la mécanique "ne rootElement" par un attribut custom qu'on pourrait avoir également dans le tableur Excel.

Mais ça me paraît OK pour le moment !

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cf https://app.asana.com/0/1208789821730581/1209151708472294

import com.hubsante.model.report.ErrorWrapper;
import com.hubsante.model.custom.CustomMessage;
import java.util.Map;
import java.util.stream.Collectors;
import java.util.stream.Stream;

@JsonTypeInfo(use = JsonTypeInfo.Id.DEDUCTION)
@JsonSubTypes({ {{ range $i, $e := (datasource "config").schemas }}{{if eq .automaticGeneration "Y"}}
@JsonSubTypes.Type({{ .rootElement | title }}Wrapper.class),{{end}}{{end}}
@JsonSubTypes({ {{ range $i, $e := (datasource "config").schemas }}{{if eq .automaticGeneration "Y"}}{{ if ne .rootElement "distributionElement" }}
@JsonSubTypes.Type({{ .rootElement | title }}Wrapper.class),{{ end }}{{end}}{{end}}
@JsonSubTypes.Type(ErrorWrapper.class),
@JsonSubTypes.Type(CustomMessage.class)
})
Expand Down
6 changes: 3 additions & 3 deletions csv_parser/csv_parser.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,8 +116,8 @@ def get_nomenclature(elem):
def is_custom_content():
return MODEL_TYPE == "customContent"

def is_allowing_additional_properties():
return is_custom_content() or MODEL_TYPE == "DistributionElement"
def is_allowing_additional_properties(name):
return is_custom_content() or MODEL_TYPE == "DistributionElement" or name == "RC-DE"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

par curiosité, pourquoi ajouter la condition sur le name ="RC-DE" ?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#222 (comment)


Path('out/' + name).mkdir(parents=True, exist_ok=True)
Comment on lines 118 to 122
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on utilise les deux, ou c'est qu'on a abandonné l'utilisation deMODEL_TYPE pour ne plus utiliser que "name" ?

Copy link
Author

@issam71100 issam71100 Jan 14, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

J'avais vu avec Saveliy qu'il fallait accepter les données supplémentaires spécialement pour le RC-DE, c'est pourquoi j'ai ajouté une condition sur le name. Quand au MODEL_TYPE je n'y ai pas spécialement touché.


Expand Down Expand Up @@ -481,7 +481,7 @@ def build_example(elem):
'required': [],
'properties': {},
'definitions': {},
'additionalProperties': is_allowing_additional_properties()
'additionalProperties': is_allowing_additional_properties(name)
}

def has_format_details(elem, details):
Expand Down
42 changes: 42 additions & 0 deletions csv_parser/json_schema2xsd/src/main/resources/schemas.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,210 +2,252 @@ schemas:
- automaticGeneration: Y
customExtendClass: ContentMessage
customExtendPackage: com.hubsante.model.edxl
file: model.xlsx
header: N
package: rcde
perimeter: 15-15
rootElement: distributionElement
schema: RC-DE
sheet: RC-DE
subschema: N
xmlns: cisu:3.0
- automaticGeneration: Y
customExtendClass: null
customExtendPackage: null
file: model.xlsx
header: Y
package: reference
perimeter: 15-15
rootElement: reference
schema: RC-REF
sheet: RC-REF
subschema: N
xmlns: cisu:3.0
- automaticGeneration: Y
customExtendClass: null
customExtendPackage: null
file: model.xlsx
header: Y
package: cisu
perimeter: 15-18
rootElement: createCase
schema: RC-EDA
sheet: RC-EDA
subschema: N
xmlns: cisu:3.0
- automaticGeneration: Y
customExtendClass: null
customExtendPackage: null
file: model.xlsx
header: Y
package: health
perimeter: 15-15
rootElement: createCaseHealth
schema: RS-EDA
sheet: RC-EDA
subschema: N
xmlns: cisu:3.0:createCaseHealth
- automaticGeneration: Y
customExtendClass: null
customExtendPackage: null
file: model.xlsx
header: Y
package: health
perimeter: 15-MAJ
rootElement: createCaseHealthUpdate
schema: RS-EDA-MAJ
sheet: RC-EDA
subschema: Y
xmlns: cisu:3.0:createCaseHealthUpdate
- automaticGeneration: Y
customExtendClass: null
customExtendPackage: null
file: model.xlsx
header: Y
package: emsi
perimeter: 15-15
rootElement: emsi
schema: EMSI
sheet: EMSI
subschema: N
xmlns: cisu:3.0:emsi
- automaticGeneration: Y
customExtendClass: null
customExtendPackage: null
file: model.xlsx
header: Y
package: resources.info
perimeter: 15-15
rootElement: resourcesInfo
schema: RS-RI
sheet: RS-RI
subschema: N
xmlns: cisu:3.0:resourcesinfo
- automaticGeneration: Y
customExtendClass: null
customExtendPackage: null
file: model.xlsx
header: Y
package: resources.info
perimeter: 15-SMUR
rootElement: resourcesEngagement
schema: RS-ER
sheet: RS-RI
subschema: Y
xmlns: cisu:3.0:resourcesengagement
- automaticGeneration: Y
customExtendClass: null
customExtendPackage: null
file: model.xlsx
header: Y
package: resources.status
perimeter: 15-15
rootElement: resourcesStatus
schema: RS-SR
sheet: RS-SR
subschema: N
xmlns: cisu:3.0:resourcesstatus
- automaticGeneration: Y
customExtendClass: null
customExtendPackage: null
file: model.xlsx
header: Y
package: resources.request
perimeter: 15-15
rootElement: resourcesRequest
schema: RS-DR
sheet: RS-DR
subschema: N
xmlns: cisu:3.0:resourcesrequest
- automaticGeneration: Y
customExtendClass: null
customExtendPackage: null
file: model.xlsx
header: Y
package: resources.response
perimeter: 15-15
rootElement: resourcesResponse
schema: RS-RR
sheet: RS-RR
subschema: N
xmlns: cisu:3.0:resourcesresponse
- automaticGeneration: Y
customExtendClass: null
customExtendPackage: null
file: model.xlsx
header: Y
package: rpis
perimeter: 15-RPIS
rootElement: rpis
schema: RS-RPIS
sheet: RS-RPIS
subschema: N
xmlns: cisu:3.0:rpis
- automaticGeneration: Y
customExtendClass: null
customExtendPackage: null
file: model.xlsx
header: Y
package: interventionreport
perimeter: 15-SMUR
rootElement: interventionReport
schema: RS-BPV
sheet: RS-BPV
subschema: N
xmlns: cisu:3.0:interventionreport
- automaticGeneration: Y
customExtendClass: null
customExtendPackage: null
file: model.xlsx
header: Y
package: documentlink
perimeter: 15-SMUR
rootElement: documentLink
schema: RS-URL
sheet: RS-URL
subschema: N
xmlns: cisu:3.0:documentlink
- automaticGeneration: Y
customExtendClass: null
customExtendPackage: null
file: model.xlsx
header: Y
package: geolocation
perimeter: 15-15
rootElement: geoPositionsUpdate
schema: GEO-POS
sheet: GEO-POS
subschema: N
xmlns: cisu:3.0:geopositionsupdate
- automaticGeneration: Y
customExtendClass: null
customExtendPackage: null
file: model.xlsx
header: Y
package: geolocation
perimeter: 15-15
rootElement: geoResourcesRequest
schema: GEO-REQ
sheet: GEO-REQ
subschema: N
xmlns: cisu:3.0:georesourcesrequest
- automaticGeneration: Y
customExtendClass: null
customExtendPackage: null
file: model.xlsx
header: Y
package: geolocation
perimeter: 15-15
rootElement: geoResourcesDetails
schema: GEO-RES
sheet: GEO-RES
subschema: N
xmlns: cisu:3.0:georesourcesdetails
- automaticGeneration: N
customExtendClass: null
customExtendPackage: null
file: model.xlsx
header: N
package: report
perimeter: 15-15
rootElement: error
schema: RS-ERROR
sheet: RS-ERROR
subschema: N
xmlns: cisu:3.0
- automaticGeneration: N
customExtendClass: null
customExtendPackage: null
file: model.xlsx
header: N
package: customContent
perimeter: 15-15
rootElement: customContent
schema: customContent
sheet: customContent
subschema: N
xmlns: cisu:3.0
- automaticGeneration: Y
customExtendClass: null
customExtendPackage: null
file: model-technical.xlsx
header: Y
package: technical
perimeter: TECHNICAL
rootElement: technical
schema: TECHNICAL
sheet: TECHNICAL
subschema: N
xmlns: cisu:3.0:technical
- automaticGeneration: Y
customExtendClass: ContentMessage
customExtendPackage: com.hubsante.model.edxl
file: model-technical.xlsx
header: N
package: technical.noreq
perimeter: TECHNICAL_NOREQ
rootElement: technicalNoreq
schema: TECHNICAL_NOREQ
sheet: TECHNICAL
subschema: N
xmlns: cisu:3.0:technicalNoreq
10 changes: 5 additions & 5 deletions csv_parser/out/RC-DE/RC-DE.uml_diagram
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,18 +9,18 @@ strict digraph {
</TR>
<TR><TD BORDER="0"><I>objet sender</I></TD></TR><TR><TD BORDER="0" >name <I>string</I> : [1..1] </TD></TR><TR><TD BORDER="0" >URI <I>string</I> : [1..1] </TD></TR>
</TABLE>>]
sender -> DistributionElement [headlabel=1 taillabel=1]
sender -> distributionElement [headlabel=1 taillabel=1]
recipient [label=<<TABLE>
<TR>
<TD ><B>recipient</B></TD>
</TR>
<TR><TD BORDER="0"><I>objet recipient</I></TD></TR><TR><TD BORDER="0" >name <I>string</I> : [1..1] </TD></TR><TR><TD BORDER="0" >URI <I>string</I> : [1..1] </TD></TR>
</TABLE>>]
recipient -> DistributionElement [headlabel=1 taillabel="1..*"]
DistributionElement [label=<<TABLE>
recipient -> distributionElement [headlabel=1 taillabel="1..*"]
distributionElement [label=<<TABLE>
<TR>
<TD ><B>DistributionElement</B></TD>
<TD ><B>distributionElement</B></TD>
</TR>
<TR><TD BORDER="0"><I>objet DistributionElement</I></TD></TR><TR><TD BORDER="0" >messageId <I>string</I> : [1..1] </TD></TR><TR><TD BORDER="0" >sentAt <I>date-time</I> : [1..1] </TD></TR><TR><TD BORDER="0" >kind <I>string</I> : [1..1] </TD></TR><TR><TD BORDER="0" >status <I>string</I> : [1..1] </TD></TR>
<TR><TD BORDER="0"><I>objet distributionElement</I></TD></TR><TR><TD BORDER="0" >messageId <I>string</I> : [1..1] </TD></TR><TR><TD BORDER="0" >sentAt <I>date-time</I> : [1..1] </TD></TR><TR><TD BORDER="0" >kind <I>string</I> : [1..1] </TD></TR><TR><TD BORDER="0" >status <I>string</I> : [1..1] </TD></TR>
</TABLE>>]
}
Loading