From ae9d5b95552ed50dd7cea4bcf9cb38baf45e610e Mon Sep 17 00:00:00 2001 From: Christian Adell Date: Fri, 24 Sep 2021 16:59:56 +0200 Subject: [PATCH 01/17] Extend Readme --- README.md | 173 +++++++++++++++++++---------------- docs/images/new_workflow.png | Bin 0 -> 50380 bytes 2 files changed, 92 insertions(+), 81 deletions(-) create mode 100644 docs/images/new_workflow.png diff --git a/README.md b/README.md index 57ce40b3..46c262fe 100644 --- a/README.md +++ b/README.md @@ -19,15 +19,32 @@ during a NANOG meeting that aimed to promote the usage of the iCalendar format. proposed iCalendar format, the parser is straight-forward and there is no need to define custom logic, but this library enables supporting other providers that are not using this proposed practice, getting the same outcome. -You can leverage on this library in your automation framework to process circuit maintenance notifications, and use the standarised output to handle your received circuit maintenance notifications in a simple way. +You can leverage on this library in your automation framework to process circuit maintenance notifications, and use the standarised [`Maintenace`](https://github.com/networktocode/circuit-maintenance-parser/blob/develop/circuit_maintenance_parser/output.py) to handle your received circuit maintenance notifications in a simple way. Every `maintenace` object contains, at least, the following attributes: + +- **provider**: identifies the provider of the service that is the subject of the maintenance notification. +- **account**: identifies an account associated with the service that is the subject of the maintenance notification. +- **maintenance_id**: contains text that uniquely identifies the maintenance that is the subject of the notification. +- **circuits**: list of circuits affected by the maintenance notification and their specific impact. +- **status**: defines the overall status or confirmation for the maintenance. +- **start**: timestamp that defines the start date of the maintenance in GMT. +- **end**: timestamp that defines the end date of the maintenance in GMT. +- **stamp**: timestamp that defines the update date of the maintenance in GMT. +- **organizer**: defines the contact information included in the original notification. + +> Please, refer to the [BCOP](https://github.com/jda/maintnote-std/blob/master/standard.md) to more details about these attributes. ## Workflow 1. We instantiate a `Provider`, directly or via the `init_provider` method, that depending on the selected type will return the corresponding instance. -2. Each `Provider` have already defined multiple `Processors` that will be used to get the `Maintenances` when the `Provider.get_maintenances(data)` method is called. -3. Each `Processor` class can have a pre defined logic to combine the data extracted from the notifications and create the final `Maintenance` object, and receives a `List` of multiple `Parsers` that will be to `parse` each type of data. -4. Each `Parser` class supports one or more data types and implements the `Parser.parse()` method used to retrieve a `Dict` with the relevant key/values. -5. When calling the `Provider.get_maintenances(data)`, the `data` argument is an instance of `NotificationData` (which is just a collection of multiple `DataParts`, each one with a `type` and a `content`) that will be used by the corresponding `Parser` when the `Processor` will try to match them. +2. Get an instance of the `NotificationData` class that groups together `DataParts` which contain some content and a specific type (that will match a specific `Parser`), and adding some factory methods to initialize it from a single content (as before) or directly from a raw email content or `email.message.EmailMessage` instance. +3. Each `Provider` have already defined multiple `Processors` that will be used to get the `Maintenances` when the `Provider.get_maintenances(data)` method is called. +4. Each `Processor` class can have a custom logic to combine the data extracted from the notifications and create the final `Maintenance` object, and receives a `List` of multiple `Parsers` that will be to `parse` each type of data. +5. Each `Parser` class supports one or more data types and implements the `Parser.parse()` method used to retrieve a `Dict` with the relevant key/values. +6. When calling the `Provider.get_maintenances(data)`, the `data` argument is an instance of `NotificationData` that will be used by the corresponding `Parser` when the `Processor` will try to match them. + +

+ +

By default, there is a `GenericProvider` that support a `SimpleProcessor` using the standard `ICal` `Parser`, being the easiest path to start using the library in case the provider uses the reference iCalendar standard. @@ -65,15 +82,39 @@ By default, there is a `GenericProvider` that support a `SimpleProcessor` using The library is available as a Python package in pypi and can be installed with pip: `pip install circuit-maintenance-parser` -## Usage +## How to use it? + +The library requires two things: + +- The `notificationdata`: this is the data that the library will check to extract the maintenance notifications. It can be simple (only one data type and content) or from an email, with multiple data parts. +- The `provider` type: used to select the proper `Provider` which contains the `processor` logic to take the proper `Parsers` and use the data that they extract. By default, the `GenericProvider`(used when no other provider type is defined) will support parsing of `iCalendar` notifications using the recommended format. + +### Python Library + +First step is to define the `Provider` that we will use to parse the notifications. As commented, there is a `GenericProvider` that implements the gold standard format and can be reused for any notification matching the expectations. + +```python +from circuit_maintenance_parser import init_provider -> Please, refer to the [BCOP](https://github.com/jda/maintnote-std/blob/master/standard.md) to understand the meaning -> of the output attributes. +generic_provider = init_provider() + +type(generic_provider) + +``` + +However, usually some `Providers` don't fully implement the standard and maybe some information is missing, for example the `organizer` email or maybe a custom logic to combine information is required, so we allow custom `Providers`: + +```python +ntt_provider = init_provider("ntt") + +type(ntt_provider) + +``` -## Python Library +Once we have the `Provider` ready, we need to initialize the data to process, we call it `NotificationData` and can be initialized from a simple content and type or from more complex structures, such as an email. ```python -from circuit_maintenance_parser import init_provider, NotificationData +from circuit_maintenance_parser import NotificationData raw_data = b"""BEGIN:VCALENDAR VERSION:2.0 @@ -97,93 +138,63 @@ END:VEVENT END:VCALENDAR """ -ntt_provider = init_provider("ntt") - data_to_process = NotificationData.init_from_raw("ical", raw_data) -maintenances = ntt_provider.get_maintenances(data_to_process) +type(data_to_process) + +``` + +Finally, with we retrieve the maintenances (it is a `List` because a notification can contain multiple maintenances) from the data calling the `get_maintenances` method from the `Provider` instance: + +```python +maintenances = generic_provider.get_maintenances(data_to_process) print(maintenances[0].to_json()) { - "account": "137.035999173", - "circuits": [ - { - "circuit_id": "acme-widgets-as-a-service", - "impact": "NO-IMPACT" - }, - { - "circuit_id": "acme-widgets-as-a-service-2", - "impact": "OUTAGE" - } - ], - "end": 1444471200, - "maintenance_id": "WorkOrder-31415", - "organizer": "mailto:noone@example.com", - "provider": "example.com", - "sequence": 1, - "stamp": 1444435800, - "start": 1444464000, - "status": "TENTATIVE", - "summary": "Maint Note Example", - "uid": "42" +"account": "137.035999173", +"circuits": [ +{ +"circuit_id": "acme-widgets-as-a-service", +"impact": "NO-IMPACT" +}, +{ +"circuit_id": "acme-widgets-as-a-service-2", +"impact": "OUTAGE" +} +], +"end": 1444471200, +"maintenance_id": "WorkOrder-31415", +"organizer": "mailto:noone@example.com", +"provider": "example.com", +"sequence": 1, +"stamp": 1444435800, +"start": 1444464000, +"status": "TENTATIVE", +"summary": "Maint Note Example", +"uid": "42" } ``` -## CLI +Notice that, either with the `GenericProvider` or `NTT` provider, we get the same result from the same data, because they are using exactly the same `Processor` and `Parser`. The only difference is that `NTT` notifications come without `organizer` and `provider` in the notification, and this info is fulfilled with some default values for the `Provider`, but in this case the original notification contains all the necessary information, so the defaults are not used. -```bash -$ circuit-maintenance-parser --data-file tests/unit/data/ical/ical1 --data-type ical -Circuit Maintenance Notification #0 -{ - "account": "137.035999173", - "circuits": [ - { - "circuit_id": "acme-widgets-as-a-service", - "impact": "NO-IMPACT" - } - ], - "end": 1444471200, - "maintenance_id": "WorkOrder-31415", - "organizer": "mailto:noone@example.com", - "provider": "example.com", - "sequence": 1, - "stamp": 1444435800, - "start": 1444464000, - "status": "TENTATIVE", - "summary": "Maint Note Example", - "uid": "42" -} +```python +ntt_maintenances = ntt_provider.get_maintenances(data_to_process) +assert maintenances_ntt == maintenances ``` -```bash -$ circuit-maintenance-parser --data-file tests/unit/data/zayo/zayo1.html --data-type html --provider-type zayo -Circuit Maintenance Notification #0 -{ - "account": "clientX", - "circuits": [ - { - "circuit_id": "/OGYX/000000/ /ZYO /", - "impact": "OUTAGE" - } - ], - "end": 1601035200, - "maintenance_id": "TTN-00000000", - "organizer": "mr@zayo.com", - "provider": "zayo", - "sequence": 1, - "stamp": 1599436800, - "start": 1601017200, - "status": "CONFIRMED", - "summary": "Zayo will implement planned maintenance to troubleshoot and restore degraded span", - "uid": "0" -} -``` +### CLI + +There is also a `cli` entrypoint `circuit-maintenance-parser` which offers easy access to the library using few arguments: + +- `data-file`: file storing the notification. +- `data-type`: `ical`, `html` or `email`, depending on the data type. +- `provider-type`: to choose the right `Provider`. If empty, the `GenericProvider` is used. ```bash circuit-maintenance-parser --data-file "/tmp/___ZAYO TTN-00000000 Planned MAINTENANCE NOTIFICATION___.eml" --data-type email --provider-type zayo Circuit Maintenance Notification #0 { - "account": "Linode", + "account": "my_account", "circuits": [ { "circuit_id": "/OGYX/000000/ /ZYO /", diff --git a/docs/images/new_workflow.png b/docs/images/new_workflow.png new file mode 100644 index 0000000000000000000000000000000000000000..14aba5ed4c20f642966f4de18dd1fb032c58cf54 GIT binary patch literal 50380 zcmdSBcT|*1x9};50THChIdqewO%M=4Bs4h-NS3Gql5-GHfhN=B93+WkkSsyTL68iR zv*eu9RG)M1d+*$D=HBnlnm=aN>b1_1eyX0Ty?0gZ{oA`bP(?`^_a4Q)8#iv?%F0Np z-ME1Xzj5Ox_1)XxFBZ?4aA3rf;ITHCzKHp}y~2{KHZXUSV+{d{{r zrFs15-SN9);ez9L#rf-9)7uoOGy4TjTlH-rnVzhS7){7TL4ZWpSdQ#ykPoE7(TGOt~xFEDfsT%eA~U9>bWP&c-MZpC(vT}i_v^bU`H$`B?aGW zA8bOV*~uD*2-ak;^CQ;&fgD9jMivIePzrvPDr^0(mZMz#8D(?%dNuYPiTv#DM;n3L zjo?+@dyBGG)8ELIpX_3-jTVQmqWpNB+b_x-tdA-O7PG47sI&-vA}=NQTfIQg-;i$Cm&bOC9gs=;ZcdA6sgM3h z|C19}Mq^XHa1OlnyJsQi*(TtY1WxnRSyqIhp++_nL@Bo)t-aq?@RG32|SNk&q2-t&3IDddy4aq3PuxUiVr12OnB1Prn zxHQzCmsz(m`%7}jrarwqAk_K#8pYx%+0rpT34tLP2;sTPX_}c2c`hA+BWBE?hhR9&k)K4wJ%!520XpqmxrWV)eCt1_Anf$#`7&@}$-;wc~ z6IJ=QzCqoUk_9um8ZM6}b{E2*+COGhdA{!xLpyUk6?-t9YDy%ux01G z{OM}?tRdvAQk1e6YYp6Id^=xPQz;qV9 z>Jkc#)7cLXeq~79sralpQqI7@K<9C^At32@mwHvuD?Iw}w`H->c$pc7RLF*06 zS>EBpy$-%u4!l?h`M9Ff&g`>P!b~mQgAo3*Bl!{M&f3y-KfiG(;@ypo*LhzPp1AF~ z`onPo3pqqLKJ0D2{cgXql@P$#GZZ>+wy~1C(KA8BA^s z+Pu!w%8&_P9mwWQB%Wc$kiasPp6W^99YzK!6vwitif;}P+Id~}WAUW0NfRQQ8XQ+6 zb@myBuT2@$atyrGv&-lV5cG|w%aj$b4RWN-3P-^PkvAYu>9JATK4gd0fQk=)diA1^ zG>ZDYV$!45B%6l3-%HI8o`xK4PMD097+K(I_kWU7|5^+rX5o8Le>|O^^U91l{Q@7g z0#-I6U$gMJh)$(dUPGLdKU7u^w&CB@;jYPdu-=XdqT;Io+i##&Ad~`nUz9zJdZ%elNx$LvgPA6e>7DQ zD)*{I=4m7(65D-s**0M8lFI7-nxC9$==S9;2BPxa3i);Y{0=pa}i8$eW-RtEx%fiAwP+ zb5=yW@%xZvSYZ*f;MsN$-}@_l#aMwMk_;k= zVt8KBkb5(};r7T(s{5=kQm7`dSKW#nd6G_-^%2EDSQ--bhEWJ6JMyj*HdqBE11k(q zOo&@{FG@Pc)+`!YWrL2Jb&+ zUszZBvFtf~3fMiC6JAAg?$uLKuo{aYl);Xk-y&aX+&c8f|6+m}?4*)-e$l80ZjY?S zB3Kwq*W?hEk)k~Bp+6)gLx!FqUcV=XmoXtm%Jr*lpHbVhic5`jA>Trc!K$-KltK8I zDkds$2A9-cjhC7PJnjyBfF0pekHuoVvD7yp+l0ab(;N!hzRyr&`W)*+L4RZgkE~Qk zfiNnMgnT$cdL*V}-Z9}(}AUV?Sx0*?U>4;hqSN?tCX+ARov!u;z@MOj9gG(RqZ_CeIpS0TSicTK3uCV}Jr}Cpll4 zis2e;G`NF;5?IUB2Ob^r7Xr~s+{xQ}(} z#InE06UT)^6Xm>RWd3uJ;Uy);sl*l`ZrLx*mcRl=$alN?7jgZNQwi=muSL|;Bc3k$ zb?rqIKNk<T%cRBo8CqVccfH-ujvO(gba4fv-N=;%Ke4Z#v92VL08*+ECg%mI>)}4EZXRkcblG z-;*IP(_B7lErXgbDliD0iH!_pMHgS#;`=&Y!nV7#`Sv=(acAIn2#(7}%zl4;vnHH| zg!D7OB+71_oSEET{Cum1O>(4PM&uOiaDpFv@+Cti=~4zXl@ewe#$)BZb@|OaayTAF z^FCaRwGi4CUv4l%YF8R}P1ZOr+dqxGn2#d#ToOK2*o&Ss*KYZfq_h`EmiOx6X&1MO zca?tkpRkQYd?P6Jw+pt|JocHQ(DkQr`Pdgz?wl+3GSpIfoyA%Z3mz-(Pi=;$Wn+=y zF4!lMA<3xu1L=3^9j{g8z`l5RLIYtf>y;wm=njv(L*q)9fP`Scf}_y`t`0yCxUn#K z4`;_*m+KPf=9VAyktEfVtCIzbBK_W$ofH`TYL)`IPOOr?Ge<|UFa)3H6;~aVY4GQ# z5JDEiap$$|O3QILJycpUGGwizSn<@4Whwih#_Gc%&3T_{M`YdbRw*s-gv6w$*g zbN#pBOZFQB2-TRhFY+n;-s(9HHsI>CFhXwW&9Q9G++xUmH`9!4E0LJO;;XJWRa#Mo zmmhr;MmuEub-8t-A2C~ccnb7UmZoe2|8h*CNdww8qp5Uy)0-EvV3VIy0KrDIbbJjj zphivMmgzs2^eON_wSHV5Dcl!MA(Cn7sG+#Bt|OU)u3MWje^j{aaHp!VLF6Q;o#`Pm z(Pje^uosN8LJ{vmjaB@I{Bj%j%!Ju!3GT-F+;vY#j2?@Q@=7_va!9Hukce94)&9oq z60-}uC|gkstw1twU7@APfTXuI2S6SOZcMN$W7H)hF8U3tZ8P>hCwtQeICi>`@%(hS z6oT7B41gJ9>Y~-4boVrqPMCQyE<^97fcKTB(-uN5t$TL|C?dAD`;^q)H_ZwXa|qu{ z+xW4k)vzxS$L~mhF@DLksa^!LohRIQI|4n)F6{JDKyNC^%0Mg#TmRY^-uS{q-$};Y zxS*E=zR*w4u6RvI!y6}xmPRu}mODm%W$7wkdTniwSdItMA4apn6H+4$lc2q(F0oJM zrYotDl#QcFJ|TE=o}Iq7Wee>^nfRu@Mafjj18;}EPnvTEY;p0Fao-Z)n_9dhXu!-Bg z!14J4yCZp{!(>$TF3*oY+m9^B-J%bz%8Gu@yd7!q+qMsRUm#^U^ch9&8h-~nLJSL> z{*w4yem$cS@ak}=3*n6CnGImPDJmOTe2J}ou0%vb3Mqw8L!B-xbFYGnEC?A zYQ}o2Y?Ym9Wpddk!j__yAQ<-JnO3bE!})QK$48Dkj_n~Pz%Qc{Sm=^MU0H|5_66Wk zf3a?i9dyoCC&DOq+czKjta@_*#U%1Uyi%;2bJ$*B5E#v`Iv1Ovd@W_LT&rGnCy+F7 zX4*4HZak6$1H4LfdW<}HQ<3>g zx6HFbkbOWIBzi+49Ml%a%U?ZM#A8gj8Q1D*KmXaWqiWJ#o@X1l0NZNP3B5Jrh5h#L zOZ1fKB*ykU_kRkkU8V@T7=tsjbC2X1!-S~Wb7>l(0(lw;{K~p7*25r@>-oX40r)&V)Ssf=)iDbnJ^yx1xl`?pDW!g zo%5~1^f3OxT;;~^9>B$Fk$5cTW@_3xQ`ip`HGM9HumY?3E_A$xJ`);BDd4Zik^q9L zBKz*I$wX{@B}UCdDGHHbyC^RvgUJpKJH- zbSaJv(-Y@(@g4@`!S|r};)b)X?ZSR+Z0rb+gvB3i%&D&8g~ceTfW3~0D&*&^@Q{&? zx`uNdQ;XH&#w#^-H85u-He*TTsYf`}A6K6lb5BC7Xml3%o?>XjEmP4P3G2YFgwmM$ z7WXW48yjj@QvwGOMde+EzX*i7|4q0je#p{z(#BW8|=}0|gs4HmV6xMYx)l zmRf1YRQx5`3SRrOBH-;-s(5dA7C}TA(s3|?$n>Ac3d=N6VWv_Bq73bLvSF4oi_U)> zR|oPA*`SY2jcoMvl65E@E@8)ChQ4U9+dY=$Lhid*ehtYw*q$*|)b50euezGa<-O`l zMUyO>4&9O2j?gh^`XcHLJNYvdwGvvZtlTrPs^n8BLRs}%GvVw!xPy2=O@z-OSlZ}F9|>I*V`}MGd0Sf^5U9jTXiU(D37ux z4;|v1P{MmT@>3TJvjS$*pI(6$7%^P4E;XfixJ5fP3~MaR@qq0u^xvSJikx$UzwNFW zfeh-#1mq?-(&<=%0D|XzQ;Mb1wL?ZggRjoqp)ibQFs%D998L`3nP1g#Vr+)<;a1L( z`*#5VWB^}r+`9vsiItNTVwX_A*)*V##`Fh!EE$4C2h+JB!_I__U{>b72>>8XwCEQY ze}J|@e{mozF|DL^>f+i^K1P^CaDS9vtWS!~>&f^aEJ1>1GQbySWkPIv0|DK? zVG&9(g~*XCphrOCEUL)tOLSMIr9&f0y$-oY7LZmx+iOvC5HHu`|!x}q9q{8KYrQMi7U5V!S0hi^5v~kF}TT>rkVAA=T{-v+G&J85K=5QUtm%;-oa4?t{|}9>VbzYr|7w zq3tmtPr$zRE_By`g#Q9vv-djN*)euC1B5X%Azkwk8QZ50K8+@$+@A>fcA1Ox;TA|B3q z)wUQ~MS7~&z0s`dALU+EGNj)DiV?YxVV9v(1CxbHku+Hr*ZTuZH#?9mADBJ)?tJsl zNFn9z2Q*aQBNOFs{e`ThOp^jI8S?=J`r@ocsL(w^qp=JqlaUHVM~(12zut6lFO8#;{ZJaE8r@<2(s$` zbSpgzS*o|G`H4Ddh3XB0#$d}{(C~A$r-}yFm7-`;eGFzI9S{<)v6<6%h^Mw zWGDg~bb@{7L0XHl$AT&!8@q8nF^&*7)BVsJ6RNK#lcR$+J7lL{>Q9O{X+mJs+kxeA zs`Wpntmj(YZ_|}HPSKe5Ia2zC_@RC{o6=ha2N?^O-Q^3m6Tz5}47u;sr(N@)l)t;0 z28t`A^m6$V=jZF!MoOI8_+(bnBx03$cPn8@P314NASH?|3W<$`HTLGxfj?Yo?~$=n z#y4$wdC!j-GXJXW74bD(rZj@Q6uyU(mPuGnE%aNYAX(z>7?ZKP@{3>uznlYZhy>>M z9*d62tAbWtc$U5tZ@HNPD?YGX0gkb-gF{YE-f*9{(44^z;y;tsk#4aMnWZ7F-fb2k zt2(W!P6MSpR??5%54|6iQm<|r>X3>8=CMQ06F_tqSTz#|*rMmGv{D_~qH|55Gn`!|y#O zTkRqpDFU?1eF?t5ozk?n&qc$zUbNVpVQ1bknJM;oX#3|RecDA{>~yWOca$PrNV}zk z+K=|8mGm61DV{r6uCm`7y=p7IgCzua++coJe+@DU#@%fd)hi7=ay!xW!lWZs*VIA| zQ!yuQwT`QoIkZpLs4pr7lw#((SIoETIPZ_2>MX#SgUKp0cT9XXSv>uFp4kA)!;tJL zctg0m(%+1{_If%}aDvCU%^7;!rXba(kg#jq%nUcac;$Myv^-W~%@`EN;@3Gynr?Z{ zw6u&NF0o3CMoc5lX)JHk479cw48I3vaiPR;(MrzxzF!TlEUX#-h%Vd-lYp z;q~)(={Fubd*3{c;R=}BK6utY&lq!KH!!{`z&k!Z{P|Lqp>sY(p+PAT@6%I!b#u-i zQ;geEerj(08`)u>cOAVHD1?bX2I+f<8#@(c9$ML4bPjh&c(X9;51*2 z2!87xu||cEI|B_rd(4mEx3%f=eOD*0^RE@ipAxWFZAk<6UE)2%?OJ9mTW~2v;>&=dbd7nt08e0acucY8!XZJq#gnAd}+Z>Nw(vYF+M&i` zy@vapbwf8F%vVk%yR8cd8(Kal5C|j#Q-gnZHccj1cMS#`G&YpUBTRVbb+Nz zOWYBH;^m&Jm_4a*;+cyQ;#{Q^vHY>$)tev_()y!xoitRrN{Vt%^%+5(a~79Q*d8pKJFMvR1zHeNiXZ6)0*hJ8VqXIh|j-ARFhQK35>F zph2YOa6X=GhHuXoW-Qq4wHOK{6YekPZ@j}Y-?+w`OP z;>T5%YZ;Ap1PzdY-hwHl*_r6weX#udB82{+gJH?*R8cPikd~8=T>zfP&bhA_vP(-W zG%;DkK_|2=OEIa}wiD`q_4wWW{#!UO#l)}vgSrX8d*J_?!eK1V*yMch)ZR)u0|6Oy z^8*sqdh)h31(-BOQ6{p~KUrALNZ39g?S=d~(Ck>d!FnWmr!WX2vlN`rxHo~)U(_zr z1z%OSs&nNtgE3MeASSr4>Oixy;3wTqz>SVia+T|KMpdvYO<^o9(wVGShPr^waM|;) zi6PaBd=d&)=^KCpW<5e!{u#uyEd&KtYbO!mJq*Uhly3P4vnBLx>?V~s2 ztOuvLQFM2@_`SEVD>iGaPUZYXyjZT%e?#^6S651`hmRxNFRS-U<~$jm{V;6b*%5Jz z%#Pt4<^0|7K6j1(Ui^y7Ot1Q}@DjI*;B4JoLsPGS-uv&_E?OK97OJe^lR z*q9Vgv90qQvlY zMwL)*%D>S^8_U&{E>Jgjj@!T)Yx-}s03P&pT#K0P=iSk~$>9tq>nCYzc-e0Z-vxYf ziEE*~bG(qxos%aVTSRKdO1yJm@*rJeR8&qy>da_`^39LypEG$-1Xi711e=x#I{90@ zbL{5iez>=+hyK?d4wgi(5TR4Ap z-!nY9sYWu@`Uo`XmQoUwc%m6H*O%dxj1*{a`*ddoSIuE=Ck$R3A*34mQA5}@N_S}mZn93fcJP``Qc%+ z>dE675s`f-;~Yu`5;F3oS9PbkyM3!t18Db&DZU{#9C^dEuKh;)p_Tu==se-W`Zm6%v^S( zp#WZ$iZgZB0xJlAN;IRTGrz*+a=I##>p79Vl#e5_BK1wg{M%uuT#lkUEv>RAZPY8S zmenX`g50Qv6RWbvO-0ga&sa#*?@dhzG?#}aX1-l9Si`T!^b5||PVaxg=l#gMFaACC zoxZ$YJ%n))DcoLc8mCXhx3DqQXHpr~d}4!}_ zo^EycwHsgSaq4G{(P%OA70Y{VJ_&;QEmkuw_wuZ3q^M6=s*1U_ACg+WJG)8P{w)|K zarQ}VchN+)!KF!1nzj)a)@uJRxTr~0Y?R@8${g&GB9h?O} z%@o_d`t04UCIgE#4auG_v?Q4t3%SpeIQ95$*Vldf=&xv~=JibBxeovPbG38p*VDDU zJZ8NF5mm!z=_@8L)r9&yf-X;JJTkJ@HOB?=goP z4NIhAN+LV;=f+^l?as7kiWp15Tv;O-NTl}RrLooc*GIbuOLCs4o_}1PsP2@Ddg#{wlzgW+;5i$6&VAIn388l;Qve%^iV#m{CW*`kofgE#K5 zEI#9cVBT-wz&~=)t#}H7{g5o;|E!rA*415wulCN4slHdNf!%qu(BgpcRgthd5u7LH z&^+8(u7yhI;tm4u9bwt*IP-6v%60b-$eM8v^N){&4S88^_cpIW5k)f*tZE57U6jNr zuR-2>=BUl+q*|HK`$g6ejU=+;k8I-CH^Z3DZ)TE?sG2w53@`myb~ax-B1LL1 zzu2lq?(>B!?$d`q*5yOXA@Sc_BIlV_ z3S0+k=Ufzg)_Kc=bzcKD8e4kPp2mSK$?ek~EPIt9A<9OxkwP@F6yGn1L^{}?|5t&A zbj3(35CVIwSMaF8n`~!Os~1%~c8Vbd-Kbv8k*UcTX1PM;JcE3iRGX2<=}454-bi)w zQ*Q%BiO#L8;)aPV5@_&C!d+OVRGqVx#GrflTQ6+dWDk$FJRo`c4p{uB3l_$$Ij+Qe-mReluPy|W6 zx3KpkCgp2Hs6rRgEb;5=L#qvDBDileh|+x)uwOG9f2_6%&PI}r8;f|Y3Ep=tEMIHM zf_c7LWOzb4Auhz6J&`Nb65SIgYRA`7#gx+{LA>ieIubF3x7Sfan#ufA4V^(4)=2vt z`s#}aQbD`y)AcMo5W0L)nrWe0x-k|Z-RqHXGVYnx7Xg=o!1gbBQXD}lsBAj>O*fuJ zz0VU%P0s+OLW(cfeW?q0PtIHwD-dEinI!Kn{eELWodD|yo;i-(^1R3Yo)!3WuQ*cF zr9y1nhh}7YKK6eY=G@}(V+e0NqWe34P0-$Ot3kFe5and4|C>)O*nbJds*_+$3~^XK zV0HA&dp5thGtk@Sd2nm%8*exPTz+Ir{QHmA&}UUUZ!2LqM&&G81h(}^?ATgIbLns@ z@rFO|E#f2wn6q^qY+{Q!QtOvuhdO&hYsSH0@BZiMwVsS#=6ttqrFQ#ugW;4#imzGg^vEA7VRtgOePR{2u3r3cZN|Yj_K%d>6tst~XEy3>*T4VEEyFH4>YTY(B|>$! z;G@ItAYD`!X~yw(-}d4e@iqvRcneefj5GWBtfUcvyI#eRoe<=1qRD1u9Pn^ye#y4H zkm%9RE}^dJHY_Oo8>wyu5}(hD=*uoE$sG9Q+0m}2a^}8$-F)^5Jw`qeSfUl16 zEl{eAI^k}Vwwu4tCKMsl9dYvSG)Y_jgfh-EBp}sORepjes}9$1-}ITET8)sOX=XtE zuQ00?%H2=nVbHMnTd!rBAW(!vkw82{KyPP;E;?%`&J5z>$HF-b7}S6TzgX1xpCCuu zPzS15Ni1PQ`G@8BKhacq>t&)e=51_1e(6e73}3BKvI*tDe6IlpGUfxo6a3wu?B$B( zO-GVEY9RVeJ-H8Pjz$%TjbpeIIDUG&FUTg+=PA%-2=4zjef@`8(~*JL@4E`m9CZ*G zAsr7+AAq2Ho_FzYU^Bn~6jl`jSO|cw{zL#fN4B!<7+%JMpvuTJgYo4GB|rmE-z+rw zdjQB^=NT4_^D9Mk3x8hu{v^<1`YPLr;bk-$6cga=dO0`Hzc=9_kP!#qcMc7&|M$S} z6`H<2`funf_esl!=Z9CV0FIgXa&@l*f1VVyE` z*g!x20GxvvTOl!=8y<_IEY_ z7v9u6dixKDnlKu))NyI};<)rdHE3ANWM7JKKnOW6)0HqiJw~Qs>`ws7qE(9YgwXZRpzJkB3^$$;yBeJn>cEjb6pEn^G#~#SzNAoe zP0X745WpX;K?OY=!e=`@I_yBYIyF&k5k7ffObV@T`~C4Fg}V8yOhtZy4;{^mseQ9D z;b;xS0S$D3XT%Ytdg3$19pt>`>2?ckfOz9k&Okx70oXnMe3wLByLSEz|ZeYPj>piy}YlNOD7%?>cx0&ID9i#5{W!;_z_m zp>y#}K<0V)LRusDfIJs0;*l4{q{a?jG>aiwKio}E)5GgaxAUU4VHtJJgm*!`2$mHo zoZZ}}C1d7F(l?%EF{sOzZ`GzM081V>p{cczd+voNCSFS`%myW_n*CjZex>ze1{yGj zCPk^0QVVrb76j|w9b+@jS`dk{r8(AW*nR!3>V~OE6yaC{8(xBgX?jHI!-r#G93S7R zssfq~QY%HwFO!g{jBWn?%JczVGyMms+YHPlXX}ctS=QwLH`JRWv6dpC+)T2;xJLh0 zNDQvRY`I~yb5Hu6|{UF%sd5UAf{Rvcvhd3UZLzSb?@Nucs zrCKzj(tWa2=3cg6(Pl&hAF-{>C#r0lTpa&ehXY1_>&`5NI1kuX7B@Kaje` zu9E=!x89BJ{)?aosJ6Ul9xqZG3)_J9*-wf4K$n?PAA4U>4P+?>>#EPJk$6AO`3vIT zl2YhOF4n96Bvof7T$V#!N_32Ne-ebR3$(YNS z-e&9DpW#ENMo~ehGor10{12hsRdZp5tAN$Ue%i;x~eA)7xBVgqFs}QBops` zzyIME1a-*i@xQFGYE`>WHc6TvSJ1x7o4cEK**NEW-bL0cO>hxoOJxQE(*$+%OxOOp z-E-PIyGx^nv*mrcQ*`rcZ0U&l_Wryn_W#VZJfbxFO!m9dY`ZzDW?VL93$1!_D za~tu*4;seP;~8(l3Cr5AtC8PxkflsVFYaLr`l2DrMvH1kD(9UoE|L$=yk_d3wOkyf z8TP+=X|pkQ$s0t(@xGz{&iwZ~UGt};>n4{uPDoZwOm${Y|K8%t_H&8$%o$-Xx=fPe zXcoT*{@%!E5_dl7w$mV44G3nq^nEfi?<^c7@VeG6^SaibC&@%H;l^H-kVT26`golm zXV|w&_t&jcR(qaCp7Eb~Ts#}{{PgJY*R7AGgwiJSe^{%}wlzYM1aA*E#{0|1ac54e z_stJ;71L{cdH#AwT%YgF*!?KcQ+!fxgO=_>;f#I?gj=7IjW6tXb(R;^ckH&`2GukJk2+IIUkSzAuQ#Qi)S|Mr2~9UPLr_7w;7m)j#Z1wFPxC<1E?S_&pg5Q=V_ zze`AW0~TUzDR^G8l$rJh9qH==?sTm3)fUcYGm6zW{1#~*YS2=;;GJ_a9W`u`NYK3D zsra4jAz^NC{8Z9Ab?9gMKZ3lBRWgwX6wzt1 zi1WQw?6}`n%4B?!yV3gSbx_=dBVx{DKMyil$@x3eY*4d<=!cA@*d4!9%n8-_8wq+I zn7@(nX8T(6g&=dYdB*=@XNkbkqcAv8iB0Rsh|gw%tWMzS3xH>GPo5Kjq(*==yQX5Y zQdhE&ym4;4fD2KOD1Kp%&z`hYz7q@7I7?#k+gq5qa#HRdihjRY8ZYB)fqxjO6lzfs zVGG5`Y!7cLs-}+LANOSV8?E*3^Q)es*#7kA#dTSZ#0|nNCT%&coeRIT7vvd!skm8I z{bKFB`TcN5q>y|N)Ld&)X~-QU4Mvt1^P&X3mZJ&OrNe2!ZKhJ`{J9vsssBLMknaB{ zver`Wv6AZgX)TIuf9|!yrIr1>qk8($PUpwf&uXji=lT0AIs!Jqk1N(0E431Cr}Dgw zq?kjkM3Jlia#5jWG&XRr5wc(G2t2-xr#0gRcgZ18pZRH+C)smoxr#|xb%a|9vve!x zJBIv{Ye^RIxE((7zXTA%5gnIDIX=hy_Qx?2 zdFnJ{@a=l~PT*4xHH!yONz((#94lRZn_#vIw)tEqyS14H6xnHigphh{i*wE9O5tZp zUERR=FBRVQ%5c%Q37&X^94Q&8!n#=(72C0wj6E^JhAl-~+G_vq0`t$Dtm%J4$2v z`*myL(w`%pS4gs%vwaf&%k_yAhCx|F?u&ao6swkTzz>j$GvH>2s8{W(vs}+M{~5?g zV!Jw69TdD8g||%ViwBaC&vEFL$o#M94L51Dxs8+ZSB3_&L!51`=>y8s>Jkwx%pFbDeH=k`@*k7-%JY=mx z9ZJck9_(*(luF*;-8*_e$anvjRLhCw1Md1?6reyy^X&_S5OYXk0ZAt_G5l3uipJ(% zW+`p{p;?NDv&yH*e`fxVF7y(1(!hZ8z($^s}n-S;}2!Bb=}1!uziBDDH(!) z{I-+`0@E!pqxkc)%1~>NeE@>zWr;;0N|~PwQ(fv6p#j4}aHw(6WZKie z8J*Q?z_0zE*8ecN1%$$L9OpDkO;T)T!l9L)Cb};t7vW@+-YhiUUnwj`yfL+TUbX4k z+*cxk?^%{;ufM1;k)RRUHd{MUMu~d)M@#i+&A_yJ^*}NgwyvM`gxoIj)o^Xh>ti|g z!)u|kC9&bI@O*wu9_quNpH40A1Pp1Dz|TCmS9^0+VDV6<=k{#_LK-b6IntM*Egf7U z+Hw3A9WTsE0J9sc+UuYo4K$g-oXJ`N?3SP0PVZn!{IE!&zl$gz8PEHwFa2#?933b; z-c@_V`PpvGv>l&_>Y3^88JqK<6q{uTIejq;I()` zqRmIS0SkVm4zlDMlVs!=k5Ae$DsD?JUz>S_g|u}PiwcPx%s;(mC{o>xfq`=Dq`Zxy z4lga5e+b@%+s5ZF!HcZHq2Ti5mz+Vp#e<5*>4O)>X>WB7> z^FR$tCy&%$;wmuiv`@nX9rma!DnFZAH1upBK&W_zxCM}6L$=v6x~7UML^$N1X!K{f zzmU-TrnO>$IaI9~K6O+(0M_Z5W& zMSz+n=1juWWOnv}@3a8{3WSSm=V$|jiad}Vg(-^$4Kx5W zL>^*~3bq@`N&-xW#f#IQ46w^GUXB~XOEk2)kBM{W1nOENO&$}R(1P9&|8K%9Je(l# zBLRQ^F2Mf-utjnUP}Bdq7FR|Npv7H)79$FjXN_D&us{n-!E&N{G>ifDZqk?tG*4wI z#h(G36)MOC1696&!z`jQY27-@E?c zcJ{+)1=j#*8y2$X$3A%L6Ix6xfK36cdO?K$KdH&}4Vcm<1T6~ZdejckRlHT+Z>X`! z7y(bsY!UZNOR+)D#)n`0UZet^sGA|5k)Q;F1S0tX`PD4*QYvimVh+B z69DKHOElv^1)f(0&xhd%Y9c|Iv>iZdR4W9UEH5w(&NH|8JwAlqY>);W>4T1@59`-C zkN@uI>_#>Q=m^3LNTpp12dH@cb|gJzd4yoIj^_r2`J$ksgWc>Ej@`eXukp$G4xazJ zJi84`iGH@?>6$_LD63tYI}n0k9N2ChyqXhl(f$2J9PVE~5P=B_RkR(^mqRP1G~)^;Q2}Po##u@gOFQy9?&S5Tl!+&vYl=6HBFO}mbPqewmh;}5Tp_)rT?-G~8xhY@NYH!GX?avT zZN!HzSVuFY%9pP)GVsAKQrwtdSb)D7$fAs_dNYU!Oa|Q5FR}tQ==#$@`mt{e6x{y; z=$8!5=H%tX5X_(c#b#?mf1FqyCDw;HgX9CjLpd;8aQ4}-`$i*}9jW|wpMmcxZdlL*GLuH4ZU93DnTKeynW_}Af&Re1 z;cUvt54{ItmPfy7UKXHlWmuwXwAwTZK8O3*I2yNwkiVX+F0IUbXiyP{QbOOS_?P_= zVUVZ93pp@>yXF-%U^Hk`y!S}c9|)inZHns@nj7X=rhGZzwjPpC^$H)<7e6BEjk-lFw}j|?e=STsPF`Wc z3HDhTIw3Gu#K$*S=3EoOh6~Db4aTa--Y8u1fhlX9>7EV>%oT3EAK#mqcL%3ki9)-A z#w=P3SmU#SIwQpHqPeqA8-oA=K8C~mhS~FCTY=rz;xAWgMLJSfAKfEYypZLU5t9ZQ zeC$4KoMd|MO>dVMvQWEiK2lpR3%f_}>#ajt&Yrw%nGfbyKpgaFE61iJWMq8!=1OU? zD}kEv-i++SM95hEwldZEI+o68jWlofYdj!ZJn(YIolid>9i*0!gjz%|Q;;QH;s?%~ zh4%biblfUq+V|uy>HJ;IZDLIS>&@alM{B*nhkT4XS0Te>gB3wJBmEq}A_UNv%gT?b z)Leqd0tBJr!|BLC{1@i(m<;%jz=|n>@Z1a{s|R-&5H{BXj|psff!G8*`skff0fz zx59^#wFl&XzR{tRP#A&5K&xYb%eIkhxD&5ALewig&AyH0@PePO@xMGj`XoE`g^bS_ ziY`NlgF#Jkr4&9k@#umiRtZQjehiHu2X#iz3pH`x3Lh&uKF`&}9lj1fvtW#LLNc2iwCn;)CsBh_Lr`khoM(W?vJ8HNrE0{L>3~0_#)?c z&%iH@?Cw`8lJU%HY+S$PX45DP8@+=kJz^zfdv+ijLjD*ZXT>YbYTV&dJLUFl>6-HC za z6TY*t|FhKMl;CSnVd&e&l;=MXsds`*n{_4H7eFapiP@kIqtc>w)tU#D=z(x=ipU4n z1!?1@kxQ8vHsYi0^IOWA9Dx;2+f62s&&C?eM5d|t;Xx$a86(Tl)yEj$1k$yF(sJeF zrcr3|$yAl7fic}+F($ihLc7l^sE7Q-&r!l6@t(=WIU)DH&6 zKXLSIsSDs=gbA1)exD%ms7 zFRc2xV`}XGVDGJ?s_wS7;hT_9IyXo+0>Y*{q*Fra+#sDIA&oR_N>J$*m2ObFL0VEk zN>Vzc<6WEE^PF?O@A=~!-*}%foJr)pK6!nsZh8@lEp<2Msh|h5&kD zVMxpSD9dhnp61RkPy5QeyZPZ4o+V06<`1?Sg0P-SvrYc4c-! zM?eOAZeBe&WB`X%5I8r*H)&5|QqUXb! zJJ(kyCJl4de5-!f5nhko$)YoOmABQ%XK$XWq;qKkkMgJV%qsF{@&&M~X=2Z8zv_;S zLVY*#kHtT(#XW2x8Xbb;JVm77W>^->oW>qmjRcwv@c?xOKiDPI z5xay^%mxAGIumStdKqls5PUGp)WAs!7Hr)|V(s)lwtIy+~1eTGpM=7#Exw8ZtnB#GV^sgp6f8^Do*|4dU zXB3+B+U@MfJfsd%SMv9AEm|oopYQ_nIfoWM$;nc$@Rjx}&{h`~6HaZML?h*%y$`^=LpO>T z4LL9x++%Yr3i)p*wgq4LhCTjy8Pcau-ssZX?YZ7dT3y8fNIdF-k4c`eyh+tk(DzD` ziw`gc4L&+xE-nv{1Wpvr+tYmalwgQ)2mc8G`bPUH4N?&J1`GM;CgQJnw4l}a$bI&| z!Mw;Vb+RD?y;hIVq!4T(6!1f^1+xel5ghC<5Ht-E7a#>Ppat%`AvC*iM1&{AQG5#g+a867hfggabqYeD(fSX);2k>$H3H}7a-i9eZ!N9I|7K}DhlueQVlIkr$LO(E zdGds4qSPeQo;Ng?3&)kH5V(gg2!T$|a23MF36WGnuwgJm7!dnLT)<1P#`?{B(|DPj zudLzlwnR=7V^tm(4tQ702OA(o3?ev#^-0-tHM+PLy!mdM6Sk2Nc4leIv=bp<`PzZ( z4XZBG_qmbgt@{D_XSmMAJfp8rvYdBl89g_wzR#?G#YqeME(eYvBg0$g8)|-dR`f8NIRKy>N`c?ZTxv zA$hdB$>GD#@7 zlG?%VQ`P>3!P0xwkW7o1C&9`lah47{8*<% zCxZUXI!gu1vhKS#&LJjPZ+*A^Uh#g8%bF5Q-(1AZ|4O-21Sgc~ z2Z0ZRmY&^ZLh+mzJE5_|7I)Z9!3#BEM`Id}_*mgfSFxvkmHgd-y(IG=(2r8eMD{Y)IP7!cQ~T^-j$ke;70AQ&Xg{O-6_i<>kEPFVhYMJW;@)*hf84`QbQxxR7i6eK3d^rsE2{sbh7 z2LU(N7o0n+CKQOmPG`8j<_DY^+rXR_eB4^ z+aJxQQ)&vR3a2k7m;qNqB!$m5W;OaoM3vinZtZkEAwuxx=@L7%E-oDXaBrR%oSLpYhHrI4P-17Z@Aeu~v8h z(Nh7^bLYV%OYBl7Y`9#Nw8*fYLQ6ayJ|5-Y(R6k3db~0oCBIFkgLnHR zF@;}A_+W{UvneSjJVIs@hOp)SR%?L8kRYR?Y5v?|C_;N+CQG_dZL02B5CQr>h)HuF zTXkz)I93OAvtEZwzTEnxbn5cmvda5CF;(dS>X)kB{HBYoLmL+TItY^*Wh{lz^QFeX z_VG27{`81_>gdj+UXz7ITU?lFrj%#N&Y__1Igv%Q8i-S(b$NLr-JJV!rfaHkU&o>+ zPGjZiQg#kx_C{>EiRW>JSLikcAvYd=<@G7deyU8W*!PoBy?f+m5Lk^6oGVO;J?vnQ zYf#Ao(Ia(1Fh6w+;MCgI@y=1MMUl?3e|W#HbnL3&%S+d_&f?m+{A&}L{;2KL2a7aA zeLrTFz=hc2fHBHgv+uni#MSUn?Z?<`r)WuVBdliWK0ZynpNVp#c$^%L*X#PNn{e-V z_-6I@RPOk2?c`rX@P~;@Yn%Kf2-#MDMV|Zb(f9!l&0*W93AVQFH?F z`{9nKs?R4gz-*iQta9R4eP>^jY{u-%5x^Ko^xrdkv9ULgp8~?P{5&=3#(N_`t(GYi zBI>cy`F1;%3$~@c+{9Lz;;yuD$b_AmEgNPzRqltYIsaOpZ{qbzV~C*J zVVw_B6G|dpUZ1UvfKaE-s#qc*RtFLfSg=Wo2`}$3EXd(kx}M?%wnN8DUjm3b-{;t8069Ls8&RmJ|ag_Wz0?>1ekQMe>`;2=pPuENd{~&f%Q*Z&-bBIuUKFQQ{=5C?_F& zvK4R!+&DKizvU}NvHr|oyw=2}S?nWHDxN zz23ZOyNti@3V3}FjlKlGx^q{GJatzi)1)bHqK(1#()VG`de2CEPbMe%#<%=?G*?#@3z zy$`^Ka!HZrEyV=~8BTuq$Zc4TG{k;pLp9PM4vs33q<;X0{$b$$-z=D?de=}?b4@Bw(Z7E6t=CXPl)p(^%310NY@u?F zCxZmo4}T!PIgS@2W8*z(9P|goecMJDDf-v{ywc436v7(?l$ZY!7{LOb=%0C|G?X|# ztGSz8`-fR8)4jZ(v0s%0zWp1yKXK1SFn^PS59^wr!)v9iEv}BAy7Z*cqZIIZD$o~N z-vuY^DB>jhA)PYuIJp0x9N|_z?uSQ}MS(&jnE@y+`#lEBYIh-|=W!tTbu9v~tw%R9 z^iS5GZuFy$*^!9xzixtPD3g?=22B(HlCOd%a!-Dq{TKZ$?NNnjg)R-K4l&dep05Jp zB?Sn>C%^I~P#%J-1DZfdRQOT(CjfW4bEt=2W}v1iv?yey>iL(xv11Zh1s1s^av%m| z!<-d{JhFTe!)VGdJiyy#x02Eu^Ae#WOVS|Ao$E#>N%EdGnqxvRU+_fl>3_DYFh^jK z40z!I@#1gknDA8nc6E{EkbkJqH)xk6ZzUlB7fe(-)4W)Ke8?mqE_r}tqZt3G_CKBc z|6xi^xYFf=fXBM!Fj<<=%egzK0A_gw+zAYfwdk??`@cfn2l8y3jNLm+8nr855MacB zX$bOkVWcfMA99r9n_|5j__;dM2jR|SbsY-}h(DIX&yAh8iI7Ew^KwNMbifSd0-(#> zVdY=m5^#MJ2l1xdulrFU`k^bN(2f;}ej9|H(wr>gq68nB3QsBD?`LlL-(4xkM~&`(Z~^{jTAke(F66mq z$~{B^BEF;JP$(Shy?9Y2KKFq!pU@O5IcCh2j5}0x?KwFG8UVI#!UUduB?7U~?f`D1 zLdVPh1`QxXd(Jfe<}Y&vogsZIQu7|+33f$XV9DL=b_2_X&S|az1j^y?ltH}YDzHcGSp;T?l1G1ZWWV&0=X7XvSbXZJWiEAay7v|?*!lZg}v z3NQJyD_ZDou1>!J4BNbipkJS~4T6ZkHOK%Y_mliunw89BMaR$oHkXU(PU>WZ*}p9g z1hPf#4?c>a6oG9`l^fnelw=7?;vSnO>41EjsOIfwK&FjK3UUc6_r4OR_~B1+bLl&h z_lxU(Ax7RmIW%%FRHzCgRkd-ieJ-9HyD-Fy{+ z4Atl%euHB3ZN3B(XE(`Ze^*)XpXcEdo4vUAT|A-eos{uy5NZ6@U!PRAV`v|@PK@cc z)5%?*rYQ=>Ua-x3@;2wgs|4&W3Ir_%<6F6D+acdL^B~gaXJfWeMKQF6+vYFhX-yS6 zD5iYZr&C@UsBa`!#DBtet&aW&ONmarnn6nePCGM8G4hH`8ozThqQ5wj_HoNmczmc# zf323bQQsfs0bg_udiXD{GdXgcz_SLP9Zpxn;o24FuwKnNRRH;h2Eh)b#phWW5Spa= zbVxJVVhs`G7HG7*wyc7j+RGHDPJ28AZIZP&SOwJ%+i8_Oh2~*%k;7Cv;G5z<^|#+X zjZoD>zW0G${=%#rzuFOa)|g_+5|tMx^Mu-WmdY+_cf^roSqIaZS~G;w$?bkD&p8A~ zKURox0F>yVl+}mY9O~!%NE3q?Ks%$a;`N`591QXgIKrXpikrKxZWCsM_1qQF4dnx~ zwP9t zRRKO1WiMy)Wf%}hz}hddSe#;ETHh~6v{~OIFmL{1Jb{|`c=q#zLZrKLnO@#ygR#sC zb2qwTSaZO>ik)&9du8(>u80x-7G{TDo=ywSq9)DduA&QBDY_#s>ueb0WU$P8bOX3U z9l?bxaPEpLC*OBqCLGr95Iwf$13Z3Lx*PY>D$N@SvoVXU*+vv|=9n(P!F4wIbTM-M zKF0H-iqKZ%Oav5ooS>Y%?b4a=Z(S;F>OPI_T8jG2#iKKtYBm-sg#F~D{(0FP!YTuY}@phUR+q|Um3^A|J}9`evU z9BRQ1bM%>sgxsx-QQ4(H){jb3BBqj^c}uysJ6e5zfDmj73?_u<2+?`-@gQm>J93%=>Sw$aABKUB5$bj43K<52&!{JIOrBM*GNi3?oFFlI`YABE zX>{*&pPt>Ij9kl7wQZ`GeqVOuV12CQo)R%LG+9uy9YW&Gj$~5hnA){nerCYq?~th9z}Bt zHXNe*87$&9hqx~LcSjJ)51qv>-}guIJ-ruhuJCS0b@!C{``*7}MhDD5a(;n?mTDbLvvb9R#)8)H;SQR98Hm>o#+O<2{m?!L; zWPfuaP&5OPI9d#Ipf33|YWi!6?6g6K?5z01 zJ<=ibrJhe;D;Ae0F7H^}Ao_Ie3Yojlma04Muq!Tsx|+9?4rw$J;tesXcTtVRFF!NV z)rGVf2h)C5KEQSK-CqhRO2)s_YBQP_b@X!mhdM?!ePjS3lZth$-jAB>aE%P|nCzZ@ z!BD}_M8aV|tj-Dfxl(H6w}9NO7$hzVM$vb5GC>1`A3A+hXWehmY*bHK{E@!jXXf2a)92*Ybw%pMGFgHJZ5RPP3HS-&_VB9?R|7pDOHq}LjU-Fg<(nU6fmhw<2qFvIV-V-l4XcX5M47u4=AZ2(=(L#ga)5 z2lT*!RLVZ&J+326agH@9A0}F7pGg}A57o#6>_|HT{O&lWBtE`2XdbJh7Wt?bRx&}( zogmF4Va0GrtHLG0-OtX1J5-JeYpY+$$xv`8_IJ*-omn1c;>fKUZtxLuEq)XX#S*lh zb4@nyITd>0jygysJs6XDZ2p)=j!rI#xb%)1B}PZIa_#=5jqaA%h#BtYNpZ_$=Hz6y zRk;VN#_QmoI5Ccccl{fCItn8%`=CS@qQaHvF|-crheCPSR{Bt;Ku8qCJhZUij?p>s zcmISoZ41=mjf%)awHVs&@@GR`%vlzGmzIO8i{t{#du+*cB#dKd ze1P{3!d?ZFf75aD9U8bl&_P1PL3R#gD4}<9BM#qMe$S_!vO)bOU zK|ekYFjw%+-x?7`P3*^wgd8rX$6L!t_M(pVFs48uNHMfQjPM;>CCT1Pa6!(?$7_j( zti+T=w)70^7vTa#Gn3Vu7x%W@F431(4CMIg+^xLNBP_8Hl9^)W)~68Mio~mm9i1E) z@EX%H)RQde9Edb#^_6Ns`nFu?<2kYjIn+cIu4uy?vLO5g2)AzAdP)A@BB^Rgn_JRk zS-eW7HSWq12O|Bass9<(|4iWjp|#4+414h8V*q&8&N>B;+!ZVEV5QIe;&?k|aCdnjlj_>>N>Tad>_?k))Ghj~WimSvs*EZHlm!-G-y{b>T_ z1p;D4GC)lf1^B7`$nx43#w?4hLbBlTt%BYd-&>9W2eMK|IpBL+AGrPi+syDdR-3K+ zxJ7pps1*RpNQ^_ekq`skv(Jv0EM)7pXl8u%wiV_k2z1pnD&VyhyDXd|<t zn3g%--tXVCwT?d)e*Gd}+yf6T#3pTG!F~Si*6yZz) zAQr3m4@O~xp8ii|irNy{^)#C-H7iM0uYh2aixGK|m05QNYb!q#A;UhP(u7cD))FFp;vaix5fr4ODVRWK`>lqa0Fufrj7f=D;f^6EEHX+G5*v3vxQ~``)s7k6!I)6pD-a#Sm>R?*H z%KhU-?70Qwb;=ixM${WTmapbttBx?q7L`$8PDMSwOWG;nPZ{D-nIh=E2;Ne434@9u z+Jy=FD4n70fg{AIqyHrN6?Xk~q{3v`wbX)|jV&Ssdm$FAw=>34#N z)02em*kP+-mNQ|SzNXPS*9*(waJ^=)2p_TEKTd-74p#1cZ#vL%V}T`NvD5%6RvE%u zO8JAy#X|H$7@E&K=1Nl$Q)|dvQ-r@7m2}vypG>J=J5f}UxRC+<8$10YN>RDvlj2E% zgBv}<+^f1gS+La15lmFeIjqfUAm~MDLJZoL^iTbjvRw;|)qxZu8%E2HtT|775|VAD zeyh)#W*r?uAR1{?|Jv-vPv@gOKi~oy<0uT+q*;7GnBoY7*^KWp{Xys_>g-PX?8cQS zA~lY2rBQNfDQuOqpWVdt)(C!VGSK=JOkScDH#8?~cNQ}43rDg+NirF*GJ%@l2!`4{ z7beYzff%3+VKi5g(gc=ACB}jCMVNwK))L5S^Bv!0T^B{nkyklq;OMwJxsEF^5X?lY zimuCPLZ-;534_q;;wN}c*;=oeAd=`NjZQ8)uuB~5KI~FpW)j#;JdOi1Q)8QuUZ(po zh*7ifA)}7u`JHIH#P;aT)}FW#Y{&1j@d75T82E%uk5%d|`IdExoaP&1w6P18+CFP> zORcc^h6G0tn0-xqf3BGH=yBa+_#;z$&<%KOK>qT!?$kE~TovKv%-Z|aHneKrp7o$Y zKieCXnl!e($&$=d&cswq_&jA>axugTdyz>etbm*kxVYv8FMdftDx)A*RxRaJnjL~~ zIlaN04Oa=E3#EUg1xij?nwSNzB&_z>#G$eS4~wpE8i`}C$!;5 z{RdU6z6>^fLE2WS)6FT`&)xS*5fRD+=|J~YfueI69j0H=%fbbp=~LW`V>H0Ie7R?c zy1i{MnJXPNeK5b~*bvEoZ#RQKUC`yV^^C*Jfk-IsI$1B8*2PL}>%p`HKv zEyR_@Ap7n)Hf31iC&T@jSKsSi{v>}@YS@-RX`qp%jt|dR3L7i2SxHvrVUBKDMDbpx z;@Pf04z!Bkt|eT+A&%F;qm32sI>u^f-#%nDmQfAlXLB^y63$<;RA;m;xAuK z&-1p|iupa7_1rIWYqw=j;<@90<3{wRaI&k#go!>``FU8h3~*1&e%1|62@SBVE|v`^ z2IT6%I!ZnZxvoiTpQNpSv-_*9EhE<=5WYTb_m>>b2EAUY$rL{MFpI$8~Huyq%B$PO(ZcTwE@R>-6! z6z5DL9-jVCSbw-ssWAgtJ2kT@7ikm(;N zH!!n!GOk>~^+uzUjP9#|$sWS_X69Ew8%2LmtYp-9VnY(BZmV z8)^$|oOz#xKUrsT{!`x>uS@lYcvFY7m6W)!&-;3+O8f5X1gTh;?f4Ap;kLmpjK0o%(iT_GgMI?qnD=JQ4bQPKStSqXRYT z_^Zgch70!2urx+Y91q$kT-$QpT&ifXN1TO)q)_-B%1t2> z!C){cG)hMcc1OwZ(Z>iKqaC5=BLAJd{RPEg!;QX#b!+`WXW|L_n_ngN*ITbJ@DKZ1 zj0OFc|6=Obhxa>vU%nfqJL2ijDA(}zyW19sON2P>%DgWu*$t~zYrMhZ6oZ*(=yxOz zJKz!Y&WBzK9d{M$!q3bWSe`sc2C;rGvc0y^>L@&^_?j=-A9cuB)!L*Vd^~q+4A&~H zEiwsFanyrW&3mvI|GMWhQ`mg8xOnAV=X=34TVatl392q(yGu4JQn5^bms?*?Gk3^8xB%}3@zJ4?I%4R@_OW=-?bOLv1`}Hc zd{HcA4#p?SRgUwjB$FhA85BTcd&|Z$5)nb^Z!bB&vt71!O1w2+NogEiXg>05yF$`q zu&P{lSwPy4-=S@fMOG!OL}DTJ*L#c9#7{=F+O8dw?T-z;!p*~u1Miu+{2d%YJtMIJ;FRQ#y^G@;M)iKGbEii*KxC8S@$4vN;jQ!SPOGTj zGe4^0htx9mJ&DU%yn2(A(2Z+eF+_0o#)&qxgAsM?%bV-d21D5~r&(eB`lWHLdL|R) zMaB*#w%y&~t1w#6(*^M_RSq;EDRsd~Dcl~cAZ+}DkTkp(ZJdj{JrS-NdAh7&*uWL{ z&n5`&t561+78LMv8EQy5+h{rzR@rwd>W&@P9l%aU4gz6T7vA4@vnr1;KJ6V{x)!N* zqBS^9Q%K^9H}VaiP6JO?(Yw6PJ1}@+Hk3*KGBS~Wm$_vpfsTK0g*4oM%rMS|$}Av{~rXV6Mw{OdTh30FP)oCZ;h@ z@Se1!3Q&_f5-E10@DbR(8Bf0|P(l@nDH)Np5K6Huc{E{1EYC@fOEsNB%4xF>ZP$KA zh9A5o!HRQlCO;SQy_{F-78hXx#~p903jHFg&%|}~3mZ*(|G}eR7&r0iUW(p99R@+;8T2H%|K|CnT|9tk`G%?Gz4Ss zS3Fbd1lanj?pVveSTT3tNUjR+6kFCo+3*Z0lnYK{j%|^bl!22}3%!-^&|(<8CQpn3bp)Hvoe<#CTRVx^2pcT`mEmt_2#1_NT>)U-T2p+f2+yd170g>OO4qLrFF+H4 zbAN04o`J|vr1m!;^eZ=HpAAtF>rR@$t+Pt!pDaFQTs|uZ2z#aR5C>sWmht~xiCrwQ z95o6kOc(3R8VAB)Gqw2}w|YzU085++pyvfWFRl> zZ&S+Ky>C+@|Kt&T1aU38P2uiHy-j)bCy$^cK?}eQ531HUmk2!qSGo76nw9|5 z@N#-QA|j$sC>|ajApDxpx1ow>eIz3E4XFr86(kWFha?Ts8;sNQ^BjX(M`t&;==J^m zpWO;?+4VY_rKP1o1P%tzmoGnAy!mmvV%)$srz>y*H+=U0#Gr;R{vCsQwz(B^+aHMN zBwj*R9XJeJ7Cm^V=;-a;-D@ap;LYK8bXLPzQaDEeMz;&t4S}LCPr{#|Aj9~&1_rPJ zM=rdIy!|Nzdtnq6vBD7_JY2z=8bQs`k6wHL&;3*hyp56AKs(A|6J(Xrm<`S3^x9@h zZ`oN*c*;q-ve&;0VG1U;Y;4MxaXPb!59yInkwpoQk%-jB0&K!JZ7LF#2;b9#uWM$; zpa}-!fYSxrs~~@&b7rt6y%wz`c**0rv=!a{F{zX)(vs~18+GwImMomOaxjTfjmC%1 za(}7Qdb)~L$BaTP4R|fOpdK4eVbH=N*FJ_4DSlgM9TmZ-t5b|j;{IG9GRWm!g8)02 z==f;@lko8hq1V^f*5C3lb|JKg4_?m_9t|hB^Z7p-@PHXO($4E~l zy&=K-iVJamI0(U#G`0Wp2LoP2%i+7nv~ROA8@*!ac1-B}W`jM?gt6o~ourpMBbHyg z=A0@j0G~X`FjPlzthdNauD=X>-jyf8aEA;fg5&B*hleZazcG7g`A9ga1HV@}A)YxY zWDNq0&mQ=NF&DH(2SukH+lRQ*my`{-wN%DZKOWD}xE4_40sqJ>dgOPzX`L1HJd5$4 zQJ?2=K+K08&Iw11PU7SQTFKYduSUB_G>nJ?v-;Qt#zt_;f~0X%aoi#5X;})lw-Fb; zgkG*@iF|N5{iZ$kU^c~#n?j2rix3$`Sid4~nQJjvy8J-JzI36U)S1f}rHOSqBXy~h zBtQBBC~R0D@@S@{=aq82y$PLL1r|R_)5R%T44Q*^I$TxL9ym=h<26MXMNo{#Yj^5M z)>5PWTCUhqh^RMLs(wZ1V<(flt;sCW zV}~5-V|o0ao?TU@kC1DAnQUPvyKx`ChKUfUzmt5CI`zr!&O*MYEn)O5wJhBHtipqk zpse<~b8c9V*_@NLh91ccV`{oXjgsx=cbn?|7}L3lKL|%;-j|`OAZc?I)rg_{_jiFw zgz8@+jQTlbeVGM1wNhDtjv}sgZ2bti{F8q0bRJRfh&QB?U zkVZK~&|e1@PD~70+MrD9KV>3aV05?>id+;;%}1s#eINR&tqkslf=@rc$NXq?1LIVg z=$%rI+*Vlli8=;kbQlChQ6h?Rw%}x9SonBO>a9_()@7_|FORH2sP0WDQZ3}F)i6c) zK0HE9`q-2kS%0i~DhmFmL&8-Swwc@u)#>E~nvgNTzTT2!Q!QR6|_cc@o45mau7<|-B+sc{m_BBH-*uw>YCC#)yyNfQ=0c*ah` z+5wgBg3ZBS>VA2Nd6(!R-saxT)E@J}>QE5-jvkj=Kkw`pKSJql+-SFE>dgQ!OkLob zEfwD&2_B#Urdy|Dm2k3twe8|UYmgv_Af=h7K--?iWrnPlCKw1PAm8uv&*-JKbn}-3 zJCIoXgoOi>&-OcBBieB!U-1`{3F&SsO7&{Xrvox@=V^u1C8I_!V!cYsPVgMCMx~`{ z@8x)*4vT&jOTp8^P>o!0wFHQhhzAj=i`m%n=-|H}ekq_vKeUp#gnb}H9} z3%PGWNZ55voqFu8%#dsN%V!!L1kQa=2*3Q^4ppRQGV9*o4!_U&u){8k$wWZ*=S1{^ zfG<^ke=0n$_ccuBCLUvoG{EQ1L5|f;#vaQ$e#B0qh8CA=8++bR5>-1hc6)q~yvwf+ z357Igx25d`4s0$5Q;Ollxyqyu zvsFUAotmxVR>Yog(mO;7fO8l)Gn0Uqu4P{{D02Cpzkoxa$R8eO%2inOKB2BAf)>y} z3-e+yJpz+GnLJxfE*t_*eMI0cBqSsj?V=ih8t?6e zUn+MRQ`uXIBQ^~OEA8x4dH(Xv(KB?wVJ4ZIua>rn<=!(ACuRJa*s`fps?k$iM)c4@ zxLU5#ePHoN&;4Glx|U4?PCzHd0|Cif11m2UU$-$L&%JNyTc4_;|HWGr-Q_@TJ~G>x z1a(B4_Bed|%4#`4!+NTMR``WeY}a@y=!?%hYr_@pY;|id}E!iNS-y_};7K3ZJf(YFX>Pr>&uw zhWG1n{~Sma(A34E5c=zl-IJKH8vv-|Y>5Yi*!jP4=)z46xIgKb^|V`ipNQc8t{hrg!a6|5OaeX@~6QonzVFD@;; zVpM*+P&@CJaQJ;_2p17@4ZNU%4}paHUL5ykM(C)+GNSmskGIObiL^SVq9d`R62c%nmzZCF$tMEes503(@LXx zw_AyT1H*%`{%LdZqnjO+g|`E$nix2D-?DMmU(bb{?l1MeNph>7q?6yFm}+q3@$&N5 z<>YZ5ljOA>-^-<2ogTM4eRkmd-SU`$`qFjD$>-@wq4h=I8%}0nzD;<*J@gf^_KT}m zolQ{^J!8Kp3*$sgcC2|IU@oNieAJ|=i1kwP-kocNI?f6j*1Kbaw=oh6A4U)|%K?{y z_o?ln*a@Y{2g|+mI*wmnY`l8#v>*%C8F|2$*hnf1Z_Jw-R7dSMyn#8<9qBU@K7>uz zII!K^Wi$FxgVdekF14 zh+)fusemV-)(|Tbr={7I4YvElb}r*84H z?yPeg}aXU;HQf&b$y9=5&@y`9P2O{800IGL!sbJdkt z-3`s_a7WjRFs+802q)8*?vEPks7b|-k3LwYp2+sE zkL0ch5LgYSMGOzC`<$)3d*0+UTZfNHz#y3^?z>FQ1{;_2$q;t4t)I;fFTY24pDZL` z92`UgQ5lGH0AIP!*x2}+@BS|q;3#u&_4zdNH(<+*WJ#guU3zRzaGb>vnhY&J<3NUf z?{M51`>ease0THYnhJAb`q}1it)0o1NF{0J8e8=I$u$TS{^MC{v>ku8N7x+xcKyjF z+U{^$5QvTne1~gX%M#e>7$C*M5NOz zhJ58lU#JYC~E#ev@1&P<^Je= z*~u>O$O{unZ%iB|q1nbsYuuqzCM(taF;(NaMh8oFXU!LemUGgJ(VR(PvI}}sQpj^ zarIBk&)!Jr$N~$gIopfzsU5X z&y&GFtE|XOl)k~2=xtx>AbtVG7_h63vno=_5WtM37HjR_=PCE8 zR_bZxY}C?zd-q;4H~P|Mx-$P&=wtCK2E;nyl8y8a9pRDdbD{}9)KjB4yN4oo+A{&Q`)bJ$y;A-vKA~@a`qlfvP$6a_vB0w1@SiZtx&I&t!w@2gq;lixS@=mzN`6pln_y39YZesxu` z%Jh?(_|OV45HZI`NnhJXBq!O!KJZhP|11#<`Jz>-))NO6L;Fp|E|9t@Qp9<@I~x|@ z8@5bg(&YSkHm;=m@sIc!dV%v7=SXh%C#BYm)6HLy*G#*xY;gK$iI%@pIgp8ueWLy? zc5Yg;W1WNki1`&_tmhtgpJsOdT$uaoYAQ!DZmQ?BY6X{p>#zbc$~3LMCdyv|sk$_>i*+2-pKCi7O* ztNd|6$CUO8j|UK*x&al3zC*=&ijQ4{n~*A5nMQv4mD(L9ng03@>i$Mw(7Z0kP~J2Z zgu8wi>Lahtm6BJlkraZb2_u_mUud4Q1wWoa+uqaJi-2T?ZjZGf$=FBgRMF3lLj1Kv z0S76TWjgkC%d8lE9F3nQkz46UwnyE_F}+S?syA-rO&>31gPn#l!?HhmYn+j$;xX*{ zi748$85hg&oy8c9^)+~7N)}GVin;V!+G=t6Gg9>98wfA z{ZngEXZW4k;U*utPkHS>pyMu-vUx8^xROnrUw=aPfUQUay`|ju!oB|dbpF0ZV^0(+ zFHP3u8wkEvpKviA1rdC|vYXa_@v_c!6Gsl4BIK$b$Fwhd;Cc1RRjIC%aV?oxqjxXL z(u2v56jILZzoJ==^rXT|tvC0Vl;BjAIn@=254JQ`HTNN}sjRfBc>`e-WrZaU!)9~|hs{;t7{3;S!XQ(2cs18ed~{ux zcMi?l()RIT%f9&ZoI98q?He+3S=YjTyyVwBO77godAA+)1vmc#X?fX?W@WT913G##`I=C9aPhN6iX+E$x-9R0u1?HL0)V07lY@d=#tOgMPhD!F2k1(a_=Kf z1^^>V)BIh5a0d!7_pEh3;(yS!;t1mYobH_%VT5Bh7EUKeKg?@RjRfG4Zq_f_Mg9?# z5_8251L>ve%eMO;5DjTy+a$<0u1KC*y6);fG2buBeZ|^Ws$vfSKhZBNG5I8yK14vG2`op8XtHqe14+&~+7{vDBy9AF<_`G% z-RJCqqm!|wXYt=zk`LJ|)Y3Om%oowib}sGEmpUSXfaz*B5u7|#?#Ap+F&_g+VZswq z(cb>oTS8xnFCXoY8&=pW!YPGGaCgK&bSd-PBZlRjI}{kR0Zx%MVe(x$Gsqz@_AxzEq0$@ch5n_hF{Pb`OYl=H0{I7x* z*l6**TRNgiaZ%)P!VidS_x0tr(0Pu+vJd>qQ){LiMC^`>o*#b2zmlgjHTmm+xF;&( z%DXK$E3ArfP~HG&f=h*C*arMEVLRI^IwiRmXU79GP_t8+**bR+Ha}4k=mxaDT}QgV zXZ}xdUl|s4*X~OS2m(WQcQ+E!ogyeHARuAT-6+CHmz01sNQ!jV5CQ^2gEUBYhrn6m zbKZBqdtdv*`F8lg#Wl>pf30<|x_@yOsrZMDf3ma8mwn^UeilP;N5ZLbmv(_s{6|0Q zuJ?B2Hmr;DrG%5zuj`~wq7{`Mxqwd%a#?9|fo=k{Jz&@iaMURxk%Klwy0A(4Ls3w3 zq4kvB`uCyStqxs-gc6H_FL7C-(;`NQ)pDLE)i~1@6l!JJFEl^mATEtK`qV0u-tD$1 z@e{Ai?n!?nE2X}}yH$3R#nVz!aMy&2>8>rdP2bcJd15Wp>@eWDJgE16@ZJZnd-kOZ zoyhm-t##En7THqgYWR0QY`t<_YQdf%SZ8ST!FDP-KI?qz_PR8y){DEnn~jND%Ck4W z7^Mk6?=l0G+`IWw%Gz89%^9JPbC3DhLL)yy37=3vMnc@R?9dEY35C5US|7eu8F9qH zlJxe+Q^y?&RA4bPa#JaG2n8o9>id}4kb$0X$k&cvF;tUE8Fp9=y-%f+=3UT8?Hku5h1jQ1QpTb|OO@t6DXb zxsnS1#RVXbGVyYeO@k-4KK9@ITn7vBF?}%8AsorsJxD=l*ZIia!sZSMl+*h{Y%Cs& zv~9)E31X5EgHf|5tgU#GH!lVfdPeM2PDYQo%+m$LhEpVK*rzbO!GtivK0JJ6YO-t) z4MvYf$eV6$$SCmCCm?#&?1J28U*Zwsrfhrr;Od+#P*{DE&hC3|;jg+V$xEY>M1|Ai zFAb;44H26MoiS418xQ0ylI#sR^o{3FkJnB=Z;WTv67eom3F{QjjuqPlQ!cMlr@F&H z0wGxa$O~RSsjw`xCIH*BgG2 z(FW7cY}#2Q<*74c=N)G4thx9E<*#bW7Q%MKD*G$r+|y^-KZMlTQB)KiBE-lfHcwuEiZ38wtX9|~u(82loJS6gCV7qBl? z^LFN*Q^QTe-0N6I4_vQueE#f~{B^KFEX=}qtUJRQSgdC~lL)a4sRk10e60czlWqbZ^?}&-@Z=?x6V}g$nd@GykiVK{^L=Ht8R@O4yv=Pi}Xl#6^Fy@JOD;6m6>%F~lM z{~oLp;-|dvp^P>eX;XF9gx($-)Nqe;|2Zq5McaKWR5*gaJRWV32g#Plj00L!#mnPW z_P8AF0;wZP#{G|?n)&%AIfj_ik}iA61N90F0+Xd*VVjV zq|3uyKiN7ocPxYk*(@xABSklZ0-tVK8Qf)u!j)*LRNFE%59h1zD!}PdiYuB%N^a&P z7G1}+KnIZ+YL{)tM`6}%xN|3o+FS3yJ1pAz5ns@o%H6MgUg7m;gTL%~a|p;9$2?SB z#>00eY@CYb04I7uD?HFWYyWhQIlO8OtwXenGxnIXPO*YLV+~hqiVh^jG58cC#Hpsk z!hh>Y9#kiZ)yG8uIcM_6x7)5CsL(YxS-K+4L)Rq(uJlC~ z_X;$!u3Z2q2)>NgVljvPeMGO};if%IsOI5ctn3aPIp@R*hw1Gt3h`2&14R5;Fi0X!46^if6oPk+7g`{hTe@I=d5$J;_kUg9Y6W1?@kCns}RffF|-p? z*3K(ZT|DV~1q4Cd1&PbcyMp*~EC3R$+0g&$6YAEfxkD8ql5H7A)+3mY4)pb}({Z-V z(j@t8=N}_5NGJe*@?unDS6gpPPAFjS3p{QE2phk%#WyyX&IvV{0dk5qJ?gj6KyZgP zxULq8mKs|VWB?gsZM5Un2bpumo6(U$0G#JyrvmqpU5dfL}9g}@}a9Z8}Gb}+3!7Pve@u`fDXI4B2g*-evTlo>S zmx?~10%SAbkAnGZZ{eZ8*YR0WhofVh9x$}2Sw8S=CIuG8U#`F?^f*i?KU5}+l;@lP zqh&e*MrOJC(R2&ic6WO&-Q;zwq)Y=R2DrQr*3=A9^yjH*^(VBTJkUuj zhW86P0HEGeBYDmo#D=L*%MMyEIPVC`Mwv!9y`*A#D^6d`~c)Dn333iGJrh zQV>}v=8+9R#EL2OGdUC>_g>CFR7y1Ie07_e$R(Z5x?g~;w`~cKmfLdSFV_Jdz=>>3xtF4S|4i;J@TO4N)%}B%34&0#787r zPc0PfN~T3Rl(bP?lw#RhB7O>)d@L8pBI=OIrIipFjNF`uJ?Cj9ek7gMMXKr{pjhHwpL zJPK!0jWfSG*(Bq)K-2mn6>0S&E(IG6v@{+c8u`CMU7xgMWIhAq@|@1 z^}Ip383l85I-p--4PY^qeG6PIYo;r0qAQT$J_LMZ$i^<>4UVww-2V;9!6QBl)c}&K z)mbUGt%S7orZHeko{6n8D;Iv~_v>iF<` zE5g7Yx3w>kJNz9wb{|H@#kR&j8GMsOgg;#-HL}mzNsofYubQ{Ni(Y|e`c2D(Ma}va&9Y`gvb2Ep$gx#?r~6fFg>@QnRq4$O}ogt+6eMK^j2V%4Py>tzou+t^XHZ zaNH;pgmM&}^55_R>B>w7Fsq1PeCK?Pk7(cjNM%d4+0Qi)AHKbZ$W9M)-pUbzT=tSE zu*iBHXqA8GZYV@}KHd5Kn^~HqJPK5J=iFX`>6aw&nnm4vK+$XA&%fVA%Wct! z4ldc5P@QOIpYNDK$x;VsH1yW6k?)>`AI#*MpylS_C^9;toj}qmV~kRfGFl_)_vfO5 z>ksjQ1%WVs(-?wC$)kN{jf@iIyJ(0+E6;>H2U$#;c&@hN4*i=}S3z;WItc2Cl|fl! zK+J#NConQmkCRzdw2L+u0#7hsF~A4h@eLj{fGrS7Kr=ZUEF}LQmKWbeF)YcSae?nK z9knDdOYY4*f!QLZunZ^$;z_#vj)%t}tyekqfF;z)MR_P)0ldl@RA5Nph!0jljG*yM z<1n%Zbfdxbm*Wtp9GzW_W+lhMHz)H4`!pny;T9ywq5PV)3n6Q57p6r_@Q$d= zqUT?T_-86=g7bh^hvSwEoxMpz@I-D8#bX?2Mc2u+wAqb#ZT=Cl@jm0DM;quIX1%DI z4K+SCruP@@_WZ|9M2+6Edvyj9kI%Yz=PhiMC|E!)zt% zp_xuw!Kw_b97`2AWNJR2S+aTN`@Qa9d4;iFDp1|>zCX4Ggjz9zePm;n&idr6XVR27 zZ5yprBBITEVRu8Q=40oTP0tnMw|d#<3Hy^~0a0+qrEU>A^W%qE$x`@R;_}bP+#CXdzC9AKGvxfHub*OSkaf9aku|i; zPP>hbVePcFJT@oOm`O%8o<;tv#TU!R3t7EeiMu%mB6k&#%Oh_(V-k;DdI>m#Vg%um z#E&W+g@NsWJH;nMy*}jiOq%bU6(8}RU7Pl1h#f0ql})!$qr0pt;nwyJp|if2901yX zpf>HrM2ldtu+}(`#Ttqu*loT(AI{gsu6sF%60q=+w(EJRy`t;x6t$Utsa6r6J#D%V2#to5%M2sI?qNJC&d2wx49VttLl+wf`A!`|M@q;p!&gSg9fHg4h?R*`v_4UuTDyPZ= zzdIV+iErq>YeRCtB&p?eFRXh(NKoUJ-2x|`W1LDj6*~RUMVy1^&Oy=o;tlOZnbC)~ zNJ^%7v&pi~RmJebi%ngjn=8(fQ~S`RN^474T77St6t(X?yzctU{<`x6$W_g2&i#}M zEA|XYpAZ^u8?u~*NExdU!B(?e|Gb=5>{9{USj8(Aw>WlB-BQLNguAS}MbRz4 zq_$W7Dd=6wK$Z8uLT`023TNgdSpmjlK*fLWRC=<-!#r?vmkb`c=@hQ5#+} zrJgKlbsfR_{phJC5878AH@GX;zR|wZm7P8Z=APrF;rE^v^?ng8H$~+p+yk0eBPkXOFuHQ#M#6_DmoB-t zO*kS$sULa6yq4hr)Hkl7(kd~DYTD~AOv-`kxHWte%pv4Ug?bqi>fUdo3^u`&T@hKx zw2$;AlX4{D@O@4}%4-&pKu}TS%(3_f2(yNA?k#oolR`N81Im6e>pOQS4h%+KFJ=fg z?uZ%(MNCy4;d!odn?uKRVPp)NMdEfs5XM&)9r_k!vvonQsKv9LPPdXr%6ackl!hM5 z>N6ZdiaBaM_OA+|n}YU|9i}Z21|P**EK*E<0D{$BiQ$Jd!|%QRgv5smZ4Ti&Y}zzW zj~5F;>WNA*@rEKU5KP)@VSbCJzlY)P4JsVDw}eWA;xh7vOI@^6uyxLzuLrq_xE^ny zLR?r+YD%IZNgk6Ty)jbq!B8wSJMbTTL}(RAYD+-cgKBBQWyY3a?IxA1qUihih0~k! z^l<0W8I_>X^5$BN)tgW}Cq^M6MM+5@trOj!|6t$*(SM)O90TfpBNVQD_U&th05>0^k^%5kdHyp9La2h^WNnnHI zN2COmQ|if%-s854-9x$aI!F;El*51Ic_HIJ@m=+G7?~iBZ8uU*c=Ffq8+ubpYNO?7 z`ca$ShEKw}ES=1%1gK&vtQ*U(5*0_~sX1qMqP(A6zayfQ|(;CjK&&gx| ziL4huIh8(y0lu;JV(yV^!l30(0!*_t&}N=)Kg+z1e*Vov9Vw1gI|z4f#$uuU(a~|o z7vK9jm}%^m))5&GQAT}PEUzx6FG?Om#$G=oM7Y8&7!VY5T^@NVK$CY{_hkX+h0&DN z?;AnombT~W4~O8BcjfUou`Ey1g;IgnTpo{Ei2)kw-2=+u{5zSn8$a$Rb#<9e`%+U* z8!>-oD`otla*JH#0_KZjc4WKTp^7<5*`Dp%k6R*1gj7^jB?+)blHqw>#CA3GRUN`Xj;Hp8`m^C@Ydj^~=nZr^1ttj1~6> zH+}GE80yspV3CB%gPW8ddvumFRrwjCfchOpRa1$ET=|hqok>$RhKT>Zc}nirX@v;Y zk!&L}z`@M2Qnm&~m@GrMfNlt!P>(5}x`Hgjpcvxk{tY6Knb5%;FbCN|;Oa#@3-XT0 z*-?!kvEk&rk$MJb^;I@yd0_A7w%hI`L7-}jY`p)T1YifdJ5s!GdhBp=Q6t*mu}9p# zetEp!Yb?|_J8$+zfpm^|+}cH_n)pLgk2X!^>{8Sj&O?CWv!xVA#wrX{e&fsKcq-n9 z26dqR>QtA*c3*+YUu&d(S3U=GRsJ0lUlmR2C)0HrD{ z*g>k+b#@BReSo;<0h1=xO;auD?$z_gVV{AAX{NQ^^Tw`oQ4C6xn^`VKgpO5E$Y*#u zxQ&SBU@sUeQVn=6ji6IWI5Ek=ppQ&o8NB6YtsP0fIkFbcQ4g=iUkkmS#E+@W{KZgEh_;jIFn;#7uRE|8tIR)kd<+>9ox0xU*D1v@fyjU z5r3K+0!9%)V7|IJ)p{#J`&BnjDZIOMxaez=#tXdyyPj}Lyw7)2mEmsbdW?bVBm-G;F&iGK<6ex`@#3YFR-^`z*YA*Z z3zj-Q6<^e-TK<`u!2X4b^37`0n%z6mejX*7YMQ^=OqvNwOFJ&%-jNrBKUR8?H*1=u zw)w=?usBhOS*v%R#1YfKGh|ET%f1BggG|u_TN+4P?pUx@QsFY^%1*2GuVj*gB4 z!)fi|92Z_3NL_6)>G+=XeL!@50}U3Uw$MoCz01D@pv&N}ic9ubj=cw=r&+F|ZLqozuHh0uwP zzwJ>X;zt#Cmnkc8h(5zln!?V1G0rjN&WB{?w#9|sD+*|z*M;ypzjL1MGyiMvhVv;W z%-K3w8r|XV4$_|E#o=qhE!}lq*!Zl2cYJqAOZXB<+J-yP)w-;fyEVI0P8Z)KM`TQh ze|hJ~9a?NGWHs9#tAair8`iPG2N7&o<%-<0Jw?dabH}9}PqJ(dJVMF%J*$W?D~~a$JVazUQyaO{&TbTnu~; zx(T3(#p?)9Lea@mBBz@7k}FHz!JFblkF107w6ssYa2o@lq+wn6}(ccrJYBx|5GOWb^S~wf}BSTE^NyDF7269n*D(HBMBgDmHfrd#n0h2ler>j1bj$ID6 zy~dQ$b=Z?OUFUuc3ZdZP?#OqbVd488vp7G!qga-zP$Rq*-@HhvWSg!cFMBhGQh{fk zsP=UA@Sf6iseXib8EnI24=%`xKMWjF;EPKsgvQStdS=c1Vx-=BCvZJeD{BE{$=z<@ zh!i0d&Pkr6u9oFj3k@q63~P;23(FgO@khsuJ$PBBl}1Om9*P5POVWQ0Usdk^+7>PE ztlp-FU`P)pOAzOo!|oK6fu#YA39Rbst903Ih2qg3|0^Nb>uLVx zh1;c=nm1xxl+w7hd-I)SX432`Fk&7@_n+sTyFTX(<1o*$l1Tsg-Bq={-g*}&=W(^U zH35zuVEg%rM||5B#R!^w+F!Hy%anSRAE~8vu$`cl*z;mTHW*4S{)#Rt<6)Scr454% zi)eiZ!z;6SvyYH!A&eU8n?figRAz6c5>;Sc%!u8PRyW6p+XUx+LV;;$c=E=Mh^?XW#FA&Vr%%UTzO+cMOKNaT*p|@S z+5yAWFx`Bdlsrtr@>&7wUM~H2RR4~R&(K2}NrM&v(0zw1U7Z`Iv|-6DfM#7K;d3c) zfAU(1rn?CNseOb2I%HXa zs*cU{6e>^Asc$jkY9GXvQrS1jmEBhJV%yvHR|jQ!bEA@XI#GJ^RLwEo$=(7gvP9I2 zh4yCKr8zS7_42rmmx&9m17@=Po$W-bjb&vFlnc9{A&%;Yqu! zGDJ~H1aIsN8UlOv3GSYfk1mHN)4FxEtPl;&4|7adLB4s2W6yRJpFNZ?P<6(A_FT+V>?|$fh zd(;2SWTpvXe7>uEO)vjD0BjDV0cLebLm`nbZ==c$Lcj%OTOxaz&yC=tYb_ymSY zPuHSa<^Ds-(8M@XfyVXCHTTuo4llNKbL(-OFn8nGu0qe=uGK}^R}-Bq6U9q4+sHfb zG5aH7;K&@h7*h-n!#ez#w{^Ua~9X$lM z=Ag-1ZhBZu-Av6%_tt`bmd00STZ3AiD)+A9f<)?6wIycL_g>Yuq4UoJ+B%i-avo_K zV~Wvl1e7 z+^6`M)RI+5Zo}eB)^VvBw0m0ouSI~MB_x682ZJd!afvY=dmMATo2XJ_9f4(41l%l? z031Vfmg^q689^b3eTqbrA@dIu`x4lH>oe@u!zR-^Go+&nze3Izu$sABi9hkZkn?GX zsTO0`%~gnlsjk)*{$n!i{Q2Cp)r;EhNrCJi;Eg@wza#Y4tM9Cz6u0yPmIp|5SYK&O zwy6NN@Wt}$@pw`m*x=i?y$WpWW0yun92ZY(2E9 z0&Www$sbTlI}OD(2<~tTHUad*6ubq=`cQ3S=_4M z_VnE6FREoPRzBZ|e~(E;`v`@7G*7;}jh9lZ`bCew?JaEIR{jMXW zP1oohZTF~FPhW+C6k#d@Fl5%vgpW@#VTiQ%nF$m8k#rw?%P2(1J^Nl<{b zR!_LGXiTg1TT3x5haBero4!Lv7k3wV01|Mcxg(cBnDm6sODwaGW@83_ei9ZK)c$CJ zr|@9lzp{+FZEP&IWt71&_2=iRbza^b!Zth;y<~jN+4oq44p83V(JB>b-%nL*gSrAU z1jol)(=QKSZH@)W%+|S54t)vPoPK&mf*z!xO28n0R@G}Jz-mVC6s&@>JpQDY*4T0X z#13>lR1ZgqPF8?4_vk=oP3@`1@%_y?qCd3!oG&?l(+JulfiG1>Erl{U^LPIT8#V&$ literal 0 HcmV?d00001 From f537c5630a0abecdc20043365fe942990e9954a1 Mon Sep 17 00:00:00 2001 From: Christian Adell Date: Fri, 24 Sep 2021 19:16:20 +0200 Subject: [PATCH 02/17] Add contribution helper --- README.md | 62 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/README.md b/README.md index 46c262fe..91323422 100644 --- a/README.md +++ b/README.md @@ -214,6 +214,68 @@ Circuit Maintenance Notification #0 } ``` +## How to Extend the Library? + +Even the library aims to include support for as many providers as possible, it's likely that not all the thousands of NSP are supported and you may need to add support for some new one. Adding a new `Provider` is quite straight forward, and in the following example we are adding support for an imaginary ABCDE provider, that uses HTML notifications. + +First step is creating a new file: `circuit_maintenance_parser/parsers/abcde.py`. This file will contain all the custom parsers needed for the provider and it will import the base classes for each parser type from `circuit_maintenance_parser.parser`. In the example, we only need to import `Html` and in the child class implement the methods required by the class, in this case `parse_html()` which will return a `dict` with all the data that this `Parser` can extract. In this case we have to helper methods, `_parse_bs` and `_parse_tables` that implement the logic to navigate the notification data. + +```python +from typing import Dict +import bs4 # type: ignore +from bs4.element import ResultSet # type: ignore +from circuit_maintenance_parser.parser import Html + +class HtmlParserABCDE1(Html): + def parse_html(self, soup) -> Dict: + data = {} + self._parse_bs(soup.find_all("b"), data) + self._parse_tables(soup.find_all("table"), data) + return [data] + + def _parse_bs(self, btags: ResultSet, data: Dict): + ... + + def _parse_tables(self, tables: ResultSet, data: Dict): + ... +``` + +Next step is to create the new `Provider` by defining a new class in `circuit_maintenance_parser/provider.py`. This class that inherits from `GenericProvider` only needs to define to attributes: + +- `_processors`: is a `list` of `Processor` instances that uses several data `Parsers`. In this example, we don't need to create a new custom `Processor` because the combined logic serves well (the most likely case), and we only need to use the new defined `HtmlParserABCDE1` and also the generic `EmailDateParser` that extract the email date. Also notice that you could have multiple `Processors` with different `Parsers` in this list, supporting several formats. +- `_default_organizer`: this is a default helper to fill the `organizer` attribute in the `Maintenance` if the information is not part of the original notification. + +```python +class ABCDE(GenericProvider): + _processors: List[GenericProcessor] = [ + CombinedProcessor(data_parsers=[EmailDateParser, HtmlParserABCDE1]), + ] + _default_organizer = "noc@abcde.com" +``` + +And expose the new `Provider` in `circuit_maintenance_parser/__init__.py`: + +```python +from .provider import ( + GenericProvider, + ABCDE, + ... +) + +SUPPORTED_PROVIDERS = ( + GenericProvider, + ABCDE, + ... +) +``` + +Last, but not least, you should update the tests! + +- Test the new `Parser` in `tests/unit/test_parsers.py` +- Test the new `Provider` logic in `tests/unit/test_e2e.py` + +... adding the necessary data samples in `tests/unit/data/abcde/`. + # Contributing Pull requests are welcomed and automatically built and tested against multiple versions of Python through Travis CI. From c8bc24ba7d70f2023fb25a2d95ac584d8f7049a9 Mon Sep 17 00:00:00 2001 From: Christian Adell Date: Sat, 25 Sep 2021 08:14:44 +0200 Subject: [PATCH 03/17] Apply suggestions from Glenn's code review Co-authored-by: Glenn Matthews --- README.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 91323422..1131ebfd 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ during a NANOG meeting that aimed to promote the usage of the iCalendar format. proposed iCalendar format, the parser is straight-forward and there is no need to define custom logic, but this library enables supporting other providers that are not using this proposed practice, getting the same outcome. -You can leverage on this library in your automation framework to process circuit maintenance notifications, and use the standarised [`Maintenace`](https://github.com/networktocode/circuit-maintenance-parser/blob/develop/circuit_maintenance_parser/output.py) to handle your received circuit maintenance notifications in a simple way. Every `maintenace` object contains, at least, the following attributes: +You can leverage this library in your automation framework to process circuit maintenance notifications, and use the standardized [`Maintenance`](https://github.com/networktocode/circuit-maintenance-parser/blob/develop/circuit_maintenance_parser/output.py) to handle your received circuit maintenance notifications in a simple way. Every `maintenance` object contains, at least, the following attributes: - **provider**: identifies the provider of the service that is the subject of the maintenance notification. - **account**: identifies an account associated with the service that is the subject of the maintenance notification. @@ -36,10 +36,10 @@ You can leverage on this library in your automation framework to process circuit ## Workflow 1. We instantiate a `Provider`, directly or via the `init_provider` method, that depending on the selected type will return the corresponding instance. -2. Get an instance of the `NotificationData` class that groups together `DataParts` which contain some content and a specific type (that will match a specific `Parser`), and adding some factory methods to initialize it from a single content (as before) or directly from a raw email content or `email.message.EmailMessage` instance. -3. Each `Provider` have already defined multiple `Processors` that will be used to get the `Maintenances` when the `Provider.get_maintenances(data)` method is called. -4. Each `Processor` class can have a custom logic to combine the data extracted from the notifications and create the final `Maintenance` object, and receives a `List` of multiple `Parsers` that will be to `parse` each type of data. -5. Each `Parser` class supports one or more data types and implements the `Parser.parse()` method used to retrieve a `Dict` with the relevant key/values. +2. Get an instance of the `NotificationData` class. This instance groups together `DataParts` which each contain some content and a specific type (that will match a specific `Parser`). For example, a `NotificationData` might describe a received email message, with `DataParts` corresponding to the subject line and body of the email. There are factory methods to initialize a `NotificationData` describing a single chunk of binary data, as well as others to initialize one directly from a raw email message or `email.message.EmailMessage` instance. +3. Each `Provider` uses one or more `Processors` that will be used to build `Maintenances` when the `Provider.get_maintenances(data)` method is called. +4. Each `Processor` class uses one or more `Parsers` to process each type of data that it handles. It can have custom logic to combine the parsed data from multiple `Parsers` to create the final `Maintenance` object. +5. Each `Parser` class supports one or a set of related data types, and implements the `Parser.parse()` method used to retrieve a `Dict` with the relevant keys/values. 6. When calling the `Provider.get_maintenances(data)`, the `data` argument is an instance of `NotificationData` that will be used by the corresponding `Parser` when the `Processor` will try to match them.

@@ -86,8 +86,8 @@ The library is available as a Python package in pypi and can be installed with p The library requires two things: -- The `notificationdata`: this is the data that the library will check to extract the maintenance notifications. It can be simple (only one data type and content) or from an email, with multiple data parts. -- The `provider` type: used to select the proper `Provider` which contains the `processor` logic to take the proper `Parsers` and use the data that they extract. By default, the `GenericProvider`(used when no other provider type is defined) will support parsing of `iCalendar` notifications using the recommended format. +- The `notificationdata`: this is the data that the library will check to extract the maintenance notifications. It can be simple (only one data type and content, such as an iCalendar notification) or more complex (with multiple data parts of different types, such as from an email). +- The `provider` identifier: used to select the proper `Provider` which contains the `processor` logic to take the proper `Parsers` and use the data that they extract. By default, the `GenericProvider` (used when no other provider type is defined) will support parsing of `iCalendar` notifications using the recommended format. ### Python Library @@ -216,7 +216,7 @@ Circuit Maintenance Notification #0 ## How to Extend the Library? -Even the library aims to include support for as many providers as possible, it's likely that not all the thousands of NSP are supported and you may need to add support for some new one. Adding a new `Provider` is quite straight forward, and in the following example we are adding support for an imaginary ABCDE provider, that uses HTML notifications. +Even though the library aims to include support for as many providers as possible, it's likely that not all the thousands of NSP are supported and you may need to add support for some new one. Adding a new `Provider` is quite straightforward, and in the following example we are adding support for an imaginary provider, ABCDE, that uses HTML notifications. First step is creating a new file: `circuit_maintenance_parser/parsers/abcde.py`. This file will contain all the custom parsers needed for the provider and it will import the base classes for each parser type from `circuit_maintenance_parser.parser`. In the example, we only need to import `Html` and in the child class implement the methods required by the class, in this case `parse_html()` which will return a `dict` with all the data that this `Parser` can extract. In this case we have to helper methods, `_parse_bs` and `_parse_tables` that implement the logic to navigate the notification data. @@ -240,7 +240,7 @@ class HtmlParserABCDE1(Html): ... ``` -Next step is to create the new `Provider` by defining a new class in `circuit_maintenance_parser/provider.py`. This class that inherits from `GenericProvider` only needs to define to attributes: +Next step is to create the new `Provider` by defining a new class in `circuit_maintenance_parser/provider.py`. This class that inherits from `GenericProvider` only needs to define two attributes: - `_processors`: is a `list` of `Processor` instances that uses several data `Parsers`. In this example, we don't need to create a new custom `Processor` because the combined logic serves well (the most likely case), and we only need to use the new defined `HtmlParserABCDE1` and also the generic `EmailDateParser` that extract the email date. Also notice that you could have multiple `Processors` with different `Parsers` in this list, supporting several formats. - `_default_organizer`: this is a default helper to fill the `organizer` attribute in the `Maintenance` if the information is not part of the original notification. From 70c6c56e9ce020eb53aa7b936eaebd015bc99a0c Mon Sep 17 00:00:00 2001 From: Christian Adell Date: Sat, 25 Sep 2021 08:22:39 +0200 Subject: [PATCH 04/17] Add suggestions from Glenn's review --- README.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/README.md b/README.md index 1131ebfd..100f0747 100644 --- a/README.md +++ b/README.md @@ -40,7 +40,6 @@ You can leverage this library in your automation framework to process circuit ma 3. Each `Provider` uses one or more `Processors` that will be used to build `Maintenances` when the `Provider.get_maintenances(data)` method is called. 4. Each `Processor` class uses one or more `Parsers` to process each type of data that it handles. It can have custom logic to combine the parsed data from multiple `Parsers` to create the final `Maintenance` object. 5. Each `Parser` class supports one or a set of related data types, and implements the `Parser.parse()` method used to retrieve a `Dict` with the relevant keys/values. -6. When calling the `Provider.get_maintenances(data)`, the `data` argument is an instance of `NotificationData` that will be used by the corresponding `Parser` when the `Processor` will try to match them.

@@ -227,7 +226,7 @@ from bs4.element import ResultSet # type: ignore from circuit_maintenance_parser.parser import Html class HtmlParserABCDE1(Html): - def parse_html(self, soup) -> Dict: + def parse_html(self, soup: ResultSet) -> Dict: data = {} self._parse_bs(soup.find_all("b"), data) self._parse_tables(soup.find_all("table"), data) From 6f2370f5df60da20585c13d93b2f7f4de43448a6 Mon Sep 17 00:00:00 2001 From: carbonarok Date: Mon, 27 Sep 2021 14:22:56 +0100 Subject: [PATCH 05/17] Inital AWS parser. --- circuit_maintenance_parser/__init__.py | 2 + circuit_maintenance_parser/parser.py | 21 ++++++ circuit_maintenance_parser/parsers/aws.py | 75 +++++++++++++++++++ circuit_maintenance_parser/provider.py | 10 +++ tests/unit/data/aws/aws1.eml | 52 +++++++++++++ tests/unit/data/aws/aws1_result.json | 41 ++++++++++ .../data/aws/aws1_subject_parser_result.json | 5 ++ .../data/aws/aws1_text_parser_result.json | 35 +++++++++ tests/unit/test_e2e.py | 7 ++ tests/unit/test_parsers.py | 12 +++ 10 files changed, 260 insertions(+) create mode 100644 circuit_maintenance_parser/parsers/aws.py create mode 100644 tests/unit/data/aws/aws1.eml create mode 100644 tests/unit/data/aws/aws1_result.json create mode 100644 tests/unit/data/aws/aws1_subject_parser_result.json create mode 100644 tests/unit/data/aws/aws1_text_parser_result.json diff --git a/circuit_maintenance_parser/__init__.py b/circuit_maintenance_parser/__init__.py index 4c6fa685..c580844f 100644 --- a/circuit_maintenance_parser/__init__.py +++ b/circuit_maintenance_parser/__init__.py @@ -7,6 +7,7 @@ from .provider import ( GenericProvider, AquaComms, + AWS, Cogent, Colt, EUNetworks, @@ -29,6 +30,7 @@ SUPPORTED_PROVIDERS = ( GenericProvider, AquaComms, + AWS, Cogent, Colt, EUNetworks, diff --git a/circuit_maintenance_parser/parser.py b/circuit_maintenance_parser/parser.py index 998a87fb..0e0115d9 100644 --- a/circuit_maintenance_parser/parser.py +++ b/circuit_maintenance_parser/parser.py @@ -226,3 +226,24 @@ def parser_hook(self, raw: bytes): def parse_csv(raw: bytes) -> List[Dict]: """Custom CSV parsing.""" raise NotImplementedError + + +class Text(Parser): + """Html parser.""" + + _data_types = ["text/plain"] + + def parser_hook(self, raw: bytes): + """Execute parsing.""" + result = [] + soup = bs4.BeautifulSoup(quopri.decodestring(raw), features="lxml") + # Even we have not noticed any HTML notification with more than one maintenance yet, we define the + # return of `parse_html` as an Iterable object to accommodate this potential case. + for data in self.parse_text(soup.text): + result.append(data) + + return result + + def parse_text(self, text) -> List[Dict]: + """Custom text parsing.""" + raise NotImplementedError diff --git a/circuit_maintenance_parser/parsers/aws.py b/circuit_maintenance_parser/parsers/aws.py new file mode 100644 index 00000000..692e8194 --- /dev/null +++ b/circuit_maintenance_parser/parsers/aws.py @@ -0,0 +1,75 @@ +"""AquaComms parser.""" +import hashlib +import logging +import re + +from dateutil import parser + +from circuit_maintenance_parser.parser import CircuitImpact, EmailSubjectParser, Impact, Status, Text + +# pylint: disable=too-many-nested-blocks, too-many-branches + +logger = logging.getLogger(__name__) + + +class SubjectParserAWS1(EmailSubjectParser): + """Subject parser for AWS notifications.""" + + def parse_subject(self, subject): + """Parse subject. + + Example: AWS Direct Connect Planned Maintenance Notification [AWS Account: 00000001] + """ + data = {} + search = re.search(r"\[AWS Account ?I?D?: ([0-9]+)\]", subject) + if search: + data["account"] = search.group(1) + return [data] + + +class TextParserAWS1(Text): + """Parse text body of email.""" + + def parse_text(self, text): + """Parse text. + + Example: + Hello, + + Planned maintenance has been scheduled on an AWS Direct Connect router in A= + Block, New York, NY from Thu, 20 May 2021 08:00:00 GMT to Thu, 20 Ma= + y 2021 14:00:00 GMT for 6 hours. During this maintenance window, your AWS D= + irect Connect services listed below may become unavailable. + + aaaaa-00000001 + aaaaa-00000002 + aaaaa-00000003 + aaaaa-00000004 + aaaaa-00000005 + aaaaa-00000006 + + This maintenance is scheduled to avoid disrupting redundant connections at = + the same time. + """ + data = {"circuits": []} + impact = "OUTAGE" + maintenace_id = "" + for line in text.splitlines(): + if "planned maintenance" in line.lower(): + data["summary"] = line + search = re.search( + r"([A-Z][a-z]{2}, [0-9]{1,2} [A-Z][a-z]{2,9} [0-9]{4} [0-9]{2}:[0-9]{2}:[0-9]{2} [A-Z]{2,3}) to ([A-Z][a-z]{2}, [0-9]{1,2} [A-Z][a-z]{2,9} [0-9]{4} [0-9]{2}:[0-9]{2}:[0-9]{2} [A-Z]{2,3})", + line, + ) + if search: + data["start"] = self.dt2ts(parser.parse(search.group(1))) + maintenace_id += str(data["start"]) + data["end"] = self.dt2ts(parser.parse(search.group(2))) + if "may become unavailable" in line.lower(): + impact = "DEGRADED" + elif re.match(r"[a-z]{5}-[a-z0-9]{8}", line): + maintenace_id += line + data["circuits"].append(CircuitImpact(circuit_id=line, impact=Impact(impact))) + data["maintenance_id"] = hashlib.md5(maintenace_id.encode("utf-8")).hexdigest() + data["status"] = Status.CONFIRMED + return [data] diff --git a/circuit_maintenance_parser/provider.py b/circuit_maintenance_parser/provider.py index 5e111943..dd0b2f51 100644 --- a/circuit_maintenance_parser/provider.py +++ b/circuit_maintenance_parser/provider.py @@ -15,6 +15,7 @@ from circuit_maintenance_parser.processor import CombinedProcessor, SimpleProcessor, GenericProcessor from circuit_maintenance_parser.parsers.aquacomms import HtmlParserAquaComms1, SubjectParserAquaComms1 +from circuit_maintenance_parser.parsers.aws import SubjectParserAWS1, TextParserAWS1 from circuit_maintenance_parser.parsers.cogent import HtmlParserCogent1 from circuit_maintenance_parser.parsers.colt import ICalParserColt1, CsvParserColt1 from circuit_maintenance_parser.parsers.gtt import HtmlParserGTT1 @@ -116,6 +117,15 @@ class AquaComms(GenericProvider): _default_organizer = "tickets@aquacomms.com" +class AWS(GenericProvider): + """AWS provider custom class.""" + + _processors: List[GenericProcessor] = [ + CombinedProcessor(data_parsers=[EmailDateParser, TextParserAWS1, SubjectParserAWS1]), + ] + _default_organizer = "aaaaaaaaaaaaaaa" + + class Cogent(GenericProvider): """Cogent provider custom class.""" diff --git a/tests/unit/data/aws/aws1.eml b/tests/unit/data/aws/aws1.eml new file mode 100644 index 00000000..585ad49b --- /dev/null +++ b/tests/unit/data/aws/aws1.eml @@ -0,0 +1,52 @@ +Subject: [rCluster Request] [rCloud AWS Notification] AWS Direct Connect + Planned Maintenance Notification [AWS Account: 0000000000001] +MIME-Version: 1.0 +Content-Type: text/plain; charset="UTF-8" +Content-Transfer-Encoding: quoted-printable +X-SM-COMMUNICATION: true +X-SM-COMMUNICATION-TYPE: AWS_DIRECTCONNECT_MAINTENANCE_SCHEDULED +X-SM-DEDUPING-ID: 7cc8bab7-00bb-44e0-a3ec-bdd1a5560b80-EMAIL--1012261942-036424c1a19ca69ca7ea459ebd6823e1 +Date: Thu, 6 May 2021 21:52:56 +0000 +Feedback-ID: 1.us-east-1.xvKJ2gIiw98/SnInpbS9SQT1XBoAzwrySbDsqgMkBQI=:AmazonSES +X-SES-Outgoing: 2021.05.06-54.240.48.83 +X-Original-Sender: no-reply-aws@amazon.com +X-Original-Authentication-Results: mx.google.com; dkim=pass + header.i=@amazon.com header.s=szqgv33erturdv5cvz4vtb5qcy53gdkn + header.b=IQc0x0aC; dkim=pass header.i=@amazonses.com + header.s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug header.b=X4gZtDlT; spf=pass + (google.com: domain of 0100017943ab6519-f09ba161-049c-45e4-8ff3-698af4d94f86-000000@amazonses.com + designates 54.240.48.83 as permitted sender) smtp.mailfrom=0100017943ab6519-f09ba161-049c-45e4-8ff3-698af4d94f86-000000@amazonses.com; + dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.com +X-Original-From: "Amazon Web Services, Inc." +Reply-To: "Amazon Web Services, Inc." +Precedence: list + +Hello, + +Planned maintenance has been scheduled on an AWS Direct Connect router in A= + Block, New York, NY from Thu, 20 May 2021 08:00:00 GMT to Thu, 20 Ma= +y 2021 14:00:00 GMT for 6 hours. During this maintenance window, your AWS D= +irect Connect services listed below may become unavailable. + +aaaaa-00000001 +aaaaa-00000002 +aaaaa-00000003 +aaaaa-00000004 +aaaaa-00000005 +aaaaa-00000006 + +This maintenance is scheduled to avoid disrupting redundant connections at = +the same time. + +If you encounter any problems with your connection after the end of this ma= +intenance window, please contact AWS Support[1]. + +[1] https://aws.amazon.com/support + +Sincerely, +Amazon Web Services + +Amazon Web Services, Inc. is a subsidiary of Amazon.com, Inc. Amazon.com is= + a registered trademark of Amazon.com, Inc. This message was produced and d= +istributed by Amazon Web Services Inc., 410 Terry Ave. North, Seattle, WA 9= +8109-5210. diff --git a/tests/unit/data/aws/aws1_result.json b/tests/unit/data/aws/aws1_result.json new file mode 100644 index 00000000..92284bcd --- /dev/null +++ b/tests/unit/data/aws/aws1_result.json @@ -0,0 +1,41 @@ +[ + { + "account": "0000000000001", + "circuits": [ + { + "circuit_id": "aaaaa-00000001", + "impact": "DEGRADED" + }, + { + "circuit_id": "aaaaa-00000002", + "impact": "DEGRADED" + }, + { + "circuit_id": "aaaaa-00000003", + "impact": "DEGRADED" + }, + { + "circuit_id": "aaaaa-00000004", + "impact": "DEGRADED" + }, + { + "circuit_id": "aaaaa-00000005", + "impact": "DEGRADED" + }, + { + "circuit_id": "aaaaa-00000006", + "impact": "DEGRADED" + } + ], + "end": 1621519200, + "maintenance_id": "21264c558a1b6a2bedaae0878d6318b0", + "organizer": "aaaaaaaaaaaaaaa", + "provider": "aws", + "sequence": 1, + "stamp": 1620337976, + "start": 1621497600, + "status": "CONFIRMED", + "summary": "Planned maintenance has been scheduled on an AWS Direct Connect router in A Block, New York, NY from Thu, 20 May 2021 08:00:00 GMT to Thu, 20 May 2021 14:00:00 GMT for 6 hours. During this maintenance window, your AWS Direct Connect services listed below may become unavailable.", + "uid": "0" + } +] \ No newline at end of file diff --git a/tests/unit/data/aws/aws1_subject_parser_result.json b/tests/unit/data/aws/aws1_subject_parser_result.json new file mode 100644 index 00000000..e9a667e6 --- /dev/null +++ b/tests/unit/data/aws/aws1_subject_parser_result.json @@ -0,0 +1,5 @@ +[ + { + "account": "0000000000001" + } +] \ No newline at end of file diff --git a/tests/unit/data/aws/aws1_text_parser_result.json b/tests/unit/data/aws/aws1_text_parser_result.json new file mode 100644 index 00000000..d1fa4d6a --- /dev/null +++ b/tests/unit/data/aws/aws1_text_parser_result.json @@ -0,0 +1,35 @@ +[ + { + "circuits": [ + { + "circuit_id": "aaaaa-00000001", + "impact": "DEGRADED" + }, + { + "circuit_id": "aaaaa-00000002", + "impact": "DEGRADED" + }, + { + "circuit_id": "aaaaa-00000003", + "impact": "DEGRADED" + }, + { + "circuit_id": "aaaaa-00000004", + "impact": "DEGRADED" + }, + { + "circuit_id": "aaaaa-00000005", + "impact": "DEGRADED" + }, + { + "circuit_id": "aaaaa-00000006", + "impact": "DEGRADED" + } + ], + "end": 1621519200, + "maintenance_id": "21264c558a1b6a2bedaae0878d6318b0", + "start": 1621497600, + "status": "CONFIRMED", + "summary": "Planned maintenance has been scheduled on an AWS Direct Connect router in A Block, New York, NY from Thu, 20 May 2021 08:00:00 GMT to Thu, 20 May 2021 14:00:00 GMT for 6 hours. During this maintenance window, your AWS Direct Connect services listed below may become unavailable." + } +] \ No newline at end of file diff --git a/tests/unit/test_e2e.py b/tests/unit/test_e2e.py index 010bc8ff..98c71107 100644 --- a/tests/unit/test_e2e.py +++ b/tests/unit/test_e2e.py @@ -13,6 +13,7 @@ from circuit_maintenance_parser.provider import ( GenericProvider, AquaComms, + AWS, Cogent, Colt, EUNetworks, @@ -48,6 +49,12 @@ [("email", Path(dir_path, "data", "aquacomms", "aquacomms1.eml")),], [Path(dir_path, "data", "aquacomms", "aquacomms1_result.json"),], ), + # AWS + ( + AWS, + [("email", Path(dir_path, "data", "aws", "aws1.eml")),], + [Path(dir_path, "data", "aws", "aws1_result.json"),], + ), # Cogent ( Cogent, diff --git a/tests/unit/test_parsers.py b/tests/unit/test_parsers.py index 53009897..f7684e38 100644 --- a/tests/unit/test_parsers.py +++ b/tests/unit/test_parsers.py @@ -8,6 +8,7 @@ from circuit_maintenance_parser.errors import ParserError from circuit_maintenance_parser.parser import ICal, EmailDateParser from circuit_maintenance_parser.parsers.aquacomms import HtmlParserAquaComms1, SubjectParserAquaComms1 +from circuit_maintenance_parser.parsers.aws import SubjectParserAWS1, TextParserAWS1 from circuit_maintenance_parser.parsers.cogent import HtmlParserCogent1 from circuit_maintenance_parser.parsers.colt import ICalParserColt1, CsvParserColt1 from circuit_maintenance_parser.parsers.gtt import HtmlParserGTT1 @@ -51,6 +52,17 @@ Path(dir_path, "data", "aquacomms", "aquacomms1.eml"), Path(dir_path, "data", "aquacomms", "aquacomms1_subject_parser_result.json"), ), + # AWS + ( + TextParserAWS1, + Path(dir_path, "data", "aws", "aws1.eml"), + Path(dir_path, "data", "aws", "aws1_text_parser_result.json"), + ), + ( + SubjectParserAWS1, + Path(dir_path, "data", "aws", "aws1.eml"), + Path(dir_path, "data", "aws", "aws1_subject_parser_result.json"), + ), # Cogent ( HtmlParserCogent1, From 70d0fda6c8a676f4632a712ecda4d794e1bf3604 Mon Sep 17 00:00:00 2001 From: carbonarok Date: Mon, 27 Sep 2021 14:29:14 +0100 Subject: [PATCH 06/17] Added bandit skip. --- circuit_maintenance_parser/parsers/aws.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/circuit_maintenance_parser/parsers/aws.py b/circuit_maintenance_parser/parsers/aws.py index 692e8194..0483ca6f 100644 --- a/circuit_maintenance_parser/parsers/aws.py +++ b/circuit_maintenance_parser/parsers/aws.py @@ -70,6 +70,6 @@ def parse_text(self, text): elif re.match(r"[a-z]{5}-[a-z0-9]{8}", line): maintenace_id += line data["circuits"].append(CircuitImpact(circuit_id=line, impact=Impact(impact))) - data["maintenance_id"] = hashlib.md5(maintenace_id.encode("utf-8")).hexdigest() + data["maintenance_id"] = hashlib.md5(maintenace_id.encode("utf-8")).hexdigest() # nosec data["status"] = Status.CONFIRMED return [data] From 7a35a8e760f56652217d4e200a29486ffddc9a3c Mon Sep 17 00:00:00 2001 From: carbonarok Date: Tue, 28 Sep 2021 13:11:08 +0100 Subject: [PATCH 07/17] Added support for cancelled notifications. Minor fixes to original parser --- circuit_maintenance_parser/parser.py | 1 - circuit_maintenance_parser/parsers/aws.py | 15 ++++--- circuit_maintenance_parser/provider.py | 2 +- tests/unit/data/aws/aws1_result.json | 16 +++---- .../data/aws/aws1_text_parser_result.json | 14 +++--- tests/unit/data/aws/aws2.eml | 45 +++++++++++++++++++ tests/unit/data/aws/aws2_result.json | 41 +++++++++++++++++ .../data/aws/aws2_subject_parser_result.json | 5 +++ .../data/aws/aws2_text_parser_result.json | 35 +++++++++++++++ tests/unit/test_e2e.py | 5 +++ tests/unit/test_parsers.py | 10 +++++ 11 files changed, 167 insertions(+), 22 deletions(-) create mode 100644 tests/unit/data/aws/aws2.eml create mode 100644 tests/unit/data/aws/aws2_result.json create mode 100644 tests/unit/data/aws/aws2_subject_parser_result.json create mode 100644 tests/unit/data/aws/aws2_text_parser_result.json diff --git a/circuit_maintenance_parser/parser.py b/circuit_maintenance_parser/parser.py index 0e0115d9..6a7e6d97 100644 --- a/circuit_maintenance_parser/parser.py +++ b/circuit_maintenance_parser/parser.py @@ -241,7 +241,6 @@ def parser_hook(self, raw: bytes): # return of `parse_html` as an Iterable object to accommodate this potential case. for data in self.parse_text(soup.text): result.append(data) - return result def parse_text(self, text) -> List[Dict]: diff --git a/circuit_maintenance_parser/parsers/aws.py b/circuit_maintenance_parser/parsers/aws.py index 0483ca6f..d626cc22 100644 --- a/circuit_maintenance_parser/parsers/aws.py +++ b/circuit_maintenance_parser/parsers/aws.py @@ -52,8 +52,9 @@ def parse_text(self, text): the same time. """ data = {"circuits": []} - impact = "OUTAGE" + impact = Impact.OUTAGE maintenace_id = "" + status = Status.CONFIRMED for line in text.splitlines(): if "planned maintenance" in line.lower(): data["summary"] = line @@ -63,13 +64,17 @@ def parse_text(self, text): ) if search: data["start"] = self.dt2ts(parser.parse(search.group(1))) - maintenace_id += str(data["start"]) data["end"] = self.dt2ts(parser.parse(search.group(2))) + maintenace_id += str(data["start"]) + maintenace_id += str(data["end"]) if "may become unavailable" in line.lower(): - impact = "DEGRADED" + impact = Impact.OUTAGE + elif "has been cancelled" in line.lower(): + impact = Impact.NO_IMPACT + status = Status.CANCELLED elif re.match(r"[a-z]{5}-[a-z0-9]{8}", line): maintenace_id += line - data["circuits"].append(CircuitImpact(circuit_id=line, impact=Impact(impact))) + data["circuits"].append(CircuitImpact(circuit_id=line, impact=impact)) data["maintenance_id"] = hashlib.md5(maintenace_id.encode("utf-8")).hexdigest() # nosec - data["status"] = Status.CONFIRMED + data["status"] = status return [data] diff --git a/circuit_maintenance_parser/provider.py b/circuit_maintenance_parser/provider.py index dd0b2f51..715cfe7b 100644 --- a/circuit_maintenance_parser/provider.py +++ b/circuit_maintenance_parser/provider.py @@ -123,7 +123,7 @@ class AWS(GenericProvider): _processors: List[GenericProcessor] = [ CombinedProcessor(data_parsers=[EmailDateParser, TextParserAWS1, SubjectParserAWS1]), ] - _default_organizer = "aaaaaaaaaaaaaaa" + _default_organizer = "aws-account-notifications@amazon.com" class Cogent(GenericProvider): diff --git a/tests/unit/data/aws/aws1_result.json b/tests/unit/data/aws/aws1_result.json index 92284bcd..1ed78df7 100644 --- a/tests/unit/data/aws/aws1_result.json +++ b/tests/unit/data/aws/aws1_result.json @@ -4,32 +4,32 @@ "circuits": [ { "circuit_id": "aaaaa-00000001", - "impact": "DEGRADED" + "impact": "OUTAGE" }, { "circuit_id": "aaaaa-00000002", - "impact": "DEGRADED" + "impact": "OUTAGE" }, { "circuit_id": "aaaaa-00000003", - "impact": "DEGRADED" + "impact": "OUTAGE" }, { "circuit_id": "aaaaa-00000004", - "impact": "DEGRADED" + "impact": "OUTAGE" }, { "circuit_id": "aaaaa-00000005", - "impact": "DEGRADED" + "impact": "OUTAGE" }, { "circuit_id": "aaaaa-00000006", - "impact": "DEGRADED" + "impact": "OUTAGE" } ], "end": 1621519200, - "maintenance_id": "21264c558a1b6a2bedaae0878d6318b0", - "organizer": "aaaaaaaaaaaaaaa", + "maintenance_id": "15faf02fcf2e999792668df97828bc76", + "organizer": "aws-account-notifications@amazon.com", "provider": "aws", "sequence": 1, "stamp": 1620337976, diff --git a/tests/unit/data/aws/aws1_text_parser_result.json b/tests/unit/data/aws/aws1_text_parser_result.json index d1fa4d6a..81397864 100644 --- a/tests/unit/data/aws/aws1_text_parser_result.json +++ b/tests/unit/data/aws/aws1_text_parser_result.json @@ -3,31 +3,31 @@ "circuits": [ { "circuit_id": "aaaaa-00000001", - "impact": "DEGRADED" + "impact": "OUTAGE" }, { "circuit_id": "aaaaa-00000002", - "impact": "DEGRADED" + "impact": "OUTAGE" }, { "circuit_id": "aaaaa-00000003", - "impact": "DEGRADED" + "impact": "OUTAGE" }, { "circuit_id": "aaaaa-00000004", - "impact": "DEGRADED" + "impact": "OUTAGE" }, { "circuit_id": "aaaaa-00000005", - "impact": "DEGRADED" + "impact": "OUTAGE" }, { "circuit_id": "aaaaa-00000006", - "impact": "DEGRADED" + "impact": "OUTAGE" } ], "end": 1621519200, - "maintenance_id": "21264c558a1b6a2bedaae0878d6318b0", + "maintenance_id": "15faf02fcf2e999792668df97828bc76", "start": 1621497600, "status": "CONFIRMED", "summary": "Planned maintenance has been scheduled on an AWS Direct Connect router in A Block, New York, NY from Thu, 20 May 2021 08:00:00 GMT to Thu, 20 May 2021 14:00:00 GMT for 6 hours. During this maintenance window, your AWS Direct Connect services listed below may become unavailable." diff --git a/tests/unit/data/aws/aws2.eml b/tests/unit/data/aws/aws2.eml new file mode 100644 index 00000000..9eda21c2 --- /dev/null +++ b/tests/unit/data/aws/aws2.eml @@ -0,0 +1,45 @@ +Subject: [rCluster Request] [rCloud AWS Notification] AWS Direct Connect + Planned Maintenance Notification [AWS Account: 0000000000001] +MIME-Version: 1.0 +Content-Type: text/plain; charset="UTF-8" +Content-Transfer-Encoding: quoted-printable +X-SM-COMMUNICATION: true +X-SM-COMMUNICATION-TYPE: AWS_DIRECTCONNECT_MAINTENANCE_SCHEDULED +X-SM-DEDUPING-ID: 7cc8bab7-00bb-44e0-a3ec-bdd1a5560b80-EMAIL--1012261942-036424c1a19ca69ca7ea459ebd6823e1 +Date: Thu, 6 May 2021 21:52:56 +0000 +Feedback-ID: 1.us-east-1.xvKJ2gIiw98/SnInpbS9SQT1XBoAzwrySbDsqgMkBQI=:AmazonSES +X-SES-Outgoing: 2021.05.06-54.240.48.83 +X-Original-Sender: no-reply-aws@amazon.com +X-Original-Authentication-Results: mx.google.com; dkim=pass + header.i=@amazon.com header.s=szqgv33erturdv5cvz4vtb5qcy53gdkn + header.b=IQc0x0aC; dkim=pass header.i=@amazonses.com + header.s=ug7nbtf4gccmlpwj322ax3p6ow6yfsug header.b=X4gZtDlT; spf=pass + (google.com: domain of 0100017943ab6519-f09ba161-049c-45e4-8ff3-698af4d94f86-000000@amazonses.com + designates 54.240.48.83 as permitted sender) smtp.mailfrom=0100017943ab6519-f09ba161-049c-45e4-8ff3-698af4d94f86-000000@amazonses.com; + dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.com +X-Original-From: "Amazon Web Services, Inc." +Reply-To: "Amazon Web Services, Inc." +Precedence: list + +Hello, + +We would like to inform you that the planned maintenance that was scheduled for AWS Direct Connect endpoint in Equinix SG2, Singapore, SGP from Mon, 13 Sep 2021 19:02:00 GMT to Tue, 14 Sep 2021 02:02:00 GMT has been cancelled. Please find below your AWS Direct Connect services that would have been affected by this planned maintenance. + +aaaaa-00000001 +aaaaa-00000002 +aaaaa-00000003 +aaaaa-00000004 +aaaaa-00000005 +aaaaa-00000006 + +We sincerely regret any inconvenience our planning process may have caused you. If you have any questions regarding the planned work, or if you would like to report a fault or adjust your contact information, please contact AWS Support[1]. + +[1] https://aws.amazon.com/support + +Sincerely, +Amazon Web Services + +Amazon Web Services, Inc. is a subsidiary of Amazon.com, Inc. Amazon.com is= + a registered trademark of Amazon.com, Inc. This message was produced and d= +istributed by Amazon Web Services Inc., 410 Terry Ave. North, Seattle, WA 9= +8109-5210. diff --git a/tests/unit/data/aws/aws2_result.json b/tests/unit/data/aws/aws2_result.json new file mode 100644 index 00000000..6e5de3ad --- /dev/null +++ b/tests/unit/data/aws/aws2_result.json @@ -0,0 +1,41 @@ +[ + { + "account": "0000000000001", + "circuits": [ + { + "circuit_id": "aaaaa-00000001", + "impact": "NO-IMPACT" + }, + { + "circuit_id": "aaaaa-00000002", + "impact": "NO-IMPACT" + }, + { + "circuit_id": "aaaaa-00000003", + "impact": "NO-IMPACT" + }, + { + "circuit_id": "aaaaa-00000004", + "impact": "NO-IMPACT" + }, + { + "circuit_id": "aaaaa-00000005", + "impact": "NO-IMPACT" + }, + { + "circuit_id": "aaaaa-00000006", + "impact": "NO-IMPACT" + } + ], + "end": 1631584920, + "maintenance_id": "47876b7d5a5198643a1a9cb7f954487a", + "organizer": "aws-account-notifications@amazon.com", + "provider": "aws", + "sequence": 1, + "stamp": 1620337976, + "start": 1631559720, + "status": "CANCELLED", + "summary": "We would like to inform you that the planned maintenance that was scheduled for AWS Direct Connect endpoint in Equinix SG2, Singapore, SGP from Mon, 13 Sep 2021 19:02:00 GMT to Tue, 14 Sep 2021 02:02:00 GMT has been cancelled. Please find below your AWS Direct Connect services that would have been affected by this planned maintenance.", + "uid": "0" + } +] \ No newline at end of file diff --git a/tests/unit/data/aws/aws2_subject_parser_result.json b/tests/unit/data/aws/aws2_subject_parser_result.json new file mode 100644 index 00000000..e9a667e6 --- /dev/null +++ b/tests/unit/data/aws/aws2_subject_parser_result.json @@ -0,0 +1,5 @@ +[ + { + "account": "0000000000001" + } +] \ No newline at end of file diff --git a/tests/unit/data/aws/aws2_text_parser_result.json b/tests/unit/data/aws/aws2_text_parser_result.json new file mode 100644 index 00000000..6ba26a40 --- /dev/null +++ b/tests/unit/data/aws/aws2_text_parser_result.json @@ -0,0 +1,35 @@ +[ + { + "circuits": [ + { + "circuit_id": "aaaaa-00000001", + "impact": "NO-IMPACT" + }, + { + "circuit_id": "aaaaa-00000002", + "impact": "NO-IMPACT" + }, + { + "circuit_id": "aaaaa-00000003", + "impact": "NO-IMPACT" + }, + { + "circuit_id": "aaaaa-00000004", + "impact": "NO-IMPACT" + }, + { + "circuit_id": "aaaaa-00000005", + "impact": "NO-IMPACT" + }, + { + "circuit_id": "aaaaa-00000006", + "impact": "NO-IMPACT" + } + ], + "end": 1631584920, + "maintenance_id": "47876b7d5a5198643a1a9cb7f954487a", + "start": 1631559720, + "status": "CANCELLED", + "summary": "We would like to inform you that the planned maintenance that was scheduled for AWS Direct Connect endpoint in Equinix SG2, Singapore, SGP from Mon, 13 Sep 2021 19:02:00 GMT to Tue, 14 Sep 2021 02:02:00 GMT has been cancelled. Please find below your AWS Direct Connect services that would have been affected by this planned maintenance." + } +] \ No newline at end of file diff --git a/tests/unit/test_e2e.py b/tests/unit/test_e2e.py index 98c71107..bc37ec12 100644 --- a/tests/unit/test_e2e.py +++ b/tests/unit/test_e2e.py @@ -55,6 +55,11 @@ [("email", Path(dir_path, "data", "aws", "aws1.eml")),], [Path(dir_path, "data", "aws", "aws1_result.json"),], ), + ( + AWS, + [("email", Path(dir_path, "data", "aws", "aws2.eml")),], + [Path(dir_path, "data", "aws", "aws2_result.json"),], + ), # Cogent ( Cogent, diff --git a/tests/unit/test_parsers.py b/tests/unit/test_parsers.py index f7684e38..fb71bfae 100644 --- a/tests/unit/test_parsers.py +++ b/tests/unit/test_parsers.py @@ -63,6 +63,16 @@ Path(dir_path, "data", "aws", "aws1.eml"), Path(dir_path, "data", "aws", "aws1_subject_parser_result.json"), ), + ( + TextParserAWS1, + Path(dir_path, "data", "aws", "aws2.eml"), + Path(dir_path, "data", "aws", "aws2_text_parser_result.json"), + ), + ( + SubjectParserAWS1, + Path(dir_path, "data", "aws", "aws2.eml"), + Path(dir_path, "data", "aws", "aws2_subject_parser_result.json"), + ), # Cogent ( HtmlParserCogent1, From 4a164f23cc71fde26b8350cb466e944574543dab Mon Sep 17 00:00:00 2001 From: carbonarok Date: Tue, 28 Sep 2021 14:33:50 +0100 Subject: [PATCH 08/17] Added new layer of absraction --- circuit_maintenance_parser/parser.py | 13 ++++++++----- circuit_maintenance_parser/parsers/aws.py | 9 +++++++++ 2 files changed, 17 insertions(+), 5 deletions(-) diff --git a/circuit_maintenance_parser/parser.py b/circuit_maintenance_parser/parser.py index 6a7e6d97..4790ca67 100644 --- a/circuit_maintenance_parser/parser.py +++ b/circuit_maintenance_parser/parser.py @@ -229,20 +229,23 @@ def parse_csv(raw: bytes) -> List[Dict]: class Text(Parser): - """Html parser.""" + """Text parser.""" _data_types = ["text/plain"] def parser_hook(self, raw: bytes): """Execute parsing.""" result = [] - soup = bs4.BeautifulSoup(quopri.decodestring(raw), features="lxml") - # Even we have not noticed any HTML notification with more than one maintenance yet, we define the - # return of `parse_html` as an Iterable object to accommodate this potential case. - for data in self.parse_text(soup.text): + text = self.get_text_hook(raw) + for data in self.parse_text(text): result.append(data) return result + @staticmethod + def get_text_hook(raw): + """Can be overwritten by subclasses.""" + return raw.decode() + def parse_text(self, text) -> List[Dict]: """Custom text parsing.""" raise NotImplementedError diff --git a/circuit_maintenance_parser/parsers/aws.py b/circuit_maintenance_parser/parsers/aws.py index d626cc22..85192d67 100644 --- a/circuit_maintenance_parser/parsers/aws.py +++ b/circuit_maintenance_parser/parsers/aws.py @@ -1,8 +1,11 @@ """AquaComms parser.""" import hashlib import logging +import quopri import re +import bs4 + from dateutil import parser from circuit_maintenance_parser.parser import CircuitImpact, EmailSubjectParser, Impact, Status, Text @@ -30,6 +33,12 @@ def parse_subject(self, subject): class TextParserAWS1(Text): """Parse text body of email.""" + @staticmethod + def get_text_hook(raw): + """Modify soup before entering `parse_text`.""" + soup = bs4.BeautifulSoup(quopri.decodestring(raw), features="lxml") + return soup.text + def parse_text(self, text): """Parse text. From 10c46a3f7b235f9d3e1482b85794cfd738b98110 Mon Sep 17 00:00:00 2001 From: carbonarok Date: Tue, 28 Sep 2021 14:42:27 +0100 Subject: [PATCH 09/17] Removed impact change for cancelled job --- circuit_maintenance_parser/parsers/aws.py | 1 - 1 file changed, 1 deletion(-) diff --git a/circuit_maintenance_parser/parsers/aws.py b/circuit_maintenance_parser/parsers/aws.py index 85192d67..ca3beb1f 100644 --- a/circuit_maintenance_parser/parsers/aws.py +++ b/circuit_maintenance_parser/parsers/aws.py @@ -79,7 +79,6 @@ def parse_text(self, text): if "may become unavailable" in line.lower(): impact = Impact.OUTAGE elif "has been cancelled" in line.lower(): - impact = Impact.NO_IMPACT status = Status.CANCELLED elif re.match(r"[a-z]{5}-[a-z0-9]{8}", line): maintenace_id += line From ab09d4959f6676d1ed8331bda02f79498fadc1c9 Mon Sep 17 00:00:00 2001 From: carbonarok Date: Tue, 28 Sep 2021 14:46:23 +0100 Subject: [PATCH 10/17] Fixed lintting error. --- circuit_maintenance_parser/parsers/aws.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/circuit_maintenance_parser/parsers/aws.py b/circuit_maintenance_parser/parsers/aws.py index ca3beb1f..539e9d9e 100644 --- a/circuit_maintenance_parser/parsers/aws.py +++ b/circuit_maintenance_parser/parsers/aws.py @@ -4,7 +4,7 @@ import quopri import re -import bs4 +import bs4 # type: ignore from dateutil import parser From 12beefa1b26624067b84a113cc5f06031992eb10 Mon Sep 17 00:00:00 2001 From: carbonarok Date: Tue, 28 Sep 2021 14:52:04 +0100 Subject: [PATCH 11/17] Fixed testing --- tests/unit/data/aws/aws2_result.json | 12 ++++++------ tests/unit/data/aws/aws2_text_parser_result.json | 12 ++++++------ 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/tests/unit/data/aws/aws2_result.json b/tests/unit/data/aws/aws2_result.json index 6e5de3ad..4e6a5c1f 100644 --- a/tests/unit/data/aws/aws2_result.json +++ b/tests/unit/data/aws/aws2_result.json @@ -4,27 +4,27 @@ "circuits": [ { "circuit_id": "aaaaa-00000001", - "impact": "NO-IMPACT" + "impact": "OUTAGE" }, { "circuit_id": "aaaaa-00000002", - "impact": "NO-IMPACT" + "impact": "OUTAGE" }, { "circuit_id": "aaaaa-00000003", - "impact": "NO-IMPACT" + "impact": "OUTAGE" }, { "circuit_id": "aaaaa-00000004", - "impact": "NO-IMPACT" + "impact": "OUTAGE" }, { "circuit_id": "aaaaa-00000005", - "impact": "NO-IMPACT" + "impact": "OUTAGE" }, { "circuit_id": "aaaaa-00000006", - "impact": "NO-IMPACT" + "impact": "OUTAGE" } ], "end": 1631584920, diff --git a/tests/unit/data/aws/aws2_text_parser_result.json b/tests/unit/data/aws/aws2_text_parser_result.json index 6ba26a40..a9e8523d 100644 --- a/tests/unit/data/aws/aws2_text_parser_result.json +++ b/tests/unit/data/aws/aws2_text_parser_result.json @@ -3,27 +3,27 @@ "circuits": [ { "circuit_id": "aaaaa-00000001", - "impact": "NO-IMPACT" + "impact": "OUTAGE" }, { "circuit_id": "aaaaa-00000002", - "impact": "NO-IMPACT" + "impact": "OUTAGE" }, { "circuit_id": "aaaaa-00000003", - "impact": "NO-IMPACT" + "impact": "OUTAGE" }, { "circuit_id": "aaaaa-00000004", - "impact": "NO-IMPACT" + "impact": "OUTAGE" }, { "circuit_id": "aaaaa-00000005", - "impact": "NO-IMPACT" + "impact": "OUTAGE" }, { "circuit_id": "aaaaa-00000006", - "impact": "NO-IMPACT" + "impact": "OUTAGE" } ], "end": 1631584920, From 40be9fec6cab2c64be5ea190866498a0a027a638 Mon Sep 17 00:00:00 2001 From: carbonarok Date: Tue, 28 Sep 2021 16:16:40 +0100 Subject: [PATCH 12/17] Added function definition --- circuit_maintenance_parser/parser.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/circuit_maintenance_parser/parser.py b/circuit_maintenance_parser/parser.py index 4790ca67..1428024d 100644 --- a/circuit_maintenance_parser/parser.py +++ b/circuit_maintenance_parser/parser.py @@ -242,7 +242,7 @@ def parser_hook(self, raw: bytes): return result @staticmethod - def get_text_hook(raw): + def get_text_hook(raw: bytes) -> str: """Can be overwritten by subclasses.""" return raw.decode() From ccd1f72f60206fbf626de5c3b14c6dd4b7b86338 Mon Sep 17 00:00:00 2001 From: carbonarok Date: Tue, 28 Sep 2021 16:38:46 +0100 Subject: [PATCH 13/17] Modifications to readme and changelog. Added comment for hash. --- CHANGELOG.md | 4 ++++ README.md | 1 + circuit_maintenance_parser/parsers/aws.py | 2 ++ 3 files changed, 7 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 674f2429..d5ac3d56 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,10 @@ - #86 - Fix `CombinedProcessor` carries over data from previous parsing +### Added + +- #84 - New parser added for text. Added new provider `AquaComms` using `Text` and `EmailSubjectParser` + ## v2.0.1 - 2021-09-16 ### Fixed diff --git a/README.md b/README.md index 57ce40b3..fef6e41a 100644 --- a/README.md +++ b/README.md @@ -43,6 +43,7 @@ By default, there is a `GenericProvider` that support a `SimpleProcessor` using #### Supported providers based on other parsers +- AWS - AquaComms - Cogent - Colt diff --git a/circuit_maintenance_parser/parsers/aws.py b/circuit_maintenance_parser/parsers/aws.py index 539e9d9e..00be0b8c 100644 --- a/circuit_maintenance_parser/parsers/aws.py +++ b/circuit_maintenance_parser/parsers/aws.py @@ -83,6 +83,8 @@ def parse_text(self, text): elif re.match(r"[a-z]{5}-[a-z0-9]{8}", line): maintenace_id += line data["circuits"].append(CircuitImpact(circuit_id=line, impact=impact)) + # No maintenance ID found in emails, so a hash value is being generated using the start, + # end and IDs of all circuits in the notification. data["maintenance_id"] = hashlib.md5(maintenace_id.encode("utf-8")).hexdigest() # nosec data["status"] = status return [data] From 08d02347c817f7c5f56587258ae746bee5770d00 Mon Sep 17 00:00:00 2001 From: Glenn Matthews Date: Tue, 28 Sep 2021 14:10:27 -0400 Subject: [PATCH 14/17] Handle more potential Lumen statuses seen in the field --- circuit_maintenance_parser/parsers/lumen.py | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/circuit_maintenance_parser/parsers/lumen.py b/circuit_maintenance_parser/parsers/lumen.py index 9d7e8a88..7c0de7ad 100644 --- a/circuit_maintenance_parser/parsers/lumen.py +++ b/circuit_maintenance_parser/parsers/lumen.py @@ -6,7 +6,7 @@ import bs4 # type: ignore from bs4.element import ResultSet # type: ignore -from circuit_maintenance_parser.parser import Html, Impact, CircuitImpact, Status +from circuit_maintenance_parser.parser import CircuitImpact, Html, Impact, Status # pylint: disable=too-many-nested-blocks, too-many-branches @@ -88,8 +88,16 @@ def parse_tables(self, tables: ResultSet, data: Dict): if "account" not in data: data["account"] = cells[idx].string if num_columns == 10: - if cells[idx + 9].string == "Completed": + status_string = cells[idx + 9].string + if status_string == "Completed": data["status"] = Status("COMPLETED") + elif status_string == "Postponed": + data["status"] = Status("RE-SCHEDULED") + elif status_string == "Not Completed": + data["status"] = Status("CANCELLED") + elif "status" not in data: + # Update to an existing ticket may not include an update to the status - make a guess + data["status"] = "CONFIRMED" data_circuit = {} data_circuit["circuit_id"] = cells[idx + 1].string From ce4173c58edd2a8666c51d4716b37e9cd209bd5f Mon Sep 17 00:00:00 2001 From: Christian Adell Date: Fri, 1 Oct 2021 14:57:32 +0200 Subject: [PATCH 15/17] Provider implements a include filter to define relevant notifications (#91) * Provider implements a include and exlcude filter to define relevant notifications --- CHANGELOG.md | 6 +++ README.md | 3 +- circuit_maintenance_parser/constants.py | 4 ++ circuit_maintenance_parser/data.py | 7 +-- circuit_maintenance_parser/parser.py | 5 +- circuit_maintenance_parser/provider.py | 57 +++++++++++++++++++++- tests/unit/data/lumen/subject_work_planned | 1 + tests/unit/test_e2e.py | 36 ++++++++------ tests/unit/test_providers.py | 50 +++++++++++++++++++ 9 files changed, 146 insertions(+), 23 deletions(-) create mode 100644 circuit_maintenance_parser/constants.py create mode 100644 tests/unit/data/lumen/subject_work_planned diff --git a/CHANGELOG.md b/CHANGELOG.md index d5ac3d56..0dc8d2f6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,11 @@ # Changelog +## v2.0.3 + +### Added + +- #91 - `Provider` now adds `_include_filter` and `_exclude_filter` attributes (using regex) to filter in and out notifications that are relevant to be parsed vs other that are not, avoiding false positives. + ## v2.0.2 - 2021-09-28 ### Fixed diff --git a/README.md b/README.md index fef6e41a..2f914955 100644 --- a/README.md +++ b/README.md @@ -184,7 +184,7 @@ Circuit Maintenance Notification #0 circuit-maintenance-parser --data-file "/tmp/___ZAYO TTN-00000000 Planned MAINTENANCE NOTIFICATION___.eml" --data-type email --provider-type zayo Circuit Maintenance Notification #0 { - "account": "Linode", + "account": "some account", "circuits": [ { "circuit_id": "/OGYX/000000/ /ZYO /", @@ -226,6 +226,7 @@ The project is following Network to Code software development guidelines and is 1. Define the `Parsers`(inheriting from some of the generic `Parsers` or a new one) that will extract the data from the notification, that could contain itself multiple `DataParts`. The `data_type` of the `Parser` and the `DataPart` have to match. The custom `Parsers` will be placed in the `parsers` folder. 2. Update the `unit/test_parsers.py` with the new parsers, providing some data to test and validate the extracted data. 3. Define a new `Provider` inheriting from the `GenericProvider`, defining the `Processors` and the respective `Parsers` to be used. Maybe you can reuse some of the generic `Processors` or maybe you will need to create a custom one. If this is the case, place it in the `processors` folder. + - The `Provider` also supports the definition of a `_include_filter` and a `_exclude_filter` to limit the notifications that are actually processed, avoiding false positive errors for notification that are not relevant. 4. Update the `unit/test_e2e.py` with the new provider, providing some data to test and validate the final `Maintenances` created. 5. **Expose the new `Provider` class** updating the map `SUPPORTED_PROVIDERS` in `circuit_maintenance_parser/__init__.py` to officially expose the `Provider`. diff --git a/circuit_maintenance_parser/constants.py b/circuit_maintenance_parser/constants.py new file mode 100644 index 00000000..da2ace39 --- /dev/null +++ b/circuit_maintenance_parser/constants.py @@ -0,0 +1,4 @@ +"""Constants used in the library.""" + +EMAIL_HEADER_SUBJECT = "email-header-subject" +EMAIL_HEADER_DATE = "email-header-date" diff --git a/circuit_maintenance_parser/data.py b/circuit_maintenance_parser/data.py index 627e0200..8385b243 100644 --- a/circuit_maintenance_parser/data.py +++ b/circuit_maintenance_parser/data.py @@ -4,6 +4,8 @@ import email from pydantic import BaseModel, Extra +from circuit_maintenance_parser.constants import EMAIL_HEADER_SUBJECT, EMAIL_HEADER_DATE + logger = logging.getLogger(__name__) @@ -73,9 +75,8 @@ def init_from_emailmessage(cls: Type["NotificationData"], email_message) -> Opti cls.walk_email(email_message, data_parts) # Adding extra headers that are interesting to be parsed - data_parts.add(DataPart("email-header-subject", email_message["Subject"].encode())) - # TODO: Date could be used to extend the "Stamp" time of a notification when not available, but we need a parser - data_parts.add(DataPart("email-header-date", email_message["Date"].encode())) + data_parts.add(DataPart(EMAIL_HEADER_SUBJECT, email_message["Subject"].encode())) + data_parts.add(DataPart(EMAIL_HEADER_DATE, email_message["Date"].encode())) return cls(data_parts=list(data_parts)) except Exception: # pylint: disable=broad-except logger.exception("Error found initializing data from email message: %s", email_message) diff --git a/circuit_maintenance_parser/parser.py b/circuit_maintenance_parser/parser.py index 1428024d..0083cca0 100644 --- a/circuit_maintenance_parser/parser.py +++ b/circuit_maintenance_parser/parser.py @@ -15,6 +15,7 @@ from circuit_maintenance_parser.errors import ParserError from circuit_maintenance_parser.output import Status, Impact, CircuitImpact +from circuit_maintenance_parser.constants import EMAIL_HEADER_SUBJECT, EMAIL_HEADER_DATE # pylint: disable=no-member @@ -177,7 +178,7 @@ def clean_line(line): class EmailDateParser(Parser): """Parser for Email Date.""" - _data_types = ["email-header-date"] + _data_types = [EMAIL_HEADER_DATE] def parser_hook(self, raw: bytes): """Execute parsing.""" @@ -190,7 +191,7 @@ def parser_hook(self, raw: bytes): class EmailSubjectParser(Parser): """Parse data from subject or email.""" - _data_types = ["email-header-subject"] + _data_types = [EMAIL_HEADER_SUBJECT] def parser_hook(self, raw: bytes): """Execute parsing.""" diff --git a/circuit_maintenance_parser/provider.py b/circuit_maintenance_parser/provider.py index 715cfe7b..e485f4a2 100644 --- a/circuit_maintenance_parser/provider.py +++ b/circuit_maintenance_parser/provider.py @@ -1,8 +1,9 @@ """Definition of Provider class as the entry point to the library.""" import logging +import re import traceback -from typing import Iterable, List +from typing import Iterable, List, Dict from pydantic import BaseModel @@ -13,6 +14,7 @@ from circuit_maintenance_parser.parser import ICal, EmailDateParser from circuit_maintenance_parser.errors import ProcessorError, ProviderError from circuit_maintenance_parser.processor import CombinedProcessor, SimpleProcessor, GenericProcessor +from circuit_maintenance_parser.constants import EMAIL_HEADER_SUBJECT from circuit_maintenance_parser.parsers.aquacomms import HtmlParserAquaComms1, SubjectParserAquaComms1 from circuit_maintenance_parser.parsers.aws import SubjectParserAWS1, TextParserAWS1 @@ -50,6 +52,14 @@ class GenericProvider(BaseModel): that will be used. Default: `[SimpleProcessor(data_parsers=[ICal])]`. _default_organizer (optional): Defines a default `organizer`, an email address, to be used to create a `Maintenance` in absence of the information in the original notification. + _include_filter (optional): Dictionary that defines matching regex per data type to take a notification into + account. + _exclude_filter (optional): Dictionary that defines matching regex per data type to NOT take a notification + into account. + + Notes: + - If a notification matches both the `_include_filter` and `_exclude_filter`, the exclusion takes precedence and + the notification will be filtered out. Examples: >>> GenericProvider() @@ -59,12 +69,55 @@ class GenericProvider(BaseModel): _processors: List[GenericProcessor] = [SimpleProcessor(data_parsers=[ICal])] _default_organizer: str = "unknown" + _include_filter: Dict[str, List[str]] = {} + _exclude_filter: Dict[str, List[str]] = {} + + def include_filter_check(self, data: NotificationData) -> bool: + """If `_include_filter` is defined, it verifies that the matching criteria is met.""" + if self._include_filter: + return self.filter_check(self._include_filter, data, "include") + return True + + def exclude_filter_check(self, data: NotificationData) -> bool: + """If `_exclude_filter` is defined, it verifies that the matching criteria is met.""" + if self._exclude_filter: + return self.filter_check(self._exclude_filter, data, "exclude") + return False + + @staticmethod + def filter_check(filter_dict: Dict, data: NotificationData, filter_type: str) -> bool: + """Generic filter check.""" + data_part_content = None + for data_part in data.data_parts: + filter_data_type = data_part.type + if filter_data_type not in filter_dict: + continue + + data_part_content = data_part.content.decode() + if any(re.search(filter_re, data_part_content) for filter_re in filter_dict[filter_data_type]): + logger.debug("Matching %s filter expression for %s.", filter_type, data_part_content) + return True + + if data_part_content: + logger.warning("Not matching any %s filter expression for %s.", filter_type, data_part_content) + else: + logger.warning( + "Not matching any %s filter expression because the notification doesn't contain the expected data_types: %s", + filter_type, + ", ".join(filter_dict.keys()), + ) + return False + def get_maintenances(self, data: NotificationData) -> Iterable[Maintenance]: """Main entry method that will use the defined `_processors` in order to extract the `Maintenances` from data.""" provider_name = self.__class__.__name__ error_message = "" related_exceptions = [] + if self.exclude_filter_check(data) or not self.include_filter_check(data): + logger.debug("Skipping notification %s due filtering policy for %s.", data, self.__class__.__name__) + return [] + for processor in self._processors: try: return processor.process(data, self.get_extended_data()) @@ -172,6 +225,8 @@ class HGC(GenericProvider): class Lumen(GenericProvider): """Lumen provider custom class.""" + _include_filter = {EMAIL_HEADER_SUBJECT: ["Scheduled Maintenance"]} + _processors: List[GenericProcessor] = [ CombinedProcessor(data_parsers=[EmailDateParser, HtmlParserLumen1]), ] diff --git a/tests/unit/data/lumen/subject_work_planned b/tests/unit/data/lumen/subject_work_planned new file mode 100644 index 00000000..b4a7cf24 --- /dev/null +++ b/tests/unit/data/lumen/subject_work_planned @@ -0,0 +1 @@ +Scheduled Maintenance Window diff --git a/tests/unit/test_e2e.py b/tests/unit/test_e2e.py index bc37ec12..982c80a9 100644 --- a/tests/unit/test_e2e.py +++ b/tests/unit/test_e2e.py @@ -7,7 +7,7 @@ from circuit_maintenance_parser.data import NotificationData from circuit_maintenance_parser.errors import ProviderError - +from circuit_maintenance_parser.constants import EMAIL_HEADER_DATE, EMAIL_HEADER_SUBJECT # pylint: disable=duplicate-code from circuit_maintenance_parser.provider import ( @@ -65,7 +65,7 @@ Cogent, [ ("html", Path(dir_path, "data", "cogent", "cogent1.html")), - ("email-header-date", Path(dir_path, "data", "date", "email_date_1")), + (EMAIL_HEADER_DATE, Path(dir_path, "data", "date", "email_date_1")), ], [ Path(dir_path, "data", "cogent", "cogent1_result.json"), @@ -76,7 +76,7 @@ Cogent, [ ("html", Path(dir_path, "data", "cogent", "cogent2.html")), - ("email-header-date", Path(dir_path, "data", "date", "email_date_1")), + (EMAIL_HEADER_DATE, Path(dir_path, "data", "date", "email_date_1")), ], [ Path(dir_path, "data", "cogent", "cogent2_result.json"), @@ -105,7 +105,8 @@ Lumen, [ ("html", Path(dir_path, "data", "lumen", "lumen1.html")), - ("email-header-date", Path(dir_path, "data", "date", "email_date_1")), + (EMAIL_HEADER_DATE, Path(dir_path, "data", "date", "email_date_1")), + (EMAIL_HEADER_SUBJECT, Path(dir_path, "data", "lumen", "subject_work_planned")), ], [ Path(dir_path, "data", "lumen", "lumen1_result.json"), @@ -116,7 +117,8 @@ Lumen, [ ("html", Path(dir_path, "data", "lumen", "lumen2.html")), - ("email-header-date", Path(dir_path, "data", "date", "email_date_1")), + (EMAIL_HEADER_DATE, Path(dir_path, "data", "date", "email_date_1")), + (EMAIL_HEADER_SUBJECT, Path(dir_path, "data", "lumen", "subject_work_planned")), ], [ Path(dir_path, "data", "lumen", "lumen2_result.json"), @@ -127,7 +129,8 @@ Lumen, [ ("html", Path(dir_path, "data", "lumen", "lumen3.html")), - ("email-header-date", Path(dir_path, "data", "date", "email_date_1")), + (EMAIL_HEADER_DATE, Path(dir_path, "data", "date", "email_date_1")), + (EMAIL_HEADER_SUBJECT, Path(dir_path, "data", "lumen", "subject_work_planned")), ], [ Path(dir_path, "data", "lumen", "lumen3_result.json"), @@ -138,7 +141,8 @@ Lumen, [ ("html", Path(dir_path, "data", "lumen", "lumen4.html")), - ("email-header-date", Path(dir_path, "data", "date", "email_date_1")), + (EMAIL_HEADER_DATE, Path(dir_path, "data", "date", "email_date_1")), + (EMAIL_HEADER_SUBJECT, Path(dir_path, "data", "lumen", "subject_work_planned")), ], [ Path(dir_path, "data", "lumen", "lumen4_result.json"), @@ -150,7 +154,7 @@ Megaport, [ ("html", Path(dir_path, "data", "megaport", "megaport1.html")), - ("email-header-date", Path(dir_path, "data", "date", "email_date_1")), + (EMAIL_HEADER_DATE, Path(dir_path, "data", "date", "email_date_1")), ], [ Path(dir_path, "data", "megaport", "megaport1_result.json"), @@ -161,7 +165,7 @@ Megaport, [ ("html", Path(dir_path, "data", "megaport", "megaport2.html")), - ("email-header-date", Path(dir_path, "data", "date", "email_date_1")), + (EMAIL_HEADER_DATE, Path(dir_path, "data", "date", "email_date_1")), ], [ Path(dir_path, "data", "megaport", "megaport2_result.json"), @@ -221,7 +225,7 @@ Telstra, [ ("html", Path(dir_path, "data", "telstra", "telstra1.html")), - ("email-header-date", Path(dir_path, "data", "date", "email_date_1")), + (EMAIL_HEADER_DATE, Path(dir_path, "data", "date", "email_date_1")), ], [ Path(dir_path, "data", "telstra", "telstra1_result.json"), @@ -232,7 +236,7 @@ Telstra, [ ("html", Path(dir_path, "data", "telstra", "telstra2.html")), - ("email-header-date", Path(dir_path, "data", "date", "email_date_1")), + (EMAIL_HEADER_DATE, Path(dir_path, "data", "date", "email_date_1")), ], [ Path(dir_path, "data", "telstra", "telstra2_result.json"), @@ -245,7 +249,7 @@ Turkcell, [ ("html", Path(dir_path, "data", "turkcell", "turkcell1.html")), - ("email-header-date", Path(dir_path, "data", "date", "email_date_1")), + (EMAIL_HEADER_DATE, Path(dir_path, "data", "date", "email_date_1")), ], [ Path(dir_path, "data", "turkcell", "turkcell1_result.json"), @@ -256,7 +260,7 @@ Turkcell, [ ("html", Path(dir_path, "data", "turkcell", "turkcell2.html")), - ("email-header-date", Path(dir_path, "data", "date", "email_date_1")), + (EMAIL_HEADER_DATE, Path(dir_path, "data", "date", "email_date_1")), ], [ Path(dir_path, "data", "turkcell", "turkcell2_result.json"), @@ -268,7 +272,7 @@ Verizon, [ ("html", Path(dir_path, "data", "verizon", "verizon1.html")), - ("email-header-date", Path(dir_path, "data", "date", "email_date_1")), + (EMAIL_HEADER_DATE, Path(dir_path, "data", "date", "email_date_1")), ], [ Path(dir_path, "data", "verizon", "verizon1_result.json"), @@ -279,7 +283,7 @@ Verizon, [ ("html", Path(dir_path, "data", "verizon", "verizon2.html")), - ("email-header-date", Path(dir_path, "data", "date", "email_date_1")), + (EMAIL_HEADER_DATE, Path(dir_path, "data", "date", "email_date_1")), ], [ Path(dir_path, "data", "verizon", "verizon2_result.json"), @@ -290,7 +294,7 @@ Verizon, [ ("html", Path(dir_path, "data", "verizon", "verizon3.html")), - ("email-header-date", Path(dir_path, "data", "date", "email_date_1")), + (EMAIL_HEADER_DATE, Path(dir_path, "data", "date", "email_date_1")), ], [ Path(dir_path, "data", "verizon", "verizon3_result.json"), diff --git a/tests/unit/test_providers.py b/tests/unit/test_providers.py index 96aa097b..638e24ca 100644 --- a/tests/unit/test_providers.py +++ b/tests/unit/test_providers.py @@ -55,3 +55,53 @@ def test_provide_get_maintenances_one_exception(provider_class): else: provider.get_maintenances(fake_data) assert mock_processor.call_count == 2 + + +def test_provider_with_include_filter(): + """Tests usage of _include_filter.""" + + class ProviderWithIncludeFilter(GenericProvider): + """Fake Provider.""" + + _include_filter = {fake_data.data_parts[0].type: [fake_data.data_parts[0].content.decode()]} + + # Because the include filter is matching with the data, we expect that we hit the `process` + with pytest.raises(ProviderError): + ProviderWithIncludeFilter().get_maintenances(fake_data) + + # With a non matching data to include, the notification will be skipped and just return empty + other_fake_data = NotificationData.init_from_raw("other type", b"other data") + assert ProviderWithIncludeFilter().get_maintenances(other_fake_data) == [] + + +def test_provider_with_exclude_filter(): + """Tests usage of _exclude_filter.""" + + class ProviderWithIncludeFilter(GenericProvider): + """Fake Provider.""" + + _exclude_filter = {fake_data.data_parts[0].type: [fake_data.data_parts[0].content.decode()]} + + # Because the exclude filter is matching with the data, we expect that we skip the processing + assert ProviderWithIncludeFilter().get_maintenances(fake_data) == [] + + # With a non matching data to exclude, the notification will be not skipped and processed + other_fake_data = NotificationData.init_from_raw("other type", b"other data") + with pytest.raises(ProviderError): + ProviderWithIncludeFilter().get_maintenances(other_fake_data) + + +def test_provider_with_include_and_exclude_filters(): + """Tests matching of include and exclude filter, where the exclude takes precedence.""" + data = NotificationData.init_from_raw("fake_type", b"fake data") + data.add_data_part("other_type", b"other data") + + class ProviderWithIncludeFilter(GenericProvider): + """Fake Provider.""" + + _include_filter = {data.data_parts[0].type: [data.data_parts[0].content.decode()]} + _exclude_filter = {data.data_parts[1].type: [data.data_parts[1].content.decode()]} + + # Because the exclude filter and the include filter are matching, we expect the exclude to take + # precedence + assert ProviderWithIncludeFilter().get_maintenances(data) == [] From 9d14c6f10c6aa61e5c2811aa099eb50f6e5916d9 Mon Sep 17 00:00:00 2001 From: Glenn Matthews Date: Fri, 1 Oct 2021 09:51:13 -0400 Subject: [PATCH 16/17] Update changelog --- CHANGELOG.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0dc8d2f6..dac24e1d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,10 @@ - #91 - `Provider` now adds `_include_filter` and `_exclude_filter` attributes (using regex) to filter in and out notifications that are relevant to be parsed vs other that are not, avoiding false positives. +### Fixed + +- #90 - Improved handling of Lumen scheduled maintenance notices + ## v2.0.2 - 2021-09-28 ### Fixed @@ -14,7 +18,7 @@ ### Added -- #84 - New parser added for text. Added new provider `AquaComms` using `Text` and `EmailSubjectParser` +- #84 - New parser added for text. Added new provider `AWS` using `Text` and `EmailSubjectParser` ## v2.0.1 - 2021-09-16 From a3c66ddde1573067b9a8350b8bd06ada43ddd83a Mon Sep 17 00:00:00 2001 From: Glenn Matthews Date: Fri, 1 Oct 2021 09:54:21 -0400 Subject: [PATCH 17/17] Release prep --- CHANGELOG.md | 7 ++----- pyproject.toml | 2 +- 2 files changed, 3 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index dac24e1d..6ff7a8c9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,9 +1,10 @@ # Changelog -## v2.0.3 +## v2.0.3 - 2021-10-01 ### Added +- #84 - New parser added for text. Added new provider `AWS` using `Text` and `EmailSubjectParser` - #91 - `Provider` now adds `_include_filter` and `_exclude_filter` attributes (using regex) to filter in and out notifications that are relevant to be parsed vs other that are not, avoiding false positives. ### Fixed @@ -16,10 +17,6 @@ - #86 - Fix `CombinedProcessor` carries over data from previous parsing -### Added - -- #84 - New parser added for text. Added new provider `AWS` using `Text` and `EmailSubjectParser` - ## v2.0.1 - 2021-09-16 ### Fixed diff --git a/pyproject.toml b/pyproject.toml index a5d919d2..327779cc 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "circuit-maintenance-parser" -version = "2.0.2" +version = "2.0.3" description = "Python library to parse Circuit Maintenance notifications and return a structured data back" authors = ["Network to Code "] license = "Apache-2.0"