From d0045afc45d6fc22e4db828fdecc43bcfe52eb97 Mon Sep 17 00:00:00 2001 From: mgcam Date: Fri, 19 Apr 2024 11:52:08 +0100 Subject: [PATCH 01/27] Added standard project files --- .gitignore | 9 + CHANGELOG.md | 7 + LICENSE.md | 675 +++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 691 insertions(+) create mode 100644 .gitignore create mode 100644 CHANGELOG.md create mode 100644 LICENSE.md diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..b77136a --- /dev/null +++ b/.gitignore @@ -0,0 +1,9 @@ +.DS_Store +*~ +__pycache__ +*.egg-info +.vscode +.eggs +build +.pytest_cache +.vscode diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..148ccf2 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,7 @@ +# Change Log for npg_porch Project + +The format is based on [Keep a Changelog](http://keepachangelog.com/). +This project adheres to [Semantic Versioning](http://semver.org/). + +## [Unreleased] + diff --git a/LICENSE.md b/LICENSE.md new file mode 100644 index 0000000..01a7a80 --- /dev/null +++ b/LICENSE.md @@ -0,0 +1,675 @@ +### GNU GENERAL PUBLIC LICENSE + +Version 3, 29 June 2007 + +Copyright (C) 2007 Free Software Foundation, Inc. + + +Everyone is permitted to copy and distribute verbatim copies of this +license document, but changing it is not allowed. + +### Preamble + +The GNU General Public License is a free, copyleft license for +software and other kinds of works. + +The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +the GNU General Public License is intended to guarantee your freedom +to share and change all versions of a program--to make sure it remains +free software for all its users. We, the Free Software Foundation, use +the GNU General Public License for most of our software; it applies +also to any other work released this way by its authors. You can apply +it to your programs, too. + +When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + +To protect your rights, we need to prevent others from denying you +these rights or asking you to surrender the rights. Therefore, you +have certain responsibilities if you distribute copies of the +software, or if you modify it: responsibilities to respect the freedom +of others. + +For example, if you distribute copies of such a program, whether +gratis or for a fee, you must pass on to the recipients the same +freedoms that you received. You must make sure that they, too, receive +or can get the source code. And you must show them these terms so they +know their rights. + +Developers that use the GNU GPL protect your rights with two steps: +(1) assert copyright on the software, and (2) offer you this License +giving you legal permission to copy, distribute and/or modify it. + +For the developers' and authors' protection, the GPL clearly explains +that there is no warranty for this free software. For both users' and +authors' sake, the GPL requires that modified versions be marked as +changed, so that their problems will not be attributed erroneously to +authors of previous versions. + +Some devices are designed to deny users access to install or run +modified versions of the software inside them, although the +manufacturer can do so. This is fundamentally incompatible with the +aim of protecting users' freedom to change the software. The +systematic pattern of such abuse occurs in the area of products for +individuals to use, which is precisely where it is most unacceptable. +Therefore, we have designed this version of the GPL to prohibit the +practice for those products. If such problems arise substantially in +other domains, we stand ready to extend this provision to those +domains in future versions of the GPL, as needed to protect the +freedom of users. + +Finally, every program is threatened constantly by software patents. +States should not allow patents to restrict development and use of +software on general-purpose computers, but in those that do, we wish +to avoid the special danger that patents applied to a free program +could make it effectively proprietary. To prevent this, the GPL +assures that patents cannot be used to render the program non-free. + +The precise terms and conditions for copying, distribution and +modification follow. + +### TERMS AND CONDITIONS + +#### 0. Definitions. + +"This License" refers to version 3 of the GNU General Public License. + +"Copyright" also means copyright-like laws that apply to other kinds +of works, such as semiconductor masks. + +"The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + +To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of +an exact copy. The resulting work is called a "modified version" of +the earlier work or a work "based on" the earlier work. + +A "covered work" means either the unmodified Program or a work based +on the Program. + +To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + +To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user +through a computer network, with no transfer of a copy, is not +conveying. + +An interactive user interface displays "Appropriate Legal Notices" to +the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + +#### 1. Source Code. + +The "source code" for a work means the preferred form of the work for +making modifications to it. "Object code" means any non-source form of +a work. + +A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + +The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + +The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + +The Corresponding Source need not include anything that users can +regenerate automatically from other parts of the Corresponding Source. + +The Corresponding Source for a work in source code form is that same +work. + +#### 2. Basic Permissions. + +All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + +You may make, run and propagate covered works that you do not convey, +without conditions so long as your license otherwise remains in force. +You may convey covered works to others for the sole purpose of having +them make modifications exclusively for you, or provide you with +facilities for running those works, provided that you comply with the +terms of this License in conveying all material for which you do not +control copyright. Those thus making or running the covered works for +you must do so exclusively on your behalf, under your direction and +control, on terms that prohibit them from making any copies of your +copyrighted material outside their relationship with you. + +Conveying under any other circumstances is permitted solely under the +conditions stated below. Sublicensing is not allowed; section 10 makes +it unnecessary. + +#### 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + +No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + +When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such +circumvention is effected by exercising rights under this License with +respect to the covered work, and you disclaim any intention to limit +operation or modification of the work as a means of enforcing, against +the work's users, your or third parties' legal rights to forbid +circumvention of technological measures. + +#### 4. Conveying Verbatim Copies. + +You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + +You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + +#### 5. Conveying Modified Source Versions. + +You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these +conditions: + +- a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. +- b) The work must carry prominent notices stating that it is + released under this License and any conditions added under + section 7. This requirement modifies the requirement in section 4 + to "keep intact all notices". +- c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. +- d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + +A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + +#### 6. Conveying Non-Source Forms. + +You may convey a covered work in object code form under the terms of +sections 4 and 5, provided that you also convey the machine-readable +Corresponding Source under the terms of this License, in one of these +ways: + +- a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. +- b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the Corresponding + Source from a network server at no charge. +- c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. +- d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. +- e) Convey the object code using peer-to-peer transmission, + provided you inform other peers where the object code and + Corresponding Source of the work are being offered to the general + public at no charge under subsection 6d. + +A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + +A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, +family, or household purposes, or (2) anything designed or sold for +incorporation into a dwelling. In determining whether a product is a +consumer product, doubtful cases shall be resolved in favor of +coverage. For a particular product received by a particular user, +"normally used" refers to a typical or common use of that class of +product, regardless of the status of the particular user or of the way +in which the particular user actually uses, or expects or is expected +to use, the product. A product is a consumer product regardless of +whether the product has substantial commercial, industrial or +non-consumer uses, unless such uses represent the only significant +mode of use of the product. + +"Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to +install and execute modified versions of a covered work in that User +Product from a modified version of its Corresponding Source. The +information must suffice to ensure that the continued functioning of +the modified object code is in no case prevented or interfered with +solely because modification has been made. + +If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + +The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or +updates for a work that has been modified or installed by the +recipient, or for the User Product in which it has been modified or +installed. Access to a network may be denied when the modification +itself materially and adversely affects the operation of the network +or violates the rules and protocols for communication across the +network. + +Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + +#### 7. Additional Terms. + +"Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + +When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + +Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders +of that material) supplement the terms of this License with terms: + +- a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or +- b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or +- c) Prohibiting misrepresentation of the origin of that material, + or requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or +- d) Limiting the use for publicity purposes of names of licensors + or authors of the material; or +- e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or +- f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions + of it) with contractual assumptions of liability to the recipient, + for any liability that these contractual assumptions directly + impose on those licensors and authors. + +All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + +If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + +Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; the +above requirements apply either way. + +#### 8. Termination. + +You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + +However, if you cease all violation of this License, then your license +from a particular copyright holder is reinstated (a) provisionally, +unless and until the copyright holder explicitly and finally +terminates your license, and (b) permanently, if the copyright holder +fails to notify you of the violation by some reasonable means prior to +60 days after the cessation. + +Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + +Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + +#### 9. Acceptance Not Required for Having Copies. + +You are not required to accept this License in order to receive or run +a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + +#### 10. Automatic Licensing of Downstream Recipients. + +Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + +An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + +You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + +#### 11. Patents. + +A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + +A contributor's "essential patent claims" are all patent claims owned +or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + +Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + +In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + +If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + +If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + +A patent license is "discriminatory" if it does not include within the +scope of its coverage, prohibits the exercise of, or is conditioned on +the non-exercise of one or more of the rights that are specifically +granted under this License. You may not convey a covered work if you +are a party to an arrangement with a third party that is in the +business of distributing software, under which you make payment to the +third party based on the extent of your activity of conveying the +work, and under which the third party grants, to any of the parties +who would receive the covered work from you, a discriminatory patent +license (a) in connection with copies of the covered work conveyed by +you (or copies made from those copies), or (b) primarily for and in +connection with specific products or compilations that contain the +covered work, unless you entered into that arrangement, or that patent +license was granted, prior to 28 March 2007. + +Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + +#### 12. No Surrender of Others' Freedom. + +If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under +this License and any other pertinent obligations, then as a +consequence you may not convey it at all. For example, if you agree to +terms that obligate you to collect a royalty for further conveying +from those to whom you convey the Program, the only way you could +satisfy both those terms and this License would be to refrain entirely +from conveying the Program. + +#### 13. Use with the GNU Affero General Public License. + +Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU Affero General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the special requirements of the GNU Affero General Public License, +section 13, concerning interaction through a network will apply to the +combination as such. + +#### 14. Revised Versions of this License. + +The Free Software Foundation may publish revised and/or new versions +of the GNU General Public License from time to time. Such new versions +will be similar in spirit to the present version, but may differ in +detail to address new problems or concerns. + +Each version is given a distinguishing version number. If the Program +specifies that a certain numbered version of the GNU General Public +License "or any later version" applies to it, you have the option of +following the terms and conditions either of that numbered version or +of any later version published by the Free Software Foundation. If the +Program does not specify a version number of the GNU General Public +License, you may choose any version ever published by the Free +Software Foundation. + +If the Program specifies that a proxy can decide which future versions +of the GNU General Public License can be used, that proxy's public +statement of acceptance of a version permanently authorizes you to +choose that version for the Program. + +Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + +#### 15. Disclaimer of Warranty. + +THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT +WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND +PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE +DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR +CORRECTION. + +#### 16. Limitation of Liability. + +IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR +CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, +INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES +ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT +NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR +LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM +TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER +PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. + +#### 17. Interpretation of Sections 15 and 16. + +If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + +END OF TERMS AND CONDITIONS + +### How to Apply These Terms to Your New Programs + +If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these +terms. + +To do so, attach the following notices to the program. It is safest to +attach them to the start of each source file to most effectively state +the exclusion of warranty; and each file should have at least the +"copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see . + +Also add information on how to contact you by electronic and paper +mail. + +If the program does terminal interaction, make it output a short +notice like this when it starts in an interactive mode: + + Copyright (C) + This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands \`show w' and \`show c' should show the +appropriate parts of the General Public License. Of course, your +program's commands might be different; for a GUI interface, you would +use an "about box". + +You should also get your employer (if you work as a programmer) or +school, if any, to sign a "copyright disclaimer" for the program, if +necessary. For more information on this, and how to apply and follow +the GNU GPL, see . + +The GNU General Public License does not permit incorporating your +program into proprietary programs. If your program is a subroutine +library, you may consider it more useful to permit linking proprietary +applications with the library. If this is what you want to do, use the +GNU Lesser General Public License instead of this License. But first, +please read . \ No newline at end of file From c61f59eb6b9791fd193650b8f1e813e7b519af5f Mon Sep 17 00:00:00 2001 From: mgcam Date: Fri, 19 Apr 2024 15:36:12 +0100 Subject: [PATCH 02/27] Initial project scaffold --- .github/workflows/test.yml | 34 ++ CHANGELOG.md | 5 + LICENSE.md | 675 ---------------------------------- pyproject.toml | 32 ++ src/npg_porch_cli/__init__.py | 0 src/npg_porch_cli/api.py | 22 ++ tests/test_api.py | 8 + 7 files changed, 101 insertions(+), 675 deletions(-) create mode 100644 .github/workflows/test.yml delete mode 100644 LICENSE.md create mode 100644 pyproject.toml create mode 100644 src/npg_porch_cli/__init__.py create mode 100644 src/npg_porch_cli/api.py create mode 100644 tests/test_api.py diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml new file mode 100644 index 0000000..4df3427 --- /dev/null +++ b/.github/workflows/test.yml @@ -0,0 +1,34 @@ +# This workflow will install Python dependencies and run tests +name: Python application + +on: + push: + branches: [master, devel] + pull_request: + branches: [master, devel] + +jobs: + + test-packaging: + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v4 + + - name: Install Poetry + run: | + pipx install poetry + + - uses: actions/setup-python@v5 + with: + python-version: '3.11' + architecture: 'x64' + + - name: Run poetry install + run: | + poetry env use '3.11' + poetry install + + - name: Run pytest + run: | + poetry run pytest diff --git a/CHANGELOG.md b/CHANGELOG.md index 148ccf2..f0ebaaa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,3 +5,8 @@ This project adheres to [Semantic Versioning](http://semver.org/). ## [Unreleased] +## [0.0.1] + +### Added + +# Initial project scaffold diff --git a/LICENSE.md b/LICENSE.md deleted file mode 100644 index 01a7a80..0000000 --- a/LICENSE.md +++ /dev/null @@ -1,675 +0,0 @@ -### GNU GENERAL PUBLIC LICENSE - -Version 3, 29 June 2007 - -Copyright (C) 2007 Free Software Foundation, Inc. - - -Everyone is permitted to copy and distribute verbatim copies of this -license document, but changing it is not allowed. - -### Preamble - -The GNU General Public License is a free, copyleft license for -software and other kinds of works. - -The licenses for most software and other practical works are designed -to take away your freedom to share and change the works. By contrast, -the GNU General Public License is intended to guarantee your freedom -to share and change all versions of a program--to make sure it remains -free software for all its users. We, the Free Software Foundation, use -the GNU General Public License for most of our software; it applies -also to any other work released this way by its authors. You can apply -it to your programs, too. - -When we speak of free software, we are referring to freedom, not -price. Our General Public Licenses are designed to make sure that you -have the freedom to distribute copies of free software (and charge for -them if you wish), that you receive source code or can get it if you -want it, that you can change the software or use pieces of it in new -free programs, and that you know you can do these things. - -To protect your rights, we need to prevent others from denying you -these rights or asking you to surrender the rights. Therefore, you -have certain responsibilities if you distribute copies of the -software, or if you modify it: responsibilities to respect the freedom -of others. - -For example, if you distribute copies of such a program, whether -gratis or for a fee, you must pass on to the recipients the same -freedoms that you received. You must make sure that they, too, receive -or can get the source code. And you must show them these terms so they -know their rights. - -Developers that use the GNU GPL protect your rights with two steps: -(1) assert copyright on the software, and (2) offer you this License -giving you legal permission to copy, distribute and/or modify it. - -For the developers' and authors' protection, the GPL clearly explains -that there is no warranty for this free software. For both users' and -authors' sake, the GPL requires that modified versions be marked as -changed, so that their problems will not be attributed erroneously to -authors of previous versions. - -Some devices are designed to deny users access to install or run -modified versions of the software inside them, although the -manufacturer can do so. This is fundamentally incompatible with the -aim of protecting users' freedom to change the software. The -systematic pattern of such abuse occurs in the area of products for -individuals to use, which is precisely where it is most unacceptable. -Therefore, we have designed this version of the GPL to prohibit the -practice for those products. If such problems arise substantially in -other domains, we stand ready to extend this provision to those -domains in future versions of the GPL, as needed to protect the -freedom of users. - -Finally, every program is threatened constantly by software patents. -States should not allow patents to restrict development and use of -software on general-purpose computers, but in those that do, we wish -to avoid the special danger that patents applied to a free program -could make it effectively proprietary. To prevent this, the GPL -assures that patents cannot be used to render the program non-free. - -The precise terms and conditions for copying, distribution and -modification follow. - -### TERMS AND CONDITIONS - -#### 0. Definitions. - -"This License" refers to version 3 of the GNU General Public License. - -"Copyright" also means copyright-like laws that apply to other kinds -of works, such as semiconductor masks. - -"The Program" refers to any copyrightable work licensed under this -License. Each licensee is addressed as "you". "Licensees" and -"recipients" may be individuals or organizations. - -To "modify" a work means to copy from or adapt all or part of the work -in a fashion requiring copyright permission, other than the making of -an exact copy. The resulting work is called a "modified version" of -the earlier work or a work "based on" the earlier work. - -A "covered work" means either the unmodified Program or a work based -on the Program. - -To "propagate" a work means to do anything with it that, without -permission, would make you directly or secondarily liable for -infringement under applicable copyright law, except executing it on a -computer or modifying a private copy. Propagation includes copying, -distribution (with or without modification), making available to the -public, and in some countries other activities as well. - -To "convey" a work means any kind of propagation that enables other -parties to make or receive copies. Mere interaction with a user -through a computer network, with no transfer of a copy, is not -conveying. - -An interactive user interface displays "Appropriate Legal Notices" to -the extent that it includes a convenient and prominently visible -feature that (1) displays an appropriate copyright notice, and (2) -tells the user that there is no warranty for the work (except to the -extent that warranties are provided), that licensees may convey the -work under this License, and how to view a copy of this License. If -the interface presents a list of user commands or options, such as a -menu, a prominent item in the list meets this criterion. - -#### 1. Source Code. - -The "source code" for a work means the preferred form of the work for -making modifications to it. "Object code" means any non-source form of -a work. - -A "Standard Interface" means an interface that either is an official -standard defined by a recognized standards body, or, in the case of -interfaces specified for a particular programming language, one that -is widely used among developers working in that language. - -The "System Libraries" of an executable work include anything, other -than the work as a whole, that (a) is included in the normal form of -packaging a Major Component, but which is not part of that Major -Component, and (b) serves only to enable use of the work with that -Major Component, or to implement a Standard Interface for which an -implementation is available to the public in source code form. A -"Major Component", in this context, means a major essential component -(kernel, window system, and so on) of the specific operating system -(if any) on which the executable work runs, or a compiler used to -produce the work, or an object code interpreter used to run it. - -The "Corresponding Source" for a work in object code form means all -the source code needed to generate, install, and (for an executable -work) run the object code and to modify the work, including scripts to -control those activities. However, it does not include the work's -System Libraries, or general-purpose tools or generally available free -programs which are used unmodified in performing those activities but -which are not part of the work. For example, Corresponding Source -includes interface definition files associated with source files for -the work, and the source code for shared libraries and dynamically -linked subprograms that the work is specifically designed to require, -such as by intimate data communication or control flow between those -subprograms and other parts of the work. - -The Corresponding Source need not include anything that users can -regenerate automatically from other parts of the Corresponding Source. - -The Corresponding Source for a work in source code form is that same -work. - -#### 2. Basic Permissions. - -All rights granted under this License are granted for the term of -copyright on the Program, and are irrevocable provided the stated -conditions are met. This License explicitly affirms your unlimited -permission to run the unmodified Program. The output from running a -covered work is covered by this License only if the output, given its -content, constitutes a covered work. This License acknowledges your -rights of fair use or other equivalent, as provided by copyright law. - -You may make, run and propagate covered works that you do not convey, -without conditions so long as your license otherwise remains in force. -You may convey covered works to others for the sole purpose of having -them make modifications exclusively for you, or provide you with -facilities for running those works, provided that you comply with the -terms of this License in conveying all material for which you do not -control copyright. Those thus making or running the covered works for -you must do so exclusively on your behalf, under your direction and -control, on terms that prohibit them from making any copies of your -copyrighted material outside their relationship with you. - -Conveying under any other circumstances is permitted solely under the -conditions stated below. Sublicensing is not allowed; section 10 makes -it unnecessary. - -#### 3. Protecting Users' Legal Rights From Anti-Circumvention Law. - -No covered work shall be deemed part of an effective technological -measure under any applicable law fulfilling obligations under article -11 of the WIPO copyright treaty adopted on 20 December 1996, or -similar laws prohibiting or restricting circumvention of such -measures. - -When you convey a covered work, you waive any legal power to forbid -circumvention of technological measures to the extent such -circumvention is effected by exercising rights under this License with -respect to the covered work, and you disclaim any intention to limit -operation or modification of the work as a means of enforcing, against -the work's users, your or third parties' legal rights to forbid -circumvention of technological measures. - -#### 4. Conveying Verbatim Copies. - -You may convey verbatim copies of the Program's source code as you -receive it, in any medium, provided that you conspicuously and -appropriately publish on each copy an appropriate copyright notice; -keep intact all notices stating that this License and any -non-permissive terms added in accord with section 7 apply to the code; -keep intact all notices of the absence of any warranty; and give all -recipients a copy of this License along with the Program. - -You may charge any price or no price for each copy that you convey, -and you may offer support or warranty protection for a fee. - -#### 5. Conveying Modified Source Versions. - -You may convey a work based on the Program, or the modifications to -produce it from the Program, in the form of source code under the -terms of section 4, provided that you also meet all of these -conditions: - -- a) The work must carry prominent notices stating that you modified - it, and giving a relevant date. -- b) The work must carry prominent notices stating that it is - released under this License and any conditions added under - section 7. This requirement modifies the requirement in section 4 - to "keep intact all notices". -- c) You must license the entire work, as a whole, under this - License to anyone who comes into possession of a copy. This - License will therefore apply, along with any applicable section 7 - additional terms, to the whole of the work, and all its parts, - regardless of how they are packaged. This License gives no - permission to license the work in any other way, but it does not - invalidate such permission if you have separately received it. -- d) If the work has interactive user interfaces, each must display - Appropriate Legal Notices; however, if the Program has interactive - interfaces that do not display Appropriate Legal Notices, your - work need not make them do so. - -A compilation of a covered work with other separate and independent -works, which are not by their nature extensions of the covered work, -and which are not combined with it such as to form a larger program, -in or on a volume of a storage or distribution medium, is called an -"aggregate" if the compilation and its resulting copyright are not -used to limit the access or legal rights of the compilation's users -beyond what the individual works permit. Inclusion of a covered work -in an aggregate does not cause this License to apply to the other -parts of the aggregate. - -#### 6. Conveying Non-Source Forms. - -You may convey a covered work in object code form under the terms of -sections 4 and 5, provided that you also convey the machine-readable -Corresponding Source under the terms of this License, in one of these -ways: - -- a) Convey the object code in, or embodied in, a physical product - (including a physical distribution medium), accompanied by the - Corresponding Source fixed on a durable physical medium - customarily used for software interchange. -- b) Convey the object code in, or embodied in, a physical product - (including a physical distribution medium), accompanied by a - written offer, valid for at least three years and valid for as - long as you offer spare parts or customer support for that product - model, to give anyone who possesses the object code either (1) a - copy of the Corresponding Source for all the software in the - product that is covered by this License, on a durable physical - medium customarily used for software interchange, for a price no - more than your reasonable cost of physically performing this - conveying of source, or (2) access to copy the Corresponding - Source from a network server at no charge. -- c) Convey individual copies of the object code with a copy of the - written offer to provide the Corresponding Source. This - alternative is allowed only occasionally and noncommercially, and - only if you received the object code with such an offer, in accord - with subsection 6b. -- d) Convey the object code by offering access from a designated - place (gratis or for a charge), and offer equivalent access to the - Corresponding Source in the same way through the same place at no - further charge. You need not require recipients to copy the - Corresponding Source along with the object code. If the place to - copy the object code is a network server, the Corresponding Source - may be on a different server (operated by you or a third party) - that supports equivalent copying facilities, provided you maintain - clear directions next to the object code saying where to find the - Corresponding Source. Regardless of what server hosts the - Corresponding Source, you remain obligated to ensure that it is - available for as long as needed to satisfy these requirements. -- e) Convey the object code using peer-to-peer transmission, - provided you inform other peers where the object code and - Corresponding Source of the work are being offered to the general - public at no charge under subsection 6d. - -A separable portion of the object code, whose source code is excluded -from the Corresponding Source as a System Library, need not be -included in conveying the object code work. - -A "User Product" is either (1) a "consumer product", which means any -tangible personal property which is normally used for personal, -family, or household purposes, or (2) anything designed or sold for -incorporation into a dwelling. In determining whether a product is a -consumer product, doubtful cases shall be resolved in favor of -coverage. For a particular product received by a particular user, -"normally used" refers to a typical or common use of that class of -product, regardless of the status of the particular user or of the way -in which the particular user actually uses, or expects or is expected -to use, the product. A product is a consumer product regardless of -whether the product has substantial commercial, industrial or -non-consumer uses, unless such uses represent the only significant -mode of use of the product. - -"Installation Information" for a User Product means any methods, -procedures, authorization keys, or other information required to -install and execute modified versions of a covered work in that User -Product from a modified version of its Corresponding Source. The -information must suffice to ensure that the continued functioning of -the modified object code is in no case prevented or interfered with -solely because modification has been made. - -If you convey an object code work under this section in, or with, or -specifically for use in, a User Product, and the conveying occurs as -part of a transaction in which the right of possession and use of the -User Product is transferred to the recipient in perpetuity or for a -fixed term (regardless of how the transaction is characterized), the -Corresponding Source conveyed under this section must be accompanied -by the Installation Information. But this requirement does not apply -if neither you nor any third party retains the ability to install -modified object code on the User Product (for example, the work has -been installed in ROM). - -The requirement to provide Installation Information does not include a -requirement to continue to provide support service, warranty, or -updates for a work that has been modified or installed by the -recipient, or for the User Product in which it has been modified or -installed. Access to a network may be denied when the modification -itself materially and adversely affects the operation of the network -or violates the rules and protocols for communication across the -network. - -Corresponding Source conveyed, and Installation Information provided, -in accord with this section must be in a format that is publicly -documented (and with an implementation available to the public in -source code form), and must require no special password or key for -unpacking, reading or copying. - -#### 7. Additional Terms. - -"Additional permissions" are terms that supplement the terms of this -License by making exceptions from one or more of its conditions. -Additional permissions that are applicable to the entire Program shall -be treated as though they were included in this License, to the extent -that they are valid under applicable law. If additional permissions -apply only to part of the Program, that part may be used separately -under those permissions, but the entire Program remains governed by -this License without regard to the additional permissions. - -When you convey a copy of a covered work, you may at your option -remove any additional permissions from that copy, or from any part of -it. (Additional permissions may be written to require their own -removal in certain cases when you modify the work.) You may place -additional permissions on material, added by you to a covered work, -for which you have or can give appropriate copyright permission. - -Notwithstanding any other provision of this License, for material you -add to a covered work, you may (if authorized by the copyright holders -of that material) supplement the terms of this License with terms: - -- a) Disclaiming warranty or limiting liability differently from the - terms of sections 15 and 16 of this License; or -- b) Requiring preservation of specified reasonable legal notices or - author attributions in that material or in the Appropriate Legal - Notices displayed by works containing it; or -- c) Prohibiting misrepresentation of the origin of that material, - or requiring that modified versions of such material be marked in - reasonable ways as different from the original version; or -- d) Limiting the use for publicity purposes of names of licensors - or authors of the material; or -- e) Declining to grant rights under trademark law for use of some - trade names, trademarks, or service marks; or -- f) Requiring indemnification of licensors and authors of that - material by anyone who conveys the material (or modified versions - of it) with contractual assumptions of liability to the recipient, - for any liability that these contractual assumptions directly - impose on those licensors and authors. - -All other non-permissive additional terms are considered "further -restrictions" within the meaning of section 10. If the Program as you -received it, or any part of it, contains a notice stating that it is -governed by this License along with a term that is a further -restriction, you may remove that term. If a license document contains -a further restriction but permits relicensing or conveying under this -License, you may add to a covered work material governed by the terms -of that license document, provided that the further restriction does -not survive such relicensing or conveying. - -If you add terms to a covered work in accord with this section, you -must place, in the relevant source files, a statement of the -additional terms that apply to those files, or a notice indicating -where to find the applicable terms. - -Additional terms, permissive or non-permissive, may be stated in the -form of a separately written license, or stated as exceptions; the -above requirements apply either way. - -#### 8. Termination. - -You may not propagate or modify a covered work except as expressly -provided under this License. Any attempt otherwise to propagate or -modify it is void, and will automatically terminate your rights under -this License (including any patent licenses granted under the third -paragraph of section 11). - -However, if you cease all violation of this License, then your license -from a particular copyright holder is reinstated (a) provisionally, -unless and until the copyright holder explicitly and finally -terminates your license, and (b) permanently, if the copyright holder -fails to notify you of the violation by some reasonable means prior to -60 days after the cessation. - -Moreover, your license from a particular copyright holder is -reinstated permanently if the copyright holder notifies you of the -violation by some reasonable means, this is the first time you have -received notice of violation of this License (for any work) from that -copyright holder, and you cure the violation prior to 30 days after -your receipt of the notice. - -Termination of your rights under this section does not terminate the -licenses of parties who have received copies or rights from you under -this License. If your rights have been terminated and not permanently -reinstated, you do not qualify to receive new licenses for the same -material under section 10. - -#### 9. Acceptance Not Required for Having Copies. - -You are not required to accept this License in order to receive or run -a copy of the Program. Ancillary propagation of a covered work -occurring solely as a consequence of using peer-to-peer transmission -to receive a copy likewise does not require acceptance. However, -nothing other than this License grants you permission to propagate or -modify any covered work. These actions infringe copyright if you do -not accept this License. Therefore, by modifying or propagating a -covered work, you indicate your acceptance of this License to do so. - -#### 10. Automatic Licensing of Downstream Recipients. - -Each time you convey a covered work, the recipient automatically -receives a license from the original licensors, to run, modify and -propagate that work, subject to this License. You are not responsible -for enforcing compliance by third parties with this License. - -An "entity transaction" is a transaction transferring control of an -organization, or substantially all assets of one, or subdividing an -organization, or merging organizations. If propagation of a covered -work results from an entity transaction, each party to that -transaction who receives a copy of the work also receives whatever -licenses to the work the party's predecessor in interest had or could -give under the previous paragraph, plus a right to possession of the -Corresponding Source of the work from the predecessor in interest, if -the predecessor has it or can get it with reasonable efforts. - -You may not impose any further restrictions on the exercise of the -rights granted or affirmed under this License. For example, you may -not impose a license fee, royalty, or other charge for exercise of -rights granted under this License, and you may not initiate litigation -(including a cross-claim or counterclaim in a lawsuit) alleging that -any patent claim is infringed by making, using, selling, offering for -sale, or importing the Program or any portion of it. - -#### 11. Patents. - -A "contributor" is a copyright holder who authorizes use under this -License of the Program or a work on which the Program is based. The -work thus licensed is called the contributor's "contributor version". - -A contributor's "essential patent claims" are all patent claims owned -or controlled by the contributor, whether already acquired or -hereafter acquired, that would be infringed by some manner, permitted -by this License, of making, using, or selling its contributor version, -but do not include claims that would be infringed only as a -consequence of further modification of the contributor version. For -purposes of this definition, "control" includes the right to grant -patent sublicenses in a manner consistent with the requirements of -this License. - -Each contributor grants you a non-exclusive, worldwide, royalty-free -patent license under the contributor's essential patent claims, to -make, use, sell, offer for sale, import and otherwise run, modify and -propagate the contents of its contributor version. - -In the following three paragraphs, a "patent license" is any express -agreement or commitment, however denominated, not to enforce a patent -(such as an express permission to practice a patent or covenant not to -sue for patent infringement). To "grant" such a patent license to a -party means to make such an agreement or commitment not to enforce a -patent against the party. - -If you convey a covered work, knowingly relying on a patent license, -and the Corresponding Source of the work is not available for anyone -to copy, free of charge and under the terms of this License, through a -publicly available network server or other readily accessible means, -then you must either (1) cause the Corresponding Source to be so -available, or (2) arrange to deprive yourself of the benefit of the -patent license for this particular work, or (3) arrange, in a manner -consistent with the requirements of this License, to extend the patent -license to downstream recipients. "Knowingly relying" means you have -actual knowledge that, but for the patent license, your conveying the -covered work in a country, or your recipient's use of the covered work -in a country, would infringe one or more identifiable patents in that -country that you have reason to believe are valid. - -If, pursuant to or in connection with a single transaction or -arrangement, you convey, or propagate by procuring conveyance of, a -covered work, and grant a patent license to some of the parties -receiving the covered work authorizing them to use, propagate, modify -or convey a specific copy of the covered work, then the patent license -you grant is automatically extended to all recipients of the covered -work and works based on it. - -A patent license is "discriminatory" if it does not include within the -scope of its coverage, prohibits the exercise of, or is conditioned on -the non-exercise of one or more of the rights that are specifically -granted under this License. You may not convey a covered work if you -are a party to an arrangement with a third party that is in the -business of distributing software, under which you make payment to the -third party based on the extent of your activity of conveying the -work, and under which the third party grants, to any of the parties -who would receive the covered work from you, a discriminatory patent -license (a) in connection with copies of the covered work conveyed by -you (or copies made from those copies), or (b) primarily for and in -connection with specific products or compilations that contain the -covered work, unless you entered into that arrangement, or that patent -license was granted, prior to 28 March 2007. - -Nothing in this License shall be construed as excluding or limiting -any implied license or other defenses to infringement that may -otherwise be available to you under applicable patent law. - -#### 12. No Surrender of Others' Freedom. - -If conditions are imposed on you (whether by court order, agreement or -otherwise) that contradict the conditions of this License, they do not -excuse you from the conditions of this License. If you cannot convey a -covered work so as to satisfy simultaneously your obligations under -this License and any other pertinent obligations, then as a -consequence you may not convey it at all. For example, if you agree to -terms that obligate you to collect a royalty for further conveying -from those to whom you convey the Program, the only way you could -satisfy both those terms and this License would be to refrain entirely -from conveying the Program. - -#### 13. Use with the GNU Affero General Public License. - -Notwithstanding any other provision of this License, you have -permission to link or combine any covered work with a work licensed -under version 3 of the GNU Affero General Public License into a single -combined work, and to convey the resulting work. The terms of this -License will continue to apply to the part which is the covered work, -but the special requirements of the GNU Affero General Public License, -section 13, concerning interaction through a network will apply to the -combination as such. - -#### 14. Revised Versions of this License. - -The Free Software Foundation may publish revised and/or new versions -of the GNU General Public License from time to time. Such new versions -will be similar in spirit to the present version, but may differ in -detail to address new problems or concerns. - -Each version is given a distinguishing version number. If the Program -specifies that a certain numbered version of the GNU General Public -License "or any later version" applies to it, you have the option of -following the terms and conditions either of that numbered version or -of any later version published by the Free Software Foundation. If the -Program does not specify a version number of the GNU General Public -License, you may choose any version ever published by the Free -Software Foundation. - -If the Program specifies that a proxy can decide which future versions -of the GNU General Public License can be used, that proxy's public -statement of acceptance of a version permanently authorizes you to -choose that version for the Program. - -Later license versions may give you additional or different -permissions. However, no additional obligations are imposed on any -author or copyright holder as a result of your choosing to follow a -later version. - -#### 15. Disclaimer of Warranty. - -THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY -APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT -HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT -WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT -LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR -A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND -PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE -DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR -CORRECTION. - -#### 16. Limitation of Liability. - -IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING -WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR -CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, -INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES -ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT -NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR -LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM -TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER -PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. - -#### 17. Interpretation of Sections 15 and 16. - -If the disclaimer of warranty and limitation of liability provided -above cannot be given local legal effect according to their terms, -reviewing courts shall apply local law that most closely approximates -an absolute waiver of all civil liability in connection with the -Program, unless a warranty or assumption of liability accompanies a -copy of the Program in return for a fee. - -END OF TERMS AND CONDITIONS - -### How to Apply These Terms to Your New Programs - -If you develop a new program, and you want it to be of the greatest -possible use to the public, the best way to achieve this is to make it -free software which everyone can redistribute and change under these -terms. - -To do so, attach the following notices to the program. It is safest to -attach them to the start of each source file to most effectively state -the exclusion of warranty; and each file should have at least the -"copyright" line and a pointer to where the full notice is found. - - - Copyright (C) - - This program is free software: you can redistribute it and/or modify - it under the terms of the GNU General Public License as published by - the Free Software Foundation, either version 3 of the License, or - (at your option) any later version. - - This program is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with this program. If not, see . - -Also add information on how to contact you by electronic and paper -mail. - -If the program does terminal interaction, make it output a short -notice like this when it starts in an interactive mode: - - Copyright (C) - This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. - This is free software, and you are welcome to redistribute it - under certain conditions; type `show c' for details. - -The hypothetical commands \`show w' and \`show c' should show the -appropriate parts of the General Public License. Of course, your -program's commands might be different; for a GUI interface, you would -use an "about box". - -You should also get your employer (if you work as a programmer) or -school, if any, to sign a "copyright disclaimer" for the program, if -necessary. For more information on this, and how to apply and follow -the GNU GPL, see . - -The GNU General Public License does not permit incorporating your -program into proprietary programs. If your program is a subroutine -library, you may consider it more useful to permit linking proprietary -applications with the library. If this is what you want to do, use the -GNU Lesser General Public License instead of this License. But first, -please read . \ No newline at end of file diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..3ae58a1 --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,32 @@ +[tool.poetry] +name = "npg_porch_cli" +version = "0.0.1" +authors = [ + "Marina Gourtovaia", + "Kieron Taylor", + "Jennifer Liddle", +] +description = "CLI client for communicating with npg_porch JSON API" +readme = "README.md" +license = "GPL-3.0-or-later" + +[tool.poetry.dependencies] +python = "^3.10" + +[tool.poetry.dev-dependencies] +black = "^22.3.0" +flake8 = "^4.0.1" +pytest = "^7.1.1" +isort = { version = "^5.10.1", extras = ["colors"] } + +[build-system] +requires = ["poetry-core>=1.0.0"] +build-backend = "poetry.core.masonry.api" + +[tool.isort] +profile = "black" + +[tool.pytest.ini_options] +addopts = [ + "--import-mode=importlib", +] diff --git a/src/npg_porch_cli/__init__.py b/src/npg_porch_cli/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/src/npg_porch_cli/api.py b/src/npg_porch_cli/api.py new file mode 100644 index 0000000..4668b38 --- /dev/null +++ b/src/npg_porch_cli/api.py @@ -0,0 +1,22 @@ +# Copyright (c) 2024 Genome Research Ltd. +# +# Author: Marina Gourtovaia +# +# This file is part of npg_langqc. +# +# npg_porch_cli is free software: you can redistribute it and/or modify it under +# the terms of the GNU General Public License as published by the Free Software +# Foundation; either version 3 of the License, or (at your option) any later +# version. +# +# This program is distributed in the hope that it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS +# FOR A PARTICULAR PURPOSE. See the GNU General Public License for more +# details. +# +# You should have received a copy of the GNU General Public License along with +# this program. If not, see . + + +def add_two(a, b): + return a + b diff --git a/tests/test_api.py b/tests/test_api.py new file mode 100644 index 0000000..9376545 --- /dev/null +++ b/tests/test_api.py @@ -0,0 +1,8 @@ +import pytest + +import npg_porch_cli.api as porchApi + + +def test_addition(): + + assert porchApi.add_two(1, 2) == 3 From 48d2abc35955842765296a6464965545245bcd83 Mon Sep 17 00:00:00 2001 From: mgcam Date: Wed, 24 Apr 2024 16:11:01 +0100 Subject: [PATCH 03/27] Run code checkers in CI --- .flake8 | 4 ++++ .github/workflows/test.yml | 8 ++++++++ tests/test_api.py | 3 --- 3 files changed, 12 insertions(+), 3 deletions(-) create mode 100644 .flake8 diff --git a/.flake8 b/.flake8 new file mode 100644 index 0000000..c201160 --- /dev/null +++ b/.flake8 @@ -0,0 +1,4 @@ +[flake8] +max-line-length = 100 +ignore = E251, E265, E261, E302, W503 +per-file-ignores = __init__.py:F401 diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 4df3427..40019bc 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -32,3 +32,11 @@ jobs: - name: Run pytest run: | poetry run pytest + + - name: Run code style checker + run: | + poetry run flake8 + + - name: Run code formatter + run: | + poetry run black --check src diff --git a/tests/test_api.py b/tests/test_api.py index 9376545..4e2a9c0 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -1,8 +1,5 @@ -import pytest - import npg_porch_cli.api as porchApi - def test_addition(): assert porchApi.add_two(1, 2) == 3 From 675c8951e1147f5c280cb31d87c185eac4257461 Mon Sep 17 00:00:00 2001 From: jenniferliddle Date: Tue, 14 Jun 2022 14:22:43 +0100 Subject: [PATCH 04/27] Imported a client script from GitLab. Imported a client script for interaction with the npg_porch server from an internal GitLab project. To avoid hardcoding pipeline details, created two additional script arguments, pipeline_url and pipeline_version. Replaced real token values with placeholders. --- pow | 114 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100755 pow diff --git a/pow b/pow new file mode 100755 index 0000000..7fc41ff --- /dev/null +++ b/pow @@ -0,0 +1,114 @@ +#!/usr/bin/env python3 + +# Copyright (C) 2022, 2024 Genome Research Ltd. +# +# Author: Jennifer Liddle +# +# npg_porch_client is free software: you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 3 of the License, or (at your option) any +# later version. +# +# This program is distributed in the hope that it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS +# FOR A PARTICULAR PURPOSE. See the GNU General Public License for more +# details. +# +# You should have received a copy of the GNU General Public License along with +# this program. If not, see . + +import requests +import argparse + +# Certification file for https requests +# if we don't have one, we can set certfile = False +certfile = '/usr/share/ca-certificates/sanger.ac.uk/Genome_Research_Ltd_Certificate_Authority-cert.pem' + +# tokens from Porch +admin_headers = {"Authorization":"Bearer a"} +pipeline_headers = {"Authorization":"Bearer b"} + +# Command line arguments +parser = argparse.ArgumentParser(description='Pipeline Orchestration Wrapper') +parser.add_argument('--baseurl', type=str, help="base URL") +parser.add_argument('--pipeline_url', type=str, help="Pipeline git project URL") +parser.add_argument('--pipeline_version', type=str, help="Pipeline version") +parser.add_argument('--pipeline', type=str, help="pipeline name") +parser.add_argument('--idrun', type=int, help="id_run") +parser.add_argument('--sample', type=str, help="sample name") +parser.add_argument('--study', type=str, help="study ID") +parser.add_argument('--status', type=str, help="new status to set", default="DONE", + choices = ['PENDING', 'CLAIMED', 'RUNNING', 'DONE', 'FAILED', 'CANCELLED']) +parser.add_argument('command', type=str, help="command to send to npg_porch", + choices = ['list', 'plist', 'register', 'add', 'claim', 'update']) +args = parser.parse_args() + +if (args.command == 'list'): + response = requests.get(args.baseurl+'/tasks', verify=certfile, headers=pipeline_headers) + if not response.ok: + print(f"\"{response.reason}\" received from {response.url}") + exit(1) + + x=response.json() + for p in x: + if (p['pipeline']['name'] == args.pipeline): + print(f"{p['task_input']}\t{p['status']}") + +if (args.command == 'plist'): + response = requests.get(args.baseurl+'/pipelines', verify=certfile, headers=admin_headers) + if not response.ok: + print(f"\"{response.reason}\" received from {response.url}") + exit(1) + + x=response.json() + pipelines = {} + for p in x: + pname = p['name'] + if pname not in pipelines: + print(pname) + pipelines[pname] = 1 + +if (args.command == 'register'): + data = { 'name': args.pipeline, 'uri': args.pipeline_url, "version": args.pipeline_version } + response = requests.post(args.baseurl+'/pipelines', json=data, verify=certfile, headers=admin_headers) + if not response.ok: + print(f"\"{response.reason}\" received from {response.url}") + exit(1) + + print(response.json()) + +if (args.command == 'add'): + data = { 'pipeline': + { 'name': args.pipeline, 'uri': args.pipeline_url, "version": args.pipeline_version }, + 'task_input': { 'id_run': args.idrun, 'sample': args.sample, 'id_study': args.study } + } + response = requests.post(args.baseurl+'/tasks', json=data, verify=certfile, headers=pipeline_headers) + if not response.ok: + print(f"\"{response.reason}\" received from {response.url}") + exit(1) + + print(response.json()) + +if (args.command == 'claim'): + data = { 'name': args.pipeline, 'uri': args.pipeline_url, "version": args.pipeline_version } + response = requests.post(args.baseurl+'/tasks/claim', json=data, verify=certfile, headers=pipeline_headers) + if not response.ok: + print(f"\"{response.reason}\" received from {response.url}") + exit(1) + + print(response.json()) + +if (args.command == 'update'): + data = { 'pipeline': + { 'name': args.pipeline, 'uri': args.pipeline_url, "version": args.pipeline_version }, + 'task_input': { 'id_run': args.idrun, 'sample': args.sample, 'id_study': args.study }, + 'status': args.status + } + response = requests.put(args.baseurl+'/tasks', json=data, verify=certfile, headers=pipeline_headers) + if not response.ok: + print(f"\"{response.reason}\" received from {response.url}") + exit(1) + + print(data) + print(response.json()) + From 9e6b1dadd8611245f9966edb25d3012439ce13a4 Mon Sep 17 00:00:00 2001 From: mgcam Date: Wed, 24 Apr 2024 18:32:24 +0100 Subject: [PATCH 05/27] Renamed and reformatted the imported script. Updated project dependencies. --- .flake8 | 2 + .github/workflows/test.yml | 6 +- pow | 114 ------------------ pyproject.toml | 5 + src/npg_porch_cli/npg_porch_client.py | 160 ++++++++++++++++++++++++++ tests/test_api.py | 1 + 6 files changed, 173 insertions(+), 115 deletions(-) delete mode 100755 pow create mode 100755 src/npg_porch_cli/npg_porch_client.py diff --git a/.flake8 b/.flake8 index c201160..45aa47f 100644 --- a/.flake8 +++ b/.flake8 @@ -1,4 +1,6 @@ [flake8] max-line-length = 100 +extend-select = B950 +extend-ignore = E203, E501, E701 ignore = E251, E265, E261, E302, W503 per-file-ignores = __init__.py:F401 diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 40019bc..b697fb7 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -33,10 +33,14 @@ jobs: run: | poetry run pytest + - name: Run import checker + run: | + poetry run isort --check . + - name: Run code style checker run: | poetry run flake8 - name: Run code formatter run: | - poetry run black --check src + poetry run black --check src tests diff --git a/pow b/pow deleted file mode 100755 index 7fc41ff..0000000 --- a/pow +++ /dev/null @@ -1,114 +0,0 @@ -#!/usr/bin/env python3 - -# Copyright (C) 2022, 2024 Genome Research Ltd. -# -# Author: Jennifer Liddle -# -# npg_porch_client is free software: you can redistribute it and/or modify it -# under the terms of the GNU General Public License as published by the Free -# Software Foundation; either version 3 of the License, or (at your option) any -# later version. -# -# This program is distributed in the hope that it will be useful, but WITHOUT -# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS -# FOR A PARTICULAR PURPOSE. See the GNU General Public License for more -# details. -# -# You should have received a copy of the GNU General Public License along with -# this program. If not, see . - -import requests -import argparse - -# Certification file for https requests -# if we don't have one, we can set certfile = False -certfile = '/usr/share/ca-certificates/sanger.ac.uk/Genome_Research_Ltd_Certificate_Authority-cert.pem' - -# tokens from Porch -admin_headers = {"Authorization":"Bearer a"} -pipeline_headers = {"Authorization":"Bearer b"} - -# Command line arguments -parser = argparse.ArgumentParser(description='Pipeline Orchestration Wrapper') -parser.add_argument('--baseurl', type=str, help="base URL") -parser.add_argument('--pipeline_url', type=str, help="Pipeline git project URL") -parser.add_argument('--pipeline_version', type=str, help="Pipeline version") -parser.add_argument('--pipeline', type=str, help="pipeline name") -parser.add_argument('--idrun', type=int, help="id_run") -parser.add_argument('--sample', type=str, help="sample name") -parser.add_argument('--study', type=str, help="study ID") -parser.add_argument('--status', type=str, help="new status to set", default="DONE", - choices = ['PENDING', 'CLAIMED', 'RUNNING', 'DONE', 'FAILED', 'CANCELLED']) -parser.add_argument('command', type=str, help="command to send to npg_porch", - choices = ['list', 'plist', 'register', 'add', 'claim', 'update']) -args = parser.parse_args() - -if (args.command == 'list'): - response = requests.get(args.baseurl+'/tasks', verify=certfile, headers=pipeline_headers) - if not response.ok: - print(f"\"{response.reason}\" received from {response.url}") - exit(1) - - x=response.json() - for p in x: - if (p['pipeline']['name'] == args.pipeline): - print(f"{p['task_input']}\t{p['status']}") - -if (args.command == 'plist'): - response = requests.get(args.baseurl+'/pipelines', verify=certfile, headers=admin_headers) - if not response.ok: - print(f"\"{response.reason}\" received from {response.url}") - exit(1) - - x=response.json() - pipelines = {} - for p in x: - pname = p['name'] - if pname not in pipelines: - print(pname) - pipelines[pname] = 1 - -if (args.command == 'register'): - data = { 'name': args.pipeline, 'uri': args.pipeline_url, "version": args.pipeline_version } - response = requests.post(args.baseurl+'/pipelines', json=data, verify=certfile, headers=admin_headers) - if not response.ok: - print(f"\"{response.reason}\" received from {response.url}") - exit(1) - - print(response.json()) - -if (args.command == 'add'): - data = { 'pipeline': - { 'name': args.pipeline, 'uri': args.pipeline_url, "version": args.pipeline_version }, - 'task_input': { 'id_run': args.idrun, 'sample': args.sample, 'id_study': args.study } - } - response = requests.post(args.baseurl+'/tasks', json=data, verify=certfile, headers=pipeline_headers) - if not response.ok: - print(f"\"{response.reason}\" received from {response.url}") - exit(1) - - print(response.json()) - -if (args.command == 'claim'): - data = { 'name': args.pipeline, 'uri': args.pipeline_url, "version": args.pipeline_version } - response = requests.post(args.baseurl+'/tasks/claim', json=data, verify=certfile, headers=pipeline_headers) - if not response.ok: - print(f"\"{response.reason}\" received from {response.url}") - exit(1) - - print(response.json()) - -if (args.command == 'update'): - data = { 'pipeline': - { 'name': args.pipeline, 'uri': args.pipeline_url, "version": args.pipeline_version }, - 'task_input': { 'id_run': args.idrun, 'sample': args.sample, 'id_study': args.study }, - 'status': args.status - } - response = requests.put(args.baseurl+'/tasks', json=data, verify=certfile, headers=pipeline_headers) - if not response.ok: - print(f"\"{response.reason}\" received from {response.url}") - exit(1) - - print(data) - print(response.json()) - diff --git a/pyproject.toml b/pyproject.toml index 3ae58a1..338747b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -12,10 +12,12 @@ license = "GPL-3.0-or-later" [tool.poetry.dependencies] python = "^3.10" +requests = "^2.31.0" [tool.poetry.dev-dependencies] black = "^22.3.0" flake8 = "^4.0.1" +flake8-bugbear = "^18.2.0" pytest = "^7.1.1" isort = { version = "^5.10.1", extras = ["colors"] } @@ -26,6 +28,9 @@ build-backend = "poetry.core.masonry.api" [tool.isort] profile = "black" +[tool.black] +line_length = 100 + [tool.pytest.ini_options] addopts = [ "--import-mode=importlib", diff --git a/src/npg_porch_cli/npg_porch_client.py b/src/npg_porch_cli/npg_porch_client.py new file mode 100755 index 0000000..abc212c --- /dev/null +++ b/src/npg_porch_cli/npg_porch_client.py @@ -0,0 +1,160 @@ +#!/usr/bin/env python3 + +# Copyright (C) 2022, 2024 Genome Research Ltd. +# +# Author: Jennifer Liddle +# +# npg_porch_client is free software: you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 3 of the License, or (at your option) any +# later version. +# +# This program is distributed in the hope that it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS +# FOR A PARTICULAR PURPOSE. See the GNU General Public License for more +# details. +# +# You should have received a copy of the GNU General Public License along with +# this program. If not, see . + +import argparse + +import requests + +# Certification file for https requests +# if we don't have one, we can set certfile = False +certfile = ( + "/usr/share/ca-certificates/sanger.ac.uk/Genome_Research_Ltd_Certificate_Authority-cert.pem" +) + +# tokens from Porch +admin_headers = {"Authorization": "Bearer a"} +pipeline_headers = {"Authorization": "Bearer b"} + +# Command line arguments +parser = argparse.ArgumentParser(description="Pipeline Orchestration Wrapper") +parser.add_argument("--baseurl", type=str, help="base URL") +parser.add_argument("--pipeline_url", type=str, help="Pipeline git project URL") +parser.add_argument("--pipeline_version", type=str, help="Pipeline version") +parser.add_argument("--pipeline", type=str, help="pipeline name") +parser.add_argument("--idrun", type=int, help="id_run") +parser.add_argument("--sample", type=str, help="sample name") +parser.add_argument("--study", type=str, help="study ID") +parser.add_argument( + "--status", + type=str, + help="new status to set", + default="DONE", + choices=["PENDING", "CLAIMED", "RUNNING", "DONE", "FAILED", "CANCELLED"], +) +parser.add_argument( + "command", + type=str, + help="command to send to npg_porch", + choices=["list", "plist", "register", "add", "claim", "update"], +) +args = parser.parse_args() + +if args.command == "list": + response = requests.get(args.baseurl + "/tasks", verify=certfile, headers=pipeline_headers) + if not response.ok: + print(f'"{response.reason}" received from {response.url}') + exit(1) + + x = response.json() + for p in x: + if p["pipeline"]["name"] == args.pipeline: + print(f"{p['task_input']}\t{p['status']}") + +if args.command == "plist": + response = requests.get(args.baseurl + "/pipelines", verify=certfile, headers=admin_headers) + if not response.ok: + print(f'"{response.reason}" received from {response.url}') + exit(1) + + x = response.json() + pipelines = {} + for p in x: + pname = p["name"] + if pname not in pipelines: + print(pname) + pipelines[pname] = 1 + +if args.command == "register": + data = { + "name": args.pipeline, + "uri": args.pipeline_url, + "version": args.pipeline_version, + } + response = requests.post( + args.baseurl + "/pipelines", json=data, verify=certfile, headers=admin_headers + ) + if not response.ok: + print(f'"{response.reason}" received from {response.url}') + exit(1) + + print(response.json()) + +if args.command == "add": + data = { + "pipeline": { + "name": args.pipeline, + "uri": args.pipeline_url, + "version": args.pipeline_version, + }, + "task_input": { + "id_run": args.idrun, + "sample": args.sample, + "id_study": args.study, + }, + } + response = requests.post( + args.baseurl + "/tasks", json=data, verify=certfile, headers=pipeline_headers + ) + if not response.ok: + print(f'"{response.reason}" received from {response.url}') + exit(1) + + print(response.json()) + +if args.command == "claim": + data = { + "name": args.pipeline, + "uri": args.pipeline_url, + "version": args.pipeline_version, + } + response = requests.post( + args.baseurl + "/tasks/claim", + json=data, + verify=certfile, + headers=pipeline_headers, + ) + if not response.ok: + print(f'"{response.reason}" received from {response.url}') + exit(1) + + print(response.json()) + +if args.command == "update": + data = { + "pipeline": { + "name": args.pipeline, + "uri": args.pipeline_url, + "version": args.pipeline_version, + }, + "task_input": { + "id_run": args.idrun, + "sample": args.sample, + "id_study": args.study, + }, + "status": args.status, + } + response = requests.put( + args.baseurl + "/tasks", json=data, verify=certfile, headers=pipeline_headers + ) + if not response.ok: + print(f'"{response.reason}" received from {response.url}') + exit(1) + + print(data) + print(response.json()) diff --git a/tests/test_api.py b/tests/test_api.py index 4e2a9c0..0ae0c2f 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -1,5 +1,6 @@ import npg_porch_cli.api as porchApi + def test_addition(): assert porchApi.add_two(1, 2) == 3 From aa594ee323a880fbe29a7be12410b7cb8b54eb73 Mon Sep 17 00:00:00 2001 From: mgcam Date: Thu, 25 Apr 2024 13:02:46 +0100 Subject: [PATCH 06/27] Manage flake8 via pyproject-flake8 And configure flake8 in pyproject.toml --- .flake8 | 6 ------ .github/workflows/test.yml | 2 +- pyproject.toml | 6 ++++-- 3 files changed, 5 insertions(+), 9 deletions(-) delete mode 100644 .flake8 diff --git a/.flake8 b/.flake8 deleted file mode 100644 index 45aa47f..0000000 --- a/.flake8 +++ /dev/null @@ -1,6 +0,0 @@ -[flake8] -max-line-length = 100 -extend-select = B950 -extend-ignore = E203, E501, E701 -ignore = E251, E265, E261, E302, W503 -per-file-ignores = __init__.py:F401 diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index b697fb7..5a44a45 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -39,7 +39,7 @@ jobs: - name: Run code style checker run: | - poetry run flake8 + poetry run pflake8 - name: Run code formatter run: | diff --git a/pyproject.toml b/pyproject.toml index 338747b..c769f61 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -16,8 +16,7 @@ requests = "^2.31.0" [tool.poetry.dev-dependencies] black = "^22.3.0" -flake8 = "^4.0.1" -flake8-bugbear = "^18.2.0" +pyproject-flake8 = "^7.0.0" pytest = "^7.1.1" isort = { version = "^5.10.1", extras = ["colors"] } @@ -31,6 +30,9 @@ profile = "black" [tool.black] line_length = 100 +[tool.flake8] +max-line-length = 100 + [tool.pytest.ini_options] addopts = [ "--import-mode=importlib", From 3d0a05dd146363311f03570c3dab682a8f2aa9e1 Mon Sep 17 00:00:00 2001 From: mgcam Date: Thu, 25 Apr 2024 13:15:27 +0100 Subject: [PATCH 07/27] Reduced max line length to 88. --- pyproject.toml | 4 ++-- src/npg_porch_cli/npg_porch_client.py | 12 +++++++----- 2 files changed, 9 insertions(+), 7 deletions(-) diff --git a/pyproject.toml b/pyproject.toml index c769f61..5078197 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -28,10 +28,10 @@ build-backend = "poetry.core.masonry.api" profile = "black" [tool.black] -line_length = 100 +line_length = 88 [tool.flake8] -max-line-length = 100 +max-line-length = 88 [tool.pytest.ini_options] addopts = [ diff --git a/src/npg_porch_cli/npg_porch_client.py b/src/npg_porch_cli/npg_porch_client.py index abc212c..1d7e57b 100755 --- a/src/npg_porch_cli/npg_porch_client.py +++ b/src/npg_porch_cli/npg_porch_client.py @@ -23,9 +23,7 @@ # Certification file for https requests # if we don't have one, we can set certfile = False -certfile = ( - "/usr/share/ca-certificates/sanger.ac.uk/Genome_Research_Ltd_Certificate_Authority-cert.pem" -) +certfile = "/usr/share/ca-certificates/sanger.ac.uk/Genome_Research_Ltd_Certificate_Authority-cert.pem" # noqa: E501 # tokens from Porch admin_headers = {"Authorization": "Bearer a"} @@ -56,7 +54,9 @@ args = parser.parse_args() if args.command == "list": - response = requests.get(args.baseurl + "/tasks", verify=certfile, headers=pipeline_headers) + response = requests.get( + args.baseurl + "/tasks", verify=certfile, headers=pipeline_headers + ) if not response.ok: print(f'"{response.reason}" received from {response.url}') exit(1) @@ -67,7 +67,9 @@ print(f"{p['task_input']}\t{p['status']}") if args.command == "plist": - response = requests.get(args.baseurl + "/pipelines", verify=certfile, headers=admin_headers) + response = requests.get( + args.baseurl + "/pipelines", verify=certfile, headers=admin_headers + ) if not response.ok: print(f'"{response.reason}" received from {response.url}') exit(1) From b5e1a4d9ca45daa4225dda19e256b04c408049a7 Mon Sep 17 00:00:00 2001 From: mgcam Date: Thu, 25 Apr 2024 13:50:15 +0100 Subject: [PATCH 08/27] The bugbear returns. --- pyproject.toml | 3 +++ src/npg_porch_cli/npg_porch_client.py | 2 +- 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/pyproject.toml b/pyproject.toml index 5078197..ded1888 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -17,6 +17,7 @@ requests = "^2.31.0" [tool.poetry.dev-dependencies] black = "^22.3.0" pyproject-flake8 = "^7.0.0" +flake8-bugbear = "^24.4.0" pytest = "^7.1.1" isort = { version = "^5.10.1", extras = ["colors"] } @@ -32,6 +33,8 @@ line_length = 88 [tool.flake8] max-line-length = 88 +extend-select = ["B950"] +extend-ignore = ["E501"] [tool.pytest.ini_options] addopts = [ diff --git a/src/npg_porch_cli/npg_porch_client.py b/src/npg_porch_cli/npg_porch_client.py index 1d7e57b..773b4a4 100755 --- a/src/npg_porch_cli/npg_porch_client.py +++ b/src/npg_porch_cli/npg_porch_client.py @@ -23,7 +23,7 @@ # Certification file for https requests # if we don't have one, we can set certfile = False -certfile = "/usr/share/ca-certificates/sanger.ac.uk/Genome_Research_Ltd_Certificate_Authority-cert.pem" # noqa: E501 +certfile = "/usr/share/ca-certificates/sanger.ac.uk/Genome_Research_Ltd_Certificate_Authority-cert.pem" # noqa: B950 # tokens from Porch admin_headers = {"Authorization": "Bearer a"} From 59bef420aa8c2b96667efb700b95bb76509f1a17 Mon Sep 17 00:00:00 2001 From: mgcam Date: Fri, 26 Apr 2024 13:48:02 +0100 Subject: [PATCH 09/27] Created porch client API. Refactored the existing client script to use this API. --- README.md | 17 ++- pyproject.toml | 3 + src/npg_porch_cli/api.py | 162 +++++++++++++++++++++++++- src/npg_porch_cli/api_cli_user.py | 113 ++++++++++++++++++ src/npg_porch_cli/npg_porch_client.py | 162 -------------------------- tests/test_api.py | 62 +++++++++- 6 files changed, 349 insertions(+), 170 deletions(-) create mode 100755 src/npg_porch_cli/api_cli_user.py delete mode 100755 src/npg_porch_cli/npg_porch_client.py diff --git a/README.md b/README.md index 370fb0d..9fdb6bd 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,17 @@ # npg_porch_cli -A command-line client to enable communication with the npg_porch API. + +A command-line client and a simple client library to enable communication +with the [npg_porch](https://github.com/wtsi-npg/npg_porch) JSON API. + +Provides a Python script, `npg_porch_client`, and a Python client API. + +Can be deployed with pip or poetry in a standard way. + +Example of using a client API: + +``` python + from npg_porch_cli.api import PorchRequest + + pr = PorchRequest(porch_url="https://myporch.com") + response = pr.send(action="list_pipelines") +``` diff --git a/pyproject.toml b/pyproject.toml index ded1888..64f06c0 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -10,6 +10,9 @@ description = "CLI client for communicating with npg_porch JSON API" readme = "README.md" license = "GPL-3.0-or-later" +[tool.poetry.scripts] +npg_porch_client = "npg_porch_cli.api_cli_user:run" + [tool.poetry.dependencies] python = "^3.10" requests = "^2.31.0" diff --git a/src/npg_porch_cli/api.py b/src/npg_porch_cli/api.py index 4668b38..5cbc446 100644 --- a/src/npg_porch_cli/api.py +++ b/src/npg_porch_cli/api.py @@ -1,8 +1,9 @@ # Copyright (c) 2024 Genome Research Ltd. # -# Author: Marina Gourtovaia +# Authors: Marina Gourtovaia +# Jennifer Liddle # -# This file is part of npg_langqc. +# This file is part of npg_porch_cli project. # # npg_porch_cli is free software: you can redistribute it and/or modify it under # the terms of the GNU General Public License as published by the Free Software @@ -17,6 +18,159 @@ # You should have received a copy of the GNU General Public License along with # this program. If not, see . +import os +from dataclasses import dataclass, field -def add_two(a, b): - return a + b +import requests + +PORCH_CLIENT_ACTIONS = { + "list_tasks": "tasks", + "list_pipelines": "pipelines", + "add_pipeline": "pipelines", + "add_task": "tasks", + "claim_task": "tasks/claim", + "update_task": "tasks", +} + +PORCH_STATUSES = ["PENDING", "CLAIMED", "RUNNING", "DONE", "FAILED", "CANCELLED"] + +CLIENT_TIMEOUT = (10, 60) + +NPG_PORCH_TOKEN_ENV_VAR = "NPG_PORCH_TOKEN" + + +class AuthException(Exception): + pass + + +class InvalidValueException(Exception): + pass + + +class ServerErrorException(Exception): + pass + + +@dataclass +class PorchRequest: + + porch_url: str + ca_cert: str | None = field(default=None) + pipeline_name: str | None = field(default=None) + pipeline_url: str | None = field(default=None) + pipeline_version: str | None = field(default=None) + + def send(self, action: str, task_input: str | None, task_status: str | None): + + if task_status is not None: + task_status = task_status.upper() + self._validate_request( + action=action, task_input=task_input, task_status=task_status + ) + + method = "GET" + pipeline_data = { + "name": self.pipeline_name, + "uri": self.pipeline_url, + "version": self.pipeline_version, + } + data = None + + if action == "update_task": + method = "PUT" + data = { + "pipeline": pipeline_data, + "task_input": task_input, + "status": task_status, + } + elif action.startswith("list") is False: + method = "POST" + data = pipeline_data + if action == "add_task": + data = {"pipeline": pipeline_data, "task_input": task_input} + + request_args = { + "headers": self._get_request_headers(action), + "timeout": CLIENT_TIMEOUT, + } + if self.ca_cert is None: + request_args["verify"] = False + else: + request_args["cert"] = self.ca_cert + if data is not None: + request_args["json"] = data + + response = requests.request( + method, self._generate_request_url(action), **request_args + ) + if not response.ok: + raise ServerErrorException( + f"Action {action} failed. " + f'Status code {response.status_code} "{response.reason}" ' + f"received from {response.url}" + ) + + response_obj = response.json() + if action == "list_tasks" and self.pipeline_name is not None: + response_obj = [ + o for o in response_obj if o["pipeline"]["name"] == self.pipeline_name + ] + + return response_obj + + def _generate_request_url(self, action: str): + return "/".join([self.porch_url, PORCH_CLIENT_ACTIONS[action]]) + + def _get_token(self): + if NPG_PORCH_TOKEN_ENV_VAR not in os.environ: + raise AuthException("Authorization token is needed") + return os.environ[NPG_PORCH_TOKEN_ENV_VAR] + + def _get_request_headers(self, action: str): + headers = {"Authorization": "Bearer " + self._get_token()} + if action.startswith("list") is False: + headers["Content-Type"] = "application/json" + return headers + + def _validate_request( + self, action: str, task_status: str | None, task_input: str | None + ): + + if action not in PORCH_CLIENT_ACTIONS: + raise InvalidValueException( + f"Action '{action}' is not valid. " + "Valid actions: " + ", ".join(sorted(PORCH_CLIENT_ACTIONS.keys())) + ) + + if task_status is not None and task_status not in PORCH_STATUSES: + raise InvalidValueException( + f"Task status '{task_status}' is not valid. " + "Valid statuses: " + ", ".join(PORCH_STATUSES) + ) + + if action.startswith("list") is False: + if ( + self.pipeline_name is None + or self.pipeline_url is None + or self.pipeline_version is None + ): + raise InvalidValueException( + f"Full pipeline details should be defined for action {action}" + ) + + if ( + action.endswith("task") is True + and action.startswith("claim") is False + and task_input is None + ): + raise InvalidValueException( + f"task_input should be defined for action {action}" + ) + + if action == "update_task": + if task_status is None: + raise InvalidValueException( + f"task_status should be defined for action {action}" + ) + + return True diff --git a/src/npg_porch_cli/api_cli_user.py b/src/npg_porch_cli/api_cli_user.py new file mode 100755 index 0000000..8c7f8e0 --- /dev/null +++ b/src/npg_porch_cli/api_cli_user.py @@ -0,0 +1,113 @@ +#!/usr/bin/env python3 + +# Copyright (C) 2022, 2024 Genome Research Ltd. +# +# Author: Jennifer Liddle +# +# This file is part of npg_porch_cli project. +# +# npg_porch_cli is free software: you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by the Free +# Software Foundation; either version 3 of the License, or (at your option) any +# later version. +# +# This program is distributed in the hope that it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS +# FOR A PARTICULAR PURPOSE. See the GNU General Public License for more +# details. +# +# You should have received a copy of the GNU General Public License along with +# this program. If not, see . + +import argparse + +from npg_porch_cli.api import PorchRequest + + +def run(): + """ + Parses command line arguments, send the request to porch server API, + prints out the server's response to STDOUT. + + The text below assumes that this code is deployed as a script called + `npg_porch_client`. + + One positional argument, the name of the action to be performed, is required. + One named argument `--base_url`, teh URL of the `porch` server, is also + required. + + npg_porch_client list_pipeline --base_url SOME_URL + + A full list of actions: + list_tasks + list_pipelines + add_pipeline + add_task + claim_task + update_task + + Though most of named arguments are optional, some actions require + certain combinations of arguments to be defined. + + All list actions do not require any optional arguments defined. If + `--pipeline_name` is defined, `list_tasks` returns a list of tasks for + this pipeline, otherwise all registered tasks are returned. + + All non-list actions require all `--pipeline`, `pipeline_url` and + `--pipeline_version` defined. + + The `add_task` and `update_task` actions require the `--task_json` to + be defined. In addition to this, for the `update_task` action `--status` + should be defined. + + NPG_PORCH_TOKEN environment variable should be set up to the value of + either an admin or project-specific token. + + """ + parser = argparse.ArgumentParser( + prog="npg_porch_client", + description="npg_porch (Pipeline Orchestration)API Client", + epilog="The server JSON reply is printed to STDOUT", + ) + + parser.add_argument( + "action", + type=str, + help="Action to send to npg_porch server API", + choices=[ + "list_tasks", + "list_pipelines", + "add_pipeline", + "add_task", + "claim_task", + "update_task", + ], + ) + parser.add_argument("--base_url", type=str, required=True, help="Base URL") + parser.add_argument( + "--certfile", type=str, help="Server CA certificate path, optional" + ) + parser.add_argument( + "--pipeline_url", type=str, help="Pipeline git project URL, optional" + ) + parser.add_argument( + "--pipeline_version", type=str, help="Pipeline version, optional" + ) + parser.add_argument("--pipeline", type=str, help="Pipeline name, optional") + parser.add_argument("--task_json", type=str, help="Task as JSON, optional") + parser.add_argument("--status", type=str, help="New status to set, optional") + + args = parser.parse_args() + + r = PorchRequest( + porch_url=args.base_url, + ca_cert=args.certfile, + pipeline_name=args.pipeline, + pipeline_url=args.pipeline_url, + pipeline_version=args.pipeline_version, + ) + + response = r.send( + action=args.action, task_input=args.task_json, task_status=args.status + ) + print(response) diff --git a/src/npg_porch_cli/npg_porch_client.py b/src/npg_porch_cli/npg_porch_client.py deleted file mode 100755 index 773b4a4..0000000 --- a/src/npg_porch_cli/npg_porch_client.py +++ /dev/null @@ -1,162 +0,0 @@ -#!/usr/bin/env python3 - -# Copyright (C) 2022, 2024 Genome Research Ltd. -# -# Author: Jennifer Liddle -# -# npg_porch_client is free software: you can redistribute it and/or modify it -# under the terms of the GNU General Public License as published by the Free -# Software Foundation; either version 3 of the License, or (at your option) any -# later version. -# -# This program is distributed in the hope that it will be useful, but WITHOUT -# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS -# FOR A PARTICULAR PURPOSE. See the GNU General Public License for more -# details. -# -# You should have received a copy of the GNU General Public License along with -# this program. If not, see . - -import argparse - -import requests - -# Certification file for https requests -# if we don't have one, we can set certfile = False -certfile = "/usr/share/ca-certificates/sanger.ac.uk/Genome_Research_Ltd_Certificate_Authority-cert.pem" # noqa: B950 - -# tokens from Porch -admin_headers = {"Authorization": "Bearer a"} -pipeline_headers = {"Authorization": "Bearer b"} - -# Command line arguments -parser = argparse.ArgumentParser(description="Pipeline Orchestration Wrapper") -parser.add_argument("--baseurl", type=str, help="base URL") -parser.add_argument("--pipeline_url", type=str, help="Pipeline git project URL") -parser.add_argument("--pipeline_version", type=str, help="Pipeline version") -parser.add_argument("--pipeline", type=str, help="pipeline name") -parser.add_argument("--idrun", type=int, help="id_run") -parser.add_argument("--sample", type=str, help="sample name") -parser.add_argument("--study", type=str, help="study ID") -parser.add_argument( - "--status", - type=str, - help="new status to set", - default="DONE", - choices=["PENDING", "CLAIMED", "RUNNING", "DONE", "FAILED", "CANCELLED"], -) -parser.add_argument( - "command", - type=str, - help="command to send to npg_porch", - choices=["list", "plist", "register", "add", "claim", "update"], -) -args = parser.parse_args() - -if args.command == "list": - response = requests.get( - args.baseurl + "/tasks", verify=certfile, headers=pipeline_headers - ) - if not response.ok: - print(f'"{response.reason}" received from {response.url}') - exit(1) - - x = response.json() - for p in x: - if p["pipeline"]["name"] == args.pipeline: - print(f"{p['task_input']}\t{p['status']}") - -if args.command == "plist": - response = requests.get( - args.baseurl + "/pipelines", verify=certfile, headers=admin_headers - ) - if not response.ok: - print(f'"{response.reason}" received from {response.url}') - exit(1) - - x = response.json() - pipelines = {} - for p in x: - pname = p["name"] - if pname not in pipelines: - print(pname) - pipelines[pname] = 1 - -if args.command == "register": - data = { - "name": args.pipeline, - "uri": args.pipeline_url, - "version": args.pipeline_version, - } - response = requests.post( - args.baseurl + "/pipelines", json=data, verify=certfile, headers=admin_headers - ) - if not response.ok: - print(f'"{response.reason}" received from {response.url}') - exit(1) - - print(response.json()) - -if args.command == "add": - data = { - "pipeline": { - "name": args.pipeline, - "uri": args.pipeline_url, - "version": args.pipeline_version, - }, - "task_input": { - "id_run": args.idrun, - "sample": args.sample, - "id_study": args.study, - }, - } - response = requests.post( - args.baseurl + "/tasks", json=data, verify=certfile, headers=pipeline_headers - ) - if not response.ok: - print(f'"{response.reason}" received from {response.url}') - exit(1) - - print(response.json()) - -if args.command == "claim": - data = { - "name": args.pipeline, - "uri": args.pipeline_url, - "version": args.pipeline_version, - } - response = requests.post( - args.baseurl + "/tasks/claim", - json=data, - verify=certfile, - headers=pipeline_headers, - ) - if not response.ok: - print(f'"{response.reason}" received from {response.url}') - exit(1) - - print(response.json()) - -if args.command == "update": - data = { - "pipeline": { - "name": args.pipeline, - "uri": args.pipeline_url, - "version": args.pipeline_version, - }, - "task_input": { - "id_run": args.idrun, - "sample": args.sample, - "id_study": args.study, - }, - "status": args.status, - } - response = requests.put( - args.baseurl + "/tasks", json=data, verify=certfile, headers=pipeline_headers - ) - if not response.ok: - print(f'"{response.reason}" received from {response.url}') - exit(1) - - print(data) - print(response.json()) diff --git a/tests/test_api.py b/tests/test_api.py index 0ae0c2f..b16de62 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -1,6 +1,62 @@ -import npg_porch_cli.api as porchApi +import pytest +from npg_porch_cli.api import AuthException, InvalidValueException, PorchRequest -def test_addition(): +url = "http://some.com" - assert porchApi.add_two(1, 2) == 3 + +def test_validation(): + + r = PorchRequest(porch_url=url) + + with pytest.raises(InvalidValueException) as e: + r._validate_request(action="list_tools", task_status=None, task_input=None) + assert ( + e.value.args[0] == "Action 'list_tools' is not valid. " + "Valid actions: add_pipeline, add_task, claim_task, list_pipelines, " + "list_tasks, update_task" + ) + assert ( + r._validate_request(action="list_tasks", task_status=None, task_input=None) + is True + ) + + with pytest.raises(InvalidValueException) as e: + r._validate_request(action="list_tasks", task_status="Some", task_input=None) + assert ( + e.value.args[0] == "Task status 'Some' is not valid. " + "Valid statuses: PENDING, CLAIMED, RUNNING, DONE, FAILED, CANCELLED" + ) + with pytest.raises(InvalidValueException) as e: + r._validate_request(action="list_tasks", task_status="Running", task_input=None) + assert ( + e.value.args[0] == "Task status 'Running' is not valid. " + "Valid statuses: PENDING, CLAIMED, RUNNING, DONE, FAILED, CANCELLED" + ) + + +def test_url_generation(): + + r = PorchRequest(porch_url=url) + assert r._generate_request_url(action="list_tasks") == "/".join([url, "tasks"]) + + +def test_header_generation(monkeypatch): + + var_name = "NPG_PORCH_TOKEN" + r = PorchRequest(porch_url=url) + + monkeypatch.delenv(var_name, raising=False) + with pytest.raises(AuthException) as e: + r._get_request_headers(action="list_tasks") + assert e.value.args[0] == "Authorization token is needed" + + monkeypatch.setenv(var_name, "token_xyz") + assert r._get_request_headers(action="list_tasks") == { + "Authorization": "Bearer token_xyz" + } + assert r._get_request_headers(action="add_tasks") == { + "Content-Type": "application/json", + "Authorization": "Bearer token_xyz", + } + monkeypatch.undo() From 31a3e72f3788e4967dd8141628840a22a910557b Mon Sep 17 00:00:00 2001 From: mgcam Date: Tue, 30 Apr 2024 15:56:43 +0100 Subject: [PATCH 10/27] Streamlined handling of CA certificates --- README.md | 14 ++++++++++++++ src/npg_porch_cli/api.py | 7 ++----- src/npg_porch_cli/api_cli_user.py | 8 ++++++-- 3 files changed, 22 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 9fdb6bd..00bd17d 100644 --- a/README.md +++ b/README.md @@ -15,3 +15,17 @@ Example of using a client API: pr = PorchRequest(porch_url="https://myporch.com") response = pr.send(action="list_pipelines") ``` + +By default the client is set up to validate the server's CA certificate. +If the server is using a custom CA certificate, set the path to the certificate. + +``` bash + export SSL_CERT_FILE=/path_to/my.pem + npg_porch_client list_pipelines --base_url https://myporch.com +``` + +It is possible, but not recommended, to disable this validation check. + +``` bash + npg_porch_client list_pipelines --base_url https://myporch.com --no-validate_ca_cert +``` diff --git a/src/npg_porch_cli/api.py b/src/npg_porch_cli/api.py index 5cbc446..f075776 100644 --- a/src/npg_porch_cli/api.py +++ b/src/npg_porch_cli/api.py @@ -55,7 +55,7 @@ class ServerErrorException(Exception): class PorchRequest: porch_url: str - ca_cert: str | None = field(default=None) + validate_ca_cert: bool = field(default=True) pipeline_name: str | None = field(default=None) pipeline_url: str | None = field(default=None) pipeline_version: str | None = field(default=None) @@ -92,11 +92,8 @@ def send(self, action: str, task_input: str | None, task_status: str | None): request_args = { "headers": self._get_request_headers(action), "timeout": CLIENT_TIMEOUT, + "verify": self.validate_ca_cert, } - if self.ca_cert is None: - request_args["verify"] = False - else: - request_args["cert"] = self.ca_cert if data is not None: request_args["json"] = data diff --git a/src/npg_porch_cli/api_cli_user.py b/src/npg_porch_cli/api_cli_user.py index 8c7f8e0..b7b0757 100755 --- a/src/npg_porch_cli/api_cli_user.py +++ b/src/npg_porch_cli/api_cli_user.py @@ -85,7 +85,11 @@ def run(): ) parser.add_argument("--base_url", type=str, required=True, help="Base URL") parser.add_argument( - "--certfile", type=str, help="Server CA certificate path, optional" + "--validate_ca_cert", + action=argparse.BooleanOptionalAction, + type=bool, + help="A flag instructing to validate server's CA SSL certificate, true by default", + default=True, ) parser.add_argument( "--pipeline_url", type=str, help="Pipeline git project URL, optional" @@ -101,7 +105,7 @@ def run(): r = PorchRequest( porch_url=args.base_url, - ca_cert=args.certfile, + validate_ca_cert=args.validate_ca_cert, pipeline_name=args.pipeline, pipeline_url=args.pipeline_url, pipeline_version=args.pipeline_version, From c7baaa11262b29c775822ad7640669a05e094147 Mon Sep 17 00:00:00 2001 From: mgcam Date: Tue, 30 Apr 2024 17:37:49 +0100 Subject: [PATCH 11/27] On the client API level represent the task as a dict. --- README.md | 30 ++++++++++++++++++++++++++++++ src/npg_porch_cli/api.py | 2 +- src/npg_porch_cli/api_cli_user.py | 7 ++++++- 3 files changed, 37 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 00bd17d..9532847 100644 --- a/README.md +++ b/README.md @@ -29,3 +29,33 @@ It is possible, but not recommended, to disable this validation check. ``` bash npg_porch_client list_pipelines --base_url https://myporch.com --no-validate_ca_cert ``` + +A valid JSON string is required for the `--task_json` script's argument, note +double quotes in the example below. + +``` bash + npg_porch_client update_task --base_url https://myporch.com \ + --pipeline Snakemake_Cardinal \ + --pipeline_url 'https://github.com/wtsi-npg/snakemake_cardinal' \ + --pipeline_version 1.0 \ + --task_json '{"id_run": 409, "sample": "Valxxxx", "id_study": "65"}' \ + --status FAILED +``` + +If using the client API directly from Python, a dictionary should be used. + +``` python + from npg_porch_cli.api import PorchRequest + + pr = PorchRequest( + porch_url="https://myporch.com", + pipeline_name="Snakemake_Cardinal", + pipeline_url="https://github.com/wtsi-npg/snakemake_cardinal", + pipeline_version="1.0", + ) + response = pr.send( + action="update_task", + task_status="FAILED", + task_input={"id_run": 409, "sample": "Valxxxx", "id_study": "65"}, + ) +``` diff --git a/src/npg_porch_cli/api.py b/src/npg_porch_cli/api.py index f075776..c8cad93 100644 --- a/src/npg_porch_cli/api.py +++ b/src/npg_porch_cli/api.py @@ -60,7 +60,7 @@ class PorchRequest: pipeline_url: str | None = field(default=None) pipeline_version: str | None = field(default=None) - def send(self, action: str, task_input: str | None, task_status: str | None): + def send(self, action: str, task_input: dict | None, task_status: str | None): if task_status is not None: task_status = task_status.upper() diff --git a/src/npg_porch_cli/api_cli_user.py b/src/npg_porch_cli/api_cli_user.py index b7b0757..63fb7aa 100755 --- a/src/npg_porch_cli/api_cli_user.py +++ b/src/npg_porch_cli/api_cli_user.py @@ -20,6 +20,7 @@ # this program. If not, see . import argparse +import json from npg_porch_cli.api import PorchRequest @@ -111,7 +112,11 @@ def run(): pipeline_version=args.pipeline_version, ) + parsed_json = None + if args.task_json is not None: + print(args.task_json) + parsed_json = json.loads(args.task_json) response = r.send( - action=args.action, task_input=args.task_json, task_status=args.status + action=args.action, task_input=parsed_json, task_status=args.status ) print(response) From 0bd5229742ca64e6fe63097ac48fb445d11453cc Mon Sep 17 00:00:00 2001 From: mgcam Date: Wed, 1 May 2024 15:37:05 +0100 Subject: [PATCH 12/27] Validate status against porch OpenAPI schema --- src/npg_porch_cli/api.py | 72 +++++++++++--- tests/data/porch_openapi.json | 1 + tests/test_api.py | 181 ++++++++++++++++++++++++++++++++-- 3 files changed, 233 insertions(+), 21 deletions(-) create mode 100644 tests/data/porch_openapi.json diff --git a/src/npg_porch_cli/api.py b/src/npg_porch_cli/api.py index c8cad93..a404989 100644 --- a/src/npg_porch_cli/api.py +++ b/src/npg_porch_cli/api.py @@ -20,6 +20,7 @@ import os from dataclasses import dataclass, field +from urllib.parse import urljoin import requests @@ -32,6 +33,9 @@ "update_task": "tasks", } +PORCH_OPENAPI_SCHEMA_URL = "api/v1/openapi.json" +PORCH_TASK_STATUS_ENUM_NAME = "TaskStateEnum" + PORCH_STATUSES = ["PENDING", "CLAIMED", "RUNNING", "DONE", "FAILED", "CANCELLED"] CLIENT_TIMEOUT = (10, 60) @@ -60,10 +64,22 @@ class PorchRequest: pipeline_url: str | None = field(default=None) pipeline_version: str | None = field(default=None) - def send(self, action: str, task_input: dict | None, task_status: str | None): + def send( + self, + action: str, + task_input: dict | None = None, + task_status: str | None = None, + ) -> dict: + """ + Sends a request to the porch API server to perform an action defined + by the `action` argument. Either of two optional arguments, if defined, + are used when constructing the request. + + The server's response is returned as a dictionary. + """ if task_status is not None: - task_status = task_status.upper() + task_status = self.validate_status(task_status=task_status) self._validate_request( action=action, task_input=task_input, task_status=task_status ) @@ -115,8 +131,46 @@ def send(self, action: str, task_input: dict | None, task_status: str | None): return response_obj + def validate_status(self, task_status: str) -> str: + """ + Retrieves OpenAPI schema for the porch server and validates the given + task status value against the values listed in the schema document. + + Returns a validated task status value. The case of this string can be + different from the input string. + """ + url = urljoin(self.porch_url, PORCH_OPENAPI_SCHEMA_URL) + response = requests.request("GET", url) + if not response.ok: + raise ServerErrorException( + f"Failed to get OpenAPI Schema. " + f'Status code {response.status_code} "{response.reason}" ' + f"received from {response.url}" + ) + + status = task_status.upper() + valid_statuses = [] + error_message = f"Failed to get enumeration of valid statuses from {url}" + try: + valid_statuses = response.json()["components"]["schemas"][ + PORCH_TASK_STATUS_ENUM_NAME + ]["enum"] + except Exception as e: + raise Exception(f"{error_message}: " + e.__str__()) + + if len(valid_statuses) == 0: + raise Exception(error_message) + + if status not in valid_statuses: + raise InvalidValueException( + f"Task status '{task_status}' is not valid. " + "Valid statuses: " + ", ".join(sorted(valid_statuses)) + ) + + return status + def _generate_request_url(self, action: str): - return "/".join([self.porch_url, PORCH_CLIENT_ACTIONS[action]]) + return urljoin(self.porch_url, PORCH_CLIENT_ACTIONS[action]) def _get_token(self): if NPG_PORCH_TOKEN_ENV_VAR not in os.environ: @@ -139,12 +193,6 @@ def _validate_request( "Valid actions: " + ", ".join(sorted(PORCH_CLIENT_ACTIONS.keys())) ) - if task_status is not None and task_status not in PORCH_STATUSES: - raise InvalidValueException( - f"Task status '{task_status}' is not valid. " - "Valid statuses: " + ", ".join(PORCH_STATUSES) - ) - if action.startswith("list") is False: if ( self.pipeline_name is None @@ -152,7 +200,7 @@ def _validate_request( or self.pipeline_version is None ): raise InvalidValueException( - f"Full pipeline details should be defined for action {action}" + f"Full pipeline details should be defined for action '{action}'" ) if ( @@ -161,13 +209,13 @@ def _validate_request( and task_input is None ): raise InvalidValueException( - f"task_input should be defined for action {action}" + f"task_input argument should be defined for action '{action}'" ) if action == "update_task": if task_status is None: raise InvalidValueException( - f"task_status should be defined for action {action}" + f"task_status argument should be defined for action '{action}'" ) return True diff --git a/tests/data/porch_openapi.json b/tests/data/porch_openapi.json new file mode 100644 index 0000000..ae11577 --- /dev/null +++ b/tests/data/porch_openapi.json @@ -0,0 +1 @@ +{"openapi":"3.1.0","info":{"title":"Pipeline Orchestration (POrch)","version":"0.1.0"},"paths":{"/pipelines/":{"get":{"tags":["pipelines"],"summary":"Get information about all pipelines.","description":"Returns a list of pydantic Pipeline models.\n A uri and/or version filter can be used.\n A valid token issued for any pipeline is required for authorisation.","operationId":"get_pipelines_pipelines__get","security":[{"HTTPBearer":[]}],"parameters":[{"name":"uri","in":"query","required":false,"schema":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"Uri"}},{"name":"version","in":"query","required":false,"schema":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"Version"}}],"responses":{"200":{"description":"Successful Response","content":{"application/json":{"schema":{"type":"array","items":{"$ref":"#/components/schemas/Pipeline"},"title":"Response Get Pipelines Pipelines Get"}}}},"403":{"description":"Not authorised"},"500":{"description":"Unexpected error"},"422":{"description":"Validation Error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/HTTPValidationError"}}}}}},"post":{"tags":["pipelines"],"summary":"Create one pipeline record.","description":"Using JSON data in the request, creates a new pipeline record.\n A valid special power user token is required for authorisation.","operationId":"create_pipeline_pipelines__post","security":[{"HTTPBearer":[]}],"requestBody":{"required":true,"content":{"application/json":{"schema":{"$ref":"#/components/schemas/Pipeline"}}}},"responses":{"201":{"description":"Pipeline was created","content":{"application/json":{"schema":{"$ref":"#/components/schemas/Pipeline"}}}},"403":{"description":"Not authorised"},"500":{"description":"Unexpected error"},"400":{"description":"Insufficient pipeline properties provided"},"409":{"description":"Pipeline already exists"},"422":{"description":"Validation Error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/HTTPValidationError"}}}}}}},"/pipelines/{pipeline_name}":{"get":{"tags":["pipelines"],"summary":"Get information about one pipeline.","description":"Returns a single pydantic Pipeline model if found.\n A valid token issued for any pipeline is required for authorisation.","operationId":"get_pipeline_pipelines__pipeline_name__get","security":[{"HTTPBearer":[]}],"parameters":[{"name":"pipeline_name","in":"path","required":true,"schema":{"type":"string","title":"Pipeline Name"}}],"responses":{"200":{"description":"Successful Response","content":{"application/json":{"schema":{"$ref":"#/components/schemas/Pipeline"}}}},"403":{"description":"Not authorised"},"500":{"description":"Unexpected error"},"404":{"description":"Not found"},"422":{"description":"Validation Error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/HTTPValidationError"}}}}}}},"/tasks/":{"get":{"tags":["tasks"],"summary":"Returns all tasks, and can be filtered to task status or pipeline name","description":"Return all tasks. The list of tasks can be filtered by supplying a pipeline\n name and/or task status","operationId":"get_tasks_tasks__get","security":[{"HTTPBearer":[]}],"parameters":[{"name":"pipeline_name","in":"query","required":false,"schema":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"Pipeline Name"}},{"name":"status","in":"query","required":false,"schema":{"anyOf":[{"$ref":"#/components/schemas/TaskStateEnum"},{"type":"null"}],"title":"Status"}}],"responses":{"200":{"description":"Successful Response","content":{"application/json":{"schema":{"type":"array","items":{"$ref":"#/components/schemas/Task"},"title":"Response Get Tasks Tasks Get"}}}},"403":{"description":"Not authorised"},"500":{"description":"Unexpected error"},"422":{"description":"Validation Error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/HTTPValidationError"}}}}}},"post":{"tags":["tasks"],"summary":"Creates one task record.","description":"Given a Task object, creates a database record for it and returns\n the same object, the response HTTP status is 201 'Created'. The\n new task is assigned pending status, ie becomes available for claiming.\n\n The pipeline specified by the `pipeline` attribute of the Task object\n should exist. If it does not exist, return status 404 'Not found'.","operationId":"create_task_tasks__post","security":[{"HTTPBearer":[]}],"requestBody":{"required":true,"content":{"application/json":{"schema":{"$ref":"#/components/schemas/Task"}}}},"responses":{"201":{"description":"Task creation was successful","content":{"application/json":{"schema":{"$ref":"#/components/schemas/Task"}}}},"403":{"description":"Not authorised"},"500":{"description":"Unexpected error"},"409":{"description":"A task with the same signature already exists"},"422":{"description":"Validation Error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/HTTPValidationError"}}}}}},"put":{"tags":["tasks"],"summary":"Update one task.","description":"Given a Task object, updates the status of the task in the database\n to the value of the status in this Task object.\n\n If the task does not exist, status 404 'Not found' is returned.","operationId":"update_task_tasks__put","security":[{"HTTPBearer":[]}],"requestBody":{"required":true,"content":{"application/json":{"schema":{"$ref":"#/components/schemas/Task"}}}},"responses":{"200":{"description":"Task was modified","content":{"application/json":{"schema":{"$ref":"#/components/schemas/Task"}}}},"403":{"description":"Not authorised"},"500":{"description":"Unexpected error"},"422":{"description":"Validation Error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/HTTPValidationError"}}}}}}},"/tasks/claim":{"post":{"tags":["tasks"],"summary":"Claim tasks for a particular pipeline.","description":"Arguments - the Pipeline object and the maximum number of tasks\n to retrieve and claim, the latter defaults to 1 if not given.\n\n If no tasks that satisfy the given criteria and are unclaimed\n are found, returns status 200 and an empty array.\n\n If any tasks are claimed, return an array of these Task objects\n and status 200.\n\n The pipeline object returned within each of the tasks is consistent\n with the pipeline object in the payload, but has all possible\n attributes defined (uri, version).","operationId":"claim_task_tasks_claim_post","security":[{"HTTPBearer":[]}],"parameters":[{"name":"num_tasks","in":"query","required":false,"schema":{"anyOf":[{"type":"integer","exclusiveMinimum":0},{"type":"null"}],"default":1,"title":"Num Tasks"}}],"requestBody":{"required":true,"content":{"application/json":{"schema":{"$ref":"#/components/schemas/Pipeline"}}}},"responses":{"200":{"description":"Receive a list of tasks that have been claimed","content":{"application/json":{"schema":{"type":"array","items":{"$ref":"#/components/schemas/Task"},"title":"Response Claim Task Tasks Claim Post"}}}},"403":{"description":"Not authorised"},"500":{"description":"Unexpected error"},"422":{"description":"Validation Error","content":{"application/json":{"schema":{"$ref":"#/components/schemas/HTTPValidationError"}}}}}}},"/":{"get":{"tags":["index"],"summary":"Web page with links to OpenAPI documentation.","operationId":"root__get","responses":{"200":{"description":"Successful Response","content":{"text/html":{"schema":{"type":"string"}}}}}}}},"components":{"schemas":{"HTTPValidationError":{"properties":{"detail":{"items":{"$ref":"#/components/schemas/ValidationError"},"type":"array","title":"Detail"}},"type":"object","title":"HTTPValidationError"},"Pipeline":{"properties":{"name":{"type":"string","title":"Pipeline Name","description":"A user-controlled name for the pipeline"},"uri":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"URI","description":"URI to bootstrap the pipeline code"},"version":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"Version","description":"Pipeline version to use with URI"}},"type":"object","title":"Pipeline"},"Task":{"properties":{"pipeline":{"$ref":"#/components/schemas/Pipeline"},"task_input_id":{"anyOf":[{"type":"string"},{"type":"null"}],"title":"Task Input ID","description":"A stringified unique identifier for a piece of work. Set by the npg_porch server, not the client"},"task_input":{"type":"object","title":"Task Input","description":"A structured parameter set that uniquely identifies a piece of work, and enables an iteration of a pipeline"},"status":{"anyOf":[{"$ref":"#/components/schemas/TaskStateEnum"},{"type":"null"}]}},"type":"object","required":["pipeline"],"title":"Task"},"TaskStateEnum":{"type":"string","enum":["PENDING","CLAIMED","RUNNING","DONE","FAILED","CANCELLED"],"title":"TaskStateEnum"},"ValidationError":{"properties":{"loc":{"items":{"anyOf":[{"type":"string"},{"type":"integer"}]},"type":"array","title":"Location"},"msg":{"type":"string","title":"Message"},"type":{"type":"string","title":"Error Type"}},"type":"object","required":["loc","msg","type"],"title":"ValidationError"}},"securitySchemes":{"HTTPBearer":{"type":"http","scheme":"bearer"}}},"tags":[{"name":"pipelines","description":"Manage pipelines."},{"name":"index","description":"Links to documentation."}]} \ No newline at end of file diff --git a/tests/test_api.py b/tests/test_api.py index b16de62..ac4e961 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -1,11 +1,32 @@ +import json + import pytest +import requests -from npg_porch_cli.api import AuthException, InvalidValueException, PorchRequest +from npg_porch_cli.api import ( + AuthException, + InvalidValueException, + PorchRequest, + ServerErrorException, +) url = "http://some.com" +var_name = "NPG_PORCH_TOKEN" + + +class MockPorchResponse: + def __init__(self, json_data, status_code): + self.json_data = json_data + self.status_code = status_code + self.reason = "Some reason" + self.url = url + self.ok = True if self.status_code == 200 else False + def json(self): + return self.json_data -def test_validation(): + +def test_request_validation(): r = PorchRequest(porch_url=url) @@ -22,16 +43,47 @@ def test_validation(): ) with pytest.raises(InvalidValueException) as e: - r._validate_request(action="list_tasks", task_status="Some", task_input=None) + r._validate_request(action="claim_task", task_status=None, task_input=None) + assert ( + e.value.args[0] + == "Full pipeline details should be defined for action 'claim_task'" + ) + r = PorchRequest( + porch_url=url, + pipeline_name="p1", + pipeline_version="0.1", + pipeline_url="https//:p1.com", + ) + assert ( + r._validate_request(action="claim_task", task_status=None, task_input=None) + is True + ) + + with pytest.raises(InvalidValueException) as e: + r._validate_request(action="add_task", task_status=None, task_input=None) assert ( - e.value.args[0] == "Task status 'Some' is not valid. " - "Valid statuses: PENDING, CLAIMED, RUNNING, DONE, FAILED, CANCELLED" + e.value.args[0] == "task_input argument should be defined for action 'add_task'" ) + assert ( + r._validate_request( + action="add_task", task_status=None, task_input={"id_run": 5} + ) + is True + ) + with pytest.raises(InvalidValueException) as e: - r._validate_request(action="list_tasks", task_status="Running", task_input=None) + r._validate_request( + action="update_task", task_status=None, task_input={"id_run": 5} + ) + assert ( + e.value.args[0] + == "task_status argument should be defined for action 'update_task'" + ) assert ( - e.value.args[0] == "Task status 'Running' is not valid. " - "Valid statuses: PENDING, CLAIMED, RUNNING, DONE, FAILED, CANCELLED" + r._validate_request( + action="update_task", task_status="PENDING", task_input={"id_run": 5} + ) + is True ) @@ -43,7 +95,6 @@ def test_url_generation(): def test_header_generation(monkeypatch): - var_name = "NPG_PORCH_TOKEN" r = PorchRequest(porch_url=url) monkeypatch.delenv(var_name, raising=False) @@ -60,3 +111,115 @@ def test_header_generation(monkeypatch): "Authorization": "Bearer token_xyz", } monkeypatch.undo() + + +def test_status_validation(monkeypatch): + + with monkeypatch.context() as m: + + def mock_get_200(*args, **kwargs): + f = open("tests/data/porch_openapi.json") + r = MockPorchResponse(json.load(f), 200) + f.close() + return r + + m.setattr(requests, "request", mock_get_200) + r = PorchRequest(porch_url=url) + assert r.validate_status("FAILED") == "FAILED" + assert r.validate_status("Failed") == "FAILED" + with pytest.raises(InvalidValueException) as e: + r.validate_status("Swimming") + assert ( + e.value.args[0] == "Task status 'Swimming' is not valid. " + "Valid statuses: CANCELLED, CLAIMED, DONE, FAILED, PENDING, RUNNING" + ) + + with monkeypatch.context() as mk: + + def mock_get_404(*args, **kwargs): + return MockPorchResponse({"Error": "Not found"}, 404) + + mk.setattr(requests, "request", mock_get_404) + r = PorchRequest(porch_url=url) + with pytest.raises(ServerErrorException) as e: + r.validate_status("FAILED") + assert ( + e.value.args[0] == "Failed to get OpenAPI Schema. Status code 404 " + '"Some reason" received from http://some.com' + ) + + with monkeypatch.context() as mkp: + + def mock_get_200(*args, **kwargs): + return MockPorchResponse( + {"openapi": "3.1.0", "info": {"title": "Pipeline", "version": "0.1.0"}}, + 200, + ) + + mkp.setattr(requests, "request", mock_get_200) + r = PorchRequest(porch_url=url) + with pytest.raises(Exception) as e: + r.validate_status("FAILED") + assert e.value.args[0].startswith( + f"Failed to get enumeration of valid statuses from {url}" + ) + + +def test_sending_request(monkeypatch): + class MockPorchRequest(PorchRequest): + # Mock status validation. + def validate_status(self, task_status: str): + return task_status + + r = MockPorchRequest( + porch_url=url, + pipeline_name="p1", + pipeline_version="0.1", + pipeline_url="https//:p1.com", + ) + + task = { + "id_run": 40954, + } + response_data = { + "pipeline": { + "name": "p1", + "uri": "https//:p1.com", + "version": "0.1", + }, + "task_input_id": "8d505b17b4f", + "task_input": task, + "status": "PENDING", + } + + monkeypatch.delenv(var_name, raising=False) + monkeypatch.setenv(var_name, "MY_TOKEN") + + with monkeypatch.context() as m: + + def mock_get_200(*args, **kwargs): + return MockPorchResponse(response_data, 200) + + m.setattr(requests, "request", mock_get_200) + assert r.send(action="add_task", task_input=task) == response_data + + with monkeypatch.context() as mk: + response_data["status"] = "CLAIMED" + + def mock_get_200(*args, **kwargs): + return MockPorchResponse(response_data, 200) + + mk.setattr(requests, "request", mock_get_200) + assert r.send(action="claim_task") == response_data + + with monkeypatch.context() as mkp: + response_data["status"] = "DONE" + + def mock_get_200(*args, **kwargs): + return MockPorchResponse(response_data, 200) + + mkp.setattr(requests, "request", mock_get_200) + assert ( + r.send(action="update_task", task_input=task, task_status="DONE") + == response_data + ) From 91ce08703d44543b684054ac366e3b14bd0ee5ad Mon Sep 17 00:00:00 2001 From: mgcam Date: Fri, 3 May 2024 11:59:30 +0100 Subject: [PATCH 13/27] Add instructions about the tokens --- README.md | 7 +++++++ src/npg_porch_cli/api_cli_user.py | 2 +- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 9532847..06f7367 100644 --- a/README.md +++ b/README.md @@ -5,6 +5,10 @@ with the [npg_porch](https://github.com/wtsi-npg/npg_porch) JSON API. Provides a Python script, `npg_porch_client`, and a Python client API. +NPG_PORCH_TOKEN environment variable should be set to the value of either +the admin or project-specific token. The token should be pre-registered in +the database that is used by the npg_porch API server. + Can be deployed with pip or poetry in a standard way. Example of using a client API: @@ -20,6 +24,7 @@ By default the client is set up to validate the server's CA certificate. If the server is using a custom CA certificate, set the path to the certificate. ``` bash + export NPG_PORCH_TOKEN='my_token' export SSL_CERT_FILE=/path_to/my.pem npg_porch_client list_pipelines --base_url https://myporch.com ``` @@ -27,6 +32,7 @@ If the server is using a custom CA certificate, set the path to the certificate. It is possible, but not recommended, to disable this validation check. ``` bash + export NPG_PORCH_TOKEN='my_token' npg_porch_client list_pipelines --base_url https://myporch.com --no-validate_ca_cert ``` @@ -34,6 +40,7 @@ A valid JSON string is required for the `--task_json` script's argument, note double quotes in the example below. ``` bash + export NPG_PORCH_TOKEN='my_token' npg_porch_client update_task --base_url https://myporch.com \ --pipeline Snakemake_Cardinal \ --pipeline_url 'https://github.com/wtsi-npg/snakemake_cardinal' \ diff --git a/src/npg_porch_cli/api_cli_user.py b/src/npg_porch_cli/api_cli_user.py index 63fb7aa..5302f4a 100755 --- a/src/npg_porch_cli/api_cli_user.py +++ b/src/npg_porch_cli/api_cli_user.py @@ -61,7 +61,7 @@ def run(): be defined. In addition to this, for the `update_task` action `--status` should be defined. - NPG_PORCH_TOKEN environment variable should be set up to the value of + NPG_PORCH_TOKEN environment variable should be set to the value of either an admin or project-specific token. """ From 85ae2829d5b676f4a6a7b923d9b66d2755e52c20 Mon Sep 17 00:00:00 2001 From: mgcam Date: Fri, 3 May 2024 12:07:59 +0100 Subject: [PATCH 14/27] Get list of valid actions from the API --- src/npg_porch_cli/api_cli_user.py | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/src/npg_porch_cli/api_cli_user.py b/src/npg_porch_cli/api_cli_user.py index 5302f4a..c306ae7 100755 --- a/src/npg_porch_cli/api_cli_user.py +++ b/src/npg_porch_cli/api_cli_user.py @@ -22,7 +22,7 @@ import argparse import json -from npg_porch_cli.api import PorchRequest +from npg_porch_cli.api import PORCH_CLIENT_ACTIONS, PorchRequest def run(): @@ -75,14 +75,7 @@ def run(): "action", type=str, help="Action to send to npg_porch server API", - choices=[ - "list_tasks", - "list_pipelines", - "add_pipeline", - "add_task", - "claim_task", - "update_task", - ], + choices=PORCH_CLIENT_ACTIONS.keys(), ) parser.add_argument("--base_url", type=str, required=True, help="Base URL") parser.add_argument( From 2ac7110faa2ff5aa60d4de59b40a4f5a8fdb7a63 Mon Sep 17 00:00:00 2001 From: mgcam Date: Fri, 3 May 2024 11:02:50 +0100 Subject: [PATCH 15/27] API refactored to consolidate the logic about the action. --- src/npg_porch_cli/api.py | 322 ++++++++++++++++++------------ src/npg_porch_cli/api_cli_user.py | 26 ++- tests/test_api.py | 233 +++++++++++---------- 3 files changed, 334 insertions(+), 247 deletions(-) diff --git a/src/npg_porch_cli/api.py b/src/npg_porch_cli/api.py index a404989..33d9c99 100644 --- a/src/npg_porch_cli/api.py +++ b/src/npg_porch_cli/api.py @@ -18,21 +18,22 @@ # You should have received a copy of the GNU General Public License along with # this program. If not, see . +import inspect +import json import os -from dataclasses import dataclass, field +from dataclasses import InitVar, asdict, dataclass, field from urllib.parse import urljoin import requests -PORCH_CLIENT_ACTIONS = { - "list_tasks": "tasks", - "list_pipelines": "pipelines", - "add_pipeline": "pipelines", - "add_task": "tasks", - "claim_task": "tasks/claim", - "update_task": "tasks", -} - +PORCH_CLIENT_ACTIONS = [ + "list_tasks", + "list_pipelines", + "add_pipeline", + "add_task", + "claim_task", + "update_task", +] PORCH_OPENAPI_SCHEMA_URL = "api/v1/openapi.json" PORCH_TASK_STATUS_ENUM_NAME = "TaskStateEnum" @@ -55,83 +56,59 @@ class ServerErrorException(Exception): pass -@dataclass -class PorchRequest: +@dataclass(kw_only=True) +class Pipeline: + + name: str + uri: str + version: str + + def __post_init__(self): + "Post-constructor hook. Ensures all fields are defined." + try: + assert self.name and self.uri and self.version + except AssertionError: + raise InvalidValueException( + "Pipeline name, uri and version should be defined" + ) + + +@dataclass(kw_only=True) +class PorchAction: porch_url: str + action: str validate_ca_cert: bool = field(default=True) - pipeline_name: str | None = field(default=None) - pipeline_url: str | None = field(default=None) - pipeline_version: str | None = field(default=None) - - def send( - self, - action: str, - task_input: dict | None = None, - task_status: str | None = None, - ) -> dict: - """ - Sends a request to the porch API server to perform an action defined - by the `action` argument. Either of two optional arguments, if defined, - are used when constructing the request. + task_json: InitVar[str | None] = field(default=None, repr=False) + task_input: dict = field(default=None) + task_status: str | None = field(default=None) - The server's response is returned as a dictionary. - """ + def __post_init__(self, task_json): + "Post-constructor hook. Ensures integrity and validity of attributes." - if task_status is not None: - task_status = self.validate_status(task_status=task_status) - self._validate_request( - action=action, task_input=task_input, task_status=task_status - ) + if self.porch_url is None: + raise InvalidValueException("'porch_url' attribute should be defined") - method = "GET" - pipeline_data = { - "name": self.pipeline_name, - "uri": self.pipeline_url, - "version": self.pipeline_version, - } - data = None - - if action == "update_task": - method = "PUT" - data = { - "pipeline": pipeline_data, - "task_input": task_input, - "status": task_status, - } - elif action.startswith("list") is False: - method = "POST" - data = pipeline_data - if action == "add_task": - data = {"pipeline": pipeline_data, "task_input": task_input} - - request_args = { - "headers": self._get_request_headers(action), - "timeout": CLIENT_TIMEOUT, - "verify": self.validate_ca_cert, - } - if data is not None: - request_args["json"] = data - - response = requests.request( - method, self._generate_request_url(action), **request_args - ) - if not response.ok: - raise ServerErrorException( - f"Action {action} failed. " - f'Status code {response.status_code} "{response.reason}" ' - f"received from {response.url}" - ) + if task_json is not None: + if self.task_input is not None: + raise InvalidValueException( + "task_json and task_input cannot be both set" + ) + self.task_input = json.loads(task_json) - response_obj = response.json() - if action == "list_tasks" and self.pipeline_name is not None: - response_obj = [ - o for o in response_obj if o["pipeline"]["name"] == self.pipeline_name - ] + self._validate_action_name() + self.task_status = self._validate_status() - return response_obj + def _validate_action_name(self): + if self.action is None: + raise InvalidValueException("'action' attribute should be defined") + if self.action not in PORCH_CLIENT_ACTIONS: + raise InvalidValueException( + f"Action '{self.action}' is not valid. " + "Valid actions: " + ", ".join(sorted(PORCH_CLIENT_ACTIONS)) + ) - def validate_status(self, task_status: str) -> str: + def _validate_status(self) -> str | None: """ Retrieves OpenAPI schema for the porch server and validates the given task status value against the values listed in the schema document. @@ -139,8 +116,12 @@ def validate_status(self, task_status: str) -> str: Returns a validated task status value. The case of this string can be different from the input string. """ + + if self.task_status is None: + return None + url = urljoin(self.porch_url, PORCH_OPENAPI_SCHEMA_URL) - response = requests.request("GET", url) + response = requests.request("GET", url, verify=self.validate_ca_cert) if not response.ok: raise ServerErrorException( f"Failed to get OpenAPI Schema. " @@ -148,7 +129,7 @@ def validate_status(self, task_status: str) -> str: f"received from {response.url}" ) - status = task_status.upper() + status = self.task_status.upper() valid_statuses = [] error_message = f"Failed to get enumeration of valid statuses from {url}" try: @@ -163,59 +144,154 @@ def validate_status(self, task_status: str) -> str: if status not in valid_statuses: raise InvalidValueException( - f"Task status '{task_status}' is not valid. " + f"Task status '{self.task_status}' is not valid. " "Valid statuses: " + ", ".join(sorted(valid_statuses)) ) return status - def _generate_request_url(self, action: str): - return urljoin(self.porch_url, PORCH_CLIENT_ACTIONS[action]) - def _get_token(self): - if NPG_PORCH_TOKEN_ENV_VAR not in os.environ: - raise AuthException("Authorization token is needed") - return os.environ[NPG_PORCH_TOKEN_ENV_VAR] +def get_token(): + if NPG_PORCH_TOKEN_ENV_VAR not in os.environ: + raise AuthException("Authorization token is needed") + return os.environ[NPG_PORCH_TOKEN_ENV_VAR] + + +def _send_request(validate_ca_cert: bool, url: str, method: str, data: dict = None): + + headers = { + "Authorization": "Bearer " + get_token(), + "Content-Type": "application/json", + "Accept": "application/json", + } + request_args = { + "headers": headers, + "timeout": CLIENT_TIMEOUT, + "verify": validate_ca_cert, + } + if data is not None: + request_args["json"] = data + + response = requests.request(method, url, **request_args) + if not response.ok: + action_name = inspect.stack()[1].function + raise ServerErrorException( + f"Action {action_name} failed. " + f'Status code {response.status_code} "{response.reason}" ' + f"received from {response.url}" + ) - def _get_request_headers(self, action: str): - headers = {"Authorization": "Bearer " + self._get_token()} - if action.startswith("list") is False: - headers["Content-Type"] = "application/json" - return headers + return response.json() - def _validate_request( - self, action: str, task_status: str | None, task_input: str | None - ): - if action not in PORCH_CLIENT_ACTIONS: - raise InvalidValueException( - f"Action '{action}' is not valid. " - "Valid actions: " + ", ".join(sorted(PORCH_CLIENT_ACTIONS.keys())) - ) +def send(action: PorchAction, pipeline: Pipeline = None) -> dict | list: + """ + Sends a request to the porch API server to perform an action defined + by the `action` attribute of the `action` argument. The context of the + query is defined by the pipeline argument. - if action.startswith("list") is False: - if ( - self.pipeline_name is None - or self.pipeline_url is None - or self.pipeline_version is None - ): - raise InvalidValueException( - f"Full pipeline details should be defined for action '{action}'" - ) + The server's response is returned as a dictionary or as a list. + """ - if ( - action.endswith("task") is True - and action.startswith("claim") is False - and task_input is None - ): - raise InvalidValueException( - f"task_input argument should be defined for action '{action}'" - ) + functions = { + "list_tasks": list_tasks, + "list_pipelines": list_pipelines, + "add_pipeline": add_pipeline, + "add_task": add_task, + "claim_task": claim_task, + "update_task": update_task, + } + + # Get function's definition and then call the function. + function = functions[action.action] + if action.action == "list_pipelines": + return function(action=action) + return function(action=action, pipeline=pipeline) + + +def list_pipelines(action: PorchAction) -> list: + "Returns a listing of all pipelines registered with the porch server." + + return _send_request( + validate_ca_cert=action.validate_ca_cert, + url=urljoin(action.porch_url, "pipelines"), + method="GET", + ) + + +def list_tasks(action: PorchAction, pipeline: Pipeline = None) -> list: + """ + In the pipeline argument is not defined, returns a listing of all tasks + registered with the porch server. If the pipeline argument is defined, + only tasks belonging to this pipeline are listed. + """ + + response_obj = _send_request( + validate_ca_cert=action.validate_ca_cert, + url=urljoin(action.porch_url, "tasks"), + method="GET", + ) + if pipeline is not None: + pipeline_dict = asdict(pipeline) + response_obj = [o for o in response_obj if o["pipeline"] == pipeline_dict] + return response_obj - if action == "update_task": - if task_status is None: - raise InvalidValueException( - f"task_status argument should be defined for action '{action}'" - ) - return True +def add_pipeline(action: PorchAction, pipeline: Pipeline): + "Registers a new pipeline with the porch server." + + return _send_request( + validate_ca_cert=action.validate_ca_cert, + method="POST", + url=urljoin(action.porch_url, "pipelines"), + data=asdict(pipeline), + ) + + +def add_task(action: PorchAction, pipeline: Pipeline): + "Registers a new task with the porch server." + + if action.task_input is None: + raise InvalidValueException( + f"task_input should be defined for action '{action.action}'" + ) + return _send_request( + validate_ca_cert=action.validate_ca_cert, + url=urljoin(action.porch_url, "tasks"), + method="POST", + data={"pipeline": asdict(pipeline), "task_input": action.task_input}, + ) + + +def claim_task(action: PorchAction, pipeline: Pipeline): + "Claims a task that belongs to the previously registered pipeline." + + return _send_request( + validate_ca_cert=action.validate_ca_cert, + url=urljoin(action.porch_url, "tasks/claim"), + method="POST", + data=asdict(pipeline), + ) + + +def update_task(action: PorchAction, pipeline: Pipeline): + "Updates a status of a task." + + if action.task_input is None: + raise InvalidValueException( + f"task_input should be defined for action '{action.action}'" + ) + if action.task_status is None: + raise InvalidValueException( + f"task_status should be defined for action '{action.action}'" + ) + return _send_request( + validate_ca_cert=action.validate_ca_cert, + url=urljoin(action.porch_url, "tasks"), + method="PUT", + data={ + "pipeline": asdict(pipeline), + "task_input": action.task_input, + "status": action.task_status, + }, + ) diff --git a/src/npg_porch_cli/api_cli_user.py b/src/npg_porch_cli/api_cli_user.py index c306ae7..f839f6a 100755 --- a/src/npg_porch_cli/api_cli_user.py +++ b/src/npg_porch_cli/api_cli_user.py @@ -22,7 +22,7 @@ import argparse import json -from npg_porch_cli.api import PORCH_CLIENT_ACTIONS, PorchRequest +from npg_porch_cli.api import PORCH_CLIENT_ACTIONS, Pipeline, PorchAction, send def run(): @@ -75,7 +75,7 @@ def run(): "action", type=str, help="Action to send to npg_porch server API", - choices=PORCH_CLIENT_ACTIONS.keys(), + choices=PORCH_CLIENT_ACTIONS, ) parser.add_argument("--base_url", type=str, required=True, help="Base URL") parser.add_argument( @@ -97,19 +97,17 @@ def run(): args = parser.parse_args() - r = PorchRequest( + action = PorchAction( porch_url=args.base_url, validate_ca_cert=args.validate_ca_cert, - pipeline_name=args.pipeline, - pipeline_url=args.pipeline_url, - pipeline_version=args.pipeline_version, + action=args.action, + task_json=args.task_json, + task_status=args.status, ) + pipeline = None + if args.pipeline is not None: + pipeline = Pipeline( + name=args.pipeline, uri=args.pipeline_url, version=args.pipeline_version + ) - parsed_json = None - if args.task_json is not None: - print(args.task_json) - parsed_json = json.loads(args.task_json) - response = r.send( - action=args.action, task_input=parsed_json, task_status=args.status - ) - print(response) + print(json.dumps(send(action=action, pipeline=pipeline), indent=2)) diff --git a/tests/test_api.py b/tests/test_api.py index ac4e961..12752a3 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -6,8 +6,11 @@ from npg_porch_cli.api import ( AuthException, InvalidValueException, - PorchRequest, + Pipeline, + PorchAction, ServerErrorException, + get_token, + send, ) url = "http://some.com" @@ -26,94 +29,67 @@ def json(self): return self.json_data -def test_request_validation(): - - r = PorchRequest(porch_url=url) - - with pytest.raises(InvalidValueException) as e: - r._validate_request(action="list_tools", task_status=None, task_input=None) - assert ( - e.value.args[0] == "Action 'list_tools' is not valid. " - "Valid actions: add_pipeline, add_task, claim_task, list_pipelines, " - "list_tasks, update_task" - ) - assert ( - r._validate_request(action="list_tasks", task_status=None, task_input=None) - is True - ) +def test_header_generation(monkeypatch): - with pytest.raises(InvalidValueException) as e: - r._validate_request(action="claim_task", task_status=None, task_input=None) - assert ( - e.value.args[0] - == "Full pipeline details should be defined for action 'claim_task'" - ) - r = PorchRequest( - porch_url=url, - pipeline_name="p1", - pipeline_version="0.1", - pipeline_url="https//:p1.com", - ) - assert ( - r._validate_request(action="claim_task", task_status=None, task_input=None) - is True - ) + monkeypatch.delenv(var_name, raising=False) + with pytest.raises(AuthException) as e: + get_token() + assert e.value.args[0] == "Authorization token is needed" - with pytest.raises(InvalidValueException) as e: - r._validate_request(action="add_task", task_status=None, task_input=None) - assert ( - e.value.args[0] == "task_input argument should be defined for action 'add_task'" - ) - assert ( - r._validate_request( - action="add_task", task_status=None, task_input={"id_run": 5} - ) - is True - ) + monkeypatch.setenv(var_name, "token_xyz") + assert get_token() == "token_xyz" - with pytest.raises(InvalidValueException) as e: - r._validate_request( - action="update_task", task_status=None, task_input={"id_run": 5} - ) - assert ( - e.value.args[0] - == "task_status argument should be defined for action 'update_task'" - ) - assert ( - r._validate_request( - action="update_task", task_status="PENDING", task_input={"id_run": 5} - ) - is True - ) + monkeypatch.undo() -def test_url_generation(): +def test_pipeline_class(): - r = PorchRequest(porch_url=url) - assert r._generate_request_url(action="list_tasks") == "/".join([url, "tasks"]) + with pytest.raises(InvalidValueException) as e: + Pipeline(name=None, uri="http://some.come", version="1.0") + assert e.value.args[0] == "Pipeline name, uri and version should be defined" -def test_header_generation(monkeypatch): +def test_porch_action_class(monkeypatch): - r = PorchRequest(porch_url=url) + with pytest.raises(InvalidValueException) as e: + PorchAction(porch_url=None, action="list_tasks") + assert e.value.args[0] == "'porch_url' attribute should be defined" - monkeypatch.delenv(var_name, raising=False) - with pytest.raises(AuthException) as e: - r._get_request_headers(action="list_tasks") - assert e.value.args[0] == "Authorization token is needed" + with pytest.raises(InvalidValueException) as e: + PorchAction(porch_url="http://some.come", action=None) + assert e.value.args[0] == "'action' attribute should be defined" - monkeypatch.setenv(var_name, "token_xyz") - assert r._get_request_headers(action="list_tasks") == { - "Authorization": "Bearer token_xyz" - } - assert r._get_request_headers(action="add_tasks") == { - "Content-Type": "application/json", - "Authorization": "Bearer token_xyz", - } - monkeypatch.undo() + with pytest.raises(InvalidValueException) as e: + PorchAction(porch_url=url, action="list_tools") + assert ( + e.value.args[0] == "Action 'list_tools' is not valid. " + "Valid actions: add_pipeline, add_task, claim_task, list_pipelines, " + "list_tasks, update_task" + ) + pa = PorchAction(porch_url=url, action="list_tasks") + assert pa.validate_ca_cert is True + assert pa.task_input is None + assert pa.task_status is None -def test_status_validation(monkeypatch): + with pytest.raises(InvalidValueException) as e: + pa = PorchAction( + porch_url=url, + action="add_task", + task_json='{"id_run": 5}', + task_input={"id_run": 5}, + ) + assert e.value.args[0] == "task_json and task_input cannot be both set" + pa = PorchAction( + validate_ca_cert=False, + porch_url=url, + action="add_task", + task_json='{"id_run": 5}', + ) + assert pa.task_input == {"id_run": 5} + assert pa.validate_ca_cert is False + pa = PorchAction(porch_url=url, action="add_task", task_input={"id_run": 5}) + assert pa.task_input == {"id_run": 5} with monkeypatch.context() as m: @@ -124,11 +100,27 @@ def mock_get_200(*args, **kwargs): return r m.setattr(requests, "request", mock_get_200) - r = PorchRequest(porch_url=url) - assert r.validate_status("FAILED") == "FAILED" - assert r.validate_status("Failed") == "FAILED" + pa = PorchAction( + task_status="FAILED", + action="update_task", + task_input='{"id_run": 5}', + porch_url=url, + ) + assert pa.task_status == "FAILED" + pa = PorchAction( + task_status="FAILED", + action="update_task", + task_input='{"id_run": 5}', + porch_url=url, + ) + assert pa.task_status == "FAILED" with pytest.raises(InvalidValueException) as e: - r.validate_status("Swimming") + PorchAction( + task_status="Swimming", + action="update_task", + task_input='{"id_run": 5}', + porch_url=url, + ) assert ( e.value.args[0] == "Task status 'Swimming' is not valid. " "Valid statuses: CANCELLED, CLAIMED, DONE, FAILED, PENDING, RUNNING" @@ -140,9 +132,13 @@ def mock_get_404(*args, **kwargs): return MockPorchResponse({"Error": "Not found"}, 404) mk.setattr(requests, "request", mock_get_404) - r = PorchRequest(porch_url=url) with pytest.raises(ServerErrorException) as e: - r.validate_status("FAILED") + PorchAction( + task_status="FAILED", + action="update_task", + task_input='{"id_run": 5}', + porch_url=url, + ) assert ( e.value.args[0] == "Failed to get OpenAPI Schema. Status code 404 " '"Some reason" received from http://some.com' @@ -157,51 +153,65 @@ def mock_get_200(*args, **kwargs): ) mkp.setattr(requests, "request", mock_get_200) - r = PorchRequest(porch_url=url) with pytest.raises(Exception) as e: - r.validate_status("FAILED") + PorchAction( + task_status="FAILED", + action="update_task", + task_input='{"id_run": 5}', + porch_url=url, + ) assert e.value.args[0].startswith( f"Failed to get enumeration of valid statuses from {url}" ) +def test_request_validation(): + + p = Pipeline(uri=url, version="1.0", name="p1") + + pa = PorchAction(porch_url=url, action="add_task") + with pytest.raises(InvalidValueException) as e: + send(action=pa, pipeline=p) + assert e.value.args[0] == "task_input should be defined for action 'add_task'" + + pa = PorchAction(porch_url=url, action="update_task") + with pytest.raises(InvalidValueException) as e: + send(action=pa, pipeline=p) + assert e.value.args[0] == "task_input should be defined for action 'update_task'" + pa = PorchAction(porch_url=url, action="update_task", task_input={"id_run": 5}) + with pytest.raises(InvalidValueException) as e: + send(action=pa, pipeline=p) + assert e.value.args[0] == "task_status should be defined for action 'update_task'" + + def test_sending_request(monkeypatch): - class MockPorchRequest(PorchRequest): - # Mock status validation. - def validate_status(self, task_status: str): - return task_status - r = MockPorchRequest( - porch_url=url, - pipeline_name="p1", - pipeline_version="0.1", - pipeline_url="https//:p1.com", - ) + monkeypatch.delenv(var_name, raising=False) + monkeypatch.setenv(var_name, "MY_TOKEN") - task = { - "id_run": 40954, - } + def all_valid(*args, **kwargs): + return "PENDING" + + monkeypatch.setattr(PorchAction, "_validate_status", all_valid) + + p = Pipeline(uri=url, version="0.1", name="p1") + task = {"id_run": 5} response_data = { - "pipeline": { - "name": "p1", - "uri": "https//:p1.com", - "version": "0.1", - }, + "pipeline": p, "task_input_id": "8d505b17b4f", "task_input": task, "status": "PENDING", } - monkeypatch.delenv(var_name, raising=False) - monkeypatch.setenv(var_name, "MY_TOKEN") - with monkeypatch.context() as m: def mock_get_200(*args, **kwargs): return MockPorchResponse(response_data, 200) m.setattr(requests, "request", mock_get_200) - assert r.send(action="add_task", task_input=task) == response_data + + pa = PorchAction(porch_url=url, action="add_task", task_input=task) + assert send(action=pa, pipeline=p) == response_data with monkeypatch.context() as mk: response_data["status"] = "CLAIMED" @@ -210,7 +220,9 @@ def mock_get_200(*args, **kwargs): return MockPorchResponse(response_data, 200) mk.setattr(requests, "request", mock_get_200) - assert r.send(action="claim_task") == response_data + + pa = PorchAction(porch_url=url, action="claim_task") + assert send(action=pa, pipeline=p) == response_data with monkeypatch.context() as mkp: response_data["status"] = "DONE" @@ -219,7 +231,8 @@ def mock_get_200(*args, **kwargs): return MockPorchResponse(response_data, 200) mkp.setattr(requests, "request", mock_get_200) - assert ( - r.send(action="update_task", task_input=task, task_status="DONE") - == response_data + + pa = PorchAction( + porch_url=url, action="update_task", task_input=task, task_status="DONE" ) + assert send(action=pa, pipeline=p) == response_data From 324e27ad9452cb9903675714e2d2144c920aa089 Mon Sep 17 00:00:00 2001 From: mgcam Date: Tue, 7 May 2024 12:30:12 +0100 Subject: [PATCH 16/27] Removed duplication in listing valid actions --- src/npg_porch_cli/api.py | 87 +++++++++++++++---------------- src/npg_porch_cli/api_cli_user.py | 4 +- tests/test_api.py | 14 ++++- 3 files changed, 58 insertions(+), 47 deletions(-) diff --git a/src/npg_porch_cli/api.py b/src/npg_porch_cli/api.py index 33d9c99..2c34c9d 100644 --- a/src/npg_porch_cli/api.py +++ b/src/npg_porch_cli/api.py @@ -26,14 +26,6 @@ import requests -PORCH_CLIENT_ACTIONS = [ - "list_tasks", - "list_pipelines", - "add_pipeline", - "add_task", - "claim_task", - "update_task", -] PORCH_OPENAPI_SCHEMA_URL = "api/v1/openapi.json" PORCH_TASK_STATUS_ENUM_NAME = "TaskStateEnum" @@ -102,10 +94,10 @@ def __post_init__(self, task_json): def _validate_action_name(self): if self.action is None: raise InvalidValueException("'action' attribute should be defined") - if self.action not in PORCH_CLIENT_ACTIONS: + if self.action not in _PORCH_CLIENT_ACTIONS: raise InvalidValueException( f"Action '{self.action}' is not valid. " - "Valid actions: " + ", ".join(sorted(PORCH_CLIENT_ACTIONS)) + "Valid actions: " + ", ".join(list_client_actions()) ) def _validate_status(self) -> str | None: @@ -157,31 +149,10 @@ def get_token(): return os.environ[NPG_PORCH_TOKEN_ENV_VAR] -def _send_request(validate_ca_cert: bool, url: str, method: str, data: dict = None): - - headers = { - "Authorization": "Bearer " + get_token(), - "Content-Type": "application/json", - "Accept": "application/json", - } - request_args = { - "headers": headers, - "timeout": CLIENT_TIMEOUT, - "verify": validate_ca_cert, - } - if data is not None: - request_args["json"] = data +def list_client_actions() -> list: + """Returns a sorted list of currently implemented client actions.""" - response = requests.request(method, url, **request_args) - if not response.ok: - action_name = inspect.stack()[1].function - raise ServerErrorException( - f"Action {action_name} failed. " - f'Status code {response.status_code} "{response.reason}" ' - f"received from {response.url}" - ) - - return response.json() + return sorted(_PORCH_CLIENT_ACTIONS.keys()) def send(action: PorchAction, pipeline: Pipeline = None) -> dict | list: @@ -193,17 +164,8 @@ def send(action: PorchAction, pipeline: Pipeline = None) -> dict | list: The server's response is returned as a dictionary or as a list. """ - functions = { - "list_tasks": list_tasks, - "list_pipelines": list_pipelines, - "add_pipeline": add_pipeline, - "add_task": add_task, - "claim_task": claim_task, - "update_task": update_task, - } - # Get function's definition and then call the function. - function = functions[action.action] + function = _PORCH_CLIENT_ACTIONS[action.action] if action.action == "list_pipelines": return function(action=action) return function(action=action, pipeline=pipeline) @@ -295,3 +257,40 @@ def update_task(action: PorchAction, pipeline: Pipeline): "status": action.task_status, }, ) + + +_PORCH_CLIENT_ACTIONS = { + "list_tasks": list_tasks, + "list_pipelines": list_pipelines, + "add_pipeline": add_pipeline, + "add_task": add_task, + "claim_task": claim_task, + "update_task": update_task, +} + + +def _send_request(validate_ca_cert: bool, url: str, method: str, data: dict = None): + + headers = { + "Authorization": "Bearer " + get_token(), + "Content-Type": "application/json", + "Accept": "application/json", + } + request_args = { + "headers": headers, + "timeout": CLIENT_TIMEOUT, + "verify": validate_ca_cert, + } + if data is not None: + request_args["json"] = data + + response = requests.request(method, url, **request_args) + if not response.ok: + action_name = inspect.stack()[1].function + raise ServerErrorException( + f"Action {action_name} failed. " + f'Status code {response.status_code} "{response.reason}" ' + f"received from {response.url}" + ) + + return response.json() diff --git a/src/npg_porch_cli/api_cli_user.py b/src/npg_porch_cli/api_cli_user.py index f839f6a..cd94ddc 100755 --- a/src/npg_porch_cli/api_cli_user.py +++ b/src/npg_porch_cli/api_cli_user.py @@ -22,7 +22,7 @@ import argparse import json -from npg_porch_cli.api import PORCH_CLIENT_ACTIONS, Pipeline, PorchAction, send +from npg_porch_cli.api import Pipeline, PorchAction, list_client_actions, send def run(): @@ -75,7 +75,7 @@ def run(): "action", type=str, help="Action to send to npg_porch server API", - choices=PORCH_CLIENT_ACTIONS, + choices=list_client_actions(), ) parser.add_argument("--base_url", type=str, required=True, help="Base URL") parser.add_argument( diff --git a/tests/test_api.py b/tests/test_api.py index 12752a3..2497292 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -10,6 +10,7 @@ PorchAction, ServerErrorException, get_token, + list_client_actions, send, ) @@ -29,7 +30,7 @@ def json(self): return self.json_data -def test_header_generation(monkeypatch): +def test_retrieving_token(monkeypatch): monkeypatch.delenv(var_name, raising=False) with pytest.raises(AuthException) as e: @@ -42,6 +43,17 @@ def test_header_generation(monkeypatch): monkeypatch.undo() +def test_listing_actions(): + assert list_client_actions() == [ + "add_pipeline", + "add_task", + "claim_task", + "list_pipelines", + "list_tasks", + "update_task", + ] + + def test_pipeline_class(): with pytest.raises(InvalidValueException) as e: From f55e97c5343144c4b4fc0f0e376aadbc928fc691 Mon Sep 17 00:00:00 2001 From: mgcam Date: Wed, 8 May 2024 12:20:57 +0100 Subject: [PATCH 17/27] Use suitable native Python Exception types. Do not use 'assert' for validation in the code. --- src/npg_porch_cli/api.py | 36 ++++++++++-------------------------- tests/test_api.py | 29 ++++++++++++++--------------- 2 files changed, 24 insertions(+), 41 deletions(-) diff --git a/src/npg_porch_cli/api.py b/src/npg_porch_cli/api.py index 2c34c9d..f56a886 100644 --- a/src/npg_porch_cli/api.py +++ b/src/npg_porch_cli/api.py @@ -40,10 +40,6 @@ class AuthException(Exception): pass -class InvalidValueException(Exception): - pass - - class ServerErrorException(Exception): pass @@ -57,12 +53,8 @@ class Pipeline: def __post_init__(self): "Post-constructor hook. Ensures all fields are defined." - try: - assert self.name and self.uri and self.version - except AssertionError: - raise InvalidValueException( - "Pipeline name, uri and version should be defined" - ) + if not (self.name and self.uri and self.version): + raise TypeError("Pipeline name, uri and version should be defined") @dataclass(kw_only=True) @@ -79,13 +71,11 @@ def __post_init__(self, task_json): "Post-constructor hook. Ensures integrity and validity of attributes." if self.porch_url is None: - raise InvalidValueException("'porch_url' attribute should be defined") + raise TypeError("'porch_url' attribute cannot be None") if task_json is not None: if self.task_input is not None: - raise InvalidValueException( - "task_json and task_input cannot be both set" - ) + raise ValueError("task_json and task_input cannot be both set") self.task_input = json.loads(task_json) self._validate_action_name() @@ -93,9 +83,9 @@ def __post_init__(self, task_json): def _validate_action_name(self): if self.action is None: - raise InvalidValueException("'action' attribute should be defined") + raise TypeError("'action' attribute cannot be None") if self.action not in _PORCH_CLIENT_ACTIONS: - raise InvalidValueException( + raise ValueError( f"Action '{self.action}' is not valid. " "Valid actions: " + ", ".join(list_client_actions()) ) @@ -135,7 +125,7 @@ def _validate_status(self) -> str | None: raise Exception(error_message) if status not in valid_statuses: - raise InvalidValueException( + raise ValueError( f"Task status '{self.task_status}' is not valid. " "Valid statuses: " + ", ".join(sorted(valid_statuses)) ) @@ -214,9 +204,7 @@ def add_task(action: PorchAction, pipeline: Pipeline): "Registers a new task with the porch server." if action.task_input is None: - raise InvalidValueException( - f"task_input should be defined for action '{action.action}'" - ) + raise TypeError(f"task_input cannot be None for action '{action.action}'") return _send_request( validate_ca_cert=action.validate_ca_cert, url=urljoin(action.porch_url, "tasks"), @@ -240,13 +228,9 @@ def update_task(action: PorchAction, pipeline: Pipeline): "Updates a status of a task." if action.task_input is None: - raise InvalidValueException( - f"task_input should be defined for action '{action.action}'" - ) + raise TypeError(f"task_input cannot be None for action '{action.action}'") if action.task_status is None: - raise InvalidValueException( - f"task_status should be defined for action '{action.action}'" - ) + raise TypeError(f"task_status cannot be None for action '{action.action}'") return _send_request( validate_ca_cert=action.validate_ca_cert, url=urljoin(action.porch_url, "tasks"), diff --git a/tests/test_api.py b/tests/test_api.py index 2497292..49266b5 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -5,7 +5,6 @@ from npg_porch_cli.api import ( AuthException, - InvalidValueException, Pipeline, PorchAction, ServerErrorException, @@ -56,22 +55,22 @@ def test_listing_actions(): def test_pipeline_class(): - with pytest.raises(InvalidValueException) as e: + with pytest.raises(TypeError) as e: Pipeline(name=None, uri="http://some.come", version="1.0") assert e.value.args[0] == "Pipeline name, uri and version should be defined" def test_porch_action_class(monkeypatch): - with pytest.raises(InvalidValueException) as e: + with pytest.raises(TypeError) as e: PorchAction(porch_url=None, action="list_tasks") - assert e.value.args[0] == "'porch_url' attribute should be defined" + assert e.value.args[0] == "'porch_url' attribute cannot be None" - with pytest.raises(InvalidValueException) as e: + with pytest.raises(TypeError) as e: PorchAction(porch_url="http://some.come", action=None) - assert e.value.args[0] == "'action' attribute should be defined" + assert e.value.args[0] == "'action' attribute cannot be None" - with pytest.raises(InvalidValueException) as e: + with pytest.raises(ValueError) as e: PorchAction(porch_url=url, action="list_tools") assert ( e.value.args[0] == "Action 'list_tools' is not valid. " @@ -84,7 +83,7 @@ def test_porch_action_class(monkeypatch): assert pa.task_input is None assert pa.task_status is None - with pytest.raises(InvalidValueException) as e: + with pytest.raises(ValueError) as e: pa = PorchAction( porch_url=url, action="add_task", @@ -126,7 +125,7 @@ def mock_get_200(*args, **kwargs): porch_url=url, ) assert pa.task_status == "FAILED" - with pytest.raises(InvalidValueException) as e: + with pytest.raises(ValueError) as e: PorchAction( task_status="Swimming", action="update_task", @@ -182,18 +181,18 @@ def test_request_validation(): p = Pipeline(uri=url, version="1.0", name="p1") pa = PorchAction(porch_url=url, action="add_task") - with pytest.raises(InvalidValueException) as e: + with pytest.raises(TypeError) as e: send(action=pa, pipeline=p) - assert e.value.args[0] == "task_input should be defined for action 'add_task'" + assert e.value.args[0] == "task_input cannot be None for action 'add_task'" pa = PorchAction(porch_url=url, action="update_task") - with pytest.raises(InvalidValueException) as e: + with pytest.raises(TypeError) as e: send(action=pa, pipeline=p) - assert e.value.args[0] == "task_input should be defined for action 'update_task'" + assert e.value.args[0] == "task_input cannot be None for action 'update_task'" pa = PorchAction(porch_url=url, action="update_task", task_input={"id_run": 5}) - with pytest.raises(InvalidValueException) as e: + with pytest.raises(TypeError) as e: send(action=pa, pipeline=p) - assert e.value.args[0] == "task_status should be defined for action 'update_task'" + assert e.value.args[0] == "task_status cannot be None for action 'update_task'" def test_sending_request(monkeypatch): From e0989606b281182e72c0c5d6b7577b13a4199555 Mon Sep 17 00:00:00 2001 From: mgcam Date: Wed, 26 Jun 2024 10:30:09 +0100 Subject: [PATCH 18/27] Renamed _send_request to send_request so that this method can be used by the code that uses other methods of this API. --- src/npg_porch_cli/api.py | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/src/npg_porch_cli/api.py b/src/npg_porch_cli/api.py index f56a886..f46611d 100644 --- a/src/npg_porch_cli/api.py +++ b/src/npg_porch_cli/api.py @@ -46,7 +46,6 @@ class ServerErrorException(Exception): @dataclass(kw_only=True) class Pipeline: - name: str uri: str version: str @@ -59,7 +58,6 @@ def __post_init__(self): @dataclass(kw_only=True) class PorchAction: - porch_url: str action: str validate_ca_cert: bool = field(default=True) @@ -164,7 +162,7 @@ def send(action: PorchAction, pipeline: Pipeline = None) -> dict | list: def list_pipelines(action: PorchAction) -> list: "Returns a listing of all pipelines registered with the porch server." - return _send_request( + return send_request( validate_ca_cert=action.validate_ca_cert, url=urljoin(action.porch_url, "pipelines"), method="GET", @@ -178,7 +176,7 @@ def list_tasks(action: PorchAction, pipeline: Pipeline = None) -> list: only tasks belonging to this pipeline are listed. """ - response_obj = _send_request( + response_obj = send_request( validate_ca_cert=action.validate_ca_cert, url=urljoin(action.porch_url, "tasks"), method="GET", @@ -192,7 +190,7 @@ def list_tasks(action: PorchAction, pipeline: Pipeline = None) -> list: def add_pipeline(action: PorchAction, pipeline: Pipeline): "Registers a new pipeline with the porch server." - return _send_request( + return send_request( validate_ca_cert=action.validate_ca_cert, method="POST", url=urljoin(action.porch_url, "pipelines"), @@ -205,7 +203,7 @@ def add_task(action: PorchAction, pipeline: Pipeline): if action.task_input is None: raise TypeError(f"task_input cannot be None for action '{action.action}'") - return _send_request( + return send_request( validate_ca_cert=action.validate_ca_cert, url=urljoin(action.porch_url, "tasks"), method="POST", @@ -216,7 +214,7 @@ def add_task(action: PorchAction, pipeline: Pipeline): def claim_task(action: PorchAction, pipeline: Pipeline): "Claims a task that belongs to the previously registered pipeline." - return _send_request( + return send_request( validate_ca_cert=action.validate_ca_cert, url=urljoin(action.porch_url, "tasks/claim"), method="POST", @@ -231,7 +229,7 @@ def update_task(action: PorchAction, pipeline: Pipeline): raise TypeError(f"task_input cannot be None for action '{action.action}'") if action.task_status is None: raise TypeError(f"task_status cannot be None for action '{action.action}'") - return _send_request( + return send_request( validate_ca_cert=action.validate_ca_cert, url=urljoin(action.porch_url, "tasks"), method="PUT", @@ -253,7 +251,8 @@ def update_task(action: PorchAction, pipeline: Pipeline): } -def _send_request(validate_ca_cert: bool, url: str, method: str, data: dict = None): +def send_request(validate_ca_cert: bool, url: str, method: str, data: dict = None): + """Sends an HTTPS request.""" headers = { "Authorization": "Bearer " + get_token(), From 7a58128642513604e43eff02c8ceba38d57d0406 Mon Sep 17 00:00:00 2001 From: mgcam Date: Fri, 28 Jun 2024 10:17:59 +0100 Subject: [PATCH 19/27] Simplified import of send_request --- pyproject.toml | 10 ++++++++++ src/npg_porch_cli/__init__.py | 1 + 2 files changed, 11 insertions(+) diff --git a/pyproject.toml b/pyproject.toml index 64f06c0..87bfb1c 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -38,6 +38,16 @@ line_length = 88 max-line-length = 88 extend-select = ["B950"] extend-ignore = ["E501"] +exclude = [ + # No need to traverse our git directory + ".git", + # There's no value in checking cache directories + "__pycache__" +] +per-file-ignores = """ + # Disable 'imported but unused' + __init__.py: F401 + """ [tool.pytest.ini_options] addopts = [ diff --git a/src/npg_porch_cli/__init__.py b/src/npg_porch_cli/__init__.py index e69de29..72d9c44 100644 --- a/src/npg_porch_cli/__init__.py +++ b/src/npg_porch_cli/__init__.py @@ -0,0 +1 @@ +from .api import send_request From 11c001483ce069d2bd7809ba02146dde6c0a5e6e Mon Sep 17 00:00:00 2001 From: mgcam Date: Mon, 8 Jul 2024 17:10:08 +0100 Subject: [PATCH 20/27] Tweaked and tested send_request method. For the method to be useful outside this git package, the authorization should be optional. Simplified the error message in the method. --- src/npg_porch_cli/api.py | 18 +++++++--- tests/test_send_request.py | 73 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 86 insertions(+), 5 deletions(-) create mode 100644 tests/test_send_request.py diff --git a/src/npg_porch_cli/api.py b/src/npg_porch_cli/api.py index f46611d..49444d6 100644 --- a/src/npg_porch_cli/api.py +++ b/src/npg_porch_cli/api.py @@ -18,7 +18,6 @@ # You should have received a copy of the GNU General Public License along with # this program. If not, see . -import inspect import json import os from dataclasses import InitVar, asdict, dataclass, field @@ -251,14 +250,25 @@ def update_task(action: PorchAction, pipeline: Pipeline): } -def send_request(validate_ca_cert: bool, url: str, method: str, data: dict = None): +def send_request( + validate_ca_cert: bool, + url: str, + method: str, + data: dict = None, + auth_type: str | None = "token", +): """Sends an HTTPS request.""" headers = { - "Authorization": "Bearer " + get_token(), "Content-Type": "application/json", "Accept": "application/json", } + if auth_type is not None: + if auth_type == "token": + headers["Authorization"] = "Bearer " + get_token() + else: + raise ValueError(f"Authorization type {auth_type} is not implemented") + request_args = { "headers": headers, "timeout": CLIENT_TIMEOUT, @@ -269,9 +279,7 @@ def send_request(validate_ca_cert: bool, url: str, method: str, data: dict = Non response = requests.request(method, url, **request_args) if not response.ok: - action_name = inspect.stack()[1].function raise ServerErrorException( - f"Action {action_name} failed. " f'Status code {response.status_code} "{response.reason}" ' f"received from {response.url}" ) diff --git a/tests/test_send_request.py b/tests/test_send_request.py new file mode 100644 index 0000000..70e4d82 --- /dev/null +++ b/tests/test_send_request.py @@ -0,0 +1,73 @@ +import pytest +import requests + +from npg_porch_cli import send_request +from npg_porch_cli.api import AuthException, ServerErrorException + +url = "http://some.com" +var_name = "NPG_PORCH_TOKEN" +json_data = {"some_data": "delivered"} + + +class MockResponseOK: + def __init__(self): + self.status_code = 200 + self.reason = "OK" + self.url = url + self.ok = True + + def json(self): + return json_data + + +class MockResponseNotFound: + def __init__(self): + self.status_code = 404 + self.reason = "NOT FOUND" + self.url = url + self.ok = False + + def json(self): + return {"Error": "Not found"} + + +def mock_get_200(*args, **kwargs): + return MockResponseOK() + + +def mock_get_404(*args, **kwargs): + return MockResponseNotFound() + + +def test_sending_request(monkeypatch): + + monkeypatch.delenv(var_name, raising=False) + + with pytest.raises(ValueError) as e: + send_request(validate_ca_cert=True, url=url, method="GET", auth_type="unknown") + assert e.value.args[0] == "Authorization type unknown is not implemented" + + with pytest.raises(AuthException) as e: + send_request(validate_ca_cert=True, url=url, method="GET") + assert e.value.args[0] == "Authorization token is needed" + + with monkeypatch.context() as m: + m.setattr(requests, "request", mock_get_200) + assert ( + send_request(validate_ca_cert=True, url=url, method="GET", auth_type=None) + == json_data + ) + + monkeypatch.setenv(var_name, "token_xyz") + + with monkeypatch.context() as m: + m.setattr(requests, "request", mock_get_200) + assert send_request(validate_ca_cert=False, url=url, method="GET") == json_data + + with monkeypatch.context() as m: + m.setattr(requests, "request", mock_get_404) + with pytest.raises(ServerErrorException) as e: + send_request(validate_ca_cert=False, url=url, method="POST", data=json_data) + assert e.value.args[0] == f'Status code 404 "NOT FOUND" received from {url}' + + monkeypatch.undo() From da55dd7843a53e0cea8e533d39c550632f568845 Mon Sep 17 00:00:00 2001 From: mgcam Date: Tue, 9 Jul 2024 10:28:55 +0100 Subject: [PATCH 21/27] Documented the functions --- src/npg_porch_cli/api.py | 44 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 42 insertions(+), 2 deletions(-) diff --git a/src/npg_porch_cli/api.py b/src/npg_porch_cli/api.py index 49444d6..211846e 100644 --- a/src/npg_porch_cli/api.py +++ b/src/npg_porch_cli/api.py @@ -254,10 +254,50 @@ def send_request( validate_ca_cert: bool, url: str, method: str, - data: dict = None, + data: dict | None = None, auth_type: str | None = "token", ): - """Sends an HTTPS request.""" + """Sends an HTTP request to a JSON API web service. + + Raises ServerErrorException if the status code of the response is not + in the 200 – 299 range. + + Args: + validate_ca_cert: + A boolean flag defining whether the server CA certificate + will be validated. If set to True, SSL_CERT_FILE environment + variable should be set. + url: + A URL to send the request to. + method: + The HTTP method to use (GET, POST, etc.) + data: + Optional payload for the request as a Python object. + auth_type: + Authorization type, defaults to 'token'. If no authorization + is required, set the value explicitly to None. Only the token + type authorization is implemented at the moment. For this type + of authorization to work, set NPG_PORCH_TOKEN environment + variable. + + Example: + + from npg_porch_cli import send_request + + url = "https://some.com/api/fruit_types" + fruit_types = send_request(validate_ca_cert=True, url=url, method="GET", auth_type=None) + + url = "https://some.com/api/add_fruit_type" + new_type = send_request( + validate_ca_cert=True, + url=url, + method="PUT", + data={"banana": {"taste": "sweet"}}, + ) + + Returns: + Server's decoded reply. + """ headers = { "Content-Type": "application/json", From 8e3d429dea2486cce9476f0253cbc76ae4dca3b9 Mon Sep 17 00:00:00 2001 From: mgcam Date: Tue, 16 Jul 2024 14:52:24 +0100 Subject: [PATCH 22/27] Expanded error message when possible --- src/npg_porch_cli/api.py | 13 ++++++++++++- tests/test_send_request.py | 28 +++++++++++++++++++++++++--- 2 files changed, 37 insertions(+), 4 deletions(-) diff --git a/src/npg_porch_cli/api.py b/src/npg_porch_cli/api.py index 211846e..792256f 100644 --- a/src/npg_porch_cli/api.py +++ b/src/npg_porch_cli/api.py @@ -319,9 +319,20 @@ def send_request( response = requests.request(method, url, **request_args) if not response.ok: - raise ServerErrorException( + detail = "" + try: + data = response.json() + if "detail" in data: + detail = data["detail"] + except Exception: + pass + + message = ( f'Status code {response.status_code} "{response.reason}" ' f"received from {response.url}" ) + if detail: + message += f".\nDetail: {detail}" + raise ServerErrorException(message) return response.json() diff --git a/tests/test_send_request.py b/tests/test_send_request.py index 70e4d82..3a37ae3 100644 --- a/tests/test_send_request.py +++ b/tests/test_send_request.py @@ -1,6 +1,5 @@ import pytest import requests - from npg_porch_cli import send_request from npg_porch_cli.api import AuthException, ServerErrorException @@ -28,7 +27,18 @@ def __init__(self): self.ok = False def json(self): - return {"Error": "Not found"} + return {"detail": "Not found in our data"} + + +class MockResponseNotFoundShort: + def __init__(self): + self.status_code = 404 + self.reason = "NOT FOUND" + self.url = url + self.ok = False + + def json(self): + return {} def mock_get_200(*args, **kwargs): @@ -39,8 +49,11 @@ def mock_get_404(*args, **kwargs): return MockResponseNotFound() -def test_sending_request(monkeypatch): +def mock_get_404_short(*args, **kwargs): + return MockResponseNotFoundShort() + +def test_sending_request(monkeypatch): monkeypatch.delenv(var_name, raising=False) with pytest.raises(ValueError) as e: @@ -66,6 +79,15 @@ def test_sending_request(monkeypatch): with monkeypatch.context() as m: m.setattr(requests, "request", mock_get_404) + with pytest.raises(ServerErrorException) as e: + send_request(validate_ca_cert=False, url=url, method="POST", data=json_data) + assert e.value.args[0] == ( + 'Status code 404 "NOT FOUND" received from ' + f"{url}.\nDetail: Not found in our data" + ) + + with monkeypatch.context() as m: + m.setattr(requests, "request", mock_get_404_short) with pytest.raises(ServerErrorException) as e: send_request(validate_ca_cert=False, url=url, method="POST", data=json_data) assert e.value.args[0] == f'Status code 404 "NOT FOUND" received from {url}' From 03ed48d4d293e4a73c66a322a785c258a354466d Mon Sep 17 00:00:00 2001 From: mgcam Date: Tue, 16 Jul 2024 14:55:26 +0100 Subject: [PATCH 23/27] Added a trailing slash to url to avoid a redirect --- src/npg_porch_cli/api.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/npg_porch_cli/api.py b/src/npg_porch_cli/api.py index 792256f..c675df7 100644 --- a/src/npg_porch_cli/api.py +++ b/src/npg_porch_cli/api.py @@ -230,7 +230,7 @@ def update_task(action: PorchAction, pipeline: Pipeline): raise TypeError(f"task_status cannot be None for action '{action.action}'") return send_request( validate_ca_cert=action.validate_ca_cert, - url=urljoin(action.porch_url, "tasks"), + url=urljoin(action.porch_url, "tasks/"), method="PUT", data={ "pipeline": asdict(pipeline), From 81caf8442ba4fc22926cc048395b964c2c7ab504 Mon Sep 17 00:00:00 2001 From: mgcam Date: Tue, 16 Jul 2024 15:04:12 +0100 Subject: [PATCH 24/27] Formatted import --- tests/test_send_request.py | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/test_send_request.py b/tests/test_send_request.py index 3a37ae3..188ceee 100644 --- a/tests/test_send_request.py +++ b/tests/test_send_request.py @@ -1,5 +1,6 @@ import pytest import requests + from npg_porch_cli import send_request from npg_porch_cli.api import AuthException, ServerErrorException From 9d45a44bcf996271433a8c78a43e8e90f9b56ffe Mon Sep 17 00:00:00 2001 From: mgcam Date: Wed, 17 Jul 2024 10:20:33 +0100 Subject: [PATCH 25/27] Explicitly set the status of the new task. Following https://github.com/wtsi-npg/npg_porch/pull/72, Status code 422 "Unprocessable Entity" error is received from the server if the status is not set. --- src/npg_porch_cli/api.py | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/src/npg_porch_cli/api.py b/src/npg_porch_cli/api.py index c675df7..a1578cc 100644 --- a/src/npg_porch_cli/api.py +++ b/src/npg_porch_cli/api.py @@ -28,7 +28,15 @@ PORCH_OPENAPI_SCHEMA_URL = "api/v1/openapi.json" PORCH_TASK_STATUS_ENUM_NAME = "TaskStateEnum" -PORCH_STATUSES = ["PENDING", "CLAIMED", "RUNNING", "DONE", "FAILED", "CANCELLED"] +INITIAL_PORCH_STATUS = "PENDING" +PORCH_STATUSES = [ + INITIAL_PORCH_STATUS, + "CLAIMED", + "RUNNING", + "DONE", + "FAILED", + "CANCELLED", +] CLIENT_TIMEOUT = (10, 60) @@ -198,7 +206,10 @@ def add_pipeline(action: PorchAction, pipeline: Pipeline): def add_task(action: PorchAction, pipeline: Pipeline): - "Registers a new task with the porch server." + """Registers a new task with the porch server. + + The new task is created with the default PENDING status. + """ if action.task_input is None: raise TypeError(f"task_input cannot be None for action '{action.action}'") @@ -206,7 +217,11 @@ def add_task(action: PorchAction, pipeline: Pipeline): validate_ca_cert=action.validate_ca_cert, url=urljoin(action.porch_url, "tasks"), method="POST", - data={"pipeline": asdict(pipeline), "task_input": action.task_input}, + data={ + "pipeline": asdict(pipeline), + "task_input": action.task_input, + "status": INITIAL_PORCH_STATUS, + }, ) From 9a538654eb7c3a492db8a0eff3e156da0a55a32a Mon Sep 17 00:00:00 2001 From: mgcam Date: Thu, 18 Jul 2024 12:58:22 +0100 Subject: [PATCH 26/27] Updated documentation --- CHANGELOG.md | 4 +- pyproject.toml | 2 +- src/npg_porch_cli/api.py | 108 +++++++++++++++++++++++++++++++++------ tests/test_api.py | 10 ++-- 4 files changed, 101 insertions(+), 23 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f0ebaaa..bcb1195 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,8 +5,8 @@ This project adheres to [Semantic Versioning](http://semver.org/). ## [Unreleased] -## [0.0.1] +## [0.1.0] ### Added -# Initial project scaffold +# Initial project scaffold, code and tests diff --git a/pyproject.toml b/pyproject.toml index 87bfb1c..049982b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "npg_porch_cli" -version = "0.0.1" +version = "0.1.0" authors = [ "Marina Gourtovaia", "Kieron Taylor", diff --git a/src/npg_porch_cli/api.py b/src/npg_porch_cli/api.py index a1578cc..11f6dba 100644 --- a/src/npg_porch_cli/api.py +++ b/src/npg_porch_cli/api.py @@ -138,25 +138,45 @@ def _validate_status(self) -> str | None: return status -def get_token(): +def get_token() -> str: + """Gets the value of the porch token from the environment variable. + + If the NPG_PORCH_TOKEN is not defined or assigned to am empty string, + raises AuthException. + + Returns: + The token. + """ if NPG_PORCH_TOKEN_ENV_VAR not in os.environ: raise AuthException("Authorization token is needed") - return os.environ[NPG_PORCH_TOKEN_ENV_VAR] + token = os.environ[NPG_PORCH_TOKEN_ENV_VAR] + if token == "": + raise AuthException("Authorization token is needed") + + return token -def list_client_actions() -> list: +def list_client_actions() -> list[str]: """Returns a sorted list of currently implemented client actions.""" return sorted(_PORCH_CLIENT_ACTIONS.keys()) def send(action: PorchAction, pipeline: Pipeline = None) -> dict | list: - """ + """Sends a request to the porch API server. + Sends a request to the porch API server to perform an action defined by the `action` attribute of the `action` argument. The context of the query is defined by the pipeline argument. - The server's response is returned as a dictionary or as a list. + Args: + action: + npg_porch_cli.api.PorchAction object + pipeline: + npg_porch_cli.api.Pipeline object + + Returns: + The server's response is returned as a Python data structure. """ # Get function's definition and then call the function. @@ -167,7 +187,16 @@ def send(action: PorchAction, pipeline: Pipeline = None) -> dict | list: def list_pipelines(action: PorchAction) -> list: - "Returns a listing of all pipelines registered with the porch server." + """Lists all pipelines registered with the porch server. + + Args: + action: + npg_porch_cli.api.PorchAction object + + Returns: + A list of dictionaries representing npg_porch_cli.api.Pipeline objects + + """ return send_request( validate_ca_cert=action.validate_ca_cert, @@ -177,10 +206,21 @@ def list_pipelines(action: PorchAction) -> list: def list_tasks(action: PorchAction, pipeline: Pipeline = None) -> list: - """ - In the pipeline argument is not defined, returns a listing of all tasks - registered with the porch server. If the pipeline argument is defined, - only tasks belonging to this pipeline are listed. + """Lists tasks. + + Args: + action: + npg_porch_cli.api.PorchAction object + pipeline: + npg_porch_cli.api.Pipeline object, optional + + Returns: + A list of Python objects, most likely dictionaries, representing registered + tasks. + + If the pipeline argument is defined, only tasks belonging to this pipeline + are listed. Otherwise the list contains all tasks registered with the + porch server. """ response_obj = send_request( @@ -194,8 +234,18 @@ def list_tasks(action: PorchAction, pipeline: Pipeline = None) -> list: return response_obj -def add_pipeline(action: PorchAction, pipeline: Pipeline): - "Registers a new pipeline with the porch server." +def add_pipeline(action: PorchAction, pipeline: Pipeline) -> dict: + """Registers a new pipeline with the porch server. + + Args: + action: + npg_porch_cli.api.PorchAction object + pipeline: + npg_porch_cli.api.Pipeline object + + Returns: + A dictionary representing npg_porch_cli.api.Pipeline object + """ return send_request( validate_ca_cert=action.validate_ca_cert, @@ -208,7 +258,15 @@ def add_pipeline(action: PorchAction, pipeline: Pipeline): def add_task(action: PorchAction, pipeline: Pipeline): """Registers a new task with the porch server. - The new task is created with the default PENDING status. + Args: + action: + npg_porch_cli.api.PorchAction object + pipeline: + npg_porch_cli.api.Pipeline object + + Returns: + A dictionary representing the new task. The status of the new task is + 'PENDING'. """ if action.task_input is None: @@ -226,7 +284,17 @@ def add_task(action: PorchAction, pipeline: Pipeline): def claim_task(action: PorchAction, pipeline: Pipeline): - "Claims a task that belongs to the previously registered pipeline." + """Claims a task that belongs to the pipeline. + + Args: + action: + npg_porch_cli.api.PorchAction object + pipeline: + npg_porch_cli.api.Pipeline object + + Returns: + A dictionary representing the claimed task. + """ return send_request( validate_ca_cert=action.validate_ca_cert, @@ -237,7 +305,17 @@ def claim_task(action: PorchAction, pipeline: Pipeline): def update_task(action: PorchAction, pipeline: Pipeline): - "Updates a status of a task." + """Updates the status of an existing task. + + Args: + action: + npg_porch_cli.api.PorchAction object + pipeline: + npg_porch_cli.api.Pipeline object + + Returns: + A dictionary representing the updated task. + """ if action.task_input is None: raise TypeError(f"task_input cannot be None for action '{action.action}'") diff --git a/tests/test_api.py b/tests/test_api.py index 49266b5..685b47b 100644 --- a/tests/test_api.py +++ b/tests/test_api.py @@ -30,12 +30,16 @@ def json(self): def test_retrieving_token(monkeypatch): - monkeypatch.delenv(var_name, raising=False) with pytest.raises(AuthException) as e: get_token() assert e.value.args[0] == "Authorization token is needed" + monkeypatch.setenv(var_name, "") + with pytest.raises(AuthException) as e: + get_token() + assert e.value.args[0] == "Authorization token is needed" + monkeypatch.setenv(var_name, "token_xyz") assert get_token() == "token_xyz" @@ -54,14 +58,12 @@ def test_listing_actions(): def test_pipeline_class(): - with pytest.raises(TypeError) as e: Pipeline(name=None, uri="http://some.come", version="1.0") assert e.value.args[0] == "Pipeline name, uri and version should be defined" def test_porch_action_class(monkeypatch): - with pytest.raises(TypeError) as e: PorchAction(porch_url=None, action="list_tasks") assert e.value.args[0] == "'porch_url' attribute cannot be None" @@ -177,7 +179,6 @@ def mock_get_200(*args, **kwargs): def test_request_validation(): - p = Pipeline(uri=url, version="1.0", name="p1") pa = PorchAction(porch_url=url, action="add_task") @@ -196,7 +197,6 @@ def test_request_validation(): def test_sending_request(monkeypatch): - monkeypatch.delenv(var_name, raising=False) monkeypatch.setenv(var_name, "MY_TOKEN") From c9e43a7111a731dc16190972b585a431296a5b51 Mon Sep 17 00:00:00 2001 From: mgcam Date: Tue, 23 Jul 2024 15:39:17 +0100 Subject: [PATCH 27/27] Prepared for release 0.1.0 --- CHANGELOG.md | 2 +- README.md | 43 +++++++++++++++++++++---------------------- 2 files changed, 22 insertions(+), 23 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index bcb1195..88746e0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,7 +5,7 @@ This project adheres to [Semantic Versioning](http://semver.org/). ## [Unreleased] -## [0.1.0] +## [0.1.0] - 2024-07-23 ### Added diff --git a/README.md b/README.md index 06f7367..97e6c2c 100644 --- a/README.md +++ b/README.md @@ -7,9 +7,9 @@ Provides a Python script, `npg_porch_client`, and a Python client API. NPG_PORCH_TOKEN environment variable should be set to the value of either the admin or project-specific token. The token should be pre-registered in -the database that is used by the npg_porch API server. +the database that is used by the `porch` API server. -Can be deployed with pip or poetry in a standard way. +The project can be deployed with pip or poetry in a standard way. Example of using a client API: @@ -18,10 +18,27 @@ Example of using a client API: pr = PorchRequest(porch_url="https://myporch.com") response = pr.send(action="list_pipelines") + + pr = PorchRequest( + porch_url="https://myporch.com", + pipeline_name="Snakemake_Cardinal", + pipeline_url="https://github.com/wtsi-npg/snakemake_cardinal", + pipeline_version="1.0", + ) + response = pr.send( + action="update_task", + task_status="FAILED", + task_input={"id_run": 409, "sample": "Valxxxx", "id_study": "65"}, + ) ``` -By default the client is set up to validate the server's CA certificate. -If the server is using a custom CA certificate, set the path to the certificate. +By default the client validates the certificate of the server's certification +authority (CA). If the server's certificate is signed by a custom CA, set the +`SSL_CERT_FILE` environment variable to the path of the CA's certificate. +Python versions starting from 3.11 seem to have increased security precautions +when validating certificates of custom CAs. It might be necessary to set the +`REQUESTS_CA_BUNDLE` environmental variable, see details +[here](https://requests.readthedocs.io/en/latest/user/advanced/#ssl-cert-verification). ``` bash export NPG_PORCH_TOKEN='my_token' @@ -48,21 +65,3 @@ double quotes in the example below. --task_json '{"id_run": 409, "sample": "Valxxxx", "id_study": "65"}' \ --status FAILED ``` - -If using the client API directly from Python, a dictionary should be used. - -``` python - from npg_porch_cli.api import PorchRequest - - pr = PorchRequest( - porch_url="https://myporch.com", - pipeline_name="Snakemake_Cardinal", - pipeline_url="https://github.com/wtsi-npg/snakemake_cardinal", - pipeline_version="1.0", - ) - response = pr.send( - action="update_task", - task_status="FAILED", - task_input={"id_run": 409, "sample": "Valxxxx", "id_study": "65"}, - ) -```