sprint production release 2021.07.19

[harshad16]

Hello, Thoth-station!

This Issue would be used for the current sprint cycle production release.
By the end of the sprint cycle, we will consolidate the information of thoth-station components features upgrade and fixes in this issue.

[fridex]

TensorFlow updates in recommendations

We have incorporated few additions to the recommender system when it comes to recent TensorFlow releases. Users of Thoth can now benefit from recommendations that take into account API symbols available also in the upcoming tensorflow~=2.6.0 release. The recommender system automatically suggests using the appropriate TensorFlow release based on API symbols used in the application (that matches API symbols available in the recommended TensorFlow release). The knowledge base was also extended with supported CUDA versions for TensorFlow 2.5 and upcoming TensorFlow 2.6.

• TensorFlow 2.5 is built with CUDA 11.2 support adviser#1959
• TensorFlow 2.6 uses same CUDA as TensorFlow 2.5, CUDA 11.2 adviser#1973
• Update TensorFlow API symbols to include symbols in 2.6 release adviser#1958
• Notify about performance degradation if ptxas are not available prescriptions#39

[fridex]

Python packages packaged as RPMs in Fedora 34 and UBI8

Thoth's knowledge was extended with information that states which Python packages are packaged as RPMs in Fedora 34 and UBI8. When asking for advice, the recommender system notifies about Python packages available as RPMs in justifications in the respective distributions.

Number of Python packages packaged in Fedora 34 as RPMs: 2244
Number of Python packages packaged in UBI as RPMs: 62

• Provide information whether a project is packaged as an RPM in distributions adviser#1961
• Notify users about Python packaged in UBI8 as RPMs prescriptions#42
• Add information about Fedora 34 RPM packages prescriptions#44
• Reorganize directory structure to have it browsable on GitHub prescriptions#45

[fridex]

Notify users about Python releases installed from PyPI

With recent changes, we provide a link to PyPI's release page for each Python package installed from PyPI. This helps users to directly navigate to project information presented on PyPI.

• Link Python package releases available on PyPI adviser#1964
• Add a wrap that notifies about Python package releases on PyPI adviser#1966

[fridex]

Resolve Python software stacks considering Python modules available in the runtime environment (container image) - declarative interface to configure resolver

The declarative interface to the resolution process now allows to state how the resolution process should look like when considering also Python packages already present in the runtime environment (container image). An example can be an adjusted resolution process when JupyterLab is installed in a specific version in the runtime environment (container image).

• Resolve Python software stacks considering Python libraries already present in the base container image adviser#1926
• documentation: https://thoth-station.ninja/docs/developers/adviser/prescription.html#should-include-runtime-environments-python-packages
• Introduce a mechanism for including unit based on Python packages adviser#1947
• Detect Python packages installed inside container images package-extract#452

[fridex]

Resolve Python software stacks considering prepared container images (runtime environments)

As we provide base container images with a prepared and tested set of packages ("predictive stacks") that are shipped with the container image, these container images can be additionally used as a base for installing packages as per user requests.

An example scenario can be a tested TensorFlow container image, that could be lacking TensorBoard package. If a user wants to install TensorBoard, the resolution process automatically detects TensorFlow release available in the container image and suggests a compatible TensorBoard release that will work with the preinstalled TensorFlow package shipped in the base image.

• Create a software stack resolution pipeline unit that can take into account Python packages shipped inside container image used adviser#1948
• Implement a pipeline unit for filtering Python packages already present in the base adviser#1957

[fridex]

Links to GitHub release notes for the most used 5000 packages on PyPI

The recommendation engine now provides links to release notes pages available on GitHub (if any) for packages that are installed. For example, users will be pointed to numpy==1.21.0 release page if the given numpy is installed in the specific version. This information was aggregated for top 5000 Python packages on PyPI (top based on number of downloads). If you are a maintainer of a repository that provides release information on GitHub and your package is not listed in Thoth's database, feel free to add it to the thoth-station/prescriptions repository following the docs.

• State information about release notes published on GitHub prescriptions#40
• https://github.com/numpy/numpy/releases/tag/v1.21.0

[fridex]

Notify users if a Python package has archived repository on GitHub

To make sure Python application developers use Python packages that are still actively developed, we warn when users use Python packages that have archived repositories on GitHub. This detection is performed for the top 5000 used packages on PyPI (based on the number of downloads).

• Notify about archived projects on GitHub prescriptions#54
• Provide warning for projects that are archived on GitHub adviser#1960
• Archived projects on GitHub prescriptions#58

[fridex]

Notify users if they consume Python packages from Operate First' Pulp instance

We inform users if they consume packages from Operate First Pulp instance. The recommender system prints this information in the provided justification together with the recommended stack.

• Notify about packages published on Operate First Pulp instance adviser#1974

[harshad16]

we have completed the release of 2021.07.05 🎉 🎊 🥳

Features

• TensorFlow updates in recommendations

We have incorporated few additions to the recommender system when it comes to recent TensorFlow releases. Users of Thoth can now benefit from recommendations that take into account API symbols available also in the upcoming tensorflow~=2.6.0 release. The recommender system automatically suggests using the appropriate TensorFlow release based on API symbols used in the application (that matches API symbols available in the recommended TensorFlow release). The knowledge base was also extended with supported CUDA versions for TensorFlow 2.5 and upcoming TensorFlow 2.6.

• TensorFlow 2.5 is built with CUDA 11.2 support adviser#1959

• TensorFlow 2.6 uses same CUDA as TensorFlow 2.5, CUDA 11.2 adviser#1973

• Update TensorFlow API symbols to include symbols in 2.6 release adviser#1958

• Notify about performance degradation if ptxas are not available prescriptions#39

• Python packages packaged as RPMs in Fedora 34 and UBI8

Thoth's knowledge was extended with information that states which Python packages are packaged as RPMs in Fedora 34 and UBI8. When asking for advice, the recommender system notifies about Python packages available as RPMs in justifications in the respective distributions.

Number of Python packages packaged in Fedora 34 as RPMs: 2244
Number of Python packages packaged in UBI as RPMs: 62

• Provide information whether a project is packaged as an RPM in distributions adviser#1961

• Notify users about Python packaged in UBI8 as RPMs prescriptions#42

• Add information about Fedora 34 RPM packages prescriptions#44

• Reorganize directory structure to have it browsable on GitHub prescriptions#45

• Notify users about Python releases installed from PyPI

With recent changes, we provide a link to PyPI's release page for each Python package installed from PyPI. This helps users to directly navigate to project information presented on PyPI.

• Link Python package releases available on PyPI adviser#1964

• Add a wrap that notifies about Python package releases on PyPI adviser#1966

• Resolve Python software stacks considering Python modules available in the runtime environment (container image) - declarative interface to configure resolver

The declarative interface to the resolution process now allows to state how the resolution process should look like when considering also Python packages already present in the runtime environment (container image). An example can be an adjusted resolution process when JupyterLab is installed in a specific version in the runtime environment (container image).

• Resolve Python software stacks considering Python libraries already present in the base container image adviser#1926

• documentation: https://thoth-station.ninja/docs/developers/adviser/prescription.html#should-include-runtime-environments-python-packages

• Introduce a mechanism for including unit based on Python packages adviser#1947

• Detect Python packages installed inside container images package-extract#452

• Resolve Python software stacks considering prepared container images (runtime environments)

As we provide base container images with a prepared and tested set of packages ("predictive stacks") that are shipped with the container image, these container images can be additionally used as a base for installing packages as per user requests.

An example scenario can be a tested TensorFlow container image, that could be lacking TensorBoard package. If a user wants to install TensorBoard, the resolution process automatically detects TensorFlow release available in the container image and suggests a compatible TensorBoard release that will work with the preinstalled TensorFlow package shipped in the base image.

• Create a software stack resolution pipeline unit that can take into account Python packages shipped inside container image used adviser#1948

• Implement a pipeline unit for filtering Python packages already present in the base adviser#1957

• Links to GitHub release notes for the most used 5000 packages on PyPI

The recommendation engine now provides links to release notes pages available on GitHub (if any) for packages that are installed. For example, users will be pointed to numpy==1.21.0 release page if the given numpy is installed in the specific version. This information was aggregated for top 5000 Python packages on PyPI (top based on number of downloads). If you are a maintainer of a repository that provides release information on GitHub and your package is not listed in Thoth's database, feel free to add it to the thoth-station/prescriptions repository following the docs.

• State information about release notes published on GitHub prescriptions#40

• https://github.com/numpy/numpy/releases/tag/v1.21.0

• Notify users if a Python package has archived repository on GitHub

To make sure Python application developers use Python packages that are still actively developed, we warn when users use Python packages that have archived repositories on GitHub. This detection is performed for the top 5000 used packages on PyPI (based on the number of downloads).

• Notify about archived projects on GitHub prescriptions#54

• Provide warning for projects that are archived on GitHub adviser#1960

• Archived projects on GitHub prescriptions#58

• Notify users if they consume Python packages from Operate First' Pulp instance

We inform users if they consume packages from Operate First Pulp instance. The recommender system prints this information in the provided justification together with the recommended stack.

• Notify about packages published on Operate First Pulp instance adviser#1974

Component Updates

• adviser: v0.36.0

  • Notify about packages published on Operate First Pulp instance adviser#1974
  • Add a wrap that notifies about Python package releases on PyPI adviser#1966
  • Use CLoader to load prescriptions to speed up loading adviser#1962
  • Implement a pipeline unit for filtering Python packages already present in the base adviser#1957
  • TensorFlow 2.5 is built with CUDA 11.2 support adviser#1959
  • Update TensorFlow API symbols to include symbols in 2.6 release adviser#1958
  • Introduce a mechanism for including unit based on Python packages adviser#1947

• prescriptions: v0.2.0

  • Archived projects on GitHub prescriptions#58
  • Archived projects on GitHub prescriptions#58
  • Notify about archived projects on GitHub prescriptions#54
  • Add prescription for Pillow==8.3.0 and NumPy incompatibilities prescriptions#53
  • Notify about UBI8 RPM packages only on UBI8/RHEL8 prescriptions#52
  • Notify about Fedora 34 RPM packages only on Fedora 34 prescriptions#51
  • Remove dashes from pipeline unit names prescriptions#47
  • Add information about Fedora 34 RPM packages prescriptions#44
  • State information about release notes published on GitHub prescriptions#40
  • Notify users about Python packaged in UBI8 as RPMs prescriptions#42
  • Fix wrong wrap unit schema used prescriptions#43

• package-extract: v1.2.0

  • Detect Python packages installed inside container images package-extract#452
  • Remove unused log parsing part and improve docs package-extract#446

• metrics-exporter: v0.17.0

  • Introduce metric on number of users of kebechet per manager metrics-exporter#750
  • Do not set metrics unless schema is up 2 date metrics-exporter#743

• slo-reporter: v0.16.0

  • Remove qeb-hwt bits slo-reporter#283

• mi: v2.6.2

  • Add missing features to schema mi#428
  • Add missing features to schema mi#428
  • Feature/kebechet analysis specific day mi#411
  • Documentation/update readme mi#394
  • Change pygithub commmit enttiy to pydriller commit mi#381

• mi-scheduler: v1.6.5

  • Fix/convert path to string mi-scheduler#170
  • Feature/cli envvars mi-scheduler#136
  • Add merge schedule, make arguments optional in Scheduler mi-scheduler#131
  • Add custom path for kebechet inspection mi-scheduler#81

Thanks for the amazing work everyone. 💯