Skip to content

Commit

Permalink
Your commit message here
Browse files Browse the repository at this point in the history
  • Loading branch information
shephinphilip committed Sep 13, 2023
1 parent b7ffb01 commit 06db220
Show file tree
Hide file tree
Showing 15 changed files with 1,473 additions and 341 deletions.
85 changes: 43 additions & 42 deletions docs/overview/contributing/open_tasks.rst

Large diffs are not rendered by default.

172 changes: 87 additions & 85 deletions docs/overview/contributing/the_basics.rst

Large diffs are not rendered by default.

32 changes: 15 additions & 17 deletions docs/overview/deep_dive/continuous_integration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@ Continuous Integration
======================

.. _`continuous integration channel`: https://discord.com/channels/799879767196958751/1028268051776413759
.. _`continuous integration forum`: https://discord.com/channels/799879767196958751/1028298018438520862
.. _`discord`: https://discord.gg/sXyFF8tDtm

We follow the practice of Continuous Integration (CI), in order to regularly build and test code at Ivy.
Expand All @@ -17,10 +16,10 @@ In order to incorporate Continuous Integration in the Ivy Repository, we follow
#. Periodic Testing
#. Manual Testing

.. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/deep_dive/continuous_integration/CI.png?raw=true
.. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/deep_dive/continuous_integration/CI.png?raw=true
:alt: CI Overview

We use GitHub Actions in order to implement and automate the process of testing. GitHub Actions allow implementing custom workflows that can build the code in the repository and run the tests. All the workflows used by Ivy are defined in the `.github/workflows <https://github.com/unifyai/ivy/tree/master/.github/workflows>`_ directory.
We use GitHub Actions in order to implement and automate the process of testing. GitHub Actions allow implementing custom workflows that can build the code in the repository and run the tests. All the workflows used by Ivy are defined in the `.github/workflows <https://github.com/unifyai/ivy/tree/main/.github/workflows>`_ directory.

Commit (Push/PR) Triggered Testing
----------------------------------
Expand All @@ -37,7 +36,7 @@ A test is defined as the triplet of (submodule, function, backend). We follow th

For example, :code:`ivy_tests/test_ivy/test_frontends/test_torch/test_tensor.py::test_torch_instance_arctan_,numpy`

The Number of such Ivy tests running on the Repository (without taking any Framework/Python Versioning into account) is 12500 (as of writing this documentation), and we are adding tests daily. Therefore, triggering all the tests on each commit is neither desirable (as it will consume a huge lot of Compute Resources, as well take a large amount of time to run) nor feasible (as Each Job in Github Actions has a time Limit of 360 Minutes, and a Memory Limit as well).
The Number of such Ivy tests running on the Repository (without taking any Framework/Python Versioning into account) is 12500 (as of writing this documentation), and we are adding tests daily. Therefore, triggering all the tests on each commit is neither desirable (as it will consume a huge lot of Compute Resources, as well as take a large amount of time to run) nor feasible (as Each Job in Github Actions has a time Limit of 360 Minutes, and a Memory Limit as well).

Further, When we consider versioning, for a single Python version, and ~40 frontend and backend versions, the tests would shoot up to 40 * 40 * 12500 = 20,000,000, and we obviously don't have resources as well as time to run those many tests on each commit.

Expand Down Expand Up @@ -91,7 +90,7 @@ But, there’s a fundamental issue here, Computing the Mapping requires determin

Now assume that we had some way to update the Mapping for a commit from the previous Mapping without having to run all the tests. Then, Given the Mapping for a single commit, we could follow this to determine and run the relevant tests for each commit as follows:

.. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/deep_dive/continuous_integration/ITRoadmap.png?raw=true
.. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/deep_dive/continuous_integration/ITRoadmap.png?raw=true
:alt: Intelligent Testing Roadmap
This is exactly what we do in order to implement Intelligent Testing. The “Update Mapping” Logic works as follows for each changed file:

Expand Down Expand Up @@ -207,7 +206,7 @@ Now, that the SSH key of the Runner has permissions to push and clone the Mappin
git clone --single-branch --depth 1 --branch "$TARGET_BRANCH" [email protected]:unifyai/Mapping.git
In case of, Pull Requests, we do not have access to :code:`SSH_DEPLOY_KEY` secret (and we don’t even want to give PRs that access), and thus we don’t use the SSH Clone Methodology and instead use the HTTP Clone Method, as follows:
In the case of, Pull Requests, we do not have access to :code:`SSH_DEPLOY_KEY` secret (and we don’t even want to give PRs that access), and thus we don’t use the SSH Clone Methodology and instead use the HTTP Clone Method, as follows:

.. code-block::
Expand Down Expand Up @@ -274,7 +273,7 @@ The Determine Test Coverage Workflow and the Intelligent Tests Workflow can run
Consider the following Case for Runner 2:

#. The Determine Test Coverage workflow has been running, and is about to complete for Runner 2. Meanwhile, a commit made on the master triggers the intelligent-tests workflow.
#. The runner 2 in the intelligent-tests workflow, pulls the Mapping from the master2 branch of unifyai/Mapping repository, and starts running the determined tests (based on changes made in the commit).
#. The runner 2 in the intelligent-tests workflow, pulls the Mapping from the master2 branch of the unifyai/Mapping repository, and starts running the determined tests (based on changes made in the commit).
#. The det-test-coverage workflow completes for runner2, which makes a push to the corresponding branch in the unifyai/Mapping Repository.
#. The runner 2 in the intelligent-tests workflow also completes, and pushes the updated repository

Expand All @@ -288,14 +287,14 @@ We handle the Race Condition as follows:

Array API Tests
---------------
The `array-api-intelligent-tests.yml (Push) <https://github.com/unifyai/ivy/blob/master/.github/workflows/array-api-intelligent-tests.yml>`_ and the `array-api-intelligent-tests-pr.yml (Pull Request) <https://github.com/unifyai/ivy/blob/master/.github/workflows/array-api-intelligent-tests-pr.yml>`_ workflows run the Array API Tests. Similar to Ivy Tests, The Array API tests are also determined intelligently and only relevant tests are triggered on each commit.
The `array-api-intelligent-tests.yml (Push) <https://github.com/unifyai/ivy/blob/main/.github/workflows/array-api-intelligent-tests.yml>`_ and the `array-api-intelligent-tests-pr.yml (Pull Request) <https://github.com/unifyai/ivy/blob/main/.github/workflows/array-api-intelligent-tests-pr.yml>`_ workflows run the Array API Tests. Similar to Ivy Tests, The Array API tests are also determined intelligently and only relevant tests are triggered on each commit.

More details about the Array API Tests are available `here <https://unify.ai/docs/ivy/overview/deep_dive/array_api_tests.html>`_.
More details about the Array API Tests are available `here <array_api_tests.rst>`_.

Periodic Testing
----------------
In order to make sure that none of the Ivy Tests are left ignored for a long time, and to decouple the rate of testing to the rate of committing to the repository, we implement periodic testing on the Ivy Repository.
The `Test Ivy Cron Workflow <https://github.com/unifyai/ivy/blob/master/.github/workflows/test-ivy-cron.yml>`_ is responsible for implementing this behavior by running Ivy tests every hour. In Each Run, It triggers 150 Ivy Tests, cycling through all of the tests.
The `Test Ivy Cron Workflow <https://github.com/unifyai/ivy/blob/main/.github/workflows/test-ivy-cron.yml>`_ is responsible for implementing this behavior by running Ivy tests every hour. In Each Run, It triggers 150 Ivy Tests, cycling through all of the tests.
This number of 150 is chosen in order to make sure that the Action completes in 1 hour most of the time.
The Test Results update the corresponding cell on the Dashboards.

Expand Down Expand Up @@ -331,18 +330,18 @@ Whenever a push is made to the repository, a variety of workflows are triggered
This can be seen on the GitHub Repository Page, with the commit message followed by a yellow dot, indicating that some workflows have been queued to run following this commit, as shown below:


.. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/deep_dive/continuous_integration/push.png?raw=true
.. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/deep_dive/continuous_integration/push.png?raw=true
:alt: Push

Clicking on the yellow dot (🟡) (which changes to a tick (✔) or cross (❌), when the tests have been completed) yields a view of the test-suite results as shown below:

.. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/deep_dive/continuous_integration/push1.png?raw=true
.. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/deep_dive/continuous_integration/push1.png?raw=true
:alt: Test-Suite

Click on the "Details" link corresponding to the failing tests, in order to identify the cause of the failure.
It redirects to the Actions Tab, showing details of the failure, as shown below:

.. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/deep_dive/continuous_integration/push2.png?raw=true
.. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/deep_dive/continuous_integration/push2.png?raw=true
:alt: Workflow Result

Click on the "Run Tests" section in order to see the logs of the failing tests for Array API Tests. For Ivy Tests, head to the "Combined Test Results" Section of the display-test-results Job, which shows the Test Logs for each of the tests in the following format:
Expand Down Expand Up @@ -381,7 +380,7 @@ Pull Request
In case of a pull request, the test suite is available on the Pull Request Page on Github, as shown below:


.. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/deep_dive/continuous_integration/pull-request1.png?raw=true
.. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/deep_dive/continuous_integration/pull-request1.png?raw=true
:alt: PR Test-Suite

Clicking on the "Details" link redirects to the Action Log.
Expand All @@ -392,15 +391,14 @@ As an added feature, the Intelligent Tests for PR Workflow has a section on "New
Dashboard
---------
In order to view the status of the tests, at any point in time, we have implemented a dashboard application that shows the results of the latest Workflow that ran each test.
The Dashboards are available on the link: https://ivy-dynamical-dashboards.onrender.com
The Dashboards are available at the link: https://ivy-dynamical-dashboards.onrender.com
You can filter tests by selecting choices from the various dropdowns. The link can also be saved for redirecting straight to the filtered tests in the future. The status badges are clickable, and will take you directly to the Action log of the latest workflow that ran the corresponding test.

**Round Up**

This should have hopefully given you a good feel for how Continuous Integration works in Ivy.

If you have any questions, please feel free to reach out on `discord`_ in the `continuous integration channel`_
or in the `continuous integration forum`_!
If you have any questions, please feel free to reach out on `discord`_ in the `continuous integration channel`_!

**Video**

Expand Down
17 changes: 8 additions & 9 deletions docs/overview/deep_dive/docstring_examples.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,12 @@ Docstring Examples
.. _`repo`: https://github.com/unifyai/ivy
.. _`discord`: https://discord.gg/sXyFF8tDtm
.. _`docstring examples channel`: https://discord.com/channels/799879767196958751/982738352103129098
.. _`docstring examples forum`: https://discord.com/channels/799879767196958751/1028297703089774705

After writing the general docstrings, the final step is to add helpful examples to the docstrings.

There are eight types of examples, which each need to be added:

**Functional** examples show the function being called like so :code:`ivy.func_name(...)`, and these should be added to docstring of the function in the Ivy API :func:`ivy.func_name`.
**Functional** examples show the function being called like so :code:`ivy.func_name(...)`, and these should be added to the docstring of the function in the Ivy API :func:`ivy.func_name`.

**Array instance method** examples show the method being called like so :code:`x.func_name(...)` on an :class:`ivy.Array` instance, and these should be added to the docstring of the :class:`ivy.Array` instance method :meth:`ivy.Array.func_name`.

Expand All @@ -35,7 +34,7 @@ These special methods in turn call the functions in the Ivy API mentioned above.

**Functional Examples**

To recap, *functional* examples show the function being called like so :code:`ivy.func_name(...)`, and these should be added to docstring of the function in the Ivy API :func:`ivy.func_name`.
To recap, *functional* examples show the function being called like so :code:`ivy.func_name(...)`, and these should be added to the docstring of the function in the Ivy API :func:`ivy.func_name`.

Firstly, we should include *functional* examples with :class:`ivy.Array` instances in the input.

Expand Down Expand Up @@ -292,7 +291,7 @@ Point 12 is not relevant as :func:`ivy.tan` is not an *operator* function.

**Container Operator Examples**

Points 13, 14 and 15 are not relevant as :func:`ivy.tan` is not an *operator* function.
Points 13, 14, and 15 are not relevant as :func:`ivy.tan` is not an *operator* function.

**Container Reverse Operator Example**

Expand Down Expand Up @@ -441,7 +440,7 @@ Point 12 is not relevant as :func:`ivy.roll` is not an *operator* function.

**Container Operator Examples**

Points 13, 14 and 15 are not relevant as :func:`ivy.roll` is not an *operator* function.
Points 13, 14, and 15 are not relevant as :func:`ivy.roll` is not an *operator* function.

**Container Reverse Operator Example**

Expand Down Expand Up @@ -722,7 +721,7 @@ It basically checks if the output upon execution of the examples that are docume
Therefore each time you make a commit, you must ensure that the :code:`test-docstrings / run-docstring-tests` are working correctly at least for the function you are making changes to.
To check whether the docstring tests are passing you need to check the logs for :code:`test-docstrings / run-docstring-tests`:

.. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/deep_dive/docstring_examples/docstring_failing_test_logs.png?raw=true
.. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/deep_dive/docstring_examples/docstring_failing_test_logs.png?raw=true
:width: 420

You will need to go through the logs and see if the list of functions for which the docstring tests are failing also has the function you are working with.
Expand All @@ -732,10 +731,10 @@ If the docstring tests are failing the logs show a list of functions having iss
:code:`output in docs: ........`
as shown below:

.. image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/master/img/externally_linked/deep_dive/docstring_examples/docstring_log.png
.. image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/deep_dive/docstring_examples/docstring_log.png
:width: 420

It can be quite tedious to go through the output diffs and spot the exact error, so you can take help of online tools like `text compare <https://text-compare.com/>`_ to spot the minutest of differences.
It can be quite tedious to go through the output diffs and spot the exact error, so you can take the help of online tools like `text compare <https://text-compare.com/>`_ to spot the minutest of differences.

Once you make the necessary changes and the function you are working on doesn't cause the docstring tests to fail, you should be good to go.
However, one of the reviewers might ask you to make additional changes involving examples.
Expand All @@ -748,7 +747,7 @@ Passing docstring tests is a necessary but not sufficient condition for the comp

These three examples should give you a good understanding of what is required when adding docstring examples.

If you have any questions, please feel free to reach out on `discord`_ in the `docstring examples channel`_ or in the `docstring examples forum`_!
If you have any questions, please feel free to reach out on `discord`_ in the `docstring examples channel`_!


**Video**
Expand Down
Loading

0 comments on commit 06db220

Please sign in to comment.