diff --git a/docs/overview/contributing/open_tasks.rst b/docs/overview/contributing/open_tasks.rst index a869a15936b8d..081f7946d6488 100644 --- a/docs/overview/contributing/open_tasks.rst +++ b/docs/overview/contributing/open_tasks.rst @@ -7,13 +7,14 @@ Open Tasks .. _`issue description`: https://github.com/unifyai/ivy/issues/1526 .. _`reference API`: https://numpy.org/doc/stable/reference/routines.linalg.html .. _`imports`: https://github.com/unifyai/ivy/blob/38dbb607334cb32eb513630c4496ad0024f80e1c/ivy/functional/frontends/numpy/__init__.py#L27 +.. _`Deep Dive`: ../deep_dive.rst Here, we explain all tasks which are currently open for contributions from the community! This section of the docs will be updated frequently, whereby new tasks will be added and completed tasks will be removed. The tasks outlined here are generally broad high-level tasks, each of which is made up of many individual sub-tasks, distributed across task-specific `ToDo List Issues `_. -Please read about `ToDo List Issues `_ in detail before continuing. +Please read about :ref:`overview/contributing/the_basics:ToDo List Issues` in detail before continuing. All tasks should be selected and allocated as described in the ToDo List Issues section. We make no mention of task selection and allocation in the explanations below, which instead focus on the steps to complete only once a sub-task has been allocated to you. @@ -32,7 +33,7 @@ Function Formatting Currently, we have many ToDo list issues `open `_ for a general function formatting task, which is explained below. -Each function in each submodule should be updated to follow the implementation instructions given in the :ref:`Deep Dive` section. +Each function in each submodule should be updated to follow the implementation instructions given in the `Deep Dive`_ section. The updates should be applied for the: #. ivy API @@ -44,35 +45,35 @@ The updates should be applied for the: #. container operators #. container reverse operators -The :ref:`Deep Dive` is an **essential** resource for learning how each of these functions/methods should be implemented. -Before starting any contribution task, you should go through the :ref:`Deep Dive`, and familiarize yourself with the content. +The `Deep Dive`_ is an **essential** resource for learning how each of these functions/methods should be implemented. +Before starting any contribution task, you should go through the `Deep Dive`_, and familiarize yourself with the content. At the time of writing, many of the functions are not implemented as they should be. -You will need to make changes to the current implementations, but you do not need to address *all* sections of the :ref:`Deep Dive` in detail. +You will need to make changes to the current implementations, but you do not need to address *all* sections of the `Deep Dive`_ in detail. Specifically, you **do not** need to address the following: #. Implement the hypothesis testing for the function #. Get the tests passing for your function, if they are failing before you start -However, everything else covered in the :ref:`Deep Dive` must be addressed. +However, everything else covered in the `Deep Dive`_ must be addressed. Some common important tasks are: #. Remove all :code:`lambda` and direct bindings for the backend functions (in :code:`ivy.functional.backends`), with each function instead defined using :code:`def`. #. Implement the following if they don't exist but should do: :class:`ivy.Array` instance method, :class:`ivy.Container` instance method, :class:`ivy.Array` special method, :class:`ivy.Array` reverse special method, :class:`ivy.Container` special method, :class:`ivy.Container` reverse special method. #. Make sure that the aforementioned methods are added into the correct category-specific parent class, such as :class:`ivy.ArrayWithElementwise`, :class:`ivy.ContainerWithManipulation` etc. -#. Correct all of the :ref:`Function Arguments` and the type hints for every function **and** its *relevant methods*, including those you did not implement yourself. -#. Add the correct :ref:`Docstrings` to every function **and** its *relevant methods*, including those you did not implement yourself. -#. Add thorough :ref:`Docstring Examples` for every function **and** its *relevant methods* and ensure they pass the docstring tests. +#. Correct all of the `Function Arguments <../deep_dive/function_arguments.rst>`_ and the type hints for every function **and** its *relevant methods*, including those you did not implement yourself. +#. Add the correct `Docstrings <../deep_dive/docstrings.rst>`_ to every function **and** its *relevant methods*, including those you did not implement yourself. +#. Add thorough `Docstring Examples <../deep_dive/docstring_examples.rst>`_ for every function **and** its *relevant methods* and ensure they pass the docstring tests. Formatting checklist ~~~~~~~~~~~~~~~~~~~~ -After creating your Pull Request on github, you should then produce the checklist for the formatting task as follows: +After creating your Pull Request on github, you should then produce the checklist for the formatting task as follows: 1. Add a comment with the following format: :code:`add_reformatting_checklist_` on your PR, where ** is the name of the category that the function belongs to. An example of this is shown below. -.. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/contributing/open_tasks/checklist_generator.png?raw=true +.. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/contributing/open_tasks/checklist_generator.png?raw=true :width: 420 Using this formatting will then trigger our github automation bots to update your comment with the proper markdown text for the checklist. @@ -80,21 +81,21 @@ These updates might take a few moments to take effect, so please be patient 🙂 2. After adding the checklist to your PR, you should then modify this checklist with the status of each item according to the symbols(emojis) within the LEGEND section. -.. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/contributing/open_tasks/checklist_legend.png?raw=true +.. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/contributing/open_tasks/checklist_legend.png?raw=true :width: 420 3. When all check items are marked as (✅, ⏩, or 🆗), you should request a review for your PR and we will start checking your implementation and marking the items as complete using the checkboxes next to them. -.. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/contributing/open_tasks/checklist_checked.png?raw=true +.. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/contributing/open_tasks/checklist_checked.png?raw=true :width: 420 4. In case you are stuck or need help with one of the checklist items, please add the 🆘 symbol next to the item on the checklist, and proceed to add a comment elaborating on your point of struggle with this item. The PR assignee will then see this comment and address your issues. -.. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/contributing/open_tasks/checklist_SOS.png?raw=true +.. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/contributing/open_tasks/checklist_SOS.png?raw=true :width: 420 -**Notes**: +**Notes**: 1. It is important that the PR author is the one to add the checklist generating comment in order to ensure they will have access to edit and update it later. 2. The checklist items' statuses should be manually updated by the PR author. @@ -106,15 +107,15 @@ The PR assignee will then see this comment and address your issues. Frontend APIs ------------- -For this task, the goal will be to implement functions for each of the frontend functional APIs (see :ref:`Ivy as a Transpiler`), with frontend APIs implemented for: :code:`JAX`, :code:`NumPy`, :code:`TensorFlow` :code:`PyTorch`, :code:`Paddle`, :code:`Scipy`, :code:`MXNet` and :code:`MindSpore`. +For this task, the goal will be to implement functions for each of the frontend functional APIs (see `Ivy as a Transpiler <../design/ivy_as_a_transpiler.rst>`_), with frontend APIs implemented for: :code:`JAX`, :code:`NumPy`, :code:`TensorFlow` :code:`PyTorch`, :code:`Paddle`, :code:`Scipy`, :code:`MXNet` and :code:`MindSpore`. Currently, we have many ToDo list issues `open `_ for this task. The general workflow for this task is: -#. Find the correct location for the function by following the :ref:`Where to place a frontend function` subsection below -#. Implement the function by following the :ref:`Ivy Frontends` guide -#. Write tests for your function by following the :ref:`Ivy Frontend Tests` guide +#. Find the correct location for the function by following the :ref:`overview/contributing/open_tasks:Where to place a frontend function` subsection below +#. Implement the function by following the `Ivy Frontends <../deep_dive/ivy_frontends.rst>`_ guide +#. Write tests for your function by following the `Ivy Frontend Tests <../deep_dive/ivy_frontends_tests.rst>`_ guide #. Verify that the tests for your function are passing If you feel as though there is an ivy function :code:`ivy.` clearly missing, which would make your frontend function much simpler to implement, then you should first do the following: @@ -128,7 +129,7 @@ At some point, a member of our team will assess whether it should be added, and After this, you then have two options for how to proceed: -#. Try to implement the function as a composition of currently present ivy functions, as explained in the "Temporary Compositions" sub-section of the :ref:`Ivy Frontends` guide, and add the :code:`#ToDo` comment in the implementation as explained. +#. Try to implement the function as a composition of currently present ivy functions, as explained in the :ref:`overview/deep_dive/ivy_frontends:Short Frontend Implementations` sub-section of the `Ivy Frontends <../deep_dive/ivy_frontends.rst>`_ guide, and add the :code:`#ToDo` comment in the implementation as explained. Once the PR is merged, your sub-task issue will then be closed as normal. #. Alternatively, if you do not want to try and implement the frontend function compositionally, or if this is not feasible, then you can simply choose another frontend function to work on. You could also choose to work on another open task entirely at this point if you wanted to. @@ -142,14 +143,14 @@ There are a few other points to take note of when working on your chosen fronten #. You should only implement **one** frontend function. #. The frontend function is framework-specific, thus it should be implemented in its respective frontend framework only. #. Each frontend function should be tested on all backends to ensure that conversions are working correctly. -#. Type hints, docstrings and examples are not required for frontend functions. +#. Type hints, docstrings, and examples are not required for frontend functions. #. Some frontend functions shown in the ToDo list issues are aliases of other functions. If you detect that this is the case, then you should add all aliases in your PR, with a single implementation and then simple bindings to this implementation, such as :code:` = `. If you notice that an alias function has already been implemented and pushed, then you can simply add this one-liner binding and get this very simple PR merged. In the case where your chosen function exists in all frameworks by default, but is not implemented in Ivy's functional API, please convert your existing GitHub issue to request for the function to be added to Ivy. Meanwhile, you can select another frontend function to work on from the ToDo list! -If you're stuck on a function which requires complex compositions, you're allowed to reselect a function too! +If you're stuck on a function that requires complex compositions, you're allowed to reselect a function too! Where to place a frontend function ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -180,7 +181,7 @@ Taking :code:`numpy.inner` as an example, we can see that there are a few :code: ufunc There is a :code:`linalg` sub-directory, so we choose this. -Then we need to choose from the files at this hierarchy: +Then we need to choose from the files in this hierarchy: .. code-block:: bash :emphasize-lines: 3 @@ -217,7 +218,7 @@ However, you can still use the checklist as a reference in cases where you do un **Notes**: -1. More details on how to update the checklist items can be found in the :ref:`Formatting checklist` part of our docs. +1. More details on how to update the checklist items can be found in the :ref:`overview/contributing/open_tasks:Formatting checklist` part of our docs. 2. Do not edit the checklist text, only the emoji symbols. 3. Please refrain from using the checkboxes next to checklist items. @@ -233,62 +234,62 @@ There is only one central ToDo list `issue `_ -#. Every function will have a different file structure according to the function type, refer to :ref:`Where to place a backend function` subsection below. -#. Implement the container instance method in :mod:`ivy/container/experimental/[relevant_submodule].py` and the array instance method +#. Analyze the function type, we have a very detailed section for it in the deep dive `Function Types Guide <../deep_dive/function_types.rst>`_ +#. Every function will have a different file structure according to the function type, refer to :ref:`overview/contributing/open_tasks:Where to place a backend function` subsection below. +#. Implement the container instance method in :mod:`ivy/container/experimental/[relevant_submodule].py` and the array instance method in :mod:`ivy/array/experimental/[relevant_submodule].py` -#. Write tests for the function using the :ref:`Ivy Tests` guide, and make sure they are passing. +#. Write tests for the function using the `Ivy Tests <../deep_dive/ivy_tests.rst>`_ guide, and make sure they are passing. A few points to keep in mind while doing this: #. Make sure all the positional arguments are positional-only and optional arguments are keyword-only. #. In case some tests require function-specific parameters, you can create composite hypothesis strategies using the :code:`draw` function in the hypothesis library. -If you’re stuck on a function which requires complex compositions, feel free to reselect a function +If you’re stuck on a function which requires complex compositions, feel free to reselect a function Extending the Ivy API ~~~~~~~~~~~~~~~~~~~~~~~ -We primarily invite contributors to work on the tasks listed as :ref:`Open Tasks`, as these are on our current roadmap. As a result of this, we prompt everyone interested in contributing to our Experimental API to do so under the `Ivy Experimental API Open Task`_. +We primarily invite contributors to work on the tasks listed as :ref:`overview/contributing/open_tasks:Open Tasks`, as these are on our current roadmap. As a result of this, we prompt everyone interested in contributing to our Experimental API to do so under the :ref:`Ivy Experimental API Open Task `. -However, if you would like to extend Ivy's functionality with a new function, you are invited to open an issue using the *Missing Function Suggestion* template as described in `Creating an Issue on Ivy’s GitHub using a Template `_. +However, if you would like to extend Ivy's functionality with a new function, you are invited to open an issue using the *Missing Function Suggestion* template as described in :ref:`overview/contributing/open_tasks:Creating an Issue on Ivy's GitHub using a Template`. In this template form, you'll be asked to fill in the reason you think we should implement the suggested function, as well as the links to any native implementations of the suggested function. -We will review your issue as soon as possible and let you know if it's been accepted or not. In case we deem that the suggested function fits our roadmap, we will add it as a subtask to the `Ivy Experimental API Open Task`_. +We will review your issue as soon as possible and let you know if it's been accepted or not. In case we deem that the suggested function fits our roadmap, we will add it as a subtask to the `Ivy Experimental API Open Task `_. Where to place a backend function ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The placement of the backend function should be in the proper location to follow the proper structure as guided below. -There are multiple types of backend functions as discussed above, we will go through 3 of those which you will encounter while adding a backend function in our Funcional API: +There are multiple types of backend functions as discussed above, we will go through 3 of those which you will encounter while adding a backend function in our Functional API: **Primary Functions** Implement the function in :mod:`ivy/functional/ivy/experimental/[relevant_submodule].py` simply deferring to their backend-specific implementation -(where ivy.current_backend(x).function_name() is called), refer the `Ivy API Guide `_ +(where ivy.current_backend(x).function_name() is called), refer to the :ref:`Ivy API Guide ` to get a clearer picture of how this must be done. Then, implement the functions in each of the backend files :mod:`ivy/functional/backends/backend_name/experimental/[relevant_submodule].py`, -you can refer the `Backend API Guide `_ for this. +you can refer to the :ref:`Backend API Guide ` for this. **Compositional Functions** Implement the function in :mod:`ivy/functional/ivy/experimental/[relevant_submodule].py`, we will not use the primary function approach in this -case, the implementaion will be a composition of functions from Ivy's functional API. You can refer to -`Compostional Functions Guide `_ for a better understanding of this. +case, the implementation will be a composition of functions from Ivy's functional API. You can refer to +:ref:`overview/deep_dive/function_types:Compositional Functions` for a better understanding of this. You don't need to add any implementation in any other file in this case. **Mixed Functions** Sometimes, a function may only be provided by some of the supported backends. In this case, we have to take a mixed approach. You can say that this is a mix of both -primary and a compositional function. For this, you have to implement the function in :mod:`ivy/functional/ivy/experimental/[relevant_submodule].py`, where the implementaion +primary and a compositional function. For this, you have to implement the function in :mod:`ivy/functional/ivy/experimental/[relevant_submodule].py`, where the implementation will be a composition of functions from Ivy's functional API. After you are done with this, you then have to implement the functions in each of the backend files :mod:`ivy/functional/backends/backend_name/experimental/[relevant_submodule].py`. **Other Function Types** -`Standalone Functions `_, `Nestable Functions `_ and -`Convenience Functions `_ are the ones which you will rarely come across +:ref:`overview/deep_dive/function_types:Standalone Functions`, :ref:`overview/deep_dive/function_types:Nestable Functions` and +:ref:`overview/deep_dive/function_types:Convenience Functions` are the ones which you will rarely come across while implementing a function from the ToDo List but they are an essential part of the Ivy API. @@ -301,9 +302,9 @@ Creating an Issue on Ivy's GitHub using a Template * Bug Report: In case you find a bug in our API, you have to provide details in the form and the issue will be assigned to one of our team members to look into. * Feature request: - If you want to suggest an idea for our project, our team is always open for suggestions. + If you want to suggest an idea for our project, our team is always open to suggestions. * Missing Function Suggestion: - In case you find a function which the other frameworks have and is missing in our API or we have some functionality missing which the other frameworks support(superset behavior). + In case you find a function that the other frameworks have and is missing in our API or we have some functionality missing that the other frameworks support(superset behavior). * Sub-Task: Reserve a sub-task from a ToDo list issue. * Questions: diff --git a/docs/overview/contributing/the_basics.rst b/docs/overview/contributing/the_basics.rst index 4f81384f9f8ae..96de0bfc02e9c 100644 --- a/docs/overview/contributing/the_basics.rst +++ b/docs/overview/contributing/the_basics.rst @@ -10,7 +10,6 @@ The Basics .. _`commit frequency channel`: https://discord.com/channels/799879767196958751/982728822317256712 .. _`PyCharm blog`: https://www.jetbrains.com/help/pycharm/finding-and-replacing-text-in-file.html .. _`Debugging`: https://www.jetbrains.com/help/pycharm/debugging-code.html -.. _`Ivy Experimental API Open Task`: https://unify.ai/docs/ivy/overview/contributing/open_tasks.html#ivy-experimental-api Getting Help ------------ @@ -54,30 +53,32 @@ We make extensive use of `ToDo list issues `_, `frontend APIs `_ and `ivy experimental API `_. +a. Find a task to work on which (i) is not marked as completed with a tick (ii) does not have an issue created and (iii) is not mentioned in the comments. Currently, there are three open tasks: :ref:`overview/contributing/open_tasks:Function Formatting`, :ref:`overview/contributing/open_tasks:Frontend APIs` and :ref:`overview/contributing/open_tasks:Ivy Experimental API`. b. Create a new issue with the title being just the name of the sub-task you would like to work on. -c. Comment on the ToDo list issue with a reference to this issue like so: +c. Comment on the ToDo list issue with a reference to your new issue like so: :code:`- [ ] #Issue_number` - Your issue will then automatically be added to the ToDo list at some point, and the comment will be deleted. - No need to wait for this to happen before progressing to the next stage. Don’t comment anything else on these ToDo issues, which should be kept clean with comments only as described above. + For example, if your issue number is 12345, then the text of your comment should be :code:`- [ ] #12345`. You could also use just the issue number (:code:`#12345`), or a link to the issue itself (:code:`https://github.com/unifyai/ivy/issues/12345`). -d. Start working on the task, and create a PR as soon as you have a full or partial solution, and then directly reference the issue in the pull request by adding the following content to the description of the PR: + At some point after your comment is made, your issue will automatically be added to the ToDo list and the comment will be deleted. + No need to wait for this to happen before progressing to the next stage. Don’t comment anything else on these ToDo issues, which should be kept clean with comments only as described above. + +d. Start working on the task, and open a PR as soon as you have a full or partial solution, when you open the PR make sure to follow the `conventional commits format `_, and then directly reference the issue in the pull request by adding the following content to the description of the PR: :code:`Close #Issue_number` - This is important, so that the merging of your PR will automatically close the associated issue. Make sure this is in the + This is important, so that the merging of your PR will automatically close the associated issue. Make sure this is in the description of the PR, otherwise it might not link correctly. If you have a partial solution, the Ivy team can help to guide you through the process of getting it working 🙂 - Also remember to make the PR name well described and if there are some details that can support your changes add them to the description of the PR. + Also, remember to make the PR name well described and if there are some details that can support your changes add them to the description of the PR. e. Wait for us to review your PR. Once we have reviewed your PR we will either merge or request changes. Every time you respond to our requested changes you must re-request a review in order for us to re-engage with the PR. -f. Once the PR is in good shape, we will merge into master, and then you become an Ivy contributor! +f. Once the PR is in good shape, we will merge into main, and then you become an Ivy contributor! In order to keep our ToDo lists moving quickly, if your PR is not created within 7 days of creating the issue, then a warning message will appear on the issue. If another 7 days pass without any changes, the issue will be closed and the task will be made free for others in the community. @@ -97,7 +98,7 @@ Please don't take it personally if your issue or PR gets closed because of this Reach out to me on discord if at any point you believe this happened to you unfairly, and we will definitely investigate! -Finally, we limit the maximum number of *open* and *incomplete* sub-task issues at *three* per person. +Finally, we limit the maximum number of *open* and *incomplete* sub-task issues to *three* per person. This is to prevent anyone from self-allocating many sub-tasks, preventing others in the community from engaging, and then not being able to complete them. Even though the limit is three, sub-tasks should only be self-assigned using **one comment per sub-task**. For example, a sequence of comments like this :code:`- [ ] #Issue_number` will register correctly whereas a single comment like this :code:`- [ ] #Issue_number, - [ ] #Issue_number, - [ ] #Issue_number` or this :code:`- [ ] #Issue_number #Issue_number #Issue_number` etc. will not. @@ -117,7 +118,7 @@ For questions, please reach out on `discord`_ in the `todo list issues channel`_ Managing Your Fork ------------------ -When contributing to Ivy, the first step is create a fork of the repository. +When contributing to Ivy, the first step is to create a fork of the repository. Then, it's best practice to create a separate branch for each new pull request (PR) you create. This can be done using: @@ -125,20 +126,20 @@ This can be done using: git checkout -b name_of_your_branch -The master branch then simply has the role of being kept up to date with upstream. -You *can* create PRs based on the master branch of your fork, but this will make things more complicated if you would then like to create additional PRs in future. +The main branch then simply has the role of being kept up to date with upstream. +You *can* create PRs based on the main branch of your fork, but this will make things more complicated if you would then like to create additional PRs in the future. For keeping any branch on your fork up to date, there is a script in the root folder of the repo `merge_with_upstream.sh `_. -To update your fork's branch to the upstream master branch, simply run :code:`./merge_with_upstream.sh name_of_your_branch`. -To update the master branch, this would then be: :code:`./merge_with_upstream.sh master`. +To update your fork's branch to the upstream main branch, simply run :code:`./merge_with_upstream.sh name_of_your_branch`. +To update the main branch, this would then be: :code:`./merge_with_upstream.sh main`. When making a PR (explained in the next sub-section), sometimes you will see that changes to upstream have caused conflicts with your PR. In this case, you will need to either resolve these conflicts in the browser, or clone your fork and make changes locally in the terminal and push once resolved. Both of these cases are explained in the following video. -You may find that once you have made changes locally and try pulling from master, the pull request is aborted as there are merge conflicts. -In order to avoid tedious merge conflict resolution, you can try 'stashing' your local changes, then pulling from master. -Once your branch is up-to-date with master, you can reinstate the most recently stashed changes, commit and push to master with no conflicts. +You may find that once you have made changes locally and try pulling from main, the pull request is aborted as there are merge conflicts. +In order to avoid tedious merge conflict resolution, you can try 'stashing' your local changes, then pulling from main. +Once your branch is up-to-date with main, you can reinstate the most recently stashed changes, commit and push to main with no conflicts. The corresponding commands are :code:`git stash` -> :code:`git fetch` -> :code:`git pull` -> :code:`git stash apply stash@{0}`. Note that this only works for uncommitted changes (staged and unstaged) and untracked files won't be stashed. For a comprehensive explanation of git stashing, check out this `Atlassian tutorial`_. @@ -182,20 +183,20 @@ With Browser: **Git Blame View** is a handy tool to view the line-by-line revision history for an entire file, or view the revision history of a single line within a file. - .. image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/master/img/externally_linked/contributing/the_basics/git_blame/git_blame_1.png?raw=true + .. image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/contributing/the_basics/git_blame/git_blame_1.png?raw=true :width: 420 This view can be toggled from the option in left vertical pane, or from the "blame" icon in top-right, as highlighted above. - .. image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/master/img/externally_linked/contributing/the_basics/git_blame/git_blame_2.png?raw=true + .. image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/contributing/the_basics/git_blame/git_blame_2.png?raw=true :width: 420 Each time you click the highlighted icon, the previous revision information for that line is shown, including who committed the change and when this happened. - .. image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/master/img/externally_linked/contributing/the_basics/git_blame/git_blame_3.png?raw=true + .. image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/contributing/the_basics/git_blame/git_blame_3.png?raw=true :width: 420 -Whenever starting a discussion or creating an issue, you are very welcome to tag members of the Ivy team using "@", selecting the person you think would most suitable to interact with, based on the information gained from the above steps. +Whenever starting a discussion or creating an issue, you are very welcome to tag members of the Ivy team using "@", selecting the person you think would be most suitable to interact with, based on the information gained from the above steps. Pull Requests ------------- @@ -212,24 +213,24 @@ This simple process makes it much simpler for us to track where and when our att Note that you cannot request a code review until you have already received at least one review from us. Therefore, all new PRs will receive a code review, so please just wait and we will check out and review your newly created PR as soon as possible! -Your PR will never be closed until we have provided at least code review on it. +Your PR will never be closed until we have provided at least a code review on it. After a new PR is made, for the tests to run, it needs an approval of someone from the ivy team for the workflows to start running. Once approved, you can see the failing and passing checks for a commit relevant to your PR by clicking on the ❌ or ✔️ or 🟤 (each for: one or more tests are failing, all tests are passing, the check has just started, respectively) icon next to the commit hash. - .. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/contributing/the_basics/pull_requests/PR_checks.png?raw=true + .. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/contributing/the_basics/pull_requests/PR_checks.png?raw=true :width: 420 Further, if you click on the details next to a check then you can see the logs for that particular test. - .. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/contributing/the_basics/pull_requests/pr_logs.png?raw=true + .. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/contributing/the_basics/pull_requests/pr_logs.png?raw=true :width: 420 Also, if you have pushed multiple commits to a PR in a relatively short time, you may want to cancel the checks for a previous commit to speedup the process, you can do that by going to the log page as described above and clicking on the `Cancel Workflow` button. Note that this option might be unavailable depending on the level of access that you have. - .. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/contributing/the_basics/pull_requests/cancel_workflow.png?raw=true + .. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/contributing/the_basics/pull_requests/cancel_workflow.png?raw=true :width: 420 Finally, all PRs must give write access to Ivy maintainers of the branch. @@ -267,11 +268,11 @@ This is for a few reasons: #. It means you avoid the mountain of conflicts to resolve when you do get around to merging. This is also why we advocate using individual pull-requests per issue in the ToDo list issues. -This keeps each of the commits on master very contained and incremental, which is the style we're going for. +This keeps each of the commits on main very contained and incremental, which is the style we're going for. Sometimes, you've already dived very deep into some substantial changes in your fork, and it might be that only some of the problems you were trying to fix are actually fixed by your local changes. -In this hypothetical situation, you should aim to get the working parts merged into master **as soon as possible**. +In this hypothetical situation, you should aim to get the working parts merged into main **as soon as possible**. Adding subsections of your local changes with :code:`git` is easy. You can add individual files using: @@ -338,16 +339,16 @@ With Docker #. With PyCharm (With or without docker): 1. PyCharm enables users to run pytest using the green button present near every function declaration inside the :code:`ivy_tests` folder. - - .. image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/master/img/externally_linked/contributing/the_basics/pytest_with_pycharm/pytest_button_pycharm.png?raw=true + + .. image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/contributing/the_basics/pytest_with_pycharm/pytest_button_pycharm.png?raw=true :width: 420 - - 2. Testing can be done for the entire project, individual submodules, individual files and individual tests. + + 2. Testing can be done for the entire project, individual submodules, individual files, and individual tests. This can be done by selecting the appropriate configuration from the top pane in PyCharm. - - .. image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/master/img/externally_linked/contributing/the_basics/pytest_with_pycharm/pytest_with_pycharm.png?raw=true + + .. image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/contributing/the_basics/pytest_with_pycharm/pytest_with_pycharm.png?raw=true :width: 420 - + #. Through the command line (With docker): 1. We need to replace the folder inside the container with the current local ivy directory to run tests on the current local code. @@ -355,29 +356,29 @@ With Docker .. code-block:: none docker exec rm -rf ivy - docker cp ivy :/ + docker cp ivy :/ 2. We need to then enter inside the docker container and change into the :code:`ivy` directory using the following command. .. code-block:: none - docker exec -it ivy_container bash + docker exec -it ivy_container bash cd ivy 3. Run the test using the pytest command. 1. Ivy Tests: - 1. For a single function: + 1. For a single function: .. code-block:: none - + pytest ivy_tests/test_ivy/test_functional/test_core/test_image.py::test_random_crop --no-header --no-summary -q - + 2. For a single file: .. code-block:: none - + pytest ivy_tests/test_ivy/test_functional/test_core/test_image.py --no-header --no-summary -q 3. For all tests: @@ -388,28 +389,28 @@ With Docker 2. Array API Tests: - 1. For a single function: + 1. For a single function: .. code-block:: none - + pytest ivy_tests/array_api_testing/test_array_api/array_api_tests/test_creation_functions.py::test_arange --no-header --no-summary -q - + 2. For a single file: .. code-block:: none - + pytest ivy_tests/array_api_testing/test_array_api/array_api_tests/test_creation_functions.py --no-header --no-summary -q - + 3. For all tests: .. code-block:: none pytest ivy_tests/array_api_testing/test_array_api/ --no-header --no-summary -q - + 3. For the entire project: .. code-block:: none - + pytest ivy_tests/ --no-header --no-summary -q #. Through the command line (Without docker): @@ -433,16 +434,16 @@ With Docker 1. Ivy Tests: - 1. For a single function: + 1. For a single function: .. code-block:: none - + python -m pytest ivy_tests/test_ivy/test_functional/test_core/test_image.py::test_random_crop --no-header --no-summary -q - + 2. For a single file: .. code-block:: none - + python -m pytest ivy_tests/test_ivy/test_functional/test_core/test_image.py --no-header --no-summary -q 3. For all tests: @@ -451,43 +452,43 @@ With Docker python -m pytest ivy_tests/test_ivy/ --no-header --no-summary -q - 2. Array API Tests + 2. Array API Tests - 1. For a single function: + 1. For a single function: .. code-block:: none - + python -m pytest ivy_tests/array_api_testing/test_array_api/array_api_tests/test_creation_functions.py::test_arange --no-header --no-summary -q - + 2. For a single file: .. code-block:: none - + python -m pytest ivy_tests/array_api_testing/test_array_api/array_api_tests/test_creation_functions.py --no-header --no-summary -q - + 3. For all tests: .. code-block:: none python -m pytest ivy_tests/array_api_testing/test_array_api/ --no-header --no-summary -q - + 3. For the entire project .. code-block:: none - + python -m pytest ivy_tests/ --no-header --no-summary -q #. Optional Flags: Various optional flags are available for running the tests such as :code:`device`, :code:`backend`, etc. - 1. :code:`device`: - 1. This flag enables setting of the device where the tests would be run. + 1. :code:`device`: + 1. This flag enables the setting of the device where the tests would be run. 2. Possible values being :code:`cpu` and :code:`gpu`. 3. Default value is :code:`cpu` 2. :code:`backend`: 1. This flag enables running the tests for particular backends. - 2. The values of this flag could be any possible combination of JAX, numpy, tensorflow and torch. + 2. The values of this flag could be any possible combination of JAX, numpy, tensorflow, and torch. 3. Default value is :code:`jax,numpy,tensorflow,torch`. 3. :code:`num-examples`: - 1. Set the maximum number for examples to be generated by Hypothesis. + 1. Set the maximum number of examples to be generated by Hypothesis. 2. The value of this flag could be any positive integer value that is greater than 1. 3. Default value is :code:`5`. @@ -496,31 +497,31 @@ Getting the most out of IDE with PyCharm ************ #. Find a text: - 1. :code:`Ctrl+F` will prompt you to type in the text to be found, if not already selected, and then find all the instances of text within current file. + 1. :code:`Ctrl+F` will prompt you to type in the text to be found, if not already selected, and then find all the instances of text within the current file. - .. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/contributing/the_basics/getting_most_out_of_IDE/find_file.png?raw=true + .. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/contributing/the_basics/getting_most_out_of_IDE/find_file.png?raw=true :align: center 2. :code:`Ctrl+Shift+F` will find all the instances of text within the project. - .. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/contributing/the_basics/getting_most_out_of_IDE/find_project_wide.png?raw=true + .. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/contributing/the_basics/getting_most_out_of_IDE/find_project_wide.png?raw=true :align: center #. Find+Replace a text: - 1. :code:`Ctrl+R` will prompt you to type in the text to be found and the text to be replaced, if not already selected, within current file. + 1. :code:`Ctrl+R` will prompt you to type in the text to be found and the text to be replaced, if not already selected, within the current file. - .. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/contributing/the_basics/getting_most_out_of_IDE/find_n_replace_file.png?raw=true + .. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/contributing/the_basics/getting_most_out_of_IDE/find_n_replace_file.png?raw=true :align: center 2. :code:`Ctrl+Shift+R` will prompt you to type in the text to be found and the text to be replaced, if not already selected, within the whole project. - .. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/contributing/the_basics/getting_most_out_of_IDE/find_and_replace_project_wide.png?raw=true + .. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/contributing/the_basics/getting_most_out_of_IDE/find_and_replace_project_wide.png?raw=true :align: center #. Find and multiply the cursor: - 1. :code:`Ctrl+Shift+Alt+J` will find all the instances of selected text and multiply the cursor to all these locations. + 1. :code:`Ctrl+Shift+Alt+J` will find all the instances of the selected text and multiply the cursor to all these locations. - .. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/contributing/the_basics/getting_most_out_of_IDE/multiple_cursor.png?raw=true + .. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/contributing/the_basics/getting_most_out_of_IDE/multiple_cursor.png?raw=true :align: center You can visit `Pycharm Blog`_ for more details on efficient coding! @@ -529,23 +530,23 @@ with PyCharm 1. add breakpoints: 1. Click the gutter at the executable line of code where you want to set the breakpoint or place the caret at the line and press :code:`Ctrl+F8` - .. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/contributing/the_basics/getting_most_out_of_IDE/adding_breakpoint.png?raw=true - :aligh: center + .. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/contributing/the_basics/getting_most_out_of_IDE/adding_breakpoint.png?raw=true + :align: center 2. Enter into the debug mode: 1. Click on Run icon and Select **Debug test** or press :code:`Shift+F9`. This will open up a Debug Window Toolbar: - .. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/contributing/the_basics/getting_most_out_of_IDE/open_in_debug_mode.png?raw=true + .. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/contributing/the_basics/getting_most_out_of_IDE/open_in_debug_mode.png?raw=true :align: center 3. Stepping through the code: - 1. Step over: + 1. Step over: Steps over the current line of code and takes you to the next line even if the highlighted line has method calls in it. 1. Click the Step Over button or press :code:`F8` - .. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/contributing/the_basics/getting_most_out_of_IDE/step_over.png?raw=true + .. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/contributing/the_basics/getting_most_out_of_IDE/step_over.png?raw=true :align: center 2. Step into: @@ -561,34 +562,35 @@ with PyCharm 1. Press :code:`Shift+F7`. This will prompt you to select the method you want to step into: - .. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/contributing/the_basics/getting_most_out_of_IDE/smart_step_into.png?raw=true + .. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/contributing/the_basics/getting_most_out_of_IDE/smart_step_into.png?raw=true :align: center 2. Click the desired method. - 4. Python Console: + 4. Python Console: 1. Click the Console option on Debug Tool Window: This currently stores variables and their values upto which the code has been executed. You can print outputs and debug the code further on. - 2. If you want to open console at certain breakpoint: + 2. If you want to open the console at a certain breakpoint: 1. Select the breakpoint-fragment of code, press :code:`Alt+shift+E` Start debugging! - .. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/contributing/the_basics/getting_most_out_of_IDE/console_coding.png?raw=true - :aligh: center + .. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/contributing/the_basics/getting_most_out_of_IDE/console_coding.png?raw=true + :align: center + 5. Using **try-except**: - 1. PyChram is great at pointing the lines of code which are causing tests to fail. + 1. PyCharm is great at pointing the lines of code which are causing tests to fail. Navigating to that line, you can add Try-Except block with breakpoints to get in depth understanding of the errors. - .. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/contributing/the_basics/getting_most_out_of_IDE/try_except.png?raw=true + .. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/contributing/the_basics/getting_most_out_of_IDE/try_except.png?raw=true :align: center 6. Dummy **test** file: 1. Create a separate dummy :code:`test.py` file wherein you can evaluate a particular test failure. Make sure you don't add or commit this dummy file while pushing your changes. - .. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/contributing/the_basics/getting_most_out_of_IDE/dummy_test.png?raw=true + .. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/contributing/the_basics/getting_most_out_of_IDE/dummy_test.png?raw=true :align: center PyCharm has a detailed blog on efficient `Debugging`_ which is quite useful. diff --git a/docs/overview/deep_dive/continuous_integration.rst b/docs/overview/deep_dive/continuous_integration.rst index bb6f7e55951d7..e639efa7bcfca 100644 --- a/docs/overview/deep_dive/continuous_integration.rst +++ b/docs/overview/deep_dive/continuous_integration.rst @@ -2,7 +2,6 @@ Continuous Integration ====================== .. _`continuous integration channel`: https://discord.com/channels/799879767196958751/1028268051776413759 -.. _`continuous integration forum`: https://discord.com/channels/799879767196958751/1028298018438520862 .. _`discord`: https://discord.gg/sXyFF8tDtm We follow the practice of Continuous Integration (CI), in order to regularly build and test code at Ivy. @@ -17,10 +16,10 @@ In order to incorporate Continuous Integration in the Ivy Repository, we follow #. Periodic Testing #. Manual Testing -.. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/deep_dive/continuous_integration/CI.png?raw=true +.. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/deep_dive/continuous_integration/CI.png?raw=true :alt: CI Overview -We use GitHub Actions in order to implement and automate the process of testing. GitHub Actions allow implementing custom workflows that can build the code in the repository and run the tests. All the workflows used by Ivy are defined in the `.github/workflows `_ directory. +We use GitHub Actions in order to implement and automate the process of testing. GitHub Actions allow implementing custom workflows that can build the code in the repository and run the tests. All the workflows used by Ivy are defined in the `.github/workflows `_ directory. Commit (Push/PR) Triggered Testing ---------------------------------- @@ -37,7 +36,7 @@ A test is defined as the triplet of (submodule, function, backend). We follow th For example, :code:`ivy_tests/test_ivy/test_frontends/test_torch/test_tensor.py::test_torch_instance_arctan_,numpy` -The Number of such Ivy tests running on the Repository (without taking any Framework/Python Versioning into account) is 12500 (as of writing this documentation), and we are adding tests daily. Therefore, triggering all the tests on each commit is neither desirable (as it will consume a huge lot of Compute Resources, as well take a large amount of time to run) nor feasible (as Each Job in Github Actions has a time Limit of 360 Minutes, and a Memory Limit as well). +The Number of such Ivy tests running on the Repository (without taking any Framework/Python Versioning into account) is 12500 (as of writing this documentation), and we are adding tests daily. Therefore, triggering all the tests on each commit is neither desirable (as it will consume a huge lot of Compute Resources, as well as take a large amount of time to run) nor feasible (as Each Job in Github Actions has a time Limit of 360 Minutes, and a Memory Limit as well). Further, When we consider versioning, for a single Python version, and ~40 frontend and backend versions, the tests would shoot up to 40 * 40 * 12500 = 20,000,000, and we obviously don't have resources as well as time to run those many tests on each commit. @@ -91,7 +90,7 @@ But, there’s a fundamental issue here, Computing the Mapping requires determin Now assume that we had some way to update the Mapping for a commit from the previous Mapping without having to run all the tests. Then, Given the Mapping for a single commit, we could follow this to determine and run the relevant tests for each commit as follows: -.. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/deep_dive/continuous_integration/ITRoadmap.png?raw=true +.. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/deep_dive/continuous_integration/ITRoadmap.png?raw=true :alt: Intelligent Testing Roadmap This is exactly what we do in order to implement Intelligent Testing. The “Update Mapping” Logic works as follows for each changed file: @@ -207,7 +206,7 @@ Now, that the SSH key of the Runner has permissions to push and clone the Mappin git clone --single-branch --depth 1 --branch "$TARGET_BRANCH" git@github.com:unifyai/Mapping.git -In case of, Pull Requests, we do not have access to :code:`SSH_DEPLOY_KEY` secret (and we don’t even want to give PRs that access), and thus we don’t use the SSH Clone Methodology and instead use the HTTP Clone Method, as follows: +In the case of, Pull Requests, we do not have access to :code:`SSH_DEPLOY_KEY` secret (and we don’t even want to give PRs that access), and thus we don’t use the SSH Clone Methodology and instead use the HTTP Clone Method, as follows: .. code-block:: @@ -274,7 +273,7 @@ The Determine Test Coverage Workflow and the Intelligent Tests Workflow can run Consider the following Case for Runner 2: #. The Determine Test Coverage workflow has been running, and is about to complete for Runner 2. Meanwhile, a commit made on the master triggers the intelligent-tests workflow. -#. The runner 2 in the intelligent-tests workflow, pulls the Mapping from the master2 branch of unifyai/Mapping repository, and starts running the determined tests (based on changes made in the commit). +#. The runner 2 in the intelligent-tests workflow, pulls the Mapping from the master2 branch of the unifyai/Mapping repository, and starts running the determined tests (based on changes made in the commit). #. The det-test-coverage workflow completes for runner2, which makes a push to the corresponding branch in the unifyai/Mapping Repository. #. The runner 2 in the intelligent-tests workflow also completes, and pushes the updated repository @@ -288,14 +287,14 @@ We handle the Race Condition as follows: Array API Tests --------------- -The `array-api-intelligent-tests.yml (Push) `_ and the `array-api-intelligent-tests-pr.yml (Pull Request) `_ workflows run the Array API Tests. Similar to Ivy Tests, The Array API tests are also determined intelligently and only relevant tests are triggered on each commit. +The `array-api-intelligent-tests.yml (Push) `_ and the `array-api-intelligent-tests-pr.yml (Pull Request) `_ workflows run the Array API Tests. Similar to Ivy Tests, The Array API tests are also determined intelligently and only relevant tests are triggered on each commit. -More details about the Array API Tests are available `here `_. +More details about the Array API Tests are available `here `_. Periodic Testing ---------------- In order to make sure that none of the Ivy Tests are left ignored for a long time, and to decouple the rate of testing to the rate of committing to the repository, we implement periodic testing on the Ivy Repository. -The `Test Ivy Cron Workflow `_ is responsible for implementing this behavior by running Ivy tests every hour. In Each Run, It triggers 150 Ivy Tests, cycling through all of the tests. +The `Test Ivy Cron Workflow `_ is responsible for implementing this behavior by running Ivy tests every hour. In Each Run, It triggers 150 Ivy Tests, cycling through all of the tests. This number of 150 is chosen in order to make sure that the Action completes in 1 hour most of the time. The Test Results update the corresponding cell on the Dashboards. @@ -331,18 +330,18 @@ Whenever a push is made to the repository, a variety of workflows are triggered This can be seen on the GitHub Repository Page, with the commit message followed by a yellow dot, indicating that some workflows have been queued to run following this commit, as shown below: -.. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/deep_dive/continuous_integration/push.png?raw=true +.. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/deep_dive/continuous_integration/push.png?raw=true :alt: Push Clicking on the yellow dot (🟡) (which changes to a tick (✔) or cross (❌), when the tests have been completed) yields a view of the test-suite results as shown below: -.. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/deep_dive/continuous_integration/push1.png?raw=true +.. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/deep_dive/continuous_integration/push1.png?raw=true :alt: Test-Suite Click on the "Details" link corresponding to the failing tests, in order to identify the cause of the failure. It redirects to the Actions Tab, showing details of the failure, as shown below: -.. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/deep_dive/continuous_integration/push2.png?raw=true +.. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/deep_dive/continuous_integration/push2.png?raw=true :alt: Workflow Result Click on the "Run Tests" section in order to see the logs of the failing tests for Array API Tests. For Ivy Tests, head to the "Combined Test Results" Section of the display-test-results Job, which shows the Test Logs for each of the tests in the following format: @@ -381,7 +380,7 @@ Pull Request In case of a pull request, the test suite is available on the Pull Request Page on Github, as shown below: -.. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/deep_dive/continuous_integration/pull-request1.png?raw=true +.. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/deep_dive/continuous_integration/pull-request1.png?raw=true :alt: PR Test-Suite Clicking on the "Details" link redirects to the Action Log. @@ -392,15 +391,14 @@ As an added feature, the Intelligent Tests for PR Workflow has a section on "New Dashboard --------- In order to view the status of the tests, at any point in time, we have implemented a dashboard application that shows the results of the latest Workflow that ran each test. -The Dashboards are available on the link: https://ivy-dynamical-dashboards.onrender.com +The Dashboards are available at the link: https://ivy-dynamical-dashboards.onrender.com You can filter tests by selecting choices from the various dropdowns. The link can also be saved for redirecting straight to the filtered tests in the future. The status badges are clickable, and will take you directly to the Action log of the latest workflow that ran the corresponding test. **Round Up** This should have hopefully given you a good feel for how Continuous Integration works in Ivy. -If you have any questions, please feel free to reach out on `discord`_ in the `continuous integration channel`_ -or in the `continuous integration forum`_! +If you have any questions, please feel free to reach out on `discord`_ in the `continuous integration channel`_! **Video** diff --git a/docs/overview/deep_dive/docstring_examples.rst b/docs/overview/deep_dive/docstring_examples.rst index 53508e925972d..b315d10c0e63a 100644 --- a/docs/overview/deep_dive/docstring_examples.rst +++ b/docs/overview/deep_dive/docstring_examples.rst @@ -4,13 +4,12 @@ Docstring Examples .. _`repo`: https://github.com/unifyai/ivy .. _`discord`: https://discord.gg/sXyFF8tDtm .. _`docstring examples channel`: https://discord.com/channels/799879767196958751/982738352103129098 -.. _`docstring examples forum`: https://discord.com/channels/799879767196958751/1028297703089774705 After writing the general docstrings, the final step is to add helpful examples to the docstrings. There are eight types of examples, which each need to be added: -**Functional** examples show the function being called like so :code:`ivy.func_name(...)`, and these should be added to docstring of the function in the Ivy API :func:`ivy.func_name`. +**Functional** examples show the function being called like so :code:`ivy.func_name(...)`, and these should be added to the docstring of the function in the Ivy API :func:`ivy.func_name`. **Array instance method** examples show the method being called like so :code:`x.func_name(...)` on an :class:`ivy.Array` instance, and these should be added to the docstring of the :class:`ivy.Array` instance method :meth:`ivy.Array.func_name`. @@ -35,7 +34,7 @@ These special methods in turn call the functions in the Ivy API mentioned above. **Functional Examples** -To recap, *functional* examples show the function being called like so :code:`ivy.func_name(...)`, and these should be added to docstring of the function in the Ivy API :func:`ivy.func_name`. +To recap, *functional* examples show the function being called like so :code:`ivy.func_name(...)`, and these should be added to the docstring of the function in the Ivy API :func:`ivy.func_name`. Firstly, we should include *functional* examples with :class:`ivy.Array` instances in the input. @@ -292,7 +291,7 @@ Point 12 is not relevant as :func:`ivy.tan` is not an *operator* function. **Container Operator Examples** -Points 13, 14 and 15 are not relevant as :func:`ivy.tan` is not an *operator* function. +Points 13, 14, and 15 are not relevant as :func:`ivy.tan` is not an *operator* function. **Container Reverse Operator Example** @@ -441,7 +440,7 @@ Point 12 is not relevant as :func:`ivy.roll` is not an *operator* function. **Container Operator Examples** -Points 13, 14 and 15 are not relevant as :func:`ivy.roll` is not an *operator* function. +Points 13, 14, and 15 are not relevant as :func:`ivy.roll` is not an *operator* function. **Container Reverse Operator Example** @@ -722,7 +721,7 @@ It basically checks if the output upon execution of the examples that are docume Therefore each time you make a commit, you must ensure that the :code:`test-docstrings / run-docstring-tests` are working correctly at least for the function you are making changes to. To check whether the docstring tests are passing you need to check the logs for :code:`test-docstrings / run-docstring-tests`: - .. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/deep_dive/docstring_examples/docstring_failing_test_logs.png?raw=true + .. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/deep_dive/docstring_examples/docstring_failing_test_logs.png?raw=true :width: 420 You will need to go through the logs and see if the list of functions for which the docstring tests are failing also has the function you are working with. @@ -732,10 +731,10 @@ If the docstring tests are failing the logs show a list of functions having iss :code:`output in docs: ........` as shown below: - .. image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/master/img/externally_linked/deep_dive/docstring_examples/docstring_log.png + .. image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/deep_dive/docstring_examples/docstring_log.png :width: 420 -It can be quite tedious to go through the output diffs and spot the exact error, so you can take help of online tools like `text compare `_ to spot the minutest of differences. +It can be quite tedious to go through the output diffs and spot the exact error, so you can take the help of online tools like `text compare `_ to spot the minutest of differences. Once you make the necessary changes and the function you are working on doesn't cause the docstring tests to fail, you should be good to go. However, one of the reviewers might ask you to make additional changes involving examples. @@ -748,7 +747,7 @@ Passing docstring tests is a necessary but not sufficient condition for the comp These three examples should give you a good understanding of what is required when adding docstring examples. -If you have any questions, please feel free to reach out on `discord`_ in the `docstring examples channel`_ or in the `docstring examples forum`_! +If you have any questions, please feel free to reach out on `discord`_ in the `docstring examples channel`_! **Video** diff --git a/docs/overview/deep_dive/gradients.rst b/docs/overview/deep_dive/gradients.rst index 72f2f644b9ae0..d33d17a7d2e7d 100644 --- a/docs/overview/deep_dive/gradients.rst +++ b/docs/overview/deep_dive/gradients.rst @@ -3,12 +3,11 @@ Gradients .. _`discord`: https://discord.gg/sXyFF8tDtm .. _`gradients channel`: https://discord.com/channels/799879767196958751/1000043921633722509 -.. _`gradients forum`: https://discord.com/channels/799879767196958751/1028299026501750826 Overview -------- -Gradients are a crucial aspect of all modern deep learning workflows. +Gradients are a crucial aspect of all modern deep learning workflows. Different frameworks provide different APIs for gradient computation and there were a few considerations to be made while building a unified gradients API in Ivy. There are a number of functions added in ivy to allow gradient computation, but we'll mainly focus on the most commonly used and the most general function :func:`ivy.execute_with_gradients`. This is because the other gradient functions such as :func:`ivy.value_and_grad` and :func:`ivy.grad` can be considered as providing a subset of the functionality that :func:`ivy.execute_with_gradients` provides. @@ -19,31 +18,31 @@ Example Usage of the Gradient API The :func:`ivy.execute_with_gradients` function signature ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Following is pseudo function signature for the :func:`ivy.execute_with_gradients` function, +Following is the pseudo function signature for the :func:`ivy.execute_with_gradients` function, .. code-block:: python - + def execute_with_gradients ( func : Callable, xs : Any arbitrary nest, xs_grad_idxs : Input indices, ret_grad_idxs : Output indices, - ) : + ) : return func_ret, grads The :code:`func` in the input can be any user-defined function that returns a single scalar or any arbitrary nest of scalars. By scalars, we are referring to zero-dimensional arrays. -So for example, following are some valid outputs by the :code:`func`, +So for example, the following are some valid outputs by the :code:`func`, .. code-block:: python - + ivy.array(12.) - + # OR ivy.Container( - a=ivy.array(12.), + a=ivy.array(12.), b=ivy.Container( c=ivy.array(15.), d=ivy.array(32.) @@ -75,8 +74,8 @@ An example using :func:`ivy.execute_with_gradients` xs = [x, y] ret, grads = ivy.execute_with_gradients( - func, - xs, + func, + xs, xs_grad_idxs=[[0]], ret_grad_idxs=[["a"]] ) @@ -86,7 +85,7 @@ Custom Gradient Functions ^^^^^^^^^^^^^^^^^^^^^^^^^ There are various scenarios where users may want to define custom gradient computation rules for their functions. -Some of these are numerical stability, smoothing and clipping of the computed gradients. +Some of these are numerical stability, smoothing, and clipping of the computed gradients. Ivy provides the :func:`ivy.bind_custom_gradient_function` function to allow users to bind custom gradient computation logic to their functions. Following is an example of usage of :func:`ivy.bind_custom_gradient_function`, @@ -127,8 +126,8 @@ Our policy on gradients * The gradient API is fully-functional in ivy. * There is no explicit variable class or any public-facing function for adding gradient support to an ivy.Array. * The gradient functions in ivy implicitly convert all arrays to support gradient computation before computing gradients and detach all arrays after computing gradients. -* We don't retain any previously tracked computations in arrays by frameworks like torch for e.g. -* This makes our gradient API disambiguous, flexible and easy to debug. +* We don't retain any previously tracked computations in arrays by frameworks like torch for e.g. +* This makes our gradient API disambiguous, flexible, and easy to debug. * Any framework-specific tracking of computations or variable classes should be handled in the corresponding frontends. Gradient APIs of frameworks @@ -150,16 +149,17 @@ Gradient APIs of frameworks General Structure of Backend-specific implementations ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Here's a high-level description of the steps followed backend-specific implementation of :func:`ivy.execute_with_gradients` -1. Get Duplicate Index Chains : indices of arrays that share the same :code:`id` -2. Convert integer arrays to floats : only for ease of use. it's *not* recommended to pass integer arrays to gradient functions -3. Get relevant inputs : based on the :code:`xs_grad_idxs`, we collect the relevant inputs for gradient computation -4. Enable gradient support : we implicitly make use of framework-specific APIs to enable gradients in arrays. Ivy doesn't need to have an explicit variable class as the gradient API is fully functional -5. Compute Results : we do the forward pass by passing the input as it is to the function -6. Get relevant outputs : based on the :code:`ret_grad_idxs`, we collect the relevant outputs for gradient computation -7. Compute gradients : we make use of the framework-specific APIs to compute the gradients for the relevant outputs with respect to the relevant inputs -8. Handle duplicates : we explicitly handle duplicate instances using the index chains captured above as different frameworks treat duplicates differently -9. Post process and detach : finally, all computed gradients are updated to deal with :code:`NaN` and :code:`inf` and the input arrays are detached (i.e. gradient propagation is stopped) +Here's a high-level description of the steps followed backend-specific implementation of :func:`ivy.execute_with_gradients`: + +#. Get Duplicate Index Chains : indices of arrays that share the same :code:`id` +#. Convert integer arrays to floats : only for ease of use. it's *not* recommended to pass integer arrays to gradient functions +#. Get relevant inputs : based on the :code:`xs_grad_idxs`, we collect the relevant inputs for gradient computation +#. Enable gradient support : we implicitly make use of framework-specific APIs to enable gradients in arrays. Ivy doesn't need to have an explicit variable class as the gradient API is fully functional +#. Compute Results : we do the forward pass by passing the input as it is to the function +#. Get relevant outputs : based on the :code:`ret_grad_idxs`, we collect the relevant outputs for gradient computation +#. Compute gradients : we make use of the framework-specific APIs to compute the gradients for the relevant outputs with respect to the relevant inputs +#. Handle duplicates : we explicitly handle duplicate instances using the index chains captured above as different frameworks treat duplicates differently +#. Post process and detach : finally, all computed gradients are updated to deal with :code:`NaN` and :code:`inf` and the input arrays are detached (i.e. gradient propagation is stopped) Framework-specific Considerations ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -172,7 +172,7 @@ Framework-specific Considerations This should have hopefully given you a good feel for how the gradient API is implemented in Ivy. -If you have any questions, please feel free to reach out on `discord`_ in the `gradients channel`_ or in the `gradients forum`_! +If you have any questions, please feel free to reach out on `discord`_ in the `gradients channel`_! **Video** @@ -181,4 +181,4 @@ If you have any questions, please feel free to reach out on `discord`_ in the `g + \ No newline at end of file diff --git a/docs/overview/deep_dive/inplace_updates.rst b/docs/overview/deep_dive/inplace_updates.rst index fd8ac855fc7e8..42df6520a9668 100644 --- a/docs/overview/deep_dive/inplace_updates.rst +++ b/docs/overview/deep_dive/inplace_updates.rst @@ -9,7 +9,6 @@ Inplace Updates .. _`jax.numpy.tan`: https://jax.readthedocs.io/en/latest/_autosummary/jax.numpy.tan.html?highlight=tan .. _`presence of this attribute`: https://github.com/unifyai/ivy/blob/8ded4a5fc13a278bcbf2d76d1fa58ab41f5797d0/ivy/func_wrapper.py#L341 .. _`by the backend function`: https://github.com/unifyai/ivy/blob/8ded4a5fc13a278bcbf2d76d1fa58ab41f5797d0/ivy/func_wrapper.py#L372 -.. _`by the wrapper`: https://github.com/unifyai/ivy/blob/8ded4a5fc13a278bcbf2d76d1fa58ab41f5797d0/ivy/func_wrapper.py#L377 .. _`handled by the wrapper`: https://github.com/unifyai/ivy/blob/8ded4a5fc13a278bcbf2d76d1fa58ab41f5797d0/ivy/func_wrapper.py#L373 .. _`_wrap_fn`: https://github.com/unifyai/ivy/blob/6497b8a3d6b0d8aac735a158cd03c8f98eb288c2/ivy/container/wrapping.py#L69 .. _`NON_WRAPPED_FUNCTIONS`: https://github.com/unifyai/ivy/blob/fdaea62380c9892e679eba37f26c14a7333013fe/ivy/func_wrapper.py#L9 @@ -17,21 +16,18 @@ Inplace Updates .. _`ivy.reshape`: https://github.com/unifyai/ivy/blob/633eb420c5006a0a17c238bfa794cf5b6add8598/ivy/functional/ivy/manipulation.py#L418 .. _`ivy.astype`: https://github.com/unifyai/ivy/blob/8482eb3fcadd0721f339a1a55c3f3b9f5c86d8ba/ivy/functional/ivy/data_type.py#L46 .. _`ivy.asarray`: https://github.com/unifyai/ivy/blob/8482eb3fcadd0721f339a1a55c3f3b9f5c86d8ba/ivy/functional/ivy/creation.py#L114 -.. _`wrapping`: -.. _`ivy.inplace_update`: https://github.com/unifyai/ivy/blob/3a21a6bef52b93989f2fa2fa90e3b0f08cc2eb1b/ivy/functional/ivy/general.py#L1137 .. _`repo`: https://github.com/unifyai/ivy .. _`discord`: https://discord.gg/sXyFF8tDtm .. _`inplace updates channel`: https://discord.com/channels/799879767196958751/1028681763947552778 -.. _`inplace updates forum`: https://discord.com/channels/799879767196958751/1028681672268464199 .. _`example`: https://github.com/unifyai/ivy/blob/0ef2888cbabeaa8f61ce8aaea4f1175071f7c396/ivy/functional/ivy/layers.py#L169-L176 Inplace updates enable users to overwrite the contents of existing arrays with new data. This enables much more control over the memory-efficiency of the program, preventing old unused arrays from being kept in memory for any longer than is strictly necessary. -The function `ivy.inplace_update`_ enables explicit inplace updates. +The function :func:`ivy.inplace_update` enables explicit inplace updates. :func:`ivy.inplace_update` is a *primary* function, and the backend-specific implementations for each backend are presented below. -We also explain the rational for why each implementation is the way it is, and the important differences. +We also explain the rationale for why each implementation is the way it is, and the important differences. This is one particular area of the Ivy code where, technically speaking, the function :func:`ivy.inplace_update` will result in subtly different behaviour for each backend, unless the :code:`ensure_in_backend` flag is set to :code:`True`. @@ -241,7 +237,7 @@ TensorFlow functions also never returns views so additional logic is added to fu PyTorch **does** natively support inplace updates, and so :code:`x_native` is updated inplace with :code:`val_native`. Following this, an inplace update is then also performed on the :class:`ivy.Array` instance, if provided in the input. -PyTorch also supports views for most manipulation and indexing operation as with NumPy but it lacks that functionality with a few functions such as :func:`flip`. +PyTorch also supports views for most manipulation and indexing operations as with NumPy but it lacks that functionality with a few functions such as :func:`flip`. Additional logic had to be added to support view functionality for those functions (described in a section below). The function :func:`ivy.inplace_update` is also *nestable*, meaning it can accept :class:`ivy.Container` instances in the input. @@ -257,14 +253,14 @@ This could for example be the input array itself, but can also be any other arra All Ivy functions which return a single array should support inplace updates via the :code:`out` argument. The type hint of the :code:`out` argument is :code:`Optional[ivy.Array]`. However, as discussed above, if the function is *nestable* then :class:`ivy.Container` instances are also supported. -:class:`ivy.Container` is omitted from the type hint in such cases, as explained in the :ref:`Function Arguments` section. +:class:`ivy.Container` is omitted from the type hint in such cases, as explained in the `Function Arguments `_ section. When the :code:`out` argument is unspecified, then the return is simply provided in a newly created :class:`ivy.Array` (or :class:`ivy.Container` if *nestable*). However, when :code:`out` is specified, then the return is provided as an inplace update of the :code:`out` argument provided. This can for example be the same as the input to the function, resulting in a simple inplace update of the input. In the case of :class:`ivy.Array` return types, the :code:`out` argument is predominantly handled in `handle_out_argument`_. -As explained in the :ref:`Function Wrapping` section, this wrapping is applied to every function with the :code:`@handle_out_argument` decorator dynamically during `backend setting`_. +As explained in the `Function Wrapping `_ section, this wrapping is applied to every function with the :code:`@handle_out_argument` decorator dynamically during `backend setting`_. **Primary Functions** @@ -307,14 +303,14 @@ The implementations of :func:`ivy.tan` for each backend are as follows. **PyTorch** (includes :code:`support_native_out` attribute): .. code-block:: python - + def tan(x: torch.Tensor, /, *, out: Optional[torch.Tensor] = None) -> torch.Tensor: x = _cast_for_unary_op(x) return torch.tan(x, out=out) tan.support_native_out = True -It's very important to ensure the :code:`support_native_out` attribute is not added to backend implementations that do not handle the :code:`out` argument, as the `presence of this attribute`_ dictates whether the argument should be handled `by the backend function`_ or `by the wrapper`_. +It's very important to ensure the :code:`support_native_out` attribute is not added to backend implementations that do not handle the :code:`out` argument, as the `presence of this attribute`_ dictates whether the argument should be handled `by the backend function`_ or `by the wrapper `_. This distinction only concerns how the inplace update is applied to the native array, which is operated upon directly by the backend. If :code:`out` is specified in an Ivy function, then an inplace update is always **also** performed on the :class:`ivy.Array` instance itself, which is how :code:`out` is provided to the function originally. @@ -352,10 +348,10 @@ Here we still have the :attr:`support_native_out` attribute since we want to tak However, in the :code:`else` statement, the last operation is :func:`torch.transpose` which does not support the :code:`out` argument, and so the native inplace update can't be performed by torch here. This is why we need to call :func:`ivy.inplace_update` explicitly here, to ensure the native inplace update is performed, as well as the :class:`ivy.Array` inplace update. -Another case where we need to use :func:`ivy.inplace_update`_ with a function that has :attr:`support_native_out` is for the example of the :code:`torch` backend implementation of the :func:`ivy.remainder` function +Another case where we need to use :func:`ivy.inplace_update` with a function that has :attr:`support_native_out` is for the example of the :code:`torch` backend implementation of the :func:`ivy.remainder` function .. code-block:: python - + def remainder( x1: Union[float, torch.Tensor], x2: Union[float, torch.Tensor], @@ -436,7 +432,7 @@ Technically, this could be handled using the `handle_out_argument`_ wrapping, bu **Mixed Functions** -As explained in the :ref:`Function Types` section, *mixed* functions can effectively behave as either compositional or primary functions, depending on the backend that is selected. We must add the :code:`handle_out_argument` to the :code:`add_wrappers`key of +As explained in the `Function Types `_ section, *mixed* functions can effectively behave as either compositional or primary functions, depending on the backend that is selected. We must add the :code:`handle_out_argument` to the :code:`add_wrappers`key of the :code:`mixed_backend_wrappers` attribute so that the decorator gets added to the primary implementation when the backend is set. Here's an `example`_ from the linear function. @@ -452,7 +448,7 @@ When :code:`copy` is not specified explicitly, then an inplace update is perform Setting :code:`copy=False` is equivalent to passing :code:`out=input_array`. If only one of :code:`copy` or :code:`out` is specified, then this specified argument is given priority. If both are specified, then priority is given to the more general :code:`out` argument. -As with the :code:`out` argument, the :code:`copy` argument is also handled `by the wrapper `_. +As with the :code:`out` argument, the :code:`copy` argument is also handled `by the wrapper `_. Views @@ -464,7 +460,7 @@ More information about these arrays can be found in `NumPy's documentation `_ page in deep dive. +For more details on wrapping, see the `Function Wrapping <../../deep_dive/function_wrapping.rst>`_ page in deep dive. Instance Methods diff --git a/docs/overview/faq.rst b/docs/overview/faq.rst index 2783fd5cd9d69..e74df1b21dff7 100644 --- a/docs/overview/faq.rst +++ b/docs/overview/faq.rst @@ -4,7 +4,6 @@ FAQ .. _`dex`: https://github.com/dexidp/dex .. _`API for distributed training`: https://github.com/unifyai/ivy/blob/a2f37b1bae232b7ba5257e59f8b46a0374cca9f1/ivy/functional/ivy/device.py#L660 .. _`fully support these`: https://pytorch.org/tutorials/prototype/vmap_recipe.html -.. _`Ivy Builder`: https://github.com/unifyai/builder .. _`README`: https://github.com/unifyai/ivy These are some of the most common technical questions that continue to arise when we're discussing Ivy with developers in the community. @@ -27,7 +26,7 @@ This is not entirely surprising, each framework has strong backward compatibilit Our CI always tests against the latest version available on PyPI, and this has been the case since we started development. We do not lock-in any versions during our continuous testing, and we will continue to always pull the latest version. -In future, we hope to add explicit testing also for previous versions, so we can guarantee backward compatibility for each backend. +In the future, we hope to add explicit testing also for previous versions, so we can guarantee backward compatibility for each backend. We will also add an option to select backend versions for the small minority of cases where changes in the backend functional APIs do cause breaking changes for Ivy. Dynamic Sizes @@ -36,7 +35,7 @@ Dynamic Sizes **Q:** Assuming a static computation graph, can tensors have sizes that dynamically change? XLA does not support dynamic sizes, because it JIT-compiles the graph, and pre-allocates all buffers in memory before the graph runs. TensorFlow and PyTorch do allow dynamic sizes, but only on certain backends. -Dynamic sizes require a dynamic memory memory manager, which CPUs/GPUs have, but XLA currently doesn't. +Dynamic sizes require a dynamic memory manager, which CPUs/GPUs have, but XLA currently doesn't. How does Ivy deal with all of this? **A:** Ivy assumes dynamic shapes are supported, but an error will be thrown if/when the function is compiled with dynamic shapes enabled, but the backend does not support dynamic shapes in the compiled graph. @@ -53,10 +52,10 @@ For some backends, shape-checking will be performed during the compilation phase GPU handling ------------ -**Q:** How does Ivy handle GPU usage? +**Q:** How does Ivy handle GPU usage? -**A:** Ivy handles GPU usage by simply wrapping the backend frameworks, and so Ivy will use GPUs in the same manner as the backend framework does. -E.g. When using a torch backend, then torch will be a dependency of Ivy, and its handling of GPU functionalities will be inherited and extended upon by Ivy. +**A:** Ivy handles GPU usage by simply wrapping the backend frameworks, so Ivy will use GPUs in the same manner as the backend framework does. +E.g. When using a torch backend, torch will be a dependency of Ivy, and its handling of GPU functionalities will be inherited and extended upon by Ivy. Model Deployment ---------------- @@ -72,7 +71,7 @@ Dynamic Control Flow Jax also has dynamic control-flow (:code:`lax.scan`, :code:`lax.while`), but support is limited; only :code:`lax.scan` is differentiable in reverse mode. Branching is also tricky, and is backend-dependent. -CPUs have branch predictors and can execute tight loops, GPUs don't, but have drivers that can schedule kernels dynamically, some other architectures do static scheduling, which limits the kinds of algorithms that can run effectively. +CPUs have branch predictors and can execute tight loops, GPUs don't, but have drivers that can schedule kernels dynamically, and some other architectures do static scheduling, which limits the kinds of algorithms that can run effectively. TensorFlow eager and PyTorch allow you to use full python control flow, (loops, branches, function calls, dynamic dispatch, recursion) but there is no static computation graph. How will Ivy handle dynamic control flow? @@ -156,7 +155,6 @@ The Pipeline **A:** We are not advocating to replace all code with Ivy. We would encourage users to continue using whatever data loaders they want to, and perhaps just use an Ivy model, or use Ivy to convert a model, or even just a single function from a library. -If users want to use Ivy more deeply, then they can use `Ivy Builder`_, which includes framework-agnostic abstract data loaders, trainers, and other higher level classes for composing full training pipelines. State ----- diff --git a/docs/overview/related_work/api_standards.rst b/docs/overview/related_work/api_standards.rst index 93289f69821bc..04d6939840436 100644 --- a/docs/overview/related_work/api_standards.rst +++ b/docs/overview/related_work/api_standards.rst @@ -5,8 +5,6 @@ API Standards .. _`Array API Standard`: https://data-apis.org/array-api/latest/ .. _`discord`: https://discord.gg/sXyFF8tDtm -.. _`related work channel`: https://discord.com/channels/799879767196958751/1034436036371157083 -.. _`related work forum`: https://discord.com/channels/799879767196958751/1034436085587120149 API standards are standardized application programming interfaces (APIs) which define the function signatures which similar libraries should adhere to for maximal interoperability between those libraries. @@ -15,31 +13,11 @@ Array API Standard The `Array API Standard`_ defines a unified application programming interface (API) for Python libraries which perform numerical operations on high dimensional arrays (tensors). This standard can be considered as “higher level” than the ML frameworks themselves, given that the standard defines the functions without implementing them, whereas the frameworks include implementations for all of the functions which fit into this standard API, with all the lower level considerations also handled within these implementations. -The Array API Standard takes a lowest common denominator approach, whereby each function in the standard represents the minimum behaviors of the function without restricting extensions to the function. +The Array API Standard takes the lowest common denominator approach, whereby each function in the standard represents the minimum behaviors of the function without restricting extensions to the function. This means that two very different libraries can adhere to the same standard, despite having very different extended behaviors for some of the functions in the standard. The standard is also not exhaustive. -For example, there are 64 functions defined in the standard, whereas the functions defined in each framework are as follows: +For example, there are functions defined in the standard, whereas the functions defined in each framework are as follows: -Table: -__________________________________________________________________________________ -| | | -| Framework | Functions | -|-------------|------------------------------------------------------------------| -| NumPy | all, any, argmax, argmin, around, clip, cumprod, cumsum, max, | -| | mean, min, prod, round, std, sum, var | -|-------------|------------------------------------------------------------------| -| Pandas | append, diff, fillna, head, isin, loc, mean, min, pct_change, | -| | prod, quantile, rolling, shift, tail, to_numpy | -|-------------|------------------------------------------------------------------| -| TensorFlow | cast, clip_by_value, equal, greater, greater_equal, less, | -| | less_equal, maximum, minimum, not_equal, ones_like, reshape, | -| | sum | -|-------------|------------------------------------------------------------------| -| PyTorch | abs, add, mean, max, min, pow, sum | -|_____________|__________________________________________________________________| + Therefore, two frameworks which adhere to the standard will still have major differences by virtue of the extra functions they support which are not present in the standard. - -**Round Up** - -If you have any questions, please feel free to reach out on `discord`_ in the `related work channel`_ or in the `related work forum`_! \ No newline at end of file diff --git a/docs/overview/related_work/compiler_infrastructure.rst b/docs/overview/related_work/compiler_infrastructure.rst index d62f948ac3727..b5a11bb113493 100644 --- a/docs/overview/related_work/compiler_infrastructure.rst +++ b/docs/overview/related_work/compiler_infrastructure.rst @@ -12,8 +12,6 @@ Compiler Infrastructure .. _`Intel`: https://www.intel.com/ .. _`OneDNN`: https://github.com/oneapi-src/oneDNN .. _`discord`: https://discord.gg/sXyFF8tDtm -.. _`related work channel`: https://discord.com/channels/799879767196958751/1034436036371157083 -.. _`related work forum`: https://discord.com/channels/799879767196958751/1034436085587120149 Compiler infrastructure generally provides carefully thought through frameworks and principles to simplify the lives of compiler designers, maximizing the reusability of tools and interoperability when deploying to various different hardware targets. This infrastructure doesn’t provide “full” solutions for compiling to hardware, but instead provides the general scaffolding to make the design of such compilers as principled and interoperable as possible, with maximal code sharing and interoperability being at the heart of their design. @@ -32,20 +30,16 @@ MLIR The `Multi Level Intermediate Representation (MLIR)`_ is an important piece of compiler infrastructure designed to represent multiple levels of abstraction, with abstractions and domain-specific IR constructs being easy to add, and with location being a first-class construct. It is part of the broader `LLVM`_ project. It aims to address software fragmentation, improve compilation for heterogeneous hardware, significantly reduce the cost of building domain specific compilers, and aid in connecting existing compilers together. -Comparing to other parts of the overall ML stack, MLIR is designed to operate at a lower level than the neural network exchange formats. +Compared to other parts of the overall ML stack, MLIR is designed to operate at a lower level than the neural network exchange formats. For example, the `Onnx-mlir`_ compiler uses the MLIR compiler infrastructure to implement a compiler which enables `ONNX`_ defined models to be compiled into native code. OneAPI ------ -`OneAPI`_ is an open standard for a unified Application Programming Interface (API) intended to be used across different compute accelerator (coprocessor) architectures, including GPUs, AI accelerators and field-programmable gate arrays, although at present the main user is `Intel`_, with them being the authors of the standard. +`OneAPI`_ is an open standard for a unified Application Programming Interface (API) intended to be used across different compute accelerator (coprocessor) architectures, including GPUs, AI accelerators, and field-programmable gate arrays, although at present the main user is `Intel`_, with them being the authors of the standard. The set of APIs spans several domains that benefit from acceleration, including libraries for linear algebra math, deep learning, machine learning, video processing, and others. -`OneDNN`_ is particularly relevant, focusing on neural networks functions for deep learning training and inference. +`OneDNN`_ is particularly relevant, focusing on neural network functions for deep learning training and inference. Intel CPUs and GPUs have accelerators for Deep Learning software, and OneDNN provides a unified interface to utilize these accelerators, with much of the hardware-specific complexity abstracted away. -In a similar manner to `MLIR`_, OneAPI is also designed to operate at a lower level than the Neural Network :ref:`Exchange Formats`. -The interface is lower level and more primitive than the neural network exchange formats, with focus on the core low-level operations such as convolutions, matrix multiplications, batch normalization etc. +In a similar manner to `MLIR`_, OneAPI is also designed to operate at a lower level than the Neural Network :ref:`overview/related_work/what_does_ivy_add:Exchange Formats`. +The interface is lower level and more primitive than the neural network exchange formats, with a focus on the core low-level operations such as convolutions, matrix multiplications, batch normalization etc. This makes OneDNN very much complementary to these formats, where OneDNN can sit below the exchange formats in the overall stack, enabling accelerators to be fully leveraged with minimal hardware-specific considerations, with this all helpfully being abstracted by the OneDNN API. -Indeed, OneAPI and MLIR can work together in tandem, and OneDNN are working to `integrate Tensor Possessing Primitives in the MLIR compilers used underneath TensorFlow `_. - -**Round Up** - -If you have any questions, please feel free to reach out on `discord`_ in the `related work channel`_ or in the `related work forum`_! \ No newline at end of file +Indeed, OneAPI and MLIR can work together in tandem, and OneDNN is working to `integrate Tensor Possessing Primitives in the MLIR compilers used underneath TensorFlow `_. diff --git a/docs/overview/related_work/frameworks.rst b/docs/overview/related_work/frameworks.rst index 8620690b38d7b..7862cd2ce2215 100644 --- a/docs/overview/related_work/frameworks.rst +++ b/docs/overview/related_work/frameworks.rst @@ -49,71 +49,69 @@ Frameworks .. _`Dex language`: https://github.com/google-research/dex-lang .. _`Haskell`: https://www.haskell.org/ .. _`discord`: https://discord.gg/sXyFF8tDtm -.. _`related work channel`: https://discord.com/channels/799879767196958751/1034436036371157083 -.. _`related work forum`: https://discord.com/channels/799879767196958751/1034436085587120149 -.. |matlab| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/master/img/externally_linked/related_work/frameworks/matlab.png +.. |matlab| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/related_work/frameworks/matlab.png :height: 20pt :class: dark-light -.. |scipy| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/master/img/externally_linked/related_work/frameworks/scipy.png +.. |scipy| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/related_work/frameworks/scipy.png :height: 20pt :class: dark-light -.. |torch| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/master/img/externally_linked/related_work/frameworks/torch.png +.. |torch| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/related_work/frameworks/torch.png :height: 20pt :class: dark-light -.. |numpy| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/master/img/externally_linked/related_work/frameworks/numpy.png +.. |numpy| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/related_work/frameworks/numpy.png :height: 20pt :class: dark-light -.. |scikit-learn| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/master/img/externally_linked/related_work/frameworks/scikit-learn.png +.. |scikit-learn| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/related_work/frameworks/scikit-learn.png :height: 15pt :class: dark-light -.. |theano| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/master/img/externally_linked/related_work/frameworks/theano.png +.. |theano| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/related_work/frameworks/theano.png :height: 10pt :class: dark-light -.. |pandas| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/master/img/externally_linked/related_work/frameworks/pandas.png +.. |pandas| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/related_work/frameworks/pandas.png :height: 22pt :class: dark-light -.. |julia| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/master/img/externally_linked/related_work/frameworks/julia.png +.. |julia| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/related_work/frameworks/julia.png :height: 20pt :class: dark-light -.. |apache-spark-mllib| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/master/img/externally_linked/related_work/frameworks/apache-spark-mllib.png +.. |apache-spark-mllib| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/related_work/frameworks/apache-spark-mllib.png :height: 20pt :class: dark-light -.. |caffe| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/master/img/externally_linked/related_work/frameworks/caffe.png +.. |caffe| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/related_work/frameworks/caffe.png :height: 10pt :class: dark-light -.. |chainer| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/master/img/externally_linked/related_work/frameworks/chainer.png +.. |chainer| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/related_work/frameworks/chainer.png :height: 20pt :class: dark-light -.. |tensorflow-1| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/master/img/externally_linked/related_work/frameworks/tensorflow-1.png +.. |tensorflow-1| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/related_work/frameworks/tensorflow-1.png :height: 20pt :class: dark-light -.. |mxnet| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/master/img/externally_linked/related_work/frameworks/mxnet.png +.. |mxnet| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/related_work/frameworks/mxnet.png :height: 20pt :class: dark-light -.. |cntk| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/master/img/externally_linked/related_work/frameworks/cntk.png +.. |cntk| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/related_work/frameworks/cntk.png :height: 20pt :class: dark-light -.. |pytorch| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/master/img/externally_linked/related_work/frameworks/pytorch.png +.. |pytorch| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/related_work/frameworks/pytorch.png :height: 22pt :class: dark-light -.. |flux| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/master/img/externally_linked/related_work/frameworks/flux.png +.. |flux| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/related_work/frameworks/flux.png :height: 22pt :class: dark-light -.. |jax| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/master/img/externally_linked/related_work/frameworks/jax.png +.. |jax| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/related_work/frameworks/jax.png :height: 20pt :class: dark-light -.. |tensorflow-2| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/master/img/externally_linked/related_work/frameworks/tensorflow-2.png +.. |tensorflow-2| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/related_work/frameworks/tensorflow-2.png :height: 20pt :class: dark-light -.. |dex-language| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/master/img/externally_linked/related_work/frameworks/dex-language.png +.. |dex-language| image:: https://raw.githubusercontent.com/unifyai/unifyai.github.io/main/img/externally_linked/related_work/frameworks/dex-language.png :height: 20pt :class: dark-light Here we list some of the most prominent frameworks for array computation. These are the individual frameworks which the wrapper frameworks mentioned above generally wrap around and abstract. -.. image:: https://github.com/unifyai/unifyai.github.io/blob/master/img/externally_linked/related_work/frameworks/ml_framework_timeline.png?raw=true +.. image:: https://github.com/unifyai/unifyai.github.io/blob/main/img/externally_linked/related_work/frameworks/ml_framework_timeline.png?raw=true :width: 100% MATLAB |matlab| @@ -127,7 +125,7 @@ As of 2020, MATLAB has more than 4 million users worldwide, who come from variou SciPy |scipy| ------------- First released in 2001, `SciPy`_ is a Python framework used for scientific computing and technical computing, with modules for optimization, linear algebra, integration, interpolation, special functions, FFT, signal and image processing, ODE solvers and other tasks common in science and engineering. -While the user interface is in Python, the backend involves Fortran, Cython, C++ and C for high runtime efficiency. +While the user interface is in Python, the backend involves Fortran, Cython, C++, and C for high runtime efficiency. It is built to work with `NumPy`_ arrays, and provides many user-friendly and efficient numerical routines, such as routines for numerical integration and optimization. Torch |torch| @@ -147,13 +145,13 @@ It has long been the go-to framework for numeric computing in Python. SciKit Learn |scikit-learn| --------------------------- -First released in 2007, `Scikit-learn`_ is a Python framework which features various classification, regression and clustering algorithms including support-vector machines, random forests, gradient boosting, k-means and DBSCAN, and is designed to interoperate with the Python numerical and scientific libraries `NumPy`_ and `SciPy`_. +First released in 2007, `Scikit-learn`_ is a Python framework which features various classification, regression, and clustering algorithms including support-vector machines, random forests, gradient boosting, k-means, and DBSCAN, and is designed to interoperate with the Python numerical and scientific libraries `NumPy`_ and `SciPy`_. Theano |theano| --------------- Initially released in 2007, `Theano`_ is a Python framework which focuses on manipulating and evaluating mathematical expressions, especially matrix-valued ones, with an inbuilt optimizing compiler. Computations are expressed using a `NumPy`_-esque syntax and are compiled to run efficiently on either CPU or GPU architectures. -Notably, it includes an extensible graph framework suitable for rapid development of custom operators and symbolic optimizations, and it implements an extensible graph transpilation framework. +Notably, it includes an extensible graph framework suitable for the rapid development of custom operators and symbolic optimizations, and it implements an extensible graph transpilation framework. It is now being continued under the name `Aesara`_. Pandas |pandas| @@ -211,7 +209,7 @@ CNTK |cntk| ----------- Originally released in 2016, the `Microsoft Cognitive Toolkit (CNTK)`_ is an open-source toolkit for commercial-grade distributed deep learning, written entirely in C++. It describes neural networks as a series of computational steps via a directed graph. -CNTK allows the user to easily realize and combine popular model types such as feed-forward DNNs, convolutional neural networks (CNNs) and recurrent neural networks (RNNs/LSTMs). +CNTK allows the user to easily realize and combine popular model types such as feed-forward DNNs, convolutional neural networks (CNNs), and recurrent neural networks (RNNs/LSTMs). CNTK implements stochastic gradient descent (SGD, error backpropagation) learning with automatic differentiation and parallelization across multiple GPUs and servers. It is no longer being actively developed, having succumbed to the increasing popularity of the frameworks using Python frontend interfaces. @@ -221,7 +219,7 @@ PyTorch |pytorch| PyTorch operates based on asynchronous scheduling on the target device, without any pre-compilation of the full computation graph on the target device required. This made it possible to combine asynchronous scheduled efficient kernels with pure Python control flow, and also made it easy to query and monitor the intermediate values in the model, with the boundaries of the “computation graph” having been broken down. This quickly made it very popular for researchers. -Generally PyTorch is the choice of the ML researcher, ML practitioner and the hobbyist. +Generally, PyTorch is the choice of the ML researcher, ML practitioner, and the hobbyist. PyTorch is very Pythonic, very simple to use, very forgiving, and has a tremendous ecosystem built around it. No other framework comes close to having anything like the `PyTorch Ecosystem`_, with a vast collection of third-party libraries in various important topics for ML research. @@ -271,7 +269,3 @@ They propose an explicit nested indexing style that mirrors application of funct The goal of the project is to explore: type systems for array programming, mathematical program transformations like differentiation and integration, user-directed compilation to parallel hardware, and interactive and incremental numerical programming and visualization. It is quite early and still in an experimental phase, but this framework would provide hugely significant fundamental improvements over all existing frameworks if it reaches a mature stage of development. The language is built on top of `Haskell`_. - -**Round Up** - -If you have any questions, please feel free to reach out on `discord`_ in the `related work channel`_ or in the `related work forum`_! \ No newline at end of file diff --git a/docs/overview/related_work/graph_tracers.rst b/docs/overview/related_work/graph_tracers.rst index e248abdb6941a..f8561bf70e0a6 100644 --- a/docs/overview/related_work/graph_tracers.rst +++ b/docs/overview/related_work/graph_tracers.rst @@ -8,8 +8,6 @@ Graph Tracers .. _`PyTorch`: https://pytorch.org/ .. _`FX`: https://pytorch.org/docs/stable/fx.html .. _`discord`: https://discord.gg/sXyFF8tDtm -.. _`related work channel`: https://discord.com/channels/799879767196958751/1034436036371157083 -.. _`related work forum`: https://discord.com/channels/799879767196958751/1034436085587120149 Graph tracers enable acyclic directed computation graphs to be extracted from functions which operate on the tensors, expressed as source code in the framework. There is inevitably some overlap with the role of the lower level compilers here, but for the purpose of this discussion, we consider tracers as being any tool which executes the function to be traced and produces a computation graph consisting solely of the lowest level functions defined within the framework itself, without going any lower. @@ -43,7 +41,7 @@ torch.fx -------- `FX`_ is a toolkit for developers to use to transform :code:`torch.nn.Module` instances in `PyTorch`_. FX consists of three main components: a symbolic tracer, an intermediate representation, and Python code generation. -The symbolic tracer performs “symbolic execution” of the Python code. +The symbolic tracer performs a “symbolic execution” of the Python code. It feeds fake values, called Proxies, through the code. Operations on these Proxies are recorded. The intermediate representation is the container for the operations that were recorded during symbolic tracing. @@ -57,7 +55,3 @@ Taken together, this pipeline of components (symbolic tracing -> intermediate re In addition, these components can be used separately. For example, symbolic tracing can be used in isolation to capture a form of the code for analysis (and not transformation) purposes. Code generation can be used for programmatically generating models, for example from a config file. - -**Round Up** - -If you have any questions, please feel free to reach out on `discord`_ in the `related work channel`_ or in the `related work forum`_! diff --git a/docs/overview/related_work/vendor_specific_compilers.rst b/docs/overview/related_work/vendor_specific_compilers.rst index 78dc38aeb4ec7..7f41c47e4ab28 100644 --- a/docs/overview/related_work/vendor_specific_compilers.rst +++ b/docs/overview/related_work/vendor_specific_compilers.rst @@ -15,8 +15,6 @@ Vendor-Specific Compilers .. _`GCC`: https://gcc.gnu.org/ .. _`Microsoft Visual C++ Compiler`: https://docs.microsoft.com/en-us/cpp/ .. _`discord`: https://discord.gg/sXyFF8tDtm -.. _`related work channel`: https://discord.com/channels/799879767196958751/1034436036371157083 -.. _`related work forum`: https://discord.com/channels/799879767196958751/1034436085587120149 Below the vendor-specific APIs are the vendor specific compilers. As far as modern machine learning practitioners go, these compilers are very rarely interacted with directly. @@ -46,7 +44,3 @@ CUDA code runs on both the CPU and GPU. NVCC separates these two parts and sends host code (the part of code which will be run on the CPU) to a C compiler like `GCC`_ or `Intel C++ Compiler Classic (ICC)`_ or `Microsoft Visual C++ Compiler`_, and sends the device code (the part which will run on the GPU) to the GPU. The device code is further compiled by NVCC. Like `ICX`_, NVCC is also based on `LLVM`_. - -**Round Up** - -If you have any questions, please feel free to reach out on `discord`_ in the `related work channel`_ or in the `related work forum`_! \ No newline at end of file diff --git a/ivy/functional/ivy/experimental/linear_algebra.py b/ivy/functional/ivy/experimental/linear_algebra.py index 60eeafbf1e6b1..d4a511c30353a 100644 --- a/ivy/functional/ivy/experimental/linear_algebra.py +++ b/ivy/functional/ivy/experimental/linear_algebra.py @@ -1,5 +1,6 @@ # global -from typing import Union, Optional, Tuple, List, Sequence +import logging +from typing import Union, Optional, Tuple, List, Sequence, Literal # local import ivy @@ -11,6 +12,8 @@ handle_array_like_without_promotion, handle_array_function, handle_device_shifting, + inputs_to_ivy_arrays, + handle_backend_invalid, ) from ivy.utils.exceptions import handle_exceptions @@ -90,7 +93,7 @@ def eigh_tridiagonal( >>> beta = ivy.array([0., 1.]) >>> y = ivy.eigh_tridiagonal(alpha, beta) >>> print(y) - ivy.array([0., 0.38196, 2.61803]) + ivy.array([0., 0.38196602, 2.61803389]) >>> alpha = ivy.array([0., 1., 2.]) >>> beta = ivy.array([0., 1.]) @@ -98,22 +101,21 @@ def eigh_tridiagonal( ... beta, select='v', ... select_range=[0.2,3.0]) >>> print(y) - ivy.array([0.38196, 2.61803]) + ivy.array([0.38196602, 2.61803389]) - >>> ivy.set_backend("tensorflow") >>> alpha = ivy.array([0., 1., 2., 3.]) >>> beta = ivy.array([2., 1., 2.]) >>> y = ivy.eigh_tridiagonal(alpha, ... beta, ... eigvals_only=False, ... select='i', - ... select_range=[1,2] + ... select_range=[1,2], ... tol=1.) >>> print(y) - (ivy.array([0.18749806, 2.81250191]), ivy.array([[ 0.350609 , -0.56713122], - [ 0.06563006, -0.74146169], - [-0.74215561, -0.0636413 ], - [ 0.56742489, 0.35291126]])) + (ivy.array([0.38196602, 2.61803389]), ivy.array([[ 0.35048741, -0.56710052], + [ 0.06693714, -0.74234426], + [-0.74234426, -0.06693714], + [ 0.56710052, 0.35048741]])) With :class:`ivy.Container` input: @@ -122,7 +124,7 @@ def eigh_tridiagonal( >>> y = ivy.eigh_tridiagonal(alpha, beta) >>> print(y) { - a: ivy.array([-0.56155, 0., 3.56155]), + a: ivy.array([-0.56155282, 0., 3.56155276]), b: ivy.array([0., 2., 4.]) } @@ -131,8 +133,8 @@ def eigh_tridiagonal( >>> y = ivy.eigh_tridiagonal(alpha, beta) >>> print(y) { - a: ivy.array([-0.56155, 0., 3.56155]), - b: ivy.array([-0.82842, 2., 4.82842]) + a: ivy.array([-0.56155282, 0., 3.56155276]), + b: ivy.array([-0.82842714, 2., 4.82842731]) } """ x = ivy.diag(alpha) @@ -159,7 +161,9 @@ def eigh_tridiagonal( return eigenvalues return eigenvalues, eigenvectors + @handle_exceptions +@handle_backend_invalid @handle_nestable @handle_array_like_without_promotion @handle_out_argument @@ -168,6 +172,7 @@ def eigh_tridiagonal( def diagflat( x: Union[ivy.Array, ivy.NativeArray], /, + *, offset: int = 0, padding_value: float = 0, align: str = "RIGHT_LEFT", @@ -180,30 +185,24 @@ def diagflat( Parameters ---------- - x : Union[ivy.Array, ivy.NativeArray] - Input data, which is flattened and set as the diagonal of the output. - offset : int, optional - Diagonal offset. Positive value means superdiagonal, 0 refers to the main diagonal, - and negative value means subdiagonal. Default is 0. - padding_value : float, optional - Value to fill the off-diagonal elements with. Default is 0. - align : str, optional - Alignment of the diagonal within the output array. Default is "RIGHT_LEFT". - num_rows : int, optional - Number of rows in the output array. If not specified (-1), it is inferred from the input data. - num_cols : int, optional - Number of columns in the output array. If not specified (-1), it is inferred from the input data. - out : Optional[Union[ivy.Array, ivy.NativeArray]], optional - Optional output array, for writing the result to. It must have a shape that the inputs broadcast to. + x + Input data, which is flattened and set as the k-th diagonal of the output. + k + Diagonal to set. + Positive value means superdiagonal, + 0 refers to the main diagonal, + and negative value means subdiagonal. + out + optional output array, for writing the result to. It must have a shape that the + inputs broadcast to. Returns ------- - ret : ivy.Array + ret The 2-D output array. - Functional Examples - ------------------ - + Examples + -------- With :class:`ivy.Array` inputs: >>> x = ivy.array([[1,2], [3,4]]) @@ -214,7 +213,7 @@ def diagflat( [0, 0, 0, 4]]) >>> x = ivy.array([1,2]) - >>> ivy.diagflat(x, offset=1) + >>> ivy.diagflat(x, k=1) ivy.array([[0, 1, 0], [0, 0, 2], [0, 0, 0]]) @@ -231,6 +230,7 @@ def diagflat( @handle_exceptions +@handle_backend_invalid @handle_nestable @handle_array_like_without_promotion @handle_out_argument @@ -273,6 +273,7 @@ def kron( @handle_exceptions +@handle_backend_invalid @handle_nestable @handle_array_like_without_promotion @handle_out_argument @@ -316,6 +317,7 @@ def matrix_exp( @handle_exceptions +@handle_backend_invalid @handle_nestable @handle_array_like_without_promotion @to_native_arrays_and_back @@ -347,8 +349,8 @@ def eig( but this function is *nestable*, and therefore also accepts :class:`ivy.Container` instances in place of any of the arguments. - Functional Examples - ------------------ + Examples + -------- With :class:`ivy.Array` inputs: >>> x = ivy.array([[1,2], [3,4]]) >>> w, v = ivy.eig(x) @@ -379,6 +381,7 @@ def eig( @handle_exceptions +@handle_backend_invalid @handle_nestable @handle_array_like_without_promotion @to_native_arrays_and_back @@ -400,8 +403,8 @@ def eigvals( w Not necessarily ordered array(..., N) of eigenvalues in complex type. - Functional Examples - ------------------ + Examples + -------- With :class:`ivy.Array` inputs: >>> x = ivy.array([[1,2], [3,4]]) >>> w = ivy.eigvals(x) @@ -422,6 +425,7 @@ def eigvals( @handle_exceptions +@handle_backend_invalid @handle_nestable @handle_array_like_without_promotion @handle_out_argument @@ -461,10 +465,11 @@ def adjoint( return current_backend(x).adjoint(x, out=out) +@handle_exceptions +@handle_backend_invalid @handle_nestable @handle_out_argument @to_native_arrays_and_back -@handle_exceptions def multi_dot( x: Sequence[Union[ivy.Array, ivy.NativeArray]], /, @@ -519,6 +524,7 @@ def multi_dot( @handle_exceptions +@handle_backend_invalid @handle_nestable @handle_array_like_without_promotion @handle_out_argument @@ -551,14 +557,1199 @@ def cond( Examples -------- - >>> x = ivy.array([[1., 2.], - [3., 4.]]) - >>> ivy.cond(x) - ivy.array(14.933034) - - >>> x = ivy.array([[1., 2.], - [3., 4.]]) - >>> ivy.cond(x, p=ivy.inf) + >>> x = ivy.array([[1., 2.], + ... [3., 4.]]) + >>> ivy.cond(x) + ivy.array(14.933034) + + >>> x = ivy.array([[1., 2.], + ... [3., 4.]]) + >>> ivy.cond(x, p=ivy.inf) ivy.array(21.0) """ return current_backend(x).cond(x, p=p, out=out) + + +# This code has been adapted from TensorLy +# https://github.com/tensorly/tensorly/blob/main/tensorly/tenalg/core_tenalg/_kronecker.py +@handle_exceptions +@handle_backend_invalid +@handle_nestable +@handle_array_like_without_promotion +@handle_out_argument +@to_native_arrays_and_back +@handle_device_shifting +def kronecker( + x: Sequence[Union[ivy.Array, ivy.NativeArray]], + skip_matrix: Optional[int] = None, + reverse: Optional[bool] = False, + out: Optional[ivy.Array] = None, +) -> ivy.Array: + """ + Kronecker product of a list of matrices. + + Parameters + ---------- + x + Sequence of matrices + + skip_matrix + if not None, index of a matrix to skip + + reverse + if True, the order of the matrices is reversed + + Returns + ------- + kronecker_product: matrix of shape ``(prod(n_rows), prod(n_columns)`` + where ``prod(n_rows) = prod([m.shape[0] for m in matrices])`` + and ``prod(n_columns) = prod([m.shape[1] for m in matrices])`` + """ + if skip_matrix is not None: + x = [x[i] for i in range(len(x)) if i != skip_matrix] + + if reverse: + order = -1 + else: + order = 1 + + for i, matrix in enumerate(x[::order]): + if not i: + res = matrix + else: + res = ivy.kron(res, matrix, out=out) + return res + + +# The code has been adapated from tensorly.khatri_rao +# https://github.com/tensorly/tensorly/blob/main/tensorly/tenalg/core_tenalg/_khatri_rao.py#L9 +@handle_nestable +@handle_exceptions +@handle_array_like_without_promotion +@inputs_to_ivy_arrays +@handle_array_function +@handle_device_shifting +def khatri_rao( + x: Sequence[Union[ivy.Array, ivy.NativeArray]], + weights: Optional[Union[ivy.Array, ivy.NativeArray]] = None, + skip_matrix: Optional[Sequence[int]] = None, + mask: Optional[Union[ivy.Array, ivy.NativeArray]] = None, + out: Optional[ivy.Array] = None, +) -> ivy.Array: + """ + Khatri-Rao product of a sequence of matrices. + + This can be seen as a column-wise kronecker product. + If one matrix only is given, that matrix is directly returned. + + Parameters + ---------- + x + Sequence of tensors with the same number of columns, i.e.:: + for i in len(x): + x[i].shape = (n_i, m) + + weights + array of weights for each rank, of length m, the number of column of the factors + (i.e. m == factor[i].shape[1] for any factor) + + skip_matrix + if not None, index of a matrix to skip + + mask + array of 1s and 0s of length m + + out + optional output array, for writing the result to. It must have a shape that the + result can broadcast to. + + Returns + ------- + khatri_rao_product: ivy.Array of shape ``(prod(n_i), m)`` + where ``prod(n_i) = prod([m.shape[0] for m in input])`` + i.e. the product of the number of rows of all the input in the product. + """ + if skip_matrix is not None: + x = [x[i] for i in range(len(x)) if i != skip_matrix] + + # Khatri-rao of only one matrix: just return that matrix + if len(x) == 1: + if ivy.exists(out): + return ivy.inplace_update(out, x[0]) + return x[0] + + if len(x[0].shape) == 2: + n_columns = x[0].shape[1] + else: + n_columns = 1 + x = [ivy.reshape(m, (-1, 1)) for m in x] + logging.warning( + "Khatri-rao of a series of vectors instead of input. " + "Considering each as a matrix with 1 column." + ) + + # Testing whether the input have the proper size + for i, matrix in enumerate(x): + if len(matrix.shape) != 2: + raise ValueError( + "All the input must have exactly 2 dimensions!" + f"Matrix {i} has dimension {len(matrix.shape)} != 2." + ) + if matrix.shape[1] != n_columns: + raise ValueError( + "All input must have same number of columns!" + f"Matrix {i} has {matrix.shape[1]} columns != {n_columns}." + ) + + for i, e in enumerate(x[1:]): + if not i: + if weights is None: + res = x[0] + else: + res = x[0] * ivy.reshape(weights, (1, -1)) + s1, s2 = ivy.shape(res) + s3, s4 = ivy.shape(e) + + a = ivy.reshape(res, (s1, 1, s2)) + b = ivy.reshape(e, (1, s3, s4)) + res = ivy.reshape(a * b, (-1, n_columns)) + + m = ivy.reshape(mask, (1, -1)) if mask is not None else 1 + + res = res * m + + if ivy.exists(out): + return ivy.inplace_update(out, res) + + return res + + +# The following code has been adapted from TensorLy +# https://github.com/tensorly/tensorly/blob/main/tensorly/tenalg/core_tenalg/n_mode_product.py#L5 +@handle_nestable +@handle_exceptions +@handle_array_like_without_promotion +@inputs_to_ivy_arrays +@handle_array_function +@handle_device_shifting +def mode_dot( + x: Union[ivy.Array, ivy.NativeArray], + /, + matrix_or_vector: Union[ivy.Array, ivy.NativeArray], + mode: int, + transpose: Optional[bool] = False, + *, + out: Optional[ivy.Array] = None, +) -> ivy.Array: + """ + N-mode product of a tensor and a matrix or vector at the specified mode. + + Parameters + ---------- + x + tensor of shape ``(i_1, ..., i_k, ..., i_N)`` + matrix_or_vector + 1D or 2D array of shape ``(J, i_k)`` or ``(i_k, )`` + matrix or vectors to which to n-mode multiply the tensor + mode + int in the range(1, N) + transpose + If True, the matrix is transposed. + For complex tensors, the conjugate transpose is used. + out + optional output array, for writing the result to. It must have a shape that the + result can broadcast to. + + Returns + ------- + ivy.Array + `mode`-mode product of `tensor` by `matrix_or_vector` + * of shape :math:`(i_1, ..., i_{k-1}, J, i_{k+1}, ..., i_N)` + if matrix_or_vector is a matrix + * of shape :math:`(i_1, ..., i_{k-1}, i_{k+1}, ..., i_N)` + if matrix_or_vector is a vector + """ + # the mode along which to fold might decrease if we take product with a vector + fold_mode = mode + new_shape = list(x.shape) + ndims = len(matrix_or_vector.shape) + + if ndims == 2: # Tensor times matrix + # Test for the validity of the operation + dim = 0 if transpose else 1 + if matrix_or_vector.shape[dim] != x.shape[mode]: + raise ValueError( + f"shapes {x.shape} and {matrix_or_vector.shape} not aligned in" + f" mode-{mode} multiplication: {x.shape[mode]} (mode {mode}) !=" + f" {matrix_or_vector.shape[dim]} (dim 1 of matrix)" + ) + + if transpose: + matrix_or_vector = ivy.conj(ivy.permute_dims(matrix_or_vector, (1, 0))) + + new_shape[mode] = matrix_or_vector.shape[0] + vec = False + + elif ndims == 1: # Tensor times vector + if matrix_or_vector.shape[0] != x.shape[mode]: + raise ValueError( + f"shapes {x.shape} and {matrix_or_vector.shape} not aligned for" + f" mode-{mode} multiplication: {x.shape[mode]} (mode {mode}) !=" + f" {matrix_or_vector.shape[0]} (vector size)" + ) + if len(new_shape) > 1: + new_shape.pop(mode) + else: + new_shape = [] + vec = True + + else: + raise ValueError( + "Can only take n_mode_product with a vector or a matrix." + f"Provided array of dimension {ndims} not in [1, 2]." + ) + + res = ivy.matmul(matrix_or_vector, ivy.unfold(x, mode)) + + if vec: # We contracted with a vector, leading to a vector + return ivy.reshape(res, new_shape, out=out) + else: # tensor times vec: refold the unfolding + return ivy.fold(res, fold_mode, new_shape, out=out) + + +# The following code has been adapated from TensorLy +# https://github.com/tensorly/tensorly/blob/main/tensorly/tenalg/core_tenalg/n_mode_product.py#L81 +@handle_nestable +@handle_exceptions +@handle_array_like_without_promotion +@inputs_to_ivy_arrays +@handle_array_function +@handle_device_shifting +def multi_mode_dot( + x: Union[ivy.Array, ivy.NativeArray], + mat_or_vec_list: Sequence[Union[ivy.Array, ivy.NativeArray]], + /, + modes: Optional[Sequence[int]] = None, + skip: Optional[Sequence[int]] = None, + transpose: Optional[bool] = False, + *, + out: Optional[ivy.Array] = None, +) -> ivy.Array: + r""" + N-mode product of a tensor and several matrices or vectors over several modes. + + Parameters + ---------- + x + the input tensor + + mat_or_vec_list + sequence of matrices or vectors of length ``tensor.ndim`` + + skip + None or int, optional, default is None + If not None, index of a matrix to skip. + + modes + None or int list, optional, default is None + + transpose + If True, the matrices or vectors in in the list are transposed. + For complex tensors, the conjugate transpose is used. + out + optional output array, for writing the result to. It must have a shape that the + result can broadcast to. + + Returns + ------- + ivy.Array + tensor times each matrix or vector in the list at mode `mode` + + Notes + ----- + If no modes are specified, just assumes there is one matrix or vector per mode and returns: + :math:`\\text{x }\\times_0 \\text{ matrix or vec list[0] }\\times_1 \\cdots \\times_n \\text{ matrix or vec list[n] }` # noqa + """ + if modes is None: + modes = range(len(mat_or_vec_list)) + + decrement = 0 # If we multiply by a vector, we diminish the dimension of the tensor + + res = x + + # Order of mode dots doesn't matter for different modes + # Sorting by mode shouldn't change order for equal modes + factors_modes = sorted(zip(mat_or_vec_list, modes), key=lambda x: x[1]) + for i, (mat_or_vec_list, mode) in enumerate(factors_modes): + ndims = len(mat_or_vec_list.shape) + if (skip is not None) and (i == skip): + continue + + if transpose and ndims == 2: + res = mode_dot( + res, + ivy.conj(ivy.permute_dims(mat_or_vec_list, (1, 0))), + mode - decrement, + ) + else: + res = mode_dot(res, mat_or_vec_list, mode - decrement) + + if ndims == 1: + decrement += 1 + + if ivy.exists(out): + return ivy.inplace_update(out, res) + + return res + + +def _svd_checks(x, n_eigenvecs=None): + """ + Run common checks to all of the SVD methods. + + Parameters + ---------- + matrix : 2D-array + n_eigenvecs : int, optional, default is None + if specified, number of eigen[vectors-values] to return + + Returns + ------- + n_eigenvecs : int + the number of eigenvectors to solve for + min_dim : int + the minimum dimension of matrix + max_dim : int + the maximum dimension of matrix + """ + # ndims = len(x.shape) + # if ndims != 2: + # raise ValueError(f"matrix be a matrix. matrix.ndim is {ndims} != 2") + + dim_1, dim_2 = ivy.shape(x)[-2:] + min_dim, max_dim = min(dim_1, dim_2), max(dim_1, dim_2) + + if n_eigenvecs is None: + n_eigenvecs = max_dim + + if n_eigenvecs > max_dim: + logging.warning( + f"Trying to compute SVD with n_eigenvecs={n_eigenvecs}, which is larger " + f"than max(matrix.shape)={max_dim}. Setting n_eigenvecs to {max_dim}." + ) + n_eigenvecs = max_dim + + return n_eigenvecs, min_dim, max_dim + + +# This function has been adapated from TensorLy +# https://github.com/tensorly/tensorly/blob/main/tensorly/tenalg/svd.py#L12 +@handle_nestable +@handle_exceptions +@handle_array_like_without_promotion +@inputs_to_ivy_arrays +@handle_array_function +@handle_device_shifting +def svd_flip( + U: Union[ivy.Array, ivy.NativeArray], + V: Union[ivy.Array, ivy.NativeArray], + /, + u_based_decision: Optional[bool] = True, +) -> Tuple[ivy.Array, ivy.Array]: + """ + Sign correction to ensure deterministic output from SVD. Adjusts the columns of u + and the rows of v such that the loadings in the columns in u that are largest in + absolute value are always positive. This function is borrowed from scikit- + learn/utils/extmath.py. + + Parameters + ---------- + U + left singular matrix output of SVD + V + right singular matrix output of SVD + u_based_decision + If True, use the columns of u as the basis for sign flipping. + Otherwise, use the rows of v. The choice of which variable to base the + decision on is generally algorithm dependent. + + Returns + ------- + u_adjusted, v_adjusted : arrays with the same dimensions as the input. + """ + if u_based_decision: + # columns of U, rows of V + max_abs_cols = ivy.argmax(ivy.abs(U), axis=0) + signs = ivy.sign( + ivy.array( + [U[i, j] for (i, j) in zip(max_abs_cols, range(ivy.shape(U)[1]))], + ) + ) + U = U * signs + if ivy.shape(V)[0] > ivy.shape(U)[1]: + signs = ivy.concat((signs, ivy.ones(ivy.shape(V)[0] - ivy.shape(U)[1]))) + V = V * signs[: ivy.shape(V)[0]][:, None] + else: + # rows of V, columns of U + max_abs_rows = ivy.argmax(ivy.abs(V), axis=1) + signs = ivy.sign( + ivy.array( + [V[i, j] for (i, j) in zip(range(ivy.shape(V)[0]), max_abs_rows)], + ) + ) + V = V * signs[:, None] + if ivy.shape(U)[1] > ivy.shape(V)[0]: + signs = ivy.concat( + ( + signs, + ivy.ones( + ivy.shape(U)[1] - ivy.shape(V)[0], + ), + ) + ) + U = U * signs[: ivy.shape(U)[1]] + + return U, V + + +# This function has been adapted from TensorLy +# https://github.com/tensorly/tensorly/blob/main/tensorly/tenalg/svd.py#L65 +@handle_nestable +@handle_exceptions +@handle_array_like_without_promotion +@inputs_to_ivy_arrays +@handle_array_function +@handle_device_shifting +def make_svd_non_negative( + x: Union[ivy.Array, ivy.NativeArray], + U: Union[ivy.Array, ivy.NativeArray], + S: Union[ivy.Array, ivy.NativeArray], + V: Union[ivy.Array, ivy.NativeArray], + /, + *, + nntype: Optional[Literal["nndsvd", "nndsvda"]] = "nndsvd", +) -> Tuple[ivy.Array, ivy.Array]: + """ + Use NNDSVD method to transform SVD results into a non-negative form. This method + leads to more efficient solving with NNMF [1]. + + Parameters + ---------- + x + tensor being decomposed. + U + left singular matrix from SVD. + S + diagonal matrix from SVD. + V + right singular matrix from SVD. + nntype + whether to fill small values with 0.0 (nndsvd), + or the tensor mean (nndsvda, default). + + [1]: Boutsidis & Gallopoulos. Pattern Recognition, 41(4): 1350-1362, 2008. + """ + W = ivy.zeros_like(U) + H = ivy.zeros_like(V) + + # The leading singular triplet is non-negative + # so it can be used as is for initialization. + W[:, 0] = ivy.sqrt(S[0]) * ivy.abs(U[:, 0]) + H[0, :] = ivy.sqrt(S[0]) * ivy.abs(V[0, :]) + + for j in range(1, len(S)): + a, b = U[:, j], V[j, :] + + # extract positive and negative parts of column vectors + a_p, b_p = ivy.where(a < 0.0, 0, a), ivy.where(b < 0.0, 0.0, b) + # a_p, b_p = ivy.clip(a, 0.0), ivy.clip(b, 0.0) + # a_n, b_n = ivy.abs(ivy.clip(a, 0.0)), ivy.abs(ivy.clip(b, 0.0)) + a_n, b_n = ivy.abs(ivy.where(a > 0.0, 0.0, a)), ivy.abs( + ivy.where(b > 0.0, 0.0, b) + ) + + # and their norms + a_p_nrm, b_p_nrm = float(ivy.vector_norm(a_p)), float(ivy.vector_norm(b_p)) + a_n_nrm, b_n_nrm = float(ivy.vector_norm(a_n)), float(ivy.vector_norm(b_n)) + + m_p, m_n = a_p_nrm * b_p_nrm, a_n_nrm * b_n_nrm + + # choose update + if m_p > m_n: + u = a_p / a_p_nrm + v = b_p / b_p_nrm + sigma = m_p + else: + u = a_n / a_n_nrm + v = b_n / b_n_nrm + sigma = m_n + + lbd = float(ivy.sqrt(S[j] * sigma)) + W[:, j] = lbd * u + H[j, :] = lbd * v + + # After this point we no longer need H + eps = ivy.finfo(x.dtype).min + + if nntype == "nndsvd": + W = ivy.soft_thresholding(W, eps) + H = ivy.soft_thresholding(H, eps) + elif nntype == "nndsvda": + avg = ivy.mean(x) + W = ivy.where(W < eps, ivy.ones(ivy.shape(W)) * avg, W) + H = ivy.where(H < eps, ivy.ones(ivy.shape(H)) * avg, H) + else: + raise ValueError( + f'Invalid nntype parameter: got {nntype} instead of one of ("nndsvd",' + ' "nndsvda")' + ) + + return W, H + + +# The following function has been adapted from TensorLy +# https://github.com/tensorly/tensorly/blob/main/tensorly/tenalg/svd.py#L206 +@handle_nestable +@handle_exceptions +@handle_array_like_without_promotion +@inputs_to_ivy_arrays +@handle_array_function +@handle_device_shifting +def truncated_svd( + x: Union[ivy.Array, ivy.NativeArray], + /, + compute_uv: bool = True, + n_eigenvecs: Optional[int] = None, +) -> Union[ivy.Array, Tuple[ivy.Array, ivy.Array, ivy.Array]]: + """ + Compute a truncated SVD on `x` using the standard SVD. + + Parameters + ---------- + x + 2D-array + compute_uv + If ``True`` then left and right singular vectors will be computed and returned + in ``U`` and ``Vh``, respectively. Otherwise, only the singular values will be + computed, which can be significantly faster. + n_eigenvecs + if specified, number of eigen[vectors-values] to return + else full matrices will be returned + + Returns + ------- + ret + a namedtuple ``(U, S, Vh)`` + Each returned array must have the same floating-point data type as ``x``. + """ + n_eigenvecs, min_dim, _ = _svd_checks(x, n_eigenvecs=n_eigenvecs) + full_matrices = True if n_eigenvecs > min_dim else False + + if compute_uv: + U, S, Vh = ivy.svd(x, full_matrices=full_matrices, compute_uv=True) + return U[:, :n_eigenvecs], S[:n_eigenvecs], Vh[:n_eigenvecs, :] + else: + S = ivy.svd(x, full_matrices=full_matrices, compute_uv=False) + return S[:n_eigenvecs] + + +# TODO uncommment the code below when these svd +# methods have been added +def _svd_interface( + matrix, + method="truncated_svd", + n_eigenvecs=None, + flip_sign=True, + u_based_flip_sign=True, + non_negative=None, + mask=None, + n_iter_mask_imputation=5, + **kwargs, +): + if method == "truncated_svd": + svd_fun = truncated_svd + # elif method == "symeig_svd": + # svd_fun = symeig_svd + # elif method == "randomized_svd": + # svd_fun = randomized_svd + elif callable(method): + svd_fun = method + else: + raise ValueError("Invalid Choice") + + U, S, V = svd_fun(matrix, n_eigenvecs=n_eigenvecs, **kwargs) + if mask is not None and n_eigenvecs is not None: + for _ in range(n_iter_mask_imputation): + S = S * ivy.eye(U.shape[-1], V.shape[-2]) + matrix = matrix * mask + (U @ S @ V) * (1 - mask) + U, S, V = svd_fun(matrix, n_eigenvecs=n_eigenvecs, **kwargs) + + if flip_sign: + U, V = svd_flip(U, V, u_based_decision=u_based_flip_sign) + + if non_negative is not False and non_negative is not None: + U, V = make_svd_non_negative(matrix, U, S, V) + + return U, S, V + + +# This function has been adapted from TensorLy +# https://github.com/tensorly/tensorly/blob/main/tensorly/decomposition/_tucker.py#L22 + + +# TODO update svd type hints when other svd methods have been added +# also update the test +@handle_nestable +@handle_exceptions +@handle_array_like_without_promotion +@inputs_to_ivy_arrays +@handle_array_function +@handle_device_shifting +def initialize_tucker( + x: Union[ivy.Array, ivy.NativeArray], + rank: Sequence[int], + modes: Sequence[int], + /, + *, + init: Optional[Union[Literal["svd", "random"], ivy.TuckerTensor]] = "svd", + seed: Optional[int] = None, + svd: Optional[Literal["truncated_svd"]] = "truncated_svd", + non_negative: Optional[bool] = False, + mask: Optional[Union[ivy.Array, ivy.NativeArray]] = None, + svd_mask_repeats: Optional[int] = 5, +) -> Tuple[ivy.Array, Sequence[ivy.Array]]: + """ + Initialize core and factors used in `tucker`. The type of initialization is set + using `init`. If `init == 'random'` then initialize factor matrices using + `random_state`. If `init == 'svd'` then initialize the `m`th factor matrix using the + `rank` left singular vectors of the `m`th unfolding of the input tensor. + + Parameters + ---------- + x + input tensor + rank + number of components + modes + modes to consider in the input tensor + seed + Used to create a random seed distribution + when init == 'random' + init + initialization scheme for tucker decomposition. + svd + function to use to compute the SVD + non_negative + if True, non-negative factors are returned + mask + array of booleans with the same shape as ``tensor`` should be 0 where + the values are missing and 1 everywhere else. Note: if tensor is + sparse, then mask should also be sparse with a fill value of 1 (or + True). + svd_mask_repeats + number of iterations for imputing the values in the SVD matrix when + mask is not None + + Returns + ------- + core + initialized core tensor + factors + list of factors + """ + try: + assert len(x.shape) >= 2 + except ValueError: + raise ValueError( + "expected x to have atleast 2 dimensions but it has only" + f" {len(x.shape)} dimension(s)" + ) + + # Initialisation + if init == "svd": + factors = [] + for index, mode in enumerate(modes): + mask_unfold = None if mask is None else ivy.unfold(mask, mode) + U, _, _ = _svd_interface( + ivy.unfold(x, mode), + n_eigenvecs=rank[index], + method=svd, + non_negative=non_negative, + mask=mask_unfold, + n_iter_mask_imputation=svd_mask_repeats, + # random_state=random_state, + ) + factors.append(U) + + # The initial core approximation is needed here for the masking step + core = multi_mode_dot(x, factors, modes=modes, transpose=True) + + elif init == "random": + core = ( + ivy.random_uniform( + shape=[rank[index] for index in range(len(modes))], + dtype=x.dtype, + seed=seed, + ) + + 0.01 + ) + factors = [ + ivy.random_uniform( + shape=(x.shape[mode], rank[index]), dtype=x.dtype, seed=seed + ) + for index, mode in enumerate(modes) + ] + + else: + (core, factors) = init + + if non_negative is True: + factors = [ivy.abs(f) for f in factors] + core = ivy.abs(core) + + return (core, factors) + + +# This function has been adpated from TensorLy +# https://github.com/tensorly/tensorly/blob/main/tensorly/decomposition/_tucker.py#L98 +@handle_nestable +@handle_exceptions +@handle_array_like_without_promotion +@inputs_to_ivy_arrays +@handle_array_function +@handle_device_shifting +def partial_tucker( + x: Union[ivy.Array, ivy.NativeArray], + rank: Optional[Sequence[int]] = None, + modes: Optional[Sequence[int]] = None, + /, + *, + n_iter_max: Optional[int] = 100, + init: Optional[Union[Literal["svd", "random"], ivy.TuckerTensor]] = "svd", + svd: Optional[Literal["truncated_svd"]] = "truncated_svd", + seed: Optional[int] = None, + mask: Optional[Union[ivy.Array, ivy.NativeArray]] = None, + svd_mask_repeats: Optional[int] = 5, + tol: Optional[float] = 10e-5, + verbose: Optional[bool] = False, + return_errors: Optional[bool] = False, +) -> Tuple[ivy.Array, Sequence[ivy.Array]]: + """ + Partial tucker decomposition via Higher Order Orthogonal Iteration (HOI) + + Decomposes `tensor` into a Tucker decomposition + exclusively along the provided modes. + + Parameters + ---------- + x + the input tensor + rank + size of the core tensor, ``(len(ranks) == tensor.ndim)`` + if int, the same rank is used for all modes + if None, original tensors size will be preserved. + modes + list of the modes on which to perform the decomposition + n_iter_max + maximum number of iteration + init + {'svd', 'random'}, or TuckerTensor optional + if a TuckerTensor is provided, this is used for initialization + svd + str, default is 'truncated_svd' + function to use to compute the SVD, + seed + Used to create a random seed distribution + when init == 'random' + mask + array of booleans with the same shape as ``tensor`` should be 0 where + the values are missing and 1 everywhere else. Note: if tensor is + sparse, then mask should also be sparse with a fill value of 1 (or + True). + svd_mask_repeats + number of iterations for imputing the values in the SVD matrix when + mask is not None + tol + tolerance: the algorithm stops when the variation in + the reconstruction error is less than the tolerance. + verbose + if True, different in reconstruction errors are returned at each + iteration. + return_erros + if True, list of reconstruction errors are returned. + + Returns + ------- + core : ndarray + core tensor of the Tucker decomposition + factors : ndarray list + list of factors of the Tucker decomposition. + with ``core.shape[i] == (tensor.shape[i], ranks[i]) for i in modes`` + """ + if modes is None: + modes = list(range(len(x.shape))) + + if rank is None: + logging.warning( + "No value given for 'rank'. The decomposition will preserve the original" + " size." + ) + rank = [ivy.shape(x)[mode] for mode in modes] + elif isinstance(rank, int): + logging.warning( + f"Given only one int for 'rank' instead of a list of {len(modes)} modes." + " Using this rank for all modes." + ) + rank = tuple(rank for _ in modes) + else: + rank = ivy.TuckerTensor.validate_tucker_rank(x.shape, rank=rank) + + # SVD init + core, factors = initialize_tucker( + x, + rank, + modes, + init=init, + svd=svd, + seed=seed, + mask=mask, + svd_mask_repeats=svd_mask_repeats, + ) + + rec_errors = [] + norm_tensor = ivy.sqrt(ivy.sum(x**2)) + + for iteration in range(n_iter_max): + if mask is not None: + x = x * mask + multi_mode_dot( + core, factors, modes=modes, transpose=False + ) * (1 - mask) + + for index, mode in enumerate(modes): + core_approximation = multi_mode_dot( + x, factors, modes=modes, skip=index, transpose=True + ) + eigenvecs, _, _ = _svd_interface( + ivy.unfold(core_approximation, mode), + n_eigenvecs=rank[index], + # random_state=random_state, + ) + factors[index] = eigenvecs + + core = multi_mode_dot(x, factors, modes=modes, transpose=True) + + # The factors are orthonormal and + # therefore do not affect the reconstructed tensor's norm + norm_core = ivy.sqrt(ivy.sum(core**2)) + rec_error = ivy.sqrt(abs(norm_tensor**2 - norm_core**2)) / norm_tensor + rec_errors.append(rec_error) + + if iteration > 1: + if verbose: + print( + f"reconstruction error={rec_errors[-1]}," + f" variation={rec_errors[-2] - rec_errors[-1]}." + ) + + if tol and abs(rec_errors[-2] - rec_errors[-1]) < tol: + if verbose: + print(f"converged in {iteration} iterations.") + break + + if return_errors: + return (core, factors), rec_errors + return (core, factors) + + +@handle_nestable +@handle_exceptions +@handle_array_like_without_promotion +@inputs_to_ivy_arrays +@handle_array_function +@handle_device_shifting +def tucker( + x: Union[ivy.Array, ivy.NativeArray], + rank: Optional[Sequence[int]] = None, + /, + *, + fixed_factors: Optional[Sequence[int]] = None, + n_iter_max: Optional[int] = 100, + init: Optional[Union[Literal["svd", "random"], ivy.TuckerTensor]] = "svd", + svd: Optional[Literal["truncated_svd"]] = "truncated_svd", + seed: Optional[int] = None, + mask: Optional[Union[ivy.Array, ivy.NativeArray]] = None, + svd_mask_repeats: Optional[int] = 5, + tol: Optional[float] = 10e-5, + verbose: Optional[bool] = False, + return_errors: Optional[bool] = False, +): + """ + Tucker decomposition via Higher Order Orthogonal Iteration (HOI) + + Decomposes `tensor` into a Tucker decomposition: + ``tensor = [| core; factors[0], ...factors[-1] |]`` [1]_ + + Parameters + ---------- + x + input tensor + rank + size of the core tensor, ``(len(ranks) == tensor.ndim)`` + if int, the same rank is used for all modes + fixed_factors + if not None, list of modes for which to keep the factors fixed. + Only valid if a Tucker tensor is provided as init. + n_iter_max + maximum number of iteration + init + {'svd', 'random'}, or TuckerTensor optional + if a TuckerTensor is provided, this is used for initialization + svd + str, default is 'truncated_svd' + function to use to compute the SVD, + seed + Used to create a random seed distribution + when init == 'random' + mask + array of booleans with the same shape as ``tensor`` should be 0 where + the values are missing and 1 everywhere else. Note: if tensor is + sparse, then mask should also be sparse with a fill value of 1 (or + True). + svd_mask_repeats + number of iterations for imputing the values in the SVD matrix when + mask is not None + tol + tolerance: the algorithm stops when the variation in + the reconstruction error is less than the tolerance + verbose + if True, different in reconstruction errors are returned at each + iteration. + + return_errors + Indicates whether the algorithm should return all reconstruction errors + and computation time of each iteration or not + Default: False + + Returns + ------- + ivy.TuckerTensor or ivy.TuckerTensor and + list of reconstruction errors if return_erros is True. + + References + ---------- + .. [1] tl.G.Kolda and B.W.Bader, "Tensor Decompositions and Applications", + SIAM REVIEW, vol. 51, n. 3, pp. 455-500, 2009. + """ + if fixed_factors: + try: + (core, factors) = init + except ValueError: + raise ValueError( + f"Got fixed_factor={fixed_factors} but no appropriate Tucker tensor was" + ' passed for "init".' + ) + if len(fixed_factors) == len(factors): + return ivy.TuckerTensor((core, factors)) + + fixed_factors = sorted(fixed_factors) + modes_fixed, factors_fixed = zip( + *[(i, f) for (i, f) in enumerate(factors) if i in fixed_factors] + ) + core = multi_mode_dot(core, factors_fixed, modes=modes_fixed) + modes, factors = zip( + *[(i, f) for (i, f) in enumerate(factors) if i not in fixed_factors] + ) + init = (core, list(factors)) + + rank = ivy.TuckerTensor.validate_tucker_rank(x.shape, rank=rank) + (core, new_factors), rec_errors = partial_tucker( + x, + rank, + modes, + n_iter_max=n_iter_max, + init=init, + svd=svd, + tol=tol, + seed=seed, + mask=mask, + verbose=verbose, + svd_mask_repeats=svd_mask_repeats, + return_errors=True, + ) + + factors = list(new_factors) + for i, e in enumerate(fixed_factors): + factors.insert(e, factors_fixed[i]) + core = multi_mode_dot(core, factors_fixed, modes=modes_fixed, transpose=True) + + if return_errors: + return ivy.TuckerTensor((core, factors)), rec_errors + return ivy.TuckerTensor((core, factors)) + + else: + modes = list(range(len(x.shape))) + rank = ivy.TuckerTensor.validate_tucker_rank(x.shape, rank=rank) + + (core, factors), rec_errors = partial_tucker( + x, + rank, + modes, + n_iter_max=n_iter_max, + init=init, + svd=svd, + tol=tol, + seed=seed, + mask=mask, + verbose=verbose, + return_errors=True, + ) + if return_errors: + return ivy.TuckerTensor((core, factors)), rec_errors + else: + return ivy.TuckerTensor((core, factors)) + + +@handle_exceptions +@handle_backend_invalid +@handle_nestable +@handle_out_argument +@to_native_arrays_and_back +def dot( + a: Union[ivy.Array, ivy.NativeArray], + b: Union[ivy.Array, ivy.NativeArray], + /, + *, + out: Optional[ivy.Array] = None, +) -> ivy.Array: + """ + Compute the dot product between two arrays `a` and `b` using the current backend's + implementation. The dot product is defined as the sum of the element-wise product of + the input arrays. + + Parameters + ---------- + a + First input array. + b + Second input array. + out + Optional output array. If provided, the output array to store the result. + + Returns + ------- + ret + The dot product of the input arrays. + + Examples + -------- + With :class:`ivy.Array` inputs: + + >>> a = ivy.array([1, 2, 3]) + >>> b = ivy.array([4, 5, 6]) + >>> result = ivy.dot(a, b) + >>> print(result) + ivy.array(32) + + >>> a = ivy.array([[1, 2], [3, 4]]) + >>> b = ivy.array([[5, 6], [7, 8]]) + >>> c = ivy.empty_like(a) + >>> ivy.dot(a, b, out=c) + >>> print(c) + ivy.array([[19, 22], + [43, 50]]) + + >>> a = ivy.array([[1.1, 2.3, -3.6]]) + >>> b = ivy.array([[-4.8], [5.2], [6.1]]) + >>> c = ivy.zeros((1, 1)) + >>> ivy.dot(a, b, out=c) + >>> print(c) + ivy.array([[-15.28]]) + """ + return current_backend(a, b).dot(a, b, out=out) + + +@handle_exceptions +@handle_nestable +@handle_array_like_without_promotion +@inputs_to_ivy_arrays +@handle_array_function +@handle_device_shifting +def general_inner_product( + a: Union[ivy.Array, ivy.NativeArray], + b: Union[ivy.Array, ivy.NativeArray], + n_modes: Optional[int] = None, + /, + *, + out: Optional[ivy.Array] = None, +) -> ivy.Array: + """ + Generalised inner products between tensors. + + Takes the inner product between the last (respectively first) + `n_modes` of `a` (respectively `b`) + + Parameters + ---------- + a + first input tensor. + b + second input tensor. + n_modes + int, default is None. If None, the traditional inner product is returned + (i.e. a float) otherwise, the product between the `n_modes` last modes of + `a` and the `n_modes` first modes of `b` is returned. The resulting tensor's + order is `len(a) - n_modes`. + out + Optional output array. If provided, the output array to store the result. + + Returns + ------- + The inner product of the input arrays. + + Examples + -------- + With :class:`ivy.Array` inputs: + + >>> a = ivy.array([1, 2, 3]) + >>> b = ivy.array([4, 5, 6]) + >>> result = ivy.general_inner_product(a, b, n_modes=1) + >>> print(result) + ivy.array(32) + + >>> a = ivy.array([1, 2]) + >>> b = ivy.array([4, 5]) + >>> result = ivy.general_inner_product(a, b) + >>> print(result) + ivy.array(14) + + >>> a = ivy.array([[1, 1], [1, 1]]) + >>> b = ivy.array([[1, 2, 3, 4],[1, 1, 1, 1]]) + >>> result = ivy.general_inner_product(a, b, n_modes=1) + >>> print(result) + ivy.array([[2, 3, 4, 5], + [2, 3, 4, 5]]) + """ + shape_a = a.shape + shape_b = b.shape + if n_modes is None: + if shape_a != shape_b: + raise ValueError( + "Taking a generalised product between two tensors without specifying" + " common modes is equivalent to taking inner product.This requires" + f" a.shape == b.shape.However, got shapes {a.shape} and {b.shape}" + ) + return ivy.sum(ivy.multiply(a, b), out=out) + + common_modes = shape_a[len(shape_a) - n_modes :] + if common_modes != shape_b[:n_modes]: + raise ValueError( + f"Incorrect shapes for inner product along {n_modes} common modes." + f"Shapes {shape_a.shape} and {shape_b.shape}" + ) + + common_size = int(ivy.prod(common_modes)) if len(common_modes) != 0 else 0 + output_shape = shape_a[:-n_modes] + shape_b[n_modes:] + inner_product = ivy.dot( + ivy.reshape(a, (-1, common_size)), ivy.reshape(b, (common_size, -1)) + ) + return ivy.reshape(inner_product, output_shape, out=out) diff --git a/ivy_tests/array_api_testing/test_array_api.diff b/ivy_tests/array_api_testing/test_array_api.diff deleted file mode 100644 index fcb72101ee80e..0000000000000 --- a/ivy_tests/array_api_testing/test_array_api.diff +++ /dev/null @@ -1,7 +0,0 @@ -diff --git a/ivy_tests/array_api_testing/test_array_api b/ivy_tests/array_api_testing/test_array_api -deleted file mode 160000 -index 852a1ddfe..000000000 ---- a/ivy_tests/array_api_testing/test_array_api -+++ /dev/null -@@ -1 +0,0 @@ --Subproject commit 852a1ddfedb14742ec94a9dfc6b3f3d4e89d0189 \ No newline at end of file