Skip to content

Latest commit

 

History

History
483 lines (349 loc) · 20.3 KB

HACKING.rst.txt

File metadata and controls

483 lines (349 loc) · 20.3 KB
Raw

Contributing to SciPy

This document aims to give an overview of how to contribute to SciPy. It tries to answer commonly asked questions, and provide some insight into how the community process works in practice. Readers who are familiar with the SciPy community and are experienced Python coders may want to jump straight to the git workflow documentation.

Contributing new code

If you have been working with the scientific Python toolstack for a while, you probably have some code lying around of which you think "this could be useful for others too". Perhaps it's a good idea then to contribute it to SciPy or another open source project. The first question to ask is then, where does this code belong? That question is hard to answer here, so we start with a more specific one: what code is suitable for putting into SciPy? Almost all of the new code added to scipy has in common that it's potentially useful in multiple scientific domains and it fits in the scope of existing scipy submodules. In principle new submodules can be added too, but this is far less common. For code that is specific to a single application, there may be an existing project that can use the code. Some scikits (scikit-learn, scikit-image, statsmodels, etc.) are good examples here; they have a narrower focus and because of that more domain-specific code than SciPy.

Now if you have code that you would like to see included in SciPy, how do you go about it? After checking that your code can be distributed in SciPy under a compatible license (see FAQ for details), the first step is to discuss on the scipy-dev mailing list. All new features, as well as changes to existing code, are discussed and decided on there. You can, and probably should, already start this discussion before your code is finished.

Assuming the outcome of the discussion on the mailing list is positive and you have a function or piece of code that does what you need it to do, what next? Before code is added to SciPy, it at least has to have good documentation, unit tests and correct code style.

  1. Unit tests
    In principle you should aim to create unit tests that exercise all the code that you are adding. This gives some degree of confidence that your code runs correctly, also on Python versions and hardware or OSes that you don't have available yourself. An extensive description of how to write unit tests is given in the NumPy testing guidelines.
  2. Documentation
    Clear and complete documentation is essential in order for users to be able to find and understand the code. Documentation for individual functions and classes -- which includes at least a basic description, type and meaning of all parameters and returns values, and usage examples in doctest format -- is put in docstrings. Those docstrings can be read within the interpreter, and are compiled into a reference guide in html and pdf format. Higher-level documentation for key (areas of) functionality is provided in tutorial format and/or in module docstrings. A guide on how to write documentation is given in how to document.
  3. Code style
    Uniformity of style in which code is written is important to others trying to understand the code. SciPy follows the standard Python guidelines for code style, PEP8. In order to check that your code conforms to PEP8, you can use the pep8 package style checker. Most IDEs and text editors have settings that can help you follow PEP8, for example by translating tabs by four spaces. Using pyflakes to check your code is also a good idea.

At the end of this document a checklist is given that may help to check if your code fulfills all requirements for inclusion in SciPy.

Another question you may have is: where exactly do I put my code? To answer this, it is useful to understand how the SciPy public API (application programming interface) is defined. For most modules the API is two levels deep, which means your new function should appear as scipy.submodule.my_new_func. my_new_func can be put in an existing or new file under /scipy/<submodule>/, its name is added to the __all__ list in that file (which lists all public functions in the file), and those public functions are then imported in /scipy/<submodule>/__init__.py. Any private functions/classes should have a leading underscore (_) in their name. A more detailed description of what the public API of SciPy is, is given in SciPy API.

Once you think your code is ready for inclusion in SciPy, you can send a pull request (PR) on Github. We won't go into the details of how to work with git here, this is described well in the git workflow section of the NumPy documentation and on the Github help pages. When you send the PR for a new feature, be sure to also mention this on the scipy-dev mailing list. This can prompt interested people to help review your PR. Assuming that you already got positive feedback before on the general idea of your code/feature, the purpose of the code review is to ensure that the code is correct, efficient and meets the requirements outlined above. In many cases the code review happens relatively quickly, but it's possible that it stalls. If you have addressed all feedback already given, it's perfectly fine to ask on the mailing list again for review (after a reasonable amount of time, say a couple of weeks, has passed). Once the review is completed, the PR is merged into the "master" branch of SciPy.

The above describes the requirements and process for adding code to SciPy. It doesn't yet answer the question though how decisions are made exactly. The basic answer is: decisions are made by consensus, by everyone who chooses to participate in the discussion on the mailing list. This includes developers, other users and yourself. Aiming for consensus in the discussion is important -- SciPy is a project by and for the scientific Python community. In those rare cases that agreement cannot be reached, the maintainers of the module in question can decide the issue.

Contributing by helping maintain existing code

The previous section talked specifically about adding new functionality to SciPy. A large part of that discussion also applies to maintenance of existing code. Maintenance means fixing bugs, improving code quality or style, documenting existing functionality better, adding missing unit tests, keeping build scripts up-to-date, etc. The SciPy issue list contains all reported bugs, build/documentation issues, etc. Fixing issues helps improve the overall quality of SciPy, and is also a good way of getting familiar with the project. You may also want to fix a bug because you ran into it and need the function in question to work correctly.

The discussion on code style and unit testing above applies equally to bug fixes. It is usually best to start by writing a unit test that shows the problem, i.e. it should pass but doesn't. Once you have that, you can fix the code so that the test does pass. That should be enough to send a PR for this issue. Unlike when adding new code, discussing this on the mailing list may not be necessary - if the old behavior of the code is clearly incorrect, no one will object to having it fixed. It may be necessary to add some warning or deprecation message for the changed behavior. This should be part of the review process.

Other ways to contribute

There are many ways to contribute other than contributing code. Participating in discussions on the scipy-user and scipy-dev mailing lists is a contribution in itself. The scipy.org website contains a lot of information on the SciPy community and can always use a new pair of hands.

Recommended development setup

Since Scipy contains parts written in C, C++, and Fortran that need to be compiled before use, make sure you have the necessary compilers and Python development headers installed. Having compiled code also means that importing Scipy from the development sources needs some additional steps, which are explained below.

First fork a copy of the main Scipy repository in Github onto your own account and then create your local repository via:

$ git clone [email protected]:YOURUSERNAME/scipy.git scipy
$ cd scipy
$ git remote add upstream git://github.com/scipy/scipy.git

To build the development version of Scipy and run tests, spawn interactive shells with the Python import paths properly set up etc., do one of:

$ python runtests.py -v
$ python runtests.py -v -s optimize
$ python runtests.py -v -t scipy/special/tests/test_basic.py:test_xlogy
$ python runtests.py --ipython
$ python runtests.py --python somescript.py
$ python runtests.py --bench

This builds Scipy first, so the first time it may take some time. If you specify -n, the tests are run against the version of Scipy (if any) found on current PYTHONPATH.

Using runtests.py is the recommended approach to running tests. There are also a number of alternatives to it, for example in-place build or installing to a virtualenv. See the FAQ below for details.

Some of the tests in Scipy are very slow and need to be separately enabled. See the FAQ below for details.

SciPy structure

All SciPy modules should follow the following conventions. In the following, a SciPy module is defined as a Python package, say yyy, that is located in the scipy/ directory.

  • Ideally, each SciPy module should be as self-contained as possible. That is, it should have minimal dependencies on other packages or modules. Even dependencies on other SciPy modules should be kept to a minimum. A dependency on NumPy is of course assumed.

  • Directory yyy/ contains:

    • A file setup.py that defines configuration(parent_package='',top_path=None) function for numpy.distutils.
    • A directory tests/ that contains files test_<name>.py corresponding to modules yyy/<name>{.py,.so,/}.

  • Private modules should be prefixed with an underscore _, for instance yyy/_somemodule.py.

  • User-visible functions should have good documentation following the Numpy documentation style, see how to document

  • The __init__.py of the module should contain the main reference documentation in its docstring. This is connected to the Sphinx documentation under doc/ via Sphinx's automodule directive.

    The reference documentation should first give a categorized list of the contents of the module using autosummary:: directives, and after that explain points essential for understanding the use of the module.

    Tutorial-style documentation with extensive examples should be separate, and put under doc/source/tutorial/

See the existing Scipy submodules for guidance.

For further details on Numpy distutils, see:

https://github.com/numpy/numpy/blob/master/doc/DISTUTILS.rst.txt

Useful links, FAQ, checklist

Checklist before submitting a PR

  • Are there unit tests with good code coverage?
  • Do all public function have docstrings including examples?
  • Is the code style correct (PEP8, pyflakes)
  • Is the commit message formatted correctly?
  • Is the new functionality tagged with .. versionadded:: X.Y.Z (with X.Y.Z the version number of the next release - can be found in setup.py)?
  • Is the new functionality mentioned in the release notes of the next release?
  • Is the new functionality added to the reference guide?
  • In case of larger additions, is there a tutorial or more extensive module-level description?
  • In case compiled code is added, is it integrated correctly via setup.py (and preferably also Bento configuration files - bento.info and bscript)?
  • If you are a first-time contributor, did you add yourself to THANKS.txt? Please note that this is perfectly normal and desirable - the aim is to give every single contributor credit, and if you don't add yourself it's simply extra work for the reviewer (or worse, the reviewer may forget).
  • Did you check that the code can be distributed under a BSD license?

Useful SciPy documents

FAQ

I based my code on existing Matlab/R/... code I found online, is this OK?

It depends. SciPy is distributed under a BSD license, so if the code that you based your code on is also BSD licensed or has a BSD-compatible license (MIT, Apache, ...) then it's OK. Code which is GPL-licensed, has no clear license, requires citation or is free for academic use only can't be included in SciPy. Therefore if you copied existing code with such a license or made a direct translation to Python of it, your code can't be included. See also license compatibility.

Why is SciPy under the BSD license and not, say, the GPL?

Like Python, SciPy uses a "permissive" open source license, which allows proprietary re-use. While this allows companies to use and modify the software without giving anything back, it is felt that the larger user base results in more contributions overall, and companies often publish their modifications anyway, without being required to. See John Hunter's BSD pitch.

How do I set up a development version of SciPy in parallel to a released version that I use to do my job/research?

One simple way to achieve this is to install the released version in site-packages, by using a binary installer or pip for example, and set up the development version in a virtualenv. First install virtualenv (optionally use virtualenvwrapper), then create your virtualenv (named scipy-dev here) with:

$ virtualenv scipy-dev

Now, whenever you want to switch to the virtual environment, you can use the command source scipy-dev/bin/activate, and deactivate to exit from the virtual environment and back to your previous shell. With scipy-dev activated, install first Scipy's dependencies:

$ pip install Numpy Nose Cython

After that, you can install a development version of Scipy, for example via:

$ python setup.py install

The installation goes to the virtual environment.

How do I set up an in-place build for development

For development, you can set up an in-place build so that changes made to .py files have effect without rebuild. First, run:

$ python setup.py build_ext -i

Then you need to point your PYTHONPATH environment variable to this directory. Some IDEs (Spyder for example) have utilities to manage PYTHONPATH. On Linux and OSX, you can run the command:

$ export PYTHONPATH=$PWD

and on Windows

$ set PYTHONPATH=/path/to/scipy

Now editing a Python source file in SciPy allows you to immediately test and use your changes (in .py files), by simply restarting the interpreter.

Can I use a programming language other than Python to speed up my code?

Yes. The languages used in SciPy are Python, Cython, C, C++ and Fortran. All of these have their pros and cons. If Python really doesn't offer enough performance, one of those languages can be used. Important concerns when using compiled languages are maintainability and portability. For maintainability, Cython is clearly preferred over C/C++/Fortran. Cython and C are more portable than C++/Fortran. A lot of the existing C and Fortran code in SciPy is older, battle-tested code that was only wrapped in (but not specifically written for) Python/SciPy. Therefore the basic advice is: use Cython. If there's specific reasons why C/C++/Fortran should be preferred, please discuss those reasons first.

How do I debug code written in C/C++/Fortran inside Scipy?

The easiest way to do this is to first write a Python script that invokes the C code whose execution you want to debug. For instance mytest.py:

from scipy.special import hyp2f1
print(hyp2f1(5.0, 1.0, -1.8, 0.95))

Now, you can run:

gdb --args python runtests.py -g --python mytest.py

If you didn't compile with debug symbols enabled before, remove the build directory first. While in the debugger:

(gdb) break cephes_hyp2f1
(gdb) run

The execution will now stop at the corresponding C function and you can step through it as usual. Instead of plain gdb you can of course use your favourite alternative debugger; run it on the python binary with arguments runtests.py -g --python mytest.py.

How do I enable additional tests in Scipy?

Some of the tests in Scipy's test suite are very slow and not enabled by default. You can run the full suite via:

$ python runtests.py -g -m full

This invokes the test suite import scipy; scipy.test("full"), enabling also slow tests.

There is an additional level of very slow tests (several minutes), which are disabled also in this case. They can be enabled by setting the environment variable SCIPY_XSLOW=1 before running the test suite.

How do I write tests with test generators?

The Nose test framework supports so-called test generators, which can come useful if you need to have multiple tests where just a parameter changes. Using test generators so that they are more useful than harmful is tricky, and we recommend the following pattern:

def test_something():
    some_array = (...)

    def check(some_param):
        c = compute_result(some_array, some_param)
        known_result = (...)
        assert_allclose(c, known_result)

    for some_param in ['a', 'b', 'c']:
        yield check, some_param

We require the following:

  • All asserts and all computation that is tested must only be reached after a yield. (Rationale: the generator body is part of no test, and a failure in it will show neither the test name nor for what parameters the test failed.)
  • Arrays must not be passed as yield parameters. Either use variables from outer scope (eg. with some index passed to yield), or capsulate test data to a class with a sensible __repr__. (Rationale: Nose truncates the printed form of arrays in test output, and this makes it impossible to know for what parameters a test failed. Arrays are big, and clutter test output unnecessarily.)
  • Test generators cannot be used in test classes inheriting from unittest.TestCase; either use object as base class, or use standalone test functions. (Rationale: Nose does not run test generators in TestCase-inheriting classes.)

If in doubt, do not use test generators. You can track for what parameter things failed also by passing err_msg=repr((param1, param2, ...)) to the various assert functions.