diff --git a/authors.txt b/authors.txt index 087255916..44d4626a6 100644 --- a/authors.txt +++ b/authors.txt @@ -2,6 +2,7 @@ Alex J. Burke Brett Cannon C\. Titus Brown Carl Meyer +Carol Willing Daniel Holth Kevin Teague Jean-Paul Calderone diff --git a/source/basics.rst b/source/basics.rst new file mode 100644 index 000000000..cbec4d64c --- /dev/null +++ b/source/basics.rst @@ -0,0 +1,49 @@ +====================== +Basics of Installation +====================== + +.. Editors, see notes at the bottom of the document for maintenance info. + +.. contents:: Contents + :local: + +Jargon +------ + +- System Python +- Selected Python +- Environments (None, virtual, conda) +- Package manager/tool (pipenv, pip, conda, miniconda, others) +- Advanced installation cases + +Installation using PyPA Tools +----------------------------- + + +Third Party Installation Tools +------------------------------ + +Best Practices Checklist +------------------------ + +- Review the installation documentation for the package +- Install the package by following the installation documentation +- Most packages will be installed with pipenv or pip. +- [Science and Data Science Packages] The installation instructions may include + references to conda and conda-forge. Follow those instructions. + +Wrap up +------- + + + +.. Editing notes: + + This document targets a novice audience especially scientists who may + not have a computer development background. It's expected + that most developers finding this document will have experience installing + applications and apps, but may not have installed a Python package before + or be familiar with the command line. + + For specific how-to information refer to the documentation of the desired + package for installation. diff --git a/source/discussions/index.rst b/source/discussions/index.rst index ab35c93e1..caecd810a 100644 --- a/source/discussions/index.rst +++ b/source/discussions/index.rst @@ -1,14 +1,21 @@ -Discussions -########### +Concepts +######## -**Discussions** are focused on providing comprehensive information about a -specific topic. If you're just trying to get stuff done, see -:doc:`/guides/index`. +The **Concepts** section discusses fundamental information about specific +topics. + +The goals of this section include providing background information, describing +important technical concepts, and highlighting usage tradeoffs. + +Contents include: .. toctree:: :maxdepth: 1 - deploying-python-applications - pip-vs-easy-install install-requires-vs-requirements + pip-vs-easy-install wheel-vs-egg + deploying-python-applications + +Alternatively, if you're trying to get a specific task done, see +:doc:`/guides/index`. \ No newline at end of file diff --git a/source/tutorials/creating-documentation.rst b/source/guides/creating-documentation.rst similarity index 100% rename from source/tutorials/creating-documentation.rst rename to source/guides/creating-documentation.rst diff --git a/source/guides/index.rst b/source/guides/index.rst index b8d25d499..8f5cdea19 100644 --- a/source/guides/index.rst +++ b/source/guides/index.rst @@ -1,9 +1,10 @@ -Guides -###### +Tasks +##### -**Guides** are focused on accomplishing a specific task and assume that you are -already familiar with the basics of Python packaging. If you're looking for an -introduction to packaging, see :doc:`/tutorials/index`. +**Tasks** guides focus on accomplishing specific tasks. + +The guides in this Tasks section assume that you are already familiar with the +basics of Python packaging. Contents include: .. toctree:: :maxdepth: 1 @@ -28,4 +29,7 @@ introduction to packaging, see :doc:`/tutorials/index`. hosting-your-own-index migrating-to-pypi-org using-testpypi + creating-documentation making-a-pypi-friendly-readme + +If you're looking for an introduction to packaging, see :doc:`/tutorials/index`. \ No newline at end of file diff --git a/source/guides/installing-scientific-packages.rst b/source/guides/installing-scientific-packages.rst index 8133ba07c..119662f3d 100644 --- a/source/guides/installing-scientific-packages.rst +++ b/source/guides/installing-scientific-packages.rst @@ -7,12 +7,24 @@ Installing scientific packages .. contents:: Contents :local: +Overview +-------- Scientific software tends to have more complex dependencies than most, and it will often have multiple build options to take advantage of different kinds of hardware, or to interoperate with different pieces of external software. +Prerequisites +------------- + +A basic understanding of the Scientific Python stack as well as installation +of packages using tools such as pip or conda and using hosting repositories +like PyPI and conda-forge. + +Complexity, NumPy, and SciPy +---------------------------- + In particular, `NumPy `__, which provides the basis for most of the software in the `scientific Python stack `__ can be configured @@ -31,8 +43,8 @@ libraries (or any other Python libraries that require a compilation environment to install from source and don't provide pre-built wheel files on PyPI). -Building from source --------------------- +Building and installing from source +----------------------------------- The same complexity which makes it difficult to distribute NumPy (and many of the projects that depend on it) as wheel files also make them difficult @@ -40,9 +52,16 @@ to build from source yourself. However, for intrepid folks that are willing to spend the time wrangling compilers and linkers for both C and FORTRAN, building from source is always an option. +Installation options by operating system +---------------------------------------- + +Scientific packages may use multiple programming languages. In addition, +the use of different operating systems increases the complexity of installation. +Therefore, the selection of installation tool and method will often depend on +several factors including those already mentioned. Linux distribution packages ---------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~ For Linux users, the system package manager will often have pre-compiled versions of various pieces of scientific software, including NumPy and @@ -54,7 +73,7 @@ installed into the system Python when using virtual environments). Windows installers ------------------- +~~~~~~~~~~~~~~~~~~ Many Python projects that don't (or can't) currently publish wheel files at least publish Windows installers, either on PyPI or on their project @@ -83,7 +102,7 @@ convert a Windows `bdist_wininst` installer to a wheel. .. _mac-os-x-installers-and-package-managers: macOS installers and package managers -------------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Similar to the situation on Windows, many projects (including NumPy) publish macOS installers that are compatible with the macOS CPython binaries @@ -105,27 +124,14 @@ end users in an easy to use and update format. Some of these distributions may not be compatible with the standard ``pip`` and ``virtualenv`` based toolchain. -Spack ------- -`Spack `_ is a flexible package manager -designed to support multiple versions, configurations, platforms, and compilers. -It was built to support the needs of large supercomputing centers and scientific -application teams, who must often build software many different ways. -Spack is not limited to Python; it can install packages for ``C``, ``C++``, -``Fortran``, ``R``, and other languages. It is non-destructive; installing -a new version of one package does not break existing installations, so many -configurations can coexist on the same system. - -Spack offers a simple but powerful syntax that allows users to specify -versions and configuration options concisely. Package files are written in -pure Python, and they are templated so that it is easy to swap compilers, -dependency implementations (like MPI), versions, and build options with a single -package file. Spack also generates *module* files so that packages can -be loaded and unloaded from the user's environment. +Third-party package managers +---------------------------- +Often a third-party package manager, maintained by groups other than PyPA, may +be an expedient solution for installing scientific and data science packages. The conda cross-platform package manager ----------------------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ `Anaconda `_ is a Python distribution published by Anaconda, Inc. It is a stable collection of Open Source @@ -145,3 +151,51 @@ written in Python, but is Python-agnostic. Conda manages Python itself as a package, so that `conda update python` is possible, in contrast to pip, which only manages Python packages. Conda is available in Anaconda and Miniconda (an easy-to-install download with just Python and conda). + +Miniconda +~~~~~~~~~ + +A lighter-weight option for conda installs and often using conda-forge as the +preferred channel. + +Spack +~~~~~ + +`Spack `_ is a flexible package manager +designed to support multiple versions, configurations, platforms, and compilers. +It was built to support the needs of large supercomputing centers and scientific +application teams, who must often build software many different ways. +Spack is not limited to Python; it can install packages for ``C``, ``C++``, +``Fortran``, ``R``, and other languages. It is non-destructive; installing +a new version of one package does not break existing installations, so many +configurations can coexist on the same system. + +Spack offers a simple but powerful syntax that allows users to specify +versions and configuration options concisely. Package files are written in +pure Python, and they are templated so that it is easy to swap compilers, +dependency implementations (like MPI), versions, and build options with a single +package file. Spack also generates *module* files so that packages can +be loaded and unloaded from the user's environment. + +Conda-forge +----------- + +Maintained and tested scientific packages by the conda-forge community. A +good source for finding up-to-date builds for package installation. + +DockerHub +--------- + +Docker and other container technologies are used within the sciences to spin +up a predetermined container of packages. This often can be an aid to +reproducibility of results. + +What's next +----------- + +A wise recommendation for installing Scientific packages would be to follow +the installation recommendations provided in the package's documentation. +Another practical next step for users who are new to installing Python Scientific +packages would be to consult a recent tutorial or talk from a prior PyCon, +SciPy, or PyData conference which describes the current state of installing +scientific packages and tools. \ No newline at end of file diff --git a/source/index.rst b/source/index.rst index db8bcf1c1..193460ca4 100644 --- a/source/index.rst +++ b/source/index.rst @@ -11,6 +11,7 @@ Python Packaging User Guide :hidden: overview + basics tutorials/index guides/index discussions/index @@ -32,26 +33,64 @@ happily accept any :doc:`contributions and feedback `. 😊 .. _Python Packaging Authority: https://pypa.io +Navigating the User Guide +========================= + +The *Python Packaging User Guide* is organized into several sections: + ++------------------------------+----------------------------------------------------------+ +| Section | Description | ++------------------------------+----------------------------------------------------------+ +| :doc:`tutorials/index` | Introduce packaging concepts through opinionated, | +| | step-by-step guides, such as *Installing Packages* | +| | and *Packaging Projects*. | ++------------------------------+----------------------------------------------------------+ +| :doc:`guides/index` | Focus on accomplishing a specific task, such as | +| | *Making a PyPI friendly README* and *Installing* | +| | *Scientific Packages*. | ++------------------------------+----------------------------------------------------------+ +| :doc:`discussions/index` | Provide comprehensive information about a specific topic | +| | such as *install_requires vs requirements files* and | +| | *Wheel vs Egg*. | ++------------------------------+----------------------------------------------------------+ +| :doc:`specifications/index` | Document interoperability and standards, such as | +| | *Entry points specification* | ++------------------------------+----------------------------------------------------------+ +| :doc:`key_projects` | Highlights PyPA maintained projects and mentions notable | +| | third-party maintained projects. | ++------------------------------+----------------------------------------------------------+ +| :doc:`glossary` | Define unfamiliar terms or jargon | ++------------------------------+----------------------------------------------------------+ + + Get started =========== -Essential tools and concepts for working within the Python -development ecosystem are covered in our :doc:`tutorials/index` section: +Many readers will find the :doc:`tutorials/index` section a good starting place +to learn more about Python packaging. It provides step-by-step information about +the essential tools and concepts of installing and packaging Python projects: + +* :doc:`tutorial on Installing Packages ` +* :doc:`tutorial on Managing Application Dependencies ` + in a version controlled project +* :doc:`tutorial on Packaging and Distributing ` + your projects + +The :doc:`Basics of Installation ` discusses the methods of +installing a Python package for different use cases, such as web, scientific +applications, devops, and more. New developers are encouraged to read this +section before the :doc:`tutorial on Installing Packages `. -* to learn how to install packages, see the - :doc:`tutorial on installing packages `. -* to learn how to manage dependencies in a version controlled project, see the - :doc:`tutorial on managing application dependencies `. -* to learn how to package and distribute your projects, see the - :doc:`tutorial on packaging and distributing ` -* to get an overview of packaging options for Python libraries and - applications, see the :doc:`Overview of Python Packaging `. +Intermediate to advanced Python developers interested in creating a Python +package will benefit from reviewing the :doc:`Overview of Python Packaging ` +and its guidance on planning development for packaging. Learn more ========== -Beyond our :doc:`tutorials/index`, this guide has several other resources: +Beyond our :doc:`tutorials/index`, this User Guide has several resource +sections with topical and in-depth information: * the :doc:`guides/index` section for walk throughs, such as :doc:`guides/installing-using-linux-tools` or :doc:`guides/packaging-binary-extensions` diff --git a/source/overview.rst b/source/overview.rst index 6a11f2c00..ca4b78310 100644 --- a/source/overview.rst +++ b/source/overview.rst @@ -1,6 +1,6 @@ -=================================== -An Overview of Packaging for Python -=================================== +================================ +Overview of Packaging for Python +================================ .. Editors, see notes at the bottom of the document for maintenance info. diff --git a/source/tutorials/index.rst b/source/tutorials/index.rst index 40b0620f3..33ab4f98a 100644 --- a/source/tutorials/index.rst +++ b/source/tutorials/index.rst @@ -11,4 +11,3 @@ topics, see :doc:`/guides/index`. installing-packages managing-dependencies packaging-projects - creating-documentation diff --git a/source/tutorials/installing-packages.rst b/source/tutorials/installing-packages.rst index 8e20d6a41..b412917f2 100644 --- a/source/tutorials/installing-packages.rst +++ b/source/tutorials/installing-packages.rst @@ -33,8 +33,8 @@ packages. Ensure you can run Python from the command line ----------------------------------------------- -Before you go any further, make sure you have Python and that the expected -version is available from your command line. You can check this by running: +Before you go any further, make sure you have Python and the expected +version available from your command line. You can check this by running: .. code-block:: bash @@ -44,7 +44,8 @@ You should get some output like ``Python 3.6.3``. If you do not have Python, please install the latest 3.x version from `python.org`_ or refer to the `Installing Python`_ section of the Hitchhiker's Guide to Python. -.. Note:: If you're a newcomer and you get an error like this: +.. Note:: If you are new to Python or the command line and the system returns + an error like this: .. code-block:: python @@ -53,26 +54,30 @@ please install the latest 3.x version from `python.org`_ or refer to the File "", line 1, in NameError: name 'python' is not defined - It's because this command and other suggested commands in this tutorial - are intended to be run in a *shell* (also called a *terminal* or + It is due to running ``python --version`` in the Python REPL + instead of a *shell*, like bash. This command and others in this + tutorial should be run in a *shell* (also called a *terminal* or *console*). See the Python for Beginners `getting started tutorial`_ for an introduction to using your operating system's shell and interacting with Python. .. Note:: If you're using an enhanced shell like IPython or the Jupyter - notebook, you can run system commands like those in this tutorial by - prefacing them with a ``!`` character: + notebook, you can run system commands, like those in this tutorial, by + prefacing the command with a ``!`` character: :: In [1]: import sys - !{sys.executable} --version + + In [2]: !{sys.executable} --version + Python 3.6.3 - It's recommended to write ``{sys.executable}`` rather than plain ``python`` in - order to ensure that commands are run in the Python installation matching - the currently running notebook (which may not be the same Python - installation that the ``python`` command refers to). + It's recommended to substitute ``{sys.executable}`` for ``python``. This + helps to ensure that commands are run in the same Python installation as + the currently running notebook. Running ``python`` as a command in the + notebook may execute in the same Python installation that the + notebook is running which will cause confusion or unexpected results. .. Note:: Due to the way most Linux distributions are handling the Python 3 migration, Linux users using the system Python without creating a virtual diff --git a/source/tutorials/packaging-projects.rst b/source/tutorials/packaging-projects.rst index a2455205a..f2da4430e 100644 --- a/source/tutorials/packaging-projects.rst +++ b/source/tutorials/packaging-projects.rst @@ -201,6 +201,9 @@ Now run this command from the same directory where :file:`setup.py` is located: python3 setup.py sdist bdist_wheel +.. TODO where to document setup.py sdist / bdist_wheel -> python -m pep517.build -s -b + probably in a guide re: scientific python + This command should output a lot of text and once completed should generate two files in the :file:`dist` directory: