diff --git a/CHANGES.rst b/CHANGES.rst index 6cf0c4bf6..9c2a59f91 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -146,7 +146,7 @@ Documentation #. "Exponential Functional" split out from "Trigonometry Functions" #. "Functional Programming" section split out. #. "Image Manipulation" has been split off from Graphics and Drawing and turned into a guide section. -#. Image examples now appear in the LaTeX and therfore the PDF doc +#. Image examples now appear in the LaTeX and therefore the PDF doc #. "Logic and Boolean Algebra" section reinstated. #. "Forms of Input and Output" is its own guide section. #. More URL links to Wiki pages added; more internal cross links added. @@ -183,7 +183,7 @@ Bugs #. Better handling of ``Infinite`` quantities. #. Improved ``Precision`` and ``Accuracy``compatibility with WMA. In particular, ``Precision[0.]`` and ``Accuracy[0.]`` #. Accuracy in numbers using the notation ``` n.nnn``acc ``` now is properly handled. -#. numeric precision in mpmath was not reset after operations that changed these. This cause huges slowdowns after an operation that set the mpmath precison high. This was the source of several-minute slowdowns in testing. +#. numeric precision in mpmath was not reset after operations that changed these. This cause huges slowdowns after an operation that set the mpmath precision high. This was the source of several-minute slowdowns in testing. #. GIF87a (```MadTeaParty.gif`` or ExampleData) image loading fixed #. Replace non-free Leena image with a a freely distributable image. Issue #728 @@ -1061,7 +1061,7 @@ New features (50+ builtins) #. ``SubsetQ`` and ``Delete[]`` #688, #784, #. ``Subsets`` #685 #. ``SystemTimeZone`` and correct ``TimeZone`` #924 -#. ``System\`Byteordering`` and ``System\`Environemnt`` #859 +#. ``System\`Byteordering`` and ``System\`Environment`` #859 #. ``$UseSansSerif`` #908 #. ``randchoice`` option for ``NoNumPyRandomEnv`` #820 #. Support for ``MATHICS_MAX_RECURSION_DEPTH`` @@ -1411,7 +1411,7 @@ New features #. ``PolarPlot`` #. IPython style (coloured) input #. ``VectorAnalysis`` Package -#. More special functions (Bessel functions and othogonal polynomials) +#. More special functions (Bessel functions and orthogonal polynomials) #. More NumberTheory functions #. ``Import``, ``Export``, ``Get``, ``Needs`` and other IO related functions #. PyPy compatibility diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md index 18c914718..b3f31aecc 100644 --- a/CODE_OF_CONDUCT.md +++ b/CODE_OF_CONDUCT.md @@ -5,7 +5,7 @@ We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender -identity and expression, level of experience, education, socio-economic status, +identity and expression, level of experience, education, socioeconomic status, nationality, personal appearance, race, religion, or sexual identity and orientation. diff --git a/Makefile b/Makefile index 57dc8c171..8d0f2da9c 100644 --- a/Makefile +++ b/Makefile @@ -10,6 +10,7 @@ PIP ?= pip3 BASH ?= bash RM ?= rm PYTEST_OPTIONS ?= +DOCTEST_OPTIONS ?= # Variable indicating Mathics3 Modules you have available on your system, in latex2doc option format MATHICS3_MODULE_OPTION ?= --load-module pymathics.graph,pymathics.natlang @@ -132,9 +133,9 @@ gstest: doctest-data: mathics/builtin/*.py mathics/doc/documentation/*.mdoc mathics/doc/documentation/images/* MATHICS_CHARACTER_ENCODING="UTF-8" $(PYTHON) mathics/docpipeline.py --output --keep-going $(MATHICS3_MODULE_OPTION) -#: Run tests that appear in docstring in the code. +#: Run tests that appear in docstring in the code. Use environment variable "DOCTEST_OPTIONS" for doctest options doctest: - MATHICS_CHARACTER_ENCODING="ASCII" SANDBOX=$(SANDBOX) $(PYTHON) mathics/docpipeline.py $o + MATHICS_CHARACTER_ENCODING="ASCII" SANDBOX=$(SANDBOX) $(PYTHON) mathics/docpipeline.py $(DOCTEST_OPTIONS) #: Make Mathics PDF manual via Asymptote and LaTeX latexdoc texdoc doc: diff --git a/PAST.rst b/PAST.rst index 20d41b9c7..39d8fead9 100644 --- a/PAST.rst +++ b/PAST.rst @@ -12,7 +12,7 @@ A fair bit of code refactoring has gone on so that we might be able to scale the code, get it to be more performant, and more in line with other interpreters. There is Greater use of Symbols as opposed to strings. -The buitin Functions have been organized into grouping akind to what is found in WMA. +The builtin Functions have been organized into grouping akind to what is found in WMA. This is not just for documentation purposes, but it better modularizes the code and keep the modules smaller while suggesting where functions below as we scale. @@ -34,14 +34,14 @@ Boxing and Formatting While some work on formatting is done has been made and the change in API reflects a little of this. However a lot more work needs to be done. -Excecution Performance +Execution Performance ---------------------- This has improved a slight bit, but not because it has been a focus, but rather because in going over the code organization, we are doing this less dumb, e.g. using Symbols more where symbols are intended. Or fixing bugs like resetting mpmath numeric precision on operations that -need to chnage it temporarily. +need to change it temporarily. Simpler Things -------------- @@ -50,6 +50,6 @@ A number of items here remain, but should not be thought as independent items, b "Forms, Boxing and Formatting". "Making StandardOutput of polynomials match WMA" is really are Forms, Boxing and Formatting issue; -"Working on Jupyter integrations" is also very dependant this. +"Working on Jupyter integrations" is also very dependent this. So the next major refactor will be on Forms, Boxing and Formatting. diff --git a/examples/symbolic_logic/gries_schneider/GS1.m b/examples/symbolic_logic/gries_schneider/GS1.m index adaa372a8..41ec8bb50 100644 --- a/examples/symbolic_logic/gries_schneider/GS1.m +++ b/examples/symbolic_logic/gries_schneider/GS1.m @@ -617,7 +617,7 @@ right-hand side of the rule now, while parsing the rule itself, only later, after doing the pattern substitutions specified by the rule." - Remember, evaluation is really aggressive. When you write a rule withe "->", + Remember, evaluation is really aggressive. When you write a rule with a "->", mathics will try to evaluate the right-hand side. Sometimes, it doesn't matter which of the two you use. In the example diff --git a/examples/symbolic_logic/gries_schneider/GS2.m b/examples/symbolic_logic/gries_schneider/GS2.m index e455f58f6..86d831e2a 100644 --- a/examples/symbolic_logic/gries_schneider/GS2.m +++ b/examples/symbolic_logic/gries_schneider/GS2.m @@ -31,7 +31,7 @@ << "../../test_driver.m" -(* Chaper 2, Boolean Expressions, page 25 +(* Chapter 2, Boolean Expressions, page 25 Section 2.1, Syntax and evaluation of Boolean expression, page 25 ___ _ ___ _ @@ -110,7 +110,7 @@ target f(a). The number of different ways to assign ||B|| values to ||A|| there are 2 ** 4 == sixteen different binary functions. I start with inert "true" and "false" to avoid evaluation leaks, i.e., to - prevent mathics from reducing expessions that have active "True" and + prevent mathics from reducing expressions that have active "True" and "False". *************************************************************************** *) diff --git a/examples/symbolic_logic/gries_schneider/GS3.m b/examples/symbolic_logic/gries_schneider/GS3.m index 5aa12ac6f..5a6fdc8ee 100644 --- a/examples/symbolic_logic/gries_schneider/GS3.m +++ b/examples/symbolic_logic/gries_schneider/GS3.m @@ -29,7 +29,7 @@ *************************************************************************** *) -(* Chaper 3, Propositional Calculus, page 41 ********************************** +(* Chapter 3, Propositional Calculus, page 41 ********************************** ___ _ _ _ _ | _ \_ _ ___ _ __ ___ __(_) |_(_)___ _ _ __ _| | | _/ '_/ _ \ '_ \/ _ (_-< | _| / _ \ ' \/ _` | | diff --git a/mathics/__init__.py b/mathics/__init__.py index ff26e6b2e..3684c6c29 100644 --- a/mathics/__init__.py +++ b/mathics/__init__.py @@ -14,7 +14,7 @@ # version_info contains a list of Python packages # and the versions infsalled or "Not installed" # if the package is not installed and "No version information" -# if we can't get version infomation. +# if we can't get version information. version_info: Dict[str, str] = { "mathics": __version__, "mpmath": mpmath.__version__, @@ -58,7 +58,7 @@ license_string = """\ -Copyright (C) 2011-2023 The Mathics Team. +Copyright (C) 2011-2024 The Mathics Team. This program comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. diff --git a/mathics/builtin/arithmetic.py b/mathics/builtin/arithmetic.py index 5f622282c..f73d0699f 100644 --- a/mathics/builtin/arithmetic.py +++ b/mathics/builtin/arithmetic.py @@ -384,7 +384,7 @@ class Conjugate(MPMathFunction): """ :Complex Conjugate: https://en.wikipedia.org/wiki/Complex_conjugate \ - (:WMA:https://reference.wolfram.com/language/ref/Conjugate.html) + :WMA link:https://reference.wolfram.com/language/ref/Conjugate.html
'Conjugate[$z$]' @@ -507,7 +507,7 @@ def eval_directed_infinity(self, direction, evaluation: Evaluation): else: normalized_direction = direction / Abs(direction) elif isinstance(ndir, Complex): - re, im = ndir.value + re, im = ndir.real, ndir.imag if abs(re.value**2 + im.value**2 - 1.0) < 1.0e-9: normalized_direction = direction else: @@ -539,7 +539,7 @@ def to_sympy(self, expr, **kwargs): class Element(Builtin): """ :Element of:https://en.wikipedia.org/wiki/Element_(mathematics) \ - (:WMA:https://reference.wolfram.com/language/ref/Element.html) + :WMA link:https://reference.wolfram.com/language/ref/Element.html
'Element[$expr$, $domain$]' diff --git a/mathics/builtin/assignments/upvalues.py b/mathics/builtin/assignments/upvalues.py index 2fb6b0d0f..7580aef7f 100644 --- a/mathics/builtin/assignments/upvalues.py +++ b/mathics/builtin/assignments/upvalues.py @@ -2,7 +2,7 @@ """ UpValue-related assignments -An UpValue is a definition associated with a symbols that does not appear directly its head. +An UpValue is a definition associated with a symbols that does not appear directly its head. See :Associating Definitions with Different Symbols: @@ -17,7 +17,7 @@ # In Mathematica 5, this appears under "Types of Values". class UpValues(Builtin): """ - :WMA: https://reference.wolfram.com/language/ref/UpValues.html + :WMA link: https://reference.wolfram.com/language/ref/UpValues.html
'UpValues[$symbol$]'
gives the list of transformation rules corresponding to upvalues \ diff --git a/mathics/builtin/atomic/numbers.py b/mathics/builtin/atomic/numbers.py index bf529591d..93b6c6807 100644 --- a/mathics/builtin/atomic/numbers.py +++ b/mathics/builtin/atomic/numbers.py @@ -152,6 +152,7 @@ class Accuracy(Builtin):
examines the number of significant digits of $expr$ after the \ decimal point in the number x.
+ Notice that the result could be slightly different than the obtained \ in WMA, due to differencs in the internal representation of the real numbers. @@ -760,14 +761,15 @@ class Precision(Builtin): """ :Precision: - https://en.wikipedia.org/wiki/Accuracy_and_precision ( - :WMA: - https://reference.wolfram.com/language/ref/Precision.html) + https://en.wikipedia.org/wiki/Accuracy_and_precision + :WMA link: + https://reference.wolfram.com/language/ref/Precision.html
'Precision[$expr$]'
examines the number of significant digits of $expr$.
+ Note that the result could be slightly different than the obtained \ in WMA, due to differencs in the internal representation of the real numbers. diff --git a/mathics/builtin/atomic/symbols.py b/mathics/builtin/atomic/symbols.py index f7301bcee..7c190e53f 100644 --- a/mathics/builtin/atomic/symbols.py +++ b/mathics/builtin/atomic/symbols.py @@ -95,7 +95,8 @@ def _get_usage_string(symbol, evaluation, is_long_form: bool, htmlout=False): class Context(Builtin): r""" - :WMA: https://reference.wolfram.com/language/ref/Context.html + :WMA link: + https://reference.wolfram.com/language/ref/Context.html
'Context[$symbol$]'
yields the name of the context where $symbol$ is defined in. @@ -133,7 +134,8 @@ def eval(self, symbol, evaluation): class Definition(Builtin): """ - :WMA: https://reference.wolfram.com/language/ref/Definition.html + :WMA link: + https://reference.wolfram.com/language/ref/Definition.html
'Definition[$symbol$]'
prints as the definitions given for $symbol$. @@ -352,14 +354,14 @@ def format_definition_input(self, symbol, evaluation): # In Mathematica 5, this appears under "Types of Values". class DownValues(Builtin): """ - :WMA: https://reference.wolfram.com/language/ref/DownValues.html + :WMA link: https://reference.wolfram.com/language/ref/DownValues.html
'DownValues[$symbol$]'
gives the list of downvalues associated with $symbol$.
'DownValues' uses 'HoldPattern' and 'RuleDelayed' to protect the \ - downvalues from being evaluated. Moreover, it has attribute \ + downvalues from being evaluated, and it has attribute \ 'HoldAll' to get the specified symbol instead of its value. >> f[x_] := x ^ 2 @@ -408,7 +410,8 @@ def eval(self, symbol, evaluation): class Information(PrefixOperator): """ - :WMA: https://reference.wolfram.com/language/ref/Information.html + :WMA link: + https://reference.wolfram.com/language/ref/Information.html
'Information[$symbol$]'
Prints information about a $symbol$ @@ -562,7 +565,8 @@ def format_definition_input(self, symbol, evaluation: Evaluation, options: dict) class Names(Builtin): """ - :WMA: https://reference.wolfram.com/language/ref/Names.html + :WMA link: + https://reference.wolfram.com/language/ref/Names.html
'Names["$pattern$"]'
returns the list of names matching $pattern$. @@ -614,7 +618,8 @@ def eval(self, pattern, evaluation): # In Mathematica 5, this appears under "Types of Values". class OwnValues(Builtin): """ - :WMA: https://reference.wolfram.com/language/ref/OwnValues.html + :WMA link: + https://reference.wolfram.com/language/ref/OwnValues.html
'OwnValues[$symbol$]'
gives the list of ownvalue associated with $symbol$. @@ -647,7 +652,8 @@ def eval(self, symbol, evaluation): class Symbol_(Builtin): """ - :WMA: https://reference.wolfram.com/language/ref/Symbol.html + :WMA link: + https://reference.wolfram.com/language/ref/Symbol.html
'Symbol'
is the head of symbols. @@ -686,7 +692,8 @@ def eval(self, string, evaluation): class SymbolName(Builtin): """ - :WMA: https://reference.wolfram.com/language/ref/SymbolName.html + :WMA link: + https://reference.wolfram.com/language/ref/SymbolName.html
'SymbolName[$s$]'
returns the name of the symbol $s$ (without any leading \ @@ -709,7 +716,8 @@ def eval(self, symbol, evaluation): class SymbolQ(Test): """ - :WMA: https://reference.wolfram.com/language/ref/SymbolName.html + :WMA link: + https://reference.wolfram.com/language/ref/SymbolName.html
'SymbolQ[$x$]'
is 'True' if $x$ is a symbol, or 'False' otherwise. @@ -731,7 +739,8 @@ def test(self, expr) -> bool: class ValueQ(Builtin): """ - :WMA: https://reference.wolfram.com/language/ref/ValueQ.html + :WMA link: + https://reference.wolfram.com/language/ref/ValueQ.html
'ValueQ[$expr$]'
returns 'True' if and only if $expr$ is defined. diff --git a/mathics/builtin/attributes.py b/mathics/builtin/attributes.py index e58a129dd..be8a7cbfa 100644 --- a/mathics/builtin/attributes.py +++ b/mathics/builtin/attributes.py @@ -7,10 +7,10 @@ specify general properties of functions and symbols. This is \ independent of the parameters they take and the values they produce. -The builtin-attributes having a predefined meaning in \Mathics which \ +The builtin-attributes having a predefined meaning in \\Mathics which \ are described below. -However in contrast to \Mathematica, you can set any symbol as an attribute. +However in contrast to \\Mathematica, you can set any symbol as an attribute. """ # This tells documentation how to sort this module diff --git a/mathics/builtin/box/__init__.py b/mathics/builtin/box/__init__.py index fcff563f7..5d2777e44 100644 --- a/mathics/builtin/box/__init__.py +++ b/mathics/builtin/box/__init__.py @@ -1,4 +1,4 @@ -""" +r""" Boxing modules. Boxes are added in formatting \Mathics Expressions. diff --git a/mathics/builtin/colors/color_operations.py b/mathics/builtin/colors/color_operations.py index 4a3688cdc..76c374025 100644 --- a/mathics/builtin/colors/color_operations.py +++ b/mathics/builtin/colors/color_operations.py @@ -205,7 +205,7 @@ def eval(self, input, colorspace, evaluation: Evaluation): class ColorNegate(Builtin): """ Color Inversion ( - :WMA: + :WMA link: https://reference.wolfram.com/language/ref/ColorNegate.html)
diff --git a/mathics/builtin/files_io/importexport.py b/mathics/builtin/files_io/importexport.py index fff53163a..9cfb74eb3 100644 --- a/mathics/builtin/files_io/importexport.py +++ b/mathics/builtin/files_io/importexport.py @@ -3,18 +3,18 @@ """ Importing and Exporting -Many kinds data formats can be read into \Mathics. Variable +Many kinds data formats can be read into \\Mathics. Variable :$ExportFormats: -/doc/reference-of-built-in-symbols/importing-and-exporting/$exportformats \ +/doc/reference-of-built-in-symbols/inputoutput-files-and-filesystem/importing-and-exporting/$exportformats \ contains a list of file formats that are supported by :Export: -/doc/reference-of-built-in-symbols/importing-and-exporting/export, \ +/doc/reference-of-built-in-symbols/inputoutput-files-and-filesystem/importing-and-exporting/export, \ while :$ImportFormats: -/doc/reference-of-built-in-symbols/importing-and-exporting/$importformats \ +/doc/reference-of-built-in-symbols/inputoutput-files-and-filesystem/importing-and-exporting/$importformats \ does the corresponding thing for :Import: -/doc/reference-of-built-in-symbols/importing-and-exporting/import. +/doc/reference-of-built-in-symbols/inputoutput-files-and-filesystem/importing-and-exporting/import. """ import base64 diff --git a/mathics/builtin/forms/output.py b/mathics/builtin/forms/output.py index 58dedfb85..be995cd8a 100644 --- a/mathics/builtin/forms/output.py +++ b/mathics/builtin/forms/output.py @@ -10,7 +10,7 @@ # than applicable to all kinds of expressions. """ -Forms which appear in '$OutputForms'. +Form Functions """ import re from math import ceil @@ -18,7 +18,7 @@ from mathics.builtin.box.layout import GridBox, RowBox, to_boxes from mathics.builtin.forms.base import FormBaseClass -from mathics.builtin.makeboxes import MakeBoxes, number_form +from mathics.builtin.makeboxes import MakeBoxes, NumberForm_to_String from mathics.builtin.tensors import get_dimensions from mathics.core.atoms import ( Integer, @@ -63,19 +63,26 @@ MULTI_NEWLINE_RE = re.compile(r"\n{2,}") -class BaseForm(Builtin): +class BaseForm(FormBaseClass): """ + + :WMA link: + https://reference.wolfram.com/language/ref/BaseForm.html +
'BaseForm[$expr$, $n$]'
prints numbers in $expr$ in base $n$.
+ A binary integer: >> BaseForm[33, 2] = 100001_2 + A hexidecimal number: >> BaseForm[234, 16] = ea_16 + A binary real number: >> BaseForm[12.3, 2] = 1100.01001100110011001_2 @@ -97,6 +104,8 @@ class BaseForm(Builtin): = BaseForm[12, 100] """ + in_outputforms = True + in_printforms = False summary_text = "print with all numbers given in a base" messages = { "intpm": ( @@ -376,6 +385,10 @@ def check_NumberSeparator(self, value, evaluation: Evaluation): class NumberForm(_NumberForm): """ + + :WMA link: + https://reference.wolfram.com/language/ref/NumberForm.html +
'NumberForm[$expr$, $n$]'
prints a real number $expr$ with $n$-digits of precision. @@ -387,8 +400,12 @@ class NumberForm(_NumberForm): >> NumberForm[N[Pi], 10] = 3.141592654 - >> NumberForm[N[Pi], {10, 5}] + >> NumberForm[N[Pi], {10, 6}] + = 3.141593 + + >> NumberForm[N[Pi]] = 3.14159 + """ options = { @@ -473,7 +490,7 @@ def eval_makeboxes(self, expr, form, evaluation, options={}): if py_n is not None: py_options["_Form"] = form.get_name() - return number_form(expr, py_n, None, evaluation, py_options) + return NumberForm_to_String(expr, py_n, None, evaluation, py_options) return Expression(SymbolMakeBoxes, expr, form) def eval_makeboxes_n(self, expr, n, form, evaluation, options={}): @@ -493,7 +510,7 @@ def eval_makeboxes_n(self, expr, n, form, evaluation, options={}): if isinstance(expr, (Integer, Real)): py_options["_Form"] = form.get_name() - return number_form(expr, py_n, None, evaluation, py_options) + return NumberForm_to_String(expr, py_n, None, evaluation, py_options) return Expression(SymbolMakeBoxes, expr, form) def eval_makeboxes_nf(self, expr, n, f, form, evaluation, options={}): @@ -515,7 +532,7 @@ def eval_makeboxes_nf(self, expr, n, f, form, evaluation, options={}): if isinstance(expr, (Integer, Real)): py_options["_Form"] = form.get_name() - return number_form(expr, py_n, py_f, evaluation, py_options) + return NumberForm_to_String(expr, py_n, py_f, evaluation, py_options) return Expression(SymbolMakeBoxes, expr, form) diff --git a/mathics/builtin/intfns/recurrence.py b/mathics/builtin/intfns/recurrence.py index c28637069..4d2d043c8 100644 --- a/mathics/builtin/intfns/recurrence.py +++ b/mathics/builtin/intfns/recurrence.py @@ -50,8 +50,8 @@ class Fibonacci(MPMathFunction): class HarmonicNumber(MPMathFunction): """ - :Harmonic Number:https://en.wikipedia.org/wiki/Harmonic_number \( - :WMA link:https://reference.wolfram.com/language/ref/HarmonicNumber.html) + :Harmonic Number:https://en.wikipedia.org/wiki/Harmonic_number \ + (:WMA link:https://reference.wolfram.com/language/ref/HarmonicNumber.html)
'HarmonicNumber[n]' diff --git a/mathics/builtin/makeboxes.py b/mathics/builtin/makeboxes.py index 58a1b2315..afdd32408 100644 --- a/mathics/builtin/makeboxes.py +++ b/mathics/builtin/makeboxes.py @@ -1,9 +1,9 @@ # -*- coding: utf-8 -*- """ -Low level Format definitions +Low-level Format definitions """ -from typing import Union +from typing import Optional, Tuple, Union import mpmath @@ -27,16 +27,22 @@ ) -def int_to_s_exp(expr, n): - n = expr.get_int_value() - if n < 0: - nonnegative = 0 - s = str(-n) +def int_to_tuple_info(integer: Integer) -> Tuple[str, int, bool]: + """ + Convert ``integer`` to a tuple representing that value. The tuple consists of: + * the string absolute value of ``integer``. + * the exponent, base 10, to be used, and + * True if the value is nonnegative or False otherwise. + """ + value = integer.value + if value < 0: + is_nonnegative = False + value = -value else: - nonnegative = 1 - s = str(n) - exp = len(s) - 1 - return s, exp, nonnegative + is_nonnegative = True + s = str(value) + exponent = len(s) - 1 + return s, exponent, is_nonnegative # FIXME: op should be a string, so remove the Union. @@ -65,95 +71,120 @@ def make_boxes_infix( return Expression(SymbolRowBox, ListExpression(*result)) -def real_to_s_exp(expr, n): - if expr.is_zero: +def real_to_tuple_info(real: Real, digits: Optional[int]) -> Tuple[str, int, bool]: + """ + Convert ``real`` to a tuple representing that value. The tuple consists of: + * the string absolute value of ``integer`` with decimal point removed from the string; + the position of the decimal point is determined by the exponent below, + * the exponent, base 10, to be used, and + * True if the value is nonnegative or False otherwise. + + If ``digits`` is None, we use the default precision. + """ + if real.is_zero: s = "0" - if expr.is_machine_precision(): - exp = 0 + if real.is_machine_precision(): + exponent = 0 else: - p = expr.get_precision() - exp = -dps(p) - nonnegative = 1 + p = real.get_precision() + exponent = -dps(p) + is_nonnegative = True else: - if n is None: - if expr.is_machine_precision(): - value = expr.get_float_value() + if digits is None: + if real.is_machine_precision(): + value = real.value s = repr(value) else: - with mpmath.workprec(expr.get_precision()): - value = expr.to_mpmath() - s = mpmath.nstr(value, dps(expr.get_precision()) + 1) + with mpmath.workprec(real.get_precision()): + value = real.to_mpmath() + s = mpmath.nstr(value, dps(real.get_precision()) + 1) else: - with mpmath.workprec(expr.get_precision()): - value = expr.to_mpmath() - s = mpmath.nstr(value, n) + with mpmath.workprec(real.get_precision()): + value = real.to_mpmath() + s = mpmath.nstr(value, digits) - # sign prefix + # Set sign prefix. if s[0] == "-": assert value < 0 - nonnegative = 0 + is_nonnegative = False s = s[1:] else: assert value >= 0 - nonnegative = 1 - - # exponent (exp is actual, pexp is printed) + is_nonnegative = True + # Set exponent. ``exponent`` is actual, ``pexp`` of ``NumberForm_to_string()`` is printed. if "e" in s: - s, exp = s.split("e") - exp = int(exp) + s, exponent = s.split("e") + exponent = int(exponent) if len(s) > 1 and s[1] == ".": # str(float) doesn't always include '.' if 'e' is present. s = s[0] + s[2:].rstrip("0") else: - exp = s.index(".") - 1 - s = s[: exp + 1] + s[exp + 2 :].rstrip("0") + exponent = s.index(".") - 1 + s = s[: exponent + 1] + s[exponent + 2 :].rstrip("0") - # consume leading '0's. + # Normalize exponent: remove leading '0's after the decimal point + # and adjust the exponent accordingly. i = 0 - while s[i] == "0": + while i < len(s) and s[i] == "0": i += 1 - exp -= 1 + exponent -= 1 s = s[i:] - # add trailing zeros for precision reals - if n is not None and not expr.is_machine_precision() and len(s) < n: - s = s + "0" * (n - len(s)) - return s, exp, nonnegative - - -def number_form(expr, n, f, evaluation: Evaluation, options: dict): + # Add trailing zeros for precision reals. + if digits is not None and not real.is_machine_precision() and len(s) < digits: + s = s + "0" * (digits - len(s)) + return s, exponent, is_nonnegative + + +# FIXME: the return type should be a NumberForm, not a String. +# when this is fixed, rename the function. +def NumberForm_to_String( + value: Union[Real, Integer], + digits: Optional[int], + digits_after_decimal_point: Optional[int], + evaluation: Evaluation, + options: dict, +) -> String: """ - Converts a Real or Integer instance to Boxes. + Converts a Real or Integer value to a String. - n digits of precision with f (can be None) digits after the decimal point. - evaluation (can be None) is used for messages. + ``digits`` is the number of digits of precision and + ``digits_after_decimal_point`` is the number of digits after the + decimal point. ``evaluation`` is used for messages. - The allowed options are python versions of the options permitted to + The allowed options are Python versions of the options permitted to NumberForm and must be supplied. See NumberForm or Real.make_boxes for correct option examples. + + If ``digits`` is None, use the default precision. If + ``digits_after_decimal_points`` is None, use all the digits we get + from the converted number, that is, otherwise the number may be + padded on the right-hand side with zeros. """ - assert isinstance(n, int) and n > 0 or n is None - assert f is None or (isinstance(f, int) and f >= 0) + assert isinstance(digits, int) and digits > 0 or digits is None + assert digits_after_decimal_point is None or ( + isinstance(digits_after_decimal_point, int) and digits_after_decimal_point >= 0 + ) is_int = False - if isinstance(expr, Integer): - assert n is not None - s, exp, nonnegative = int_to_s_exp(expr, n) - if f is None: + if isinstance(value, Integer): + assert digits is not None + s, exp, is_nonnegative = int_to_tuple_info(value) + if digits_after_decimal_point is None: is_int = True - elif isinstance(expr, Real): - if n is not None: - n = min(n, dps(expr.get_precision()) + 1) - s, exp, nonnegative = real_to_s_exp(expr, n) - if n is None: - n = len(s) + elif isinstance(value, Real): + if digits is not None: + digits = min(digits, dps(value.get_precision()) + 1) + s, exp, is_nonnegative = real_to_tuple_info(value, digits) + if digits is None: + digits = len(s) else: raise ValueError("Expected Real or Integer.") - assert isinstance(n, int) and n > 0 + assert isinstance(digits, int) and digits > 0 - sign_prefix = options["NumberSigns"][nonnegative] + sign_prefix = options["NumberSigns"][1 if is_nonnegative else 0] # round exponent to ExponentStep rexp = (exp // options["ExponentStep"]) * options["ExponentStep"] @@ -198,14 +229,18 @@ def _round(number, ndigits): return number # pad with NumberPadding - if f is not None: - if len(right) < f: + if digits_after_decimal_point is not None: + if len(right) < digits_after_decimal_point: # pad right - right = right + (f - len(right)) * options["NumberPadding"][1] - elif len(right) > f: + right = ( + right + + (digits_after_decimal_point - len(right)) + * options["NumberPadding"][1] + ) + elif len(right) > digits_after_decimal_point: # round right tmp = int(left + right) - tmp = _round(tmp, f - len(right)) + tmp = _round(tmp, digits_after_decimal_point - len(right)) tmp = str(tmp) left, right = tmp[: exp + 1], tmp[exp + 1 :] @@ -227,8 +262,8 @@ def split_string(s, start, step): left_padding = 0 max_sign_len = max(len(options["NumberSigns"][0]), len(options["NumberSigns"][1])) i = len(sign_prefix) + len(left) + len(right) - max_sign_len - if i < n: - left_padding = n - i + if i < digits: + left_padding = digits - i elif len(sign_prefix) < max_sign_len: left_padding = max_sign_len - len(sign_prefix) left_padding = left_padding * options["NumberPadding"][0] diff --git a/mathics/builtin/messages.py b/mathics/builtin/messages.py index b5c420552..8e9815ee9 100644 --- a/mathics/builtin/messages.py +++ b/mathics/builtin/messages.py @@ -578,7 +578,7 @@ class Syntax(Builtin): : "1.5`" cannot be followed by "`" (line 1 of ""). """ - # Extension: MMA does not provide lineno and filename in its error messages + # Extension: WMA does not provide lineno and filename in its error messages messages = { "snthex": r"4 hexadecimal digits are required after \: to construct a 16-bit character (line `4` of `5`).", "sntoct1": r"3 octal digits are required after \ to construct an 8-bit character (line `4` of `5`).", diff --git a/mathics/builtin/numbers/constants.py b/mathics/builtin/numbers/constants.py index 26e39a974..119f94330 100644 --- a/mathics/builtin/numbers/constants.py +++ b/mathics/builtin/numbers/constants.py @@ -246,7 +246,7 @@ class ComplexInfinity(_SympyConstant): is an infinite number in the complex plane whose complex argument \ is unknown or undefined. ( :SymPy: - https://docs.sympy.org/latest/modules/core.html?highlight=zoo#complexinfinity, + https://docs.sympy.org/latest/modules/core.html#sympy.core.numbers.ComplexInfinity, :MathWorld: https://mathworld.wolfram.com/ComplexInfinity.html, :WMA: @@ -257,10 +257,19 @@ class ComplexInfinity(_SympyConstant):
represents an infinite complex quantity of undetermined direction.
+ ComplexInfinity can appear as the result of a computation such as dividing by zero: + >> 1 / 0 + : Infinite expression 1 / 0 encountered. + = ComplexInfinity + + But it can be used as an explicit value in an expression: >> 1 / ComplexInfinity = 0 + >> ComplexInfinity * Infinity = ComplexInfinity + + ComplexInfinity though is a special case of DirectedInfinity: >> FullForm[ComplexInfinity] = DirectedInfinity[] """ diff --git a/mathics/builtin/numbers/numbertheory.py b/mathics/builtin/numbers/numbertheory.py index 9e9972238..f0c734f49 100644 --- a/mathics/builtin/numbers/numbertheory.py +++ b/mathics/builtin/numbers/numbertheory.py @@ -3,9 +3,9 @@ """ Number theoretic functions """ - import mpmath import sympy +from packaging.version import Version from mathics.core.atoms import Integer, Integer0, Integer10, Rational, Real from mathics.core.attributes import ( @@ -476,7 +476,8 @@ def to_int_value(x): result = n.to_python() for i in range(-py_k): try: - result = sympy.ntheory.prevprime(result) + # from sympy 1.13, the previous prime to 2 fails... + result = -2 if result == 2 else sympy.ntheory.prevprime(result) except ValueError: # No earlier primes return Integer(-1 * sympy.ntheory.nextprime(0, py_k - i)) @@ -500,7 +501,11 @@ class PartitionsP(SympyFunction): attributes = A_LISTABLE | A_NUMERIC_FUNCTION | A_ORDERLESS | A_PROTECTED summary_text = "number of unrestricted partitions" - sympy_name = "npartitions" + # The name of this function changed in Sympy version 1.13.0. + # This supports backward compatibility. + sympy_name = ( + "npartitions" if Version(sympy.__version__) < Version("1.13.0") else "partition" + ) def eval(self, n, evaluation: Evaluation): "PartitionsP[n_Integer]" @@ -580,13 +585,13 @@ class PrimePi(SympyFunction): attributes = A_LISTABLE | A_NUMERIC_FUNCTION | A_PROTECTED mpmath_name = "primepi" summary_text = "amount of prime numbers less than or equal" - sympy_name = "ntheory.primepi" + sympy_name = "primepi" # TODO: Traditional Form def eval(self, n, evaluation: Evaluation): "PrimePi[n_?NumericQ]" - result = sympy.ntheory.primepi(eval_N(n, evaluation).to_python()) + result = sympy.primepi(eval_N(n, evaluation).to_python()) return Integer(result) diff --git a/mathics/builtin/numeric.py b/mathics/builtin/numeric.py index 63bf5d88e..73d625284 100644 --- a/mathics/builtin/numeric.py +++ b/mathics/builtin/numeric.py @@ -625,8 +625,9 @@ class RealValuedNumberQ(Builtin): # No docstring since this is internal and it will mess up documentation. # FIXME: Perhaps in future we will have a more explicite way to indicate not # to add something to the docs. + no_doc = True context = "Internal`" - + summary_text = "test whether an expression is a real number" rules = { "Internal`RealValuedNumberQ[x_Real]": "True", "Internal`RealValuedNumberQ[x_Integer]": "True", @@ -639,6 +640,7 @@ class RealValuedNumericQ(Builtin): # No docstring since this is internal and it will mess up documentation. # FIXME: Perhaps in future we will have a more explicite way to indicate not # to add something to the docs. + no_doc = True context = "Internal`" rules = { diff --git a/mathics/builtin/testing_expressions/numerical_properties.py b/mathics/builtin/testing_expressions/numerical_properties.py index ce16aa491..02205eaa3 100644 --- a/mathics/builtin/testing_expressions/numerical_properties.py +++ b/mathics/builtin/testing_expressions/numerical_properties.py @@ -38,16 +38,22 @@ class CoprimeQ(Builtin): >> CoprimeQ[12, 15] = False - CoprimeQ also works for complex numbers - >> CoprimeQ[1+2I, 1-I] - = True - - >> CoprimeQ[4+2I, 6+3I] - = True + ## + ## CoprimeQ also works for complex numbers + ## >> CoprimeQ[1+2I, 1-I] + ## = True + + ## This test case is commenteted out because the result produced by sympy is wrong: + ## In this case, both numbers can be factorized as 2 (2 + I) and 3 (2 + I): + ## >> CoprimeQ[4+2I, 6+3I] + ## = False + + For more than two arguments, CoprimeQ checks if any pair or arguments are coprime: >> CoprimeQ[2, 3, 5] = True + In this case, since 2 divides 4, the result is False: >> CoprimeQ[2, 4, 5] = False """ diff --git a/mathics/builtin/vectors/__init__.py b/mathics/builtin/vectors/__init__.py index 4e1ad3afe..96fe8eef5 100644 --- a/mathics/builtin/vectors/__init__.py +++ b/mathics/builtin/vectors/__init__.py @@ -8,7 +8,7 @@ In computer science, it is an array data structure consisting of collection \ of elements identified by at least on array index or key. -In \Mathics vectors as are Lists. One never needs to distinguish between row \ +In \\Mathics vectors as are Lists. One never needs to distinguish between row \ and column vectors. As with other objects vectors can mix number and symbolic elements. Vectors can be long, dense, or sparse. diff --git a/mathics/core/atoms.py b/mathics/core/atoms.py index f55abb451..a265b52c1 100644 --- a/mathics/core/atoms.py +++ b/mathics/core/atoms.py @@ -29,8 +29,9 @@ ) from mathics.core.systemsymbols import SymbolFullForm, SymbolInfinity, SymbolInputForm -# Imperical number that seems to work. -# We have to be able to match mpmath values with sympy values +# The below value is an empirical number for comparison precedence +# that seems to work. We have to be able to match mpmath values with +# sympy values COMPARE_PREC = 50 SymbolI = Symbol("I") @@ -85,17 +86,22 @@ def is_literal(self) -> bool: return True def is_numeric(self, evaluation=None) -> bool: + # Anything that is in a number class is Numeric, so return True. return True - def to_mpmath(self): + def to_mpmath(self, precision: Optional[int] = None) -> mpmath.ctx_mp_python.mpf: """ - Convert self._value to an mnpath number. + Convert self._value to an mpmath number with precision ``precision`` + If ``precision`` is None, use mpmath's default precision. - This is the default implementation for Number. + A mpmath number is the default implementation for Number. There are kinds of numbers, like Rational, or Complex, that need to work differently than this default, and they will change the implementation accordingly. """ + if precision is not None: + with mpmath.workprec(precision): + return mpmath.mpf(self._value) return mpmath.mpf(self._value) @property @@ -250,8 +256,8 @@ def make_boxes(self, form) -> "String": # obtained from an integer is limited, and for longer # numbers, this exception is raised. # The idea is to represent the number by its - # more significative digits, the lowest significative digits, - # and a placeholder saying the number of ommited digits. + # more significant digits, the lowest significant digits, + # and a placeholder saying the number of omitted digits. from mathics.eval.makeboxes import int_to_string_shorter_repr return int_to_string_shorter_repr(self._value, form) @@ -446,14 +452,14 @@ def is_machine_precision(self) -> bool: return True def make_boxes(self, form): - from mathics.builtin.makeboxes import number_form + from mathics.builtin.makeboxes import NumberForm_to_String _number_form_options["_Form"] = form # passed to _NumberFormat if form in ("System`InputForm", "System`FullForm"): n = None else: n = 6 - return number_form(self, n, None, None, _number_form_options) + return NumberForm_to_String(self, n, None, None, _number_form_options) @property def is_zero(self) -> bool: @@ -467,7 +473,7 @@ def round(self, d: Optional[int] = None) -> "MachineReal": def sameQ(self, other) -> bool: """Mathics SameQ for MachineReal. - If the other comparision value is a MachineReal, the values + If the other comparison value is a MachineReal, the values have to be equal. If the other value is a PrecisionReal though, then the two values have to be within 1/2 ** (precision) of other-value's precision. For any other type, sameQ is False. @@ -479,7 +485,7 @@ def sameQ(self, other) -> bool: value = self.to_sympy() # If sympy fixes the issue, this comparison would be # enough - if value == other_value: + if (value - other_value).is_zero: return True # this handles the issue... diff = abs(value - other_value) @@ -551,13 +557,14 @@ def get_precision(self) -> int: @property def is_zero(self) -> bool: - return self.value == 0.0 + # self.value == 0 does not work for sympy >=1.13 + return self.value.is_zero def make_boxes(self, form): - from mathics.builtin.makeboxes import number_form + from mathics.builtin.makeboxes import NumberForm_to_String _number_form_options["_Form"] = form # passed to _NumberFormat - return number_form( + return NumberForm_to_String( self, dps(self.get_precision()), None, None, _number_form_options ) @@ -578,7 +585,7 @@ def sameQ(self, other) -> bool: value = self.value # If sympy would handle properly # the precision, this wold be enough - if value == other_value: + if (value - other_value).is_zero: return True # in the meantime, let's use this comparison. value = self.value @@ -718,10 +725,17 @@ def __new__(cls, real, imag): if isinstance(real, MachineReal) and not isinstance(imag, MachineReal): imag = imag.round() - if isinstance(imag, MachineReal) and not isinstance(real, MachineReal): + prec = FP_MANTISA_BINARY_DIGITS + elif isinstance(imag, MachineReal) and not isinstance(real, MachineReal): real = real.round() + prec = FP_MANTISA_BINARY_DIGITS + else: + prec = min( + (u for u in (x.get_precision() for x in (real, imag)) if u is not None), + default=None, + ) - value = (real, imag) + value = (real, imag, prec) self = cls._complex_numbers.get(value) if self is None: self = super().__new__(cls) diff --git a/mathics/core/convert/__init__.py b/mathics/core/convert/__init__.py index 13cc331bb..dba17738a 100644 --- a/mathics/core/convert/__init__.py +++ b/mathics/core/convert/__init__.py @@ -1,7 +1,7 @@ # -*- coding: utf-8 -*- """ Routines here convert between various internal representations such as -between ``Expressions``, LLVM functions, SymPy Arguments, MPMath datattypes and -so on. However this does not include the inital conversion a parsed string into +between ``Expressions``, LLVM functions, SymPy Arguments, MPMath datatypes and +so on. However this does not include the initial conversion a parsed string into one of the internal representations. That is done in the parser. """ diff --git a/mathics/core/convert/python.py b/mathics/core/convert/python.py index 5fc1061f5..35863424f 100644 --- a/mathics/core/convert/python.py +++ b/mathics/core/convert/python.py @@ -31,7 +31,7 @@ def from_bool(arg: bool) -> BooleanType: # Expression class which tried to handle anything given it using # conversions. # Also, through vague or lazy coding this cause a lot of -# unecessary conversions. +# unnecessary conversions. # We may be out of those days, but we should still # be mindful that this routine can be the source @@ -43,7 +43,7 @@ def from_python(arg: Any) -> BaseElement: """Converts a Python expression into a Mathics expression. TODO: I think there are number of subtleties to be explained here. - In particular, the expression might beeen the result of evaluation + In particular, the expression might been the result of evaluation a sympy expression which contains sympy symbols. If the end result is to go back into Mathics for further @@ -62,7 +62,7 @@ def from_python(arg: Any) -> BaseElement: number_type = get_type(arg) # We should investigate whether this could be sped up - # using a disctionary lookup on type. + # using a dictionary lookup on type. if arg is None: return SymbolNull if isinstance(arg, bool): diff --git a/mathics/core/convert/regex.py b/mathics/core/convert/regex.py index 7c0a3262f..45e302d11 100644 --- a/mathics/core/convert/regex.py +++ b/mathics/core/convert/regex.py @@ -98,7 +98,7 @@ def to_regex_internal( def recurse(x: Expression, quantifiers=q) -> Tuple[Optional[str], str]: """ - Shortend way to call to_regexp_internal - + Shortened way to call to_regexp_internal - only the expr and quantifiers change here. """ return to_regex_internal( diff --git a/mathics/core/evaluation.py b/mathics/core/evaluation.py index f1f8b826c..3d132a5dc 100644 --- a/mathics/core/evaluation.py +++ b/mathics/core/evaluation.py @@ -184,11 +184,11 @@ def __init__( # ``mathics.builtin.numeric.N``. self._preferred_n_method = [] - def parse(self, query): + def parse(self, query, src_name: str = ""): "Parse a single expression and print the messages." from mathics.core.parser import MathicsSingleLineFeeder - return self.parse_feeder(MathicsSingleLineFeeder(query)) + return self.parse_feeder(MathicsSingleLineFeeder(query, src_name)) def parse_evaluate(self, query, timeout=None): expr = self.parse(query) @@ -635,7 +635,7 @@ def get_data(self): class Output(ABC): """ - Base class for Mathics ouput history. + Base class for Mathics output history. This needs to be subclassed. """ diff --git a/mathics/core/parser/__init__.py b/mathics/core/parser/__init__.py index 39e5238c1..8ad4654db 100644 --- a/mathics/core/parser/__init__.py +++ b/mathics/core/parser/__init__.py @@ -6,7 +6,7 @@ There is a separate `README `_ -for decribing how this works. +for describing how this works. """ diff --git a/mathics/core/systemsymbols.py b/mathics/core/systemsymbols.py index ef26ec447..8ce706595 100644 --- a/mathics/core/systemsymbols.py +++ b/mathics/core/systemsymbols.py @@ -39,6 +39,7 @@ SymbolAssumptions = Symbol("System`$Assumptions") SymbolAttributes = Symbol("System`Attributes") SymbolAutomatic = Symbol("System`Automatic") +SymbolBaseForm = Symbol("System`BaseForm") SymbolBlank = Symbol("System`Blank") SymbolBlankNullSequence = Symbol("System`BlankNullSequence") SymbolBlankSequence = Symbol("System`BlankSequence") diff --git a/mathics/doc/__init__.py b/mathics/doc/__init__.py index 26efa89b8..1be93c620 100644 --- a/mathics/doc/__init__.py +++ b/mathics/doc/__init__.py @@ -1,8 +1,29 @@ # -*- coding: utf-8 -*- """ -Module for handling Mathics-style documentation. +A module and library that assists in organizing document data +located in static files and docstrings from +Mathics3 Builtin Modules. Builtin Modules are written in Python and +reside either in the Mathics3 core (mathics.builtin) or are packaged outside, +in Mathics3 Modules e.g. pymathics.natlang. -Right now this covers common LaTeX/PDF and routines common to -Mathics Django. When this code is moved out, perhaps it will -include the Mathics Django-specific piece. +This data is stored in a way that facilitates: +* organizing information to produce a LaTeX file +* running documentation tests +* producing HTML-based documentation + +The command-line utility ``docpipeline.py``, loads the data from +Python modules and static files, accesses the functions here. + +Mathics Django also uses this library for its HTML-based documentation. + +The Mathics3 builtin function ``Information[]`` also uses to provide the +information it reports. +As with reading in data, final assembly to a LaTeX file or running +documentation tests is done elsewhere. + +FIXME: This code should be replaced by Sphinx and autodoc. +Things are such a mess, that it is too difficult to contemplate this right now. +Also there higher-priority flaws that are more more pressing. +In the shorter, we might we move code for extracting printing to a +separate package. """ diff --git a/mathics/doc/common_doc.py b/mathics/doc/common_doc.py index e772a1035..7b845b934 100644 --- a/mathics/doc/common_doc.py +++ b/mathics/doc/common_doc.py @@ -1,1465 +1,60 @@ # -*- coding: utf-8 -*- """ -A module and library that assists in organizing document data -located in static files and docstrings from -Mathics3 Builtin Modules. Builtin Modules are written in Python and -reside either in the Mathics3 core (mathics.builtin) or are packaged outside, -in Mathics3 Modules e.g. pymathics.natlang. -This data is stored in a way that facilitates: -* organizing information to produce a LaTeX file -* running documentation tests -* producing HTML-based documentation +common_doc -The command-line utility ``docpipeline.py``, loads the data from -Python modules and static files, accesses the functions here. +This module is kept for backward compatibility. -Mathics Django also uses this library for its HTML-based documentation. +The module was split into +* mathics.doc.doc_entries: classes containing the documentation entries and doctests. +* mathics.doc.structure: the classes describing the elements in the documentation organization +* mathics.doc.gather: functions to gather information from modules to build the + documentation reference. -The Mathics3 builtin function ``Information[]`` also uses to provide the -information it reports. -As with reading in data, final assembly to a LaTeX file or running -documentation tests is done elsewhere. - - -FIXME: This code should be replaced by Sphinx and autodoc. -Things are such a mess, that it is too difficult to contemplate this right now. Also there -higher-priority flaws that are more more pressing. -In the shorter, we might we move code for extracting printing to a separate package. """ -import importlib -import logging -import os.path as osp -import pkgutil -import re -from os import environ, getenv, listdir -from types import ModuleType -from typing import Callable, Iterator, List, Optional, Tuple - -from mathics import settings -from mathics.core.builtin import check_requires_list -from mathics.core.evaluation import Message, Print -from mathics.core.load_builtin import ( - builtins_by_module as global_builtins_by_module, - mathics3_builtins_modules, -) -from mathics.core.util import IS_PYPY -from mathics.doc.utils import slugify -from mathics.eval.pymathics import pymathics_builtins_by_module, pymathics_modules -# These are all the XML/HTML-like tags that documentation supports. -ALLOWED_TAGS = ( - "dl", - "dd", - "dt", - "em", - "url", - "ul", - "i", - "ol", - "li", - "con", - "console", - "img", - "imgpng", - "ref", - "subsection", +from mathics.doc.doc_entries import ( + ALLOWED_TAGS, + ALLOWED_TAGS_RE, + CONSOLE_RE, + DL_ITEM_RE, + DL_RE, + HYPERTEXT_RE, + IMG_PNG_RE, + IMG_RE, + LATEX_RE, + LIST_ITEM_RE, + LIST_RE, + MATHICS_RE, + PYTHON_RE, + QUOTATIONS_RE, + REF_RE, + SPECIAL_COMMANDS, + DocTest, + DocTests, + DocText, + DocumentationEntry, + Tests, + get_results_by_test, + parse_docstring_to_DocumentationEntry_items, + post_sub, + pre_sub, ) -ALLOWED_TAGS_RE = dict( - (allowed, re.compile("<(%s.*?)>" % allowed)) for allowed in ALLOWED_TAGS -) - -# This string is used, so we can indicate a trailing blank at the end of a line by -# adding this string to the end of the line which gets stripped off. -# Some editors and formatters like to strip off trailing blanks at the ends of lines. -END_LINE_SENTINAL = "#<--#" - -# The regular expressions below (strings ending with _RE -# pull out information from docstring or text in a file. Ghetto parsing. - -CHAPTER_RE = re.compile('(?s)(.*?)') -CONSOLE_RE = re.compile(r"(?s)<(?Pcon|console)>(?P.*?)") -DL_ITEM_RE = re.compile( - r"(?s)<(?Pd[td])>(?P.*?)(?:|)\s*(?:(?=)|$)" -) -DL_RE = re.compile(r"(?s)
(.*?)
") -HYPERTEXT_RE = re.compile( - r"(?s)<(?Pem|url)>(\s*:(?P.*?):\s*)?(?P.*?)" -) -IMG_PNG_RE = re.compile( - r'' -) -IMG_RE = re.compile( - r'' -) -# Preserve space before and after in-line code variables. -LATEX_RE = re.compile(r"(\s?)\$(\w+?)\$(\s?)") - -LIST_ITEM_RE = re.compile(r"(?s)
  • (.*?)(?:
    • |(?=
  • )|$)") -LIST_RE = re.compile(r"(?s)<(?Pul|ol)>(?P.*?)") -MATHICS_RE = re.compile(r"(?(.*?)") -QUOTATIONS_RE = re.compile(r"\"([\w\s,]*?)\"") -REF_RE = re.compile(r'') -SECTION_RE = re.compile('(?s)(.*?)
    (.*?)
    ') -SPECIAL_COMMANDS = { - "LaTeX": (r"LaTeX", r"\LaTeX{}"), - "Mathematica": ( - r"Mathematica®", - r"\emph{Mathematica}\textregistered{}", - ), - "Mathics": (r"Mathics3", r"\emph{Mathics3}"), - "Mathics3": (r"Mathics3", r"\emph{Mathics3}"), - "Sage": (r"Sage", r"\emph{Sage}"), - "Wolfram": (r"Wolfram", r"\emph{Wolfram}"), - "skip": (r"

    ", r"\bigskip"), -} -SUBSECTION_END_RE = re.compile("") -SUBSECTION_RE = re.compile('(?s)') - -TESTCASE_RE = re.compile( - r"""(?mx)^ # re.MULTILINE (multi-line match) - # and re.VERBOSE (readable regular expressions - ((?:.|\n)*?) - ^\s+([>#SX])>[ ](.*) # test-code indicator - ((?:\n\s*(?:[:|=.][ ]|\.).*)*) # test-code results""" -) -TESTCASE_OUT_RE = re.compile(r"^\s*([:|=])(.*)$") - -# Used for getting test results by test expresson and chapter/section information. -test_result_map = {} - -# Debug flags. - -# Set to True if want to follow the process -# The first phase is building the documentation data structure -# based on docstrings: - -MATHICS_DEBUG_DOC_BUILD: bool = "MATHICS_DEBUG_DOC_BUILD" in environ - -# After building the doc structure, we extract test cases. -MATHICS_DEBUG_TEST_CREATE: bool = "MATHICS_DEBUG_TEST_CREATE" in environ - -# Name of the Mathics3 Module part of the document. -MATHICS3_MODULES_TITLE = "Mathics3 Modules" - - -def get_module_doc(module: ModuleType) -> Tuple[str, str]: - """ - Determine the title and text associated to the documentation - of a module. - If the module has a module docstring, extract the information - from it. If not, pick the title from the name of the module. - """ - doc = module.__doc__ - if doc is not None: - doc = doc.strip() - if doc: - title = doc.splitlines()[0] - text = "\n".join(doc.splitlines()[1:]) - else: - # FIXME: Extend me for Mathics3 modules. - title = module.__name__ - for prefix in ("mathics.builtin.", "mathics.optional."): - if title.startswith(prefix): - title = title[len(prefix) :] - title = title.capitalize() - text = "" - return title, text - - -def get_results_by_test(test_expr: str, full_test_key: list, doc_data: dict) -> dict: - """ - Sometimes test numbering is off, either due to bugs or changes since the - data was read. - - Here, we compensate for this by looking up the test by its chapter and section name - portion stored in `full_test_key` along with the and the test expression data - stored in `test_expr`. - - This new key is looked up in `test_result_map` its value is returned. - - `doc_data` is only first time this is called to populate `test_result_map`. - """ - - # Strip off the test index form new key with this and the test string. - # Add to any existing value for that "result". This is now what we want to - # use as a tee in test_result_map to look for. - test_section = list(full_test_key)[:-1] - search_key = tuple(test_section) - - if not test_result_map: - # Populate test_result_map from doc_data - for key, result in doc_data.items(): - test_section = list(key)[:-1] - new_test_key = tuple(test_section) - next_result = test_result_map.get(new_test_key, None) - if next_result is None: - next_result = [result] - else: - next_result.append(result) - - test_result_map[new_test_key] = next_result - - results = test_result_map.get(search_key, None) - result = {} - if results: - for result_candidate in results: - if result_candidate["query"] == test_expr: - if result: - # Already found something - print(f"Warning, multiple results appear under {search_key}.") - return {} - else: - result = result_candidate - - return result - - -def get_submodule_names(obj) -> list: - """Many builtins are organized into modules which, from a documentation - standpoint, are like Mathematica Online Guide Docs. - - "List Functions", "Colors", or "Distance and Similarity Measures" - are some examples Guide Documents group group various Builtin Functions, - under submodules relate to that general classification. - - Here, we want to return a list of the Python modules under a "Guide Doc" - module. - - As an example of a "Guide Doc" and its submodules, consider the - module named mathics.builtin.colors. It collects code and documentation pertaining - to the builtin functions that would be found in the Guide documentation for "Colors". - - The `mathics.builtin.colors` module has a submodule - `mathics.builtin.colors.named_colors`. - - The builtin functions defined in `named_colors` then are those found in the - "Named Colors" group of the "Colors" Guide Doc. - - So in this example then, in the list the modules returned for - Python module `mathics.builtin.colors` would be the - `mathics.builtin.colors.named_colors` module which contains the - definition and docs for the "Named Colors" Mathics Bultin - Functions. - """ - modpkgs = [] - if hasattr(obj, "__path__"): - for importer, modname, ispkg in pkgutil.iter_modules(obj.__path__): - modpkgs.append(modname) - modpkgs.sort() - return modpkgs - - -def filter_comments(doc: str) -> str: - """Remove docstring documentation comments. These are lines - that start with ##""" - return "\n".join( - line for line in doc.splitlines() if not line.lstrip().startswith("##") - ) - - -def get_doc_name_from_module(module) -> str: - """ - Get the title associated to the module. - If the module has a docstring, pick the name from - its first line (the title). Otherwise, use the - name of the module. - """ - name = "???" - if module.__doc__: - lines = module.__doc__.strip() - if not lines: - name = module.__name__ - else: - name = lines.split("\n")[0] - return name - - -POST_SUBSTITUTION_TAG = "_POST_SUBSTITUTION%d_" - - -def pre_sub(regexp, text: str, repl_func): - post_substitutions = [] - - def repl_pre(match): - repl = repl_func(match) - index = len(post_substitutions) - post_substitutions.append(repl) - return POST_SUBSTITUTION_TAG % index - - text = regexp.sub(repl_pre, text) - - return text, post_substitutions - - -def post_sub(text: str, post_substitutions) -> str: - for index, sub in enumerate(post_substitutions): - text = text.replace(POST_SUBSTITUTION_TAG % index, sub) - return text - - -def skip_doc(cls) -> bool: - """Returns True if we should skip cls in docstring extraction.""" - return cls.__name__.endswith("Box") or (hasattr(cls, "no_doc") and cls.no_doc) - - -def skip_module_doc(module, must_be_skipped) -> bool: - return ( - module.__doc__ is None - or module in must_be_skipped - or module.__name__.split(".")[0] not in ("mathics", "pymathics") - or hasattr(module, "no_doc") - and module.no_doc - ) - - -def parse_docstring_to_DocumentationEntry_items( - doc: str, - test_collection_constructor: Callable, - test_case_constructor: Callable, - text_constructor: Callable, - key_part=None, -) -> list: - """ - This parses string `doc` (using regular expressions) into Python objects. - test_collection_fn() is the class construtorto call to create an object for the - test collection. Each test is created via test_case_fn(). - Text within the test is stored via text_constructor. - """ - # Remove commented lines. - doc = filter_comments(doc).strip(r"\s") - - # Remove leading
    ...
    - # doc = DL_RE.sub("", doc) - - # pre-substitute Python code because it might contain tests - doc, post_substitutions = pre_sub( - PYTHON_RE, doc, lambda m: "%s" % m.group(1) - ) - - # HACK: Artificially construct a last testcase to get the "intertext" - # after the last (real) testcase. Ignore the test, of course. - doc += "\n >> test\n = test" - testcases = TESTCASE_RE.findall(doc) - - tests = None - items = [] - for index in range(len(testcases)): - testcase = list(testcases[index]) - text = testcase.pop(0).strip() - if text: - if tests is not None: - items.append(tests) - tests = None - text = post_sub(text, post_substitutions) - items.append(text_constructor(text)) - tests = None - if index < len(testcases) - 1: - test = test_case_constructor(index, testcase, key_part) - if tests is None: - tests = test_collection_constructor() - tests.tests.append(test) - - # If the last block in the loop was not a Text block, append the - # last set of tests. - if tests is not None: - items.append(tests) - tests = None - return items - - -class DocTest: - """ - Class to hold a single doctest. - - DocTest formatting rules: - - * `>>` Marks test case; it will also appear as part of - the documentation. - * `#>` Marks test private or one that does not appear as part of - the documentation. - * `X>` Shows the example in the docs, but disables testing the example. - * `S>` Shows the example in the docs, but disables testing if environment - variable SANDBOX is set. - * `=` Compares the result text. - * `:` Compares an (error) message. - `|` Prints output. - """ - - def __init__( - self, index: int, testcase: List[str], key_prefix: Optional[tuple] = None - ): - def strip_sentinal(line: str): - """Remove END_LINE_SENTINAL from the end of a line if it appears. - - Some editors like to strip blanks at the end of a line. - Since the line ends in END_LINE_SENTINAL which isn't blank, - any blanks that appear before will be preserved. - - Some tests require some lines to be blank or entry because - Mathics3 output can be that way - """ - if line.endswith(END_LINE_SENTINAL): - line = line[: -len(END_LINE_SENTINAL)] - - # Also remove any remaining trailing blanks since that - # seems *also* what we want to do. - return line.strip() - - self.index = index - self.outs = [] - self.result = None - - # Private test cases are executed, but NOT shown as part of the docs - self.private = testcase[0] == "#" - - # Ignored test cases are NOT executed, but shown as part of the docs - # Sandboxed test cases are NOT executed if environment SANDBOX is set - if testcase[0] == "X" or (testcase[0] == "S" and getenv("SANDBOX", False)): - self.ignore = True - # substitute '>' again so we get the correct formatting - testcase[0] = ">" - else: - self.ignore = False - - self.test = strip_sentinal(testcase[1]) - - self.key = None - if key_prefix: - self.key = tuple(key_prefix + (index,)) - - outs = testcase[2].splitlines() - for line in outs: - line = strip_sentinal(line) - if line: - if line.startswith("."): - text = line[1:] - if text.startswith(" "): - text = text[1:] - text = "\n" + text - if self.result is not None: - self.result += text - elif self.outs: - self.outs[-1].text += text - continue - - match = TESTCASE_OUT_RE.match(line) - if not match: - continue - symbol, text = match.group(1), match.group(2) - text = text.strip() - if symbol == "=": - self.result = text - elif symbol == ":": - out = Message("", "", text) - self.outs.append(out) - elif symbol == "|": - out = Print(text) - self.outs.append(out) - - def __str__(self) -> str: - return self.test - - -# Tests has to appear before Documentation which uses it. -# FIXME: Turn into a NamedTuple? Or combine with another class? -class Tests: - """ - A group of tests in the same section or subsection. - """ - - def __init__( - self, - part_name: str, - chapter_name: str, - section_name: str, - doctests: List[DocTest], - subsection_name: Optional[str] = None, - ): - self.part = part_name - self.chapter = chapter_name - self.section = section_name - self.subsection = subsection_name - self.tests = doctests - - -# DocSection has to appear before DocGuideSection which uses it. -class DocSection: - """An object for a Documented Section. - A Section is part of a Chapter. It can contain subsections. - """ - - def __init__( - self, - chapter, - title: str, - text: str, - operator, - installed: bool = True, - in_guide: bool = False, - summary_text: str = "", - ): - self.chapter = chapter - self.in_guide = in_guide - self.installed = installed - self.items = [] # tests in section when this is under a guide section - self.operator = operator - self.slug = slugify(title) - self.subsections = [] - self.subsections_by_slug = {} - self.summary_text = summary_text - self.tests = None # tests in section when not under a guide section - self.title = title - - if text.count("
    ") != text.count("
    "): - raise ValueError( - "Missing opening or closing
    tag in " - "{} documentation".format(title) - ) - - # Needs to come after self.chapter is initialized since - # DocumentationEntry uses self.chapter. - self.doc = DocumentationEntry(text, title, self) - - chapter.sections_by_slug[self.slug] = self - if MATHICS_DEBUG_DOC_BUILD: - print(" DEBUG Creating Section", title) - - # Add __eq__ and __lt__ so we can sort Sections. - def __eq__(self, other) -> bool: - return self.title == other.title - - def __lt__(self, other) -> bool: - return self.title < other.title - - def __str__(self) -> str: - return f" == {self.title} ==\n{self.doc}" - - def get_tests(self): - """yield tests""" - if self.installed: - for test in self.doc.get_tests(): - yield test - - -# DocChapter has to appear before DocGuideSection which uses it. -class DocChapter: - """An object for a Documented Chapter. - A Chapter is part of a Part[dChapter. It can contain (Guide or plain) Sections. - """ - - def __init__(self, part, title, doc=None, chapter_order: Optional[int] = None): - self.chapter_order = chapter_order - self.doc = doc - self.guide_sections = [] - self.part = part - self.title = title - self.slug = slugify(title) - self.sections = [] - self.sections_by_slug = {} - self.sort_order = None - - part.chapters_by_slug[self.slug] = self - - if MATHICS_DEBUG_DOC_BUILD: - print(" DEBUG Creating Chapter", title) - - def __str__(self) -> str: - """ - A DocChapter is represented as the index of its sections - and subsections. - """ - sections_descr = "" - for section in self.all_sections: - sec_class = "@>" if isinstance(section, DocGuideSection) else "@ " - sections_descr += f" {sec_class} " + section.title + "\n" - for subsection in section.subsections: - sections_descr += " * " + subsection.title + "\n" - - return f" = {self.part.title}: {self.title} =\n\n{sections_descr}" - - @property - def all_sections(self): - return sorted(self.sections + self.guide_sections) - - -class DocGuideSection(DocSection): - """An object for a Documented Guide Section. - A Guide Section is part of a Chapter. "Colors" or "Special Functions" - are examples of Guide Sections, and each contains a number of Sections. - like NamedColors or Orthogonal Polynomials. - """ - - def __init__( - self, - chapter: DocChapter, - title: str, - text: str, - submodule, - installed: bool = True, - ): - self.chapter = chapter - self.doc = DocumentationEntry(text, title, None) - self.in_guide = False - self.installed = installed - self.section = submodule - self.slug = slugify(title) - self.subsections = [] - self.subsections_by_slug = {} - self.title = title - - # FIXME: Sections never are operators. Subsections can have - # operators though. Fix up the view and searching code not to - # look for the operator field of a section. - self.operator = False - - if text.count("
    ") != text.count("
    "): - raise ValueError( - "Missing opening or closing
    tag in " - "{} documentation".format(title) - ) - if MATHICS_DEBUG_DOC_BUILD: - print(" DEBUG Creating Guide Section", title) - chapter.sections_by_slug[self.slug] = self - - # FIXME: turn into a @property tests? - def get_tests(self): - # FIXME: The below is a little weird for Guide Sections. - # Figure out how to make this clearer. - # A guide section's subsection are Sections without the Guide. - # it is *their* subsections where we generally find tests. - for section in self.subsections: - if not section.installed: - continue - for subsection in section.subsections: - # FIXME we are omitting the section title here... - if not subsection.installed: - continue - for doctests in subsection.items: - yield doctests.get_tests() - - -def sorted_chapters(chapters: List[DocChapter]) -> List[DocChapter]: - """Return chapters sorted by title""" - return sorted( - chapters, - key=lambda chapter: str(chapter.sort_order) - if chapter.sort_order is not None - else chapter.title, - ) - - -def sorted_modules(modules) -> list: - """Return modules sorted by the ``sort_order`` attribute if that - exists, or the module's name if not.""" - return sorted( - modules, - key=lambda module: module.sort_order - if hasattr(module, "sort_order") - else module.__name__, - ) - - -class DocPart: - """ - Represents one of the main parts of the document. Parts - can be loaded from a mdoc file, generated automatically from - the docstrings of Builtin objects under `mathics.builtin`. - """ - - chapter_class = DocChapter - - def __init__(self, doc, title, is_reference=False): - self.doc = doc - self.title = title - self.chapters = [] - self.chapters_by_slug = {} - self.is_reference = is_reference - self.is_appendix = False - self.slug = slugify(title) - doc.parts_by_slug[self.slug] = self - if MATHICS_DEBUG_DOC_BUILD: - print("DEBUG Creating Part", title) - - def __str__(self) -> str: - return f" Part {self.title}\n\n" + "\n\n".join( - str(chapter) for chapter in sorted_chapters(self.chapters) - ) - - -class DocTests: - """ - A bunch of consecutive ``DocTest`` objects extracted from a Builtin docstring. - """ - - def __init__(self): - self.tests = [] - self.text = "" - - def get_tests(self) -> list: - """ - Returns lists test objects. - """ - return self.tests - - def is_private(self) -> bool: - return all(test.private for test in self.tests) - - def __str__(self) -> str: - return "\n".join(str(test) for test in self.tests) - - def test_indices(self) -> List[int]: - return [test.index for test in self.tests] - - -class Documentation: - """ - `Documentation` describes an object containing the whole documentation system. - Documentation - | - +--------0> Parts - | - +-----0> Chapters - | - +-----0>Sections - | | - | +------0> SubSections - | - +---->0>GuideSections - | - +-----0>Sections - | - +------0> SubSections - - (with 0>) meaning "aggregation". - - Each element contains a title, a collection of elements of the following class - in the hierarchy. Parts, Chapters, Guide Sections, Sections and SubSections contains a doc - attribute describing the content to be shown after the title, and before - the elements of the subsequent terms in the hierarchy. - """ - - def __init__(self): - # This is a way to load the default classes - # without defining these attributes as class - # attributes. - self._set_classes() - self.parts = [] - self.appendix = [] - self.parts_by_slug = {} - self.title = "Title" - - def _set_classes(self): - """ - Set the classes of the subelements. Must be overloaded - by the subclasses. - """ - if not hasattr(self, "part_class"): - self.chapter_class = DocChapter - self.doc_class = DocumentationEntry - self.guide_section_class = DocGuideSection - self.part_class = DocPart - self.section_class = DocSection - self.subsection_class = DocSubsection - - def __str__(self): - result = self.title + "\n" + len(self.title) * "~" + "\n" - return ( - result + "\n\n".join([str(part) for part in self.parts]) + "\n" + 60 * "-" - ) - - def add_section( - self, - chapter, - section_name: str, - section_object, - operator, - is_guide: bool = False, - in_guide: bool = False, - summary_text="", - ): - """ - Adds a DocSection or DocGuideSection - object to the chapter, a DocChapter object. - "section_object" is either a Python module or a Class object instance. - """ - if section_object is not None: - required_libs = getattr(section_object, "requires", []) - installed = check_requires_list(required_libs) if required_libs else True - # FIXME add an additional mechanism in the module - # to allow a docstring and indicate it is not to go in the - # user manual - if not section_object.__doc__: - return - - else: - installed = True - - if is_guide: - section = self.guide_section_class( - chapter, - section_name, - section_object.__doc__, - section_object, - installed=installed, - ) - chapter.guide_sections.append(section) - else: - section = self.section_class( - chapter, - section_name, - section_object.__doc__, - operator=operator, - installed=installed, - in_guide=in_guide, - summary_text=summary_text, - ) - chapter.sections.append(section) - - return section - - def add_subsection( - self, - chapter, - section, - subsection_name: str, - instance, - operator=None, - in_guide=False, - ): - """ - Append a subsection for ``instance`` into ``section.subsections`` - """ - - required_libs = getattr(instance, "requires", []) - installed = check_requires_list(required_libs) if required_libs else True - - # FIXME add an additional mechanism in the module - # to allow a docstring and indicate it is not to go in the - # user manual - if not instance.__doc__: - return - summary_text = ( - instance.summary_text if hasattr(instance, "summary_text") else "" - ) - subsection = self.subsection_class( - chapter, - section, - subsection_name, - instance.__doc__, - operator=operator, - installed=installed, - in_guide=in_guide, - summary_text=summary_text, - ) - section.subsections.append(subsection) - - def doc_part(self, title, modules, builtins_by_module, start): - """ - Build documentation structure for a "Part" - Reference - section or collection of Mathics3 Modules. - """ - - builtin_part = self.part_class(self, title, is_reference=start) - - # This is used to ensure that we pass just once over each module. - # The algorithm we use to walk all the modules without repetitions - # relies on this, which in my opinion is hard to test and susceptible - # to errors. I guess we include it as a temporal fixing to handle - # packages inside ``mathics.builtin``. - modules_seen = set([]) - - def filter_toplevel_modules(module_list): - """ - Keep just the modules at the top level. - """ - if len(module_list) == 0: - return module_list - - modules_and_levels = sorted( - ((module.__name__.count("."), module) for module in module_list), - key=lambda x: x[0], - ) - top_level = modules_and_levels[0][0] - return (entry[1] for entry in modules_and_levels if entry[0] == top_level) - - # The loop to load chapters must be run over the top-level modules. Otherwise, - # modules like ``mathics.builtin.functional.apply_fns_to_lists`` are loaded - # as chapters and sections of a GuideSection, producing duplicated tests. - # - # Also, this provides a more deterministic way to walk the module hierarchy, - # which can be decomposed in the way proposed in #984. - - modules = filter_toplevel_modules(modules) - for module in sorted_modules(modules): - if skip_module_doc(module, modules_seen): - continue - chapter = self.doc_chapter(module, builtin_part, builtins_by_module) - if chapter is None: - continue - builtin_part.chapters.append(chapter) - - self.parts.append(builtin_part) - - def doc_chapter(self, module, part, builtins_by_module) -> Optional[DocChapter]: - """ - Build documentation structure for a "Chapter" - reference section which - might be a Mathics Module. - """ - modules_seen = set([]) - - title, text = get_module_doc(module) - chapter = self.chapter_class(part, title, self.doc_class(text, title, None)) - builtins = builtins_by_module.get(module.__name__) - if module.__file__.endswith("__init__.py"): - # We have a Guide Section. - - # This is used to check if a symbol is not duplicated inside - # a guide. - submodule_names_seen = set([]) - name = get_doc_name_from_module(module) - guide_section = self.add_section( - chapter, name, module, operator=None, is_guide=True - ) - submodules = [ - value - for value in module.__dict__.values() - if isinstance(value, ModuleType) - ] - - # Add sections in the guide section... - for submodule in sorted_modules(submodules): - if skip_module_doc(submodule, modules_seen): - continue - elif IS_PYPY and submodule.__name__ == "builtins": - # PyPy seems to add this module on its own, - # but it is not something that can be importable - continue - - submodule_name = get_doc_name_from_module(submodule) - if submodule_name in submodule_names_seen: - continue - section = self.add_section( - chapter, - submodule_name, - submodule, - operator=None, - is_guide=False, - in_guide=True, - ) - modules_seen.add(submodule) - submodule_names_seen.add(submodule_name) - guide_section.subsections.append(section) - - builtins = builtins_by_module.get(submodule.__name__, []) - subsections = [builtin for builtin in builtins] - for instance in subsections: - if hasattr(instance, "no_doc") and instance.no_doc: - continue - - name = instance.get_name(short=True) - if name in submodule_names_seen: - continue - - submodule_names_seen.add(name) - modules_seen.add(instance) - - self.add_subsection( - chapter, - section, - name, - instance, - instance.get_operator(), - in_guide=True, - ) - else: - if not builtins: - return None - sections = [ - builtin for builtin in builtins if not skip_doc(builtin.__class__) - ] - self.doc_sections(sections, modules_seen, chapter) - return chapter - - def doc_sections(self, sections, modules_seen, chapter): - for instance in sections: - if instance not in modules_seen and ( - not hasattr(instance, "no_doc") or not instance.no_doc - ): - name = instance.get_name(short=True) - summary_text = ( - instance.summary_text if hasattr(instance, "summary_text") else "" - ) - self.add_section( - chapter, - name, - instance, - instance.get_operator(), - is_guide=False, - in_guide=False, - summary_text=summary_text, - ) - modules_seen.add(instance) - - def get_part(self, part_slug): - return self.parts_by_slug.get(part_slug) - - def get_chapter(self, part_slug, chapter_slug): - part = self.parts_by_slug.get(part_slug) - if part: - return part.chapters_by_slug.get(chapter_slug) - return None - - def get_section(self, part_slug, chapter_slug, section_slug): - part = self.parts_by_slug.get(part_slug) - if part: - chapter = part.chapters_by_slug.get(chapter_slug) - if chapter: - return chapter.sections_by_slug.get(section_slug) - return None - - def get_subsection(self, part_slug, chapter_slug, section_slug, subsection_slug): - part = self.parts_by_slug.get(part_slug) - if part: - chapter = part.chapters_by_slug.get(chapter_slug) - if chapter: - section = chapter.sections_by_slug.get(section_slug) - if section: - return section.subsections_by_slug.get(subsection_slug) - - return None - - # FIXME: turn into a @property tests? - def get_tests(self) -> Iterator: - """ - Returns a generator to extracts lists test objects. - """ - for part in self.parts: - for chapter in sorted_chapters(part.chapters): - if MATHICS_DEBUG_TEST_CREATE: - print(f"DEBUG Gathering tests for Chapter {chapter.title}") - - tests = chapter.doc.get_tests() - if tests: - yield Tests(part.title, chapter.title, "", tests) - - for section in chapter.all_sections: - if section.installed: - if MATHICS_DEBUG_TEST_CREATE: - if isinstance(section, DocGuideSection): - print( - f"DEBUG Gathering tests for Guide Section {section.title}" - ) - else: - print( - f"DEBUG Gathering tests for Section {section.title}" - ) - - if isinstance(section, DocGuideSection): - for docsection in section.subsections: - for docsubsection in docsection.subsections: - # FIXME: Something is weird here where tests for subsection items - # appear not as a collection but individually and need to be - # iterated below. Probably some other code is faulty and - # when fixed the below loop and collection into doctest_list[] - # will be removed. - if not docsubsection.installed: - continue - doctest_list = [] - index = 1 - for doctests in docsubsection.items: - doctest_list += list(doctests.get_tests()) - for test in doctest_list: - test.index = index - index += 1 - - if doctest_list: - yield Tests( - section.chapter.part.title, - section.chapter.title, - docsubsection.title, - doctest_list, - ) - else: - tests = section.doc.get_tests() - if tests: - yield Tests( - part.title, chapter.title, section.title, tests - ) - pass - pass - pass - pass - pass - pass - return - - def load_documentation_sources(self): - """ - Extract doctest data from various static XML-like doc files, Mathics3 Built-in functions - (inside mathics.builtin), and external Mathics3 Modules. - - The extracted structure is stored in ``self``. - """ - assert ( - len(self.parts) == 0 - ), "The documentation must be empty to call this function." - - # First gather data from static XML-like files. This constitutes "Part 1" of the - # documentation. - files = listdir(self.doc_dir) - files.sort() - - chapter_order = 0 - for file in files: - part_title = file[2:] - if part_title.endswith(".mdoc"): - part_title = part_title[: -len(".mdoc")] - # If the filename start with a number, then is a main part. Otherwise - # is an appendix. - is_appendix = not file[0].isdigit() - chapter_order = self.load_part_from_file( - osp.join(self.doc_dir, file), - part_title, - chapter_order, - is_appendix, - ) - - # Next extract data that has been loaded into Mathics3 when it runs. - # This is information from `mathics.builtin`. - # This is Part 2 of the documentation. - - # Notice that in order to generate the documentation - # from the builtin classes, it is needed to call first to - # import_and_load_builtins() - - for title, modules, builtins_by_module, start in [ - ( - "Reference of Built-in Symbols", - mathics3_builtins_modules, - global_builtins_by_module, - True, - ) - ]: - self.doc_part(title, modules, builtins_by_module, start) - - # Next extract external Mathics3 Modules that have been loaded via - # LoadModule, or eval_LoadModule. This is Part 3 of the documentation. - - self.doc_part( - MATHICS3_MODULES_TITLE, - pymathics_modules, - pymathics_builtins_by_module, - True, - ) - - # Finally, extract Appendix information. This include License text - # This is the final Part of the documentation. - - for part in self.appendix: - self.parts.append(part) - - # Via the wanderings above, collect all tests that have been - # seen. - # - # Each test is accessble by its part + chapter + section and test number - # in that section. - for tests in self.get_tests(): - for test in tests.tests: - test.key = (tests.part, tests.chapter, tests.section, test.index) - return - - def load_part_from_file( - self, filename: str, title: str, chapter_order: int, is_appendix: bool = False - ) -> int: - """Load a markdown file as a part of the documentation""" - part = self.part_class(self, title) - text = open(filename, "rb").read().decode("utf8") - text = filter_comments(text) - chapters = CHAPTER_RE.findall(text) - for title, text in chapters: - chapter = self.chapter_class(part, title, chapter_order=chapter_order) - chapter_order += 1 - text += '
    ' - section_texts = SECTION_RE.findall(text) - for pre_text, title, text in section_texts: - if title: - section = self.section_class( - chapter, title, text, operator=None, installed=True - ) - chapter.sections.append(section) - subsections = SUBSECTION_RE.findall(text) - for subsection_title in subsections: - subsection = self.subsection_class( - chapter, - section, - subsection_title, - text, - ) - section.subsections.append(subsection) - pass - pass - else: - section = None - if not chapter.doc: - chapter.doc = self.doc_class(pre_text, title, section) - pass - - part.chapters.append(chapter) - if is_appendix: - part.is_appendix = True - self.appendix.append(part) - else: - self.parts.append(part) - return chapter_order - - -class DocSubsection: - """An object for a Documented Subsection. - A Subsection is part of a Section. - """ - - def __init__( - self, - chapter, - section, - title, - text, - operator=None, - installed=True, - in_guide=False, - summary_text="", - ): - """ - Information that goes into a subsection object. This can be a written text, or - text extracted from the docstring of a builtin module or class. - - About some of the parameters... - - Some subsections are contained in a grouping module and need special work to - get the grouping module name correct. - - For example the Chapter "Colors" is a module so the docstring text for it is in - mathics/builtin/colors/__init__.py . In mathics/builtin/colors/named-colors.py we have - the "section" name for the class Read (the subsection) inside it. - """ - title_summary_text = re.split(" -- ", title) - n = len(title_summary_text) - self.title = title_summary_text[0] if n > 0 else "" - self.summary_text = title_summary_text[1] if n > 1 else summary_text - - self.doc = DocumentationEntry(text, title, section) - self.chapter = chapter - self.in_guide = in_guide - self.installed = installed - self.operator = operator - - self.section = section - self.slug = slugify(title) - self.subsections = [] - self.title = title - - if section: - chapter = section.chapter - part = chapter.part - # Note: we elide section.title - key_prefix = (part.title, chapter.title, title) - else: - key_prefix = None - - if in_guide: - # Tests haven't been picked out yet from the doc string yet. - # Gather them here. - self.items = parse_docstring_to_DocumentationEntry_items( - text, DocTests, DocTest, DocText, key_prefix - ) - else: - self.items = [] - - if text.count("
    ") != text.count("
    "): - raise ValueError( - "Missing opening or closing
    tag in " - "{} documentation".format(title) - ) - self.section.subsections_by_slug[self.slug] = self - if MATHICS_DEBUG_DOC_BUILD: - print(" DEBUG Creating Subsection", title) - - def __str__(self) -> str: - return f"=== {self.title} ===\n{self.doc}" - - def get_tests(self): - """yield tests""" - if self.installed: - for test in self.doc.get_tests(): - yield test - - -class MathicsMainDocumentation(Documentation): - """ - MathicsMainDocumentation specializes ``Documentation`` by providing the attributes - and methods needed to generate the documentation from the Mathics library. - - The parts of the documentation are loaded from the Markdown files contained - in the path specified by ``self.doc_dir``. Files with names starting in numbers - are considered parts of the main text, while those that starts with other characters - are considered as appendix parts. - - In addition to the parts loaded from markdown files, a ``Reference of Builtin-Symbols`` part - and a part for the loaded Pymathics modules are automatically generated. - - In the ``Reference of Built-in Symbols`` tom-level modules and files in ``mathics.builtin`` - are associated to Chapters. For single file submodules (like ``mathics.builtin.procedure``) - The chapter contains a Section for each Symbol in the module. For sub-packages - (like ``mathics.builtin.arithmetic``) sections are given by the sub-module files, - and the symbols in these sub-packages defines the Subsections. ``__init__.py`` in - subpackages are associated to GuideSections. - - In a similar way, in the ``Pymathics`` part, each ``pymathics`` module defines a Chapter, - files in the module defines Sections, and Symbols defines Subsections. - - - ``MathicsMainDocumentation`` is also used for creating test data and saving it to a - Python Pickle file and running tests that appear in the documentation (doctests). - - There are other classes DjangoMathicsDocumentation and LaTeXMathicsDocumentation - that format the data accumulated here. In fact I think those can sort of serve - instead of this. - - """ - - def __init__(self): - super().__init__() - - self.doc_dir = settings.DOC_DIR - self.doctest_latex_pcl_path = settings.DOCTEST_LATEX_DATA_PCL - self.pymathics_doc_loaded = False - self.doc_data_file = settings.get_doctest_latex_data_path( - should_be_readable=True - ) - self.title = "Mathics Main Documentation" - - def gather_doctest_data(self): - """ - Populates the documentatation. - (deprecated) - """ - logging.warn( - "gather_doctest_data is deprecated. Use load_documentation_sources" - ) - return self.load_documentation_sources() - - -class DocText: - """ - Class to hold some (non-test) text. - - Some of the kinds of tags you may find here are showin in global ALLOWED_TAGS. - Some text may be marked with surrounding "$" or "'". - - The code here however does not make use of any of the tagging. - - """ - - def __init__(self, text): - self.text = text - - def __str__(self) -> str: - return self.text - - def get_tests(self) -> list: - """ - Return tests in a DocText item - there never are any. - """ - return [] - - def is_private(self) -> bool: - return False - - def test_indices(self) -> List[int]: - return [] - - -# Former XMLDoc -class DocumentationEntry: - """ - A class to hold the content of a documentation entry, - in our internal markdown-like format data. - - Describes the contain of an entry in the documentation system, as a - sequence (list) of items of the clase `DocText` and `DocTests`. - ``DocText`` items contains an internal XML-like formatted text. ``DocTests`` entries - contain one or more `DocTest` element. - Each level of the Documentation hierarchy contains an XMLDoc, describing the - content after the title and before the elements of the next level. For example, - in ``DocChapter``, ``DocChapter.doc`` contains the text coming after the title - of the chapter, and before the sections in `DocChapter.sections`. - Specialized classes like LaTeXDoc or and DjangoDoc provide methods for - getting formatted output. For LaTeXDoc ``latex()`` is added while for - DjangoDoc ``html()`` is added - Mathics core also uses this in getting usage strings (`??`). - - """ - - def __init__(self, doc_str: str, title: str, section: Optional[DocSection] = None): - self._set_classes() - self.title = title - if section: - chapter = section.chapter - part = chapter.part - # Note: we elide section.title - key_prefix = (part.title, chapter.title, title) - else: - key_prefix = None - - self.rawdoc = doc_str - self.items = parse_docstring_to_DocumentationEntry_items( - self.rawdoc, - self.docTest_collection_class, - self.docTest_class, - self.docText_class, - key_prefix, - ) - - def _set_classes(self): - """ - Tells to the initializator the classes to be used to build the items. - This must be overloaded by the daughter classes. - """ - if not hasattr(self, "docTest_collection_class"): - self.docTest_collection_class = DocTests - self.docTest_class = DocTest - self.docText_class = DocText - - def __str__(self) -> str: - return "\n\n".join(str(item) for item in self.items) - - def text(self) -> str: - # used for introspection - # TODO parse XML and pretty print - # HACK - item = str(self.items[0]) - item = "\n".join(line.strip() for line in item.split("\n")) - item = item.replace("
    ", "") - item = item.replace("
    ", "") - item = item.replace("
    ", " ") - item = item.replace("
    ", "") - item = item.replace("
    ", " ") - item = item.replace("
    ", "") - item = "\n".join(line for line in item.split("\n") if not line.isspace()) - return item - - def get_tests(self) -> list: - tests = [] - for item in self.items: - tests.extend(item.get_tests()) - return tests - - -# Backward compatibility gather_tests = parse_docstring_to_DocumentationEntry_items XMLDOC = DocumentationEntry + +from mathics.doc.structure import ( + MATHICS3_MODULES_TITLE, + SUBSECTION_END_RE, + SUBSECTION_RE, + DocChapter, + DocGuideSection, + DocPart, + DocSection, + DocSubsection, + Documentation, + MathicsMainDocumentation, + sorted_chapters, +) diff --git a/mathics/doc/doc_entries.py b/mathics/doc/doc_entries.py new file mode 100644 index 000000000..114343180 --- /dev/null +++ b/mathics/doc/doc_entries.py @@ -0,0 +1,612 @@ +""" +Documentation entries and doctests + +This module contains the objects representing the entries in the documentation +system, and the functions used to parse docstrings into these objects. + + +""" + +import logging +import re +from os import getenv +from typing import Callable, List, Optional + +from mathics.core.evaluation import Message, Print + +# Used for getting test results by test expression and chapter/section information. +test_result_map = {} + + +# These are all the XML/HTML-like tags that documentation supports. +ALLOWED_TAGS = ( + "dl", + "dd", + "dt", + "em", + "url", + "ul", + "i", + "ol", + "li", + "con", + "console", + "img", + "imgpng", + "ref", + "subsection", +) +ALLOWED_TAGS_RE = dict( + (allowed, re.compile("<(%s.*?)>" % allowed)) for allowed in ALLOWED_TAGS +) + +# This string is used, so we can indicate a trailing blank at the end of a line by +# adding this string to the end of the line which gets stripped off. +# Some editors and formatters like to strip off trailing blanks at the ends of lines. +END_LINE_SENTINAL = "#<--#" + +# The regular expressions below (strings ending with _RE +# pull out information from docstring or text in a file. Ghetto parsing. + +CONSOLE_RE = re.compile(r"(?s)<(?Pcon|console)>(?P.*?)") +DL_ITEM_RE = re.compile( + r"(?s)<(?Pd[td])>(?P.*?)(?:|)\s*(?:(?=)|$)" +) +DL_RE = re.compile(r"(?s)
    (.*?)
    ") +HYPERTEXT_RE = re.compile( + r"(?s)<(?Pem|url)>(\s*:(?P.*?):\s*)?(?P.*?)" +) +IMG_PNG_RE = re.compile( + r'' +) +IMG_RE = re.compile( + r'' +) +# Preserve space before and after in-line code variables. +LATEX_RE = re.compile(r"(\s?)\$(\w+?)\$(\s?)") + +LIST_ITEM_RE = re.compile(r"(?s)
  • (.*?)(?:
    • |(?=
  • )|$)") +LIST_RE = re.compile(r"(?s)<(?Pul|ol)>(?P.*?)") +MATHICS_RE = re.compile(r"(?(.*?)") +QUOTATIONS_RE = re.compile(r"\"([\w\s,]*?)\"") +REF_RE = re.compile(r'') +SPECIAL_COMMANDS = { + "LaTeX": (r"LaTeX", r"\LaTeX{}"), + "Mathematica": ( + r"Mathematica®", + r"\emph{Mathematica}\textregistered{}", + ), + "Mathics": (r"Mathics3", r"\emph{Mathics3}"), + "Mathics3": (r"Mathics3", r"\emph{Mathics3}"), + "Sage": (r"Sage", r"\emph{Sage}"), + "Wolfram": (r"Wolfram", r"\emph{Wolfram}"), + "skip": (r"

    ", r"\bigskip"), +} + + +TESTCASE_RE = re.compile( + r"""(?mx)^ # re.MULTILINE (multi-line match) + # and re.VERBOSE (readable regular expressions + ((?:.|\n)*?) + ^\s+([>#SX])>[ ](.*) # test-code indicator + ((?:\n\s*(?:[:|=.][ ]|\.).*)*) # test-code results""" +) +TESTCASE_OUT_RE = re.compile(r"^\s*([:|=])(.*)$") + + +def get_results_by_test(test_expr: str, full_test_key: list, doc_data: dict) -> dict: + """ + Sometimes test numbering is off, either due to bugs or changes since the + data was read. + + Here, we compensate for this by looking up the test by its chapter and section name + portion stored in `full_test_key` along with the and the test expression data + stored in `test_expr`. + + This new key is looked up in `test_result_map` its value is returned. + + `doc_data` is only first time this is called to populate `test_result_map`. + """ + + # Strip off the test index form new key with this and the test string. + # Add to any existing value for that "result". This is now what we want to + # use as a tee in test_result_map to look for. + test_section = list(full_test_key)[:-1] + search_key = tuple(test_section) + + if not test_result_map: + # Populate test_result_map from doc_data + for key, result in doc_data.items(): + test_section = list(key)[:-1] + new_test_key = tuple(test_section) + next_result = test_result_map.get(new_test_key, None) + if next_result is None: + next_result = [result] + else: + next_result.append(result) + + test_result_map[new_test_key] = next_result + + results = test_result_map.get(search_key, None) + result = {} + if results: + for result_candidate in results: + if result_candidate["query"] == test_expr: + if result: + # Already found something + logging.warning( + f"Warning, multiple results appear under {search_key}." + ) + return {} + + result = result_candidate + + return result + + +def filter_comments(doc: str) -> str: + """Remove docstring documentation comments. These are lines + that start with ##""" + return "\n".join( + line for line in doc.splitlines() if not line.lstrip().startswith("##") + ) + + +POST_SUBSTITUTION_TAG = "_POST_SUBSTITUTION%d_" + + +def pre_sub(regexp, text: str, repl_func): + """apply substitutions previous to parse the text""" + post_substitutions = [] + + def repl_pre(match): + repl = repl_func(match) + index = len(post_substitutions) + post_substitutions.append(repl) + return POST_SUBSTITUTION_TAG % index + + text = regexp.sub(repl_pre, text) + + return text, post_substitutions + + +def post_sub(text: str, post_substitutions) -> str: + """apply substitutions after parsing the doctests.""" + for index, sub in enumerate(post_substitutions): + text = text.replace(POST_SUBSTITUTION_TAG % index, sub) + return text + + +def parse_docstring_to_DocumentationEntry_items( + doc: str, + test_collection_constructor: Callable, + test_case_constructor: Callable, + text_constructor: Callable, + key_part=None, +) -> list: + """ + This parses string `doc` (using regular expressions) into Python objects. + The function returns a list of ``DocText`` and ``DocTests`` objects which + are contained in a ``DocumentationElement``. + + test_collection_constructor() is the class constructor call to create an + object for the test collection. + Each test is created via test_case_constructor(). + Text within the test is stored via text_constructor. + """ + # This function is used to populate a ``DocumentEntry`` element, that + # in principle is not associated to any container + # (``DocChapter``/``DocSection``/``DocSubsection``) + # of the documentation system. + # + # The ``key_part`` parameter was used to set the ``key`` of the + # ``DocTest`` elements. This attribute + # should be set just after the ``DocumentationEntry`` ( + # to which the tests belongs) is associated + # to a container, by calling ``container.set_parent_path``. + # However, the parameter is still used in MathicsDjango, so let's + # keep it and discard its value. + # + if key_part: + logging.warning("``key_part`` is deprecated. Its value is discarded.") + + # Remove commented lines. + doc = filter_comments(doc).strip(r"\s") + + # Remove leading
    ...
    + # doc = DL_RE.sub("", doc) + + # pre-substitute Python code because it might contain tests + doc, post_substitutions = pre_sub( + PYTHON_RE, doc, lambda m: "%s" % m.group(1) + ) + + # HACK: Artificially construct a last testcase to get the "intertext" + # after the last (real) testcase. Ignore the test, of course. + doc += "\n >> test\n = test" + testcases = TESTCASE_RE.findall(doc) + + tests = None + items = [] + for index, test_case in enumerate(testcases): + testcase = list(test_case) + text = testcase.pop(0).strip() + if text: + if tests is not None: + items.append(tests) + tests = None + text = post_sub(text, post_substitutions) + items.append(text_constructor(text)) + tests = None + if index < len(testcases) - 1: + test = test_case_constructor(index, testcase, None) + if tests is None: + tests = test_collection_constructor() + tests.tests.append(test) + + # If the last block in the loop was not a Text block, append the + # last set of tests. + if tests is not None: + items.append(tests) + tests = None + return items + + +class DocTest: + """ + Class to hold a single doctest. + + DocTest formatting rules: + + * `>>` Marks test case; it will also appear as part of + the documentation. + * `#>` Marks test private or one that does not appear as part of + the documentation. + * `X>` Shows the example in the docs, but disables testing the example. + * `S>` Shows the example in the docs, but disables testing if environment + variable SANDBOX is set. + * `=` Compares the result text. + * `:` Compares an (error) message. + `|` Prints output. + """ + + def __init__( + self, + index: int, + testcase: List[str], + key_prefix: Optional[tuple] = None, + ): + def strip_sentinal(line: str): + """Remove END_LINE_SENTINAL from the end of a line if it appears. + + Some editors like to strip blanks at the end of a line. + Since the line ends in END_LINE_SENTINAL which isn't blank, + any blanks that appear before will be preserved. + + Some tests require some lines to be blank or entry because + Mathics3 output can be that way + """ + if line.endswith(END_LINE_SENTINAL): + line = line[: -len(END_LINE_SENTINAL)] + + # Also remove any remaining trailing blanks since that + # seems *also* what we want to do. + return line.strip() + + self.index = index + self.outs = [] + self.result = None + + # Private test cases are executed, but NOT shown as part of the docs + self.private = testcase[0] == "#" + + # Ignored test cases are NOT executed, but shown as part of the docs + # Sandboxed test cases are NOT executed if environment SANDBOX is set + if testcase[0] == "X" or (testcase[0] == "S" and getenv("SANDBOX", False)): + self.ignore = True + # substitute '>' again so we get the correct formatting + testcase[0] = ">" + else: + self.ignore = False + + self.test = strip_sentinal(testcase[1]) + self._key = key_prefix + (index,) if key_prefix else None + + outs = testcase[2].splitlines() + for line in outs: + line = strip_sentinal(line) + if line: + if line.startswith("."): + text = line[1:] + if text.startswith(" "): + text = text[1:] + text = "\n" + text + if self.result is not None: + self.result += text + elif self.outs: + self.outs[-1].text += text + continue + + match = TESTCASE_OUT_RE.match(line) + if not match: + continue + symbol, text = match.group(1), match.group(2) + text = text.strip() + if symbol == "=": + self.result = text + elif symbol == ":": + out = Message("", "", text) + self.outs.append(out) + elif symbol == "|": + out = Print(text) + self.outs.append(out) + + def __str__(self) -> str: + return self.test + + def compare(self, result: Optional[str], out: Optional[tuple] = tuple()) -> bool: + """ + Performs a doctest comparison between ``result`` and ``wanted`` and returns + True if the test should be considered a success. + """ + return self.compare_result(result) and self.compare_out(out) + + def compare_result(self, result: Optional[str]): + """Compare a result with the expected result""" + wanted = self.result + # Check result + if wanted in ("...", result): + return True + + if result is None or wanted is None: + return False + result_list = result.splitlines() + wanted_list = wanted.splitlines() + if result_list == [] and wanted_list == ["#<--#"]: + return True + + if len(result_list) != len(wanted_list): + return False + + for res, want in zip(result_list, wanted_list): + wanted_re = re.escape(want.strip()) + wanted_re = wanted_re.replace("\\.\\.\\.", ".*?") + wanted_re = f"^{wanted_re}$" + if not re.match(wanted_re, res.strip()): + return False + return True + + def compare_out(self, outs: tuple = tuple()) -> bool: + """Compare messages and warnings produced during the evaluation of + the test with the expected messages and warnings.""" + # Check out + wanted_outs = self.outs + if len(wanted_outs) == 1 and wanted_outs[0].text == "...": + # If we have ... don't check + return True + if len(outs) != len(wanted_outs): + # Mismatched number of output lines, and we don't have "..." + return False + + # Need to check all output line by line + for got, wanted in zip(outs, wanted_outs): + if wanted.text == "...": + return True + if not got == wanted: + return False + + return True + + @property + def key(self): + """key identifier of the test""" + return self._key if hasattr(self, "_key") else None + + @key.setter + def key(self, value): + """setter for the key identifier of the test""" + assert self.key is None + self._key = value + return self._key + + +class DocTests: + """ + A bunch of consecutive ``DocTest`` objects extracted from a Builtin docstring. + """ + + def __init__(self): + self.tests = [] + self.text = "" + + def get_tests(self) -> list: + """ + Returns lists test objects. + """ + return self.tests + + def is_private(self) -> bool: + """Returns True if this test is "private" not supposed to be visible as example documentation.""" + return all(test.private for test in self.tests) + + def __str__(self) -> str: + return "\n".join(str(test) for test in self.tests) + + def test_indices(self) -> List[int]: + """indices of the tests""" + return [test.index for test in self.tests] + + +class DocText: + """ + Class to hold some (non-test) text. + + Some of the kinds of tags you may find here are showing in global ALLOWED_TAGS. + Some text may be marked with surrounding "$" or "'". + + The code here however does not make use of any of the tagging. + + """ + + def __init__(self, text): + self.text = text + + def __str__(self) -> str: + return self.text + + def get_tests(self) -> list: + """ + Return tests in a DocText item - there never are any. + """ + return [] + + def is_private(self) -> bool: + """the test is private, meaning that it will not be included in the + documentation, but tested in the doctest cycle.""" + return False + + def test_indices(self) -> List[int]: + """indices of the tests""" + return [] + + +# Former XMLDoc +class DocumentationEntry: + """ + A class to hold the content of a documentation entry, + in our custom XML-like format. + + Describes the contain of an entry in the documentation system, as a + sequence (list) of items of the clase `DocText` and `DocTests`. + ``DocText`` items contains an internal XML-like formatted text. ``DocTests`` entries + contain one or more `DocTest` element. + Each level of the Documentation hierarchy contains an XMLDoc, describing the + content after the title and before the elements of the next level. For example, + in ``DocChapter``, ``DocChapter.doc`` contains the text coming after the title + of the chapter, and before the sections in `DocChapter.sections`. + Specialized classes like LaTeXDoc or and DjangoDoc provide methods for + getting formatted output. For LaTeXDoc ``latex()`` is added while for + DjangoDoc ``html()`` is added + Mathics core also uses this in getting usage strings (`??`). + + """ + + def __init__( + self, doc_str: str, title: str, section: Optional["DocSection"] = None + ): + self._set_classes() + self.title = title + self.path = None + if section: + chapter = section.chapter + part = chapter.part + # Note: we elide section.title + key_prefix = (part.title, chapter.title, title) + else: + key_prefix = None + + self.key_prefix = key_prefix + self.rawdoc = doc_str + self.items = parse_docstring_to_DocumentationEntry_items( + self.rawdoc, + self.docTest_collection_class, + self.docTest_class, + self.docText_class, + None, + ) + + def _set_classes(self): + """ + Tells to the initializator the classes to be used to build the items. + This must be overloaded by the daughter classes. + """ + if not hasattr(self, "docTest_collection_class"): + self.docTest_collection_class = DocTests + self.docTest_class = DocTest + self.docText_class = DocText + + def __str__(self) -> str: + return "\n\n".join(str(item) for item in self.items) + + def text(self) -> str: + """text version of the documentation entry""" + # used for introspection + # TODO parse XML and pretty print + # HACK + item = str(self.items[0]) + item = "\n".join(line.strip() for line in item.split("\n")) + item = item.replace("
    ", "") + item = item.replace("
    ", "") + item = item.replace("
    ", " ") + item = item.replace("
    ", "") + item = item.replace("
    ", " ") + item = item.replace("
    ", "") + item = "\n".join(line for line in item.split("\n") if not line.isspace()) + return item + + def get_tests(self) -> list: + """retrieve a list of tests in the documentation entry""" + tests = [] + for item in self.items: + tests.extend(item.get_tests()) + return tests + + def set_parent_path(self, parent): + """Set the parent path""" + self.path = None + path = [] + while hasattr(parent, "parent"): + path = [parent.title] + path + parent = parent.parent + + if hasattr(parent, "title"): + path = [parent.title] + path + + if path: + self.path = path + # Set the key on each test + for test in self.get_tests(): + assert test.key is None + # For backward compatibility, we need + # to reduce this to three fields. + # TODO: remove me and ensure that this + # works here and in Mathics Django + if len(path) > 3: + path = path[:2] + [path[-1]] + test.key = tuple(path) + (test.index,) + + return self + + +class Tests: + """ + A group of tests in the same section or subsection. + """ + + def __init__( + self, + part_name: str, + chapter_name: str, + section_name: str, + doctests: List[DocTest], + subsection_name: Optional[str] = None, + ): + self.part = part_name + self.chapter = chapter_name + self.section = section_name + self.subsection = subsection_name + self.tests = doctests + self._key = None + + @property + def key(self): + """key of the tests""" + return self._key + + @key.setter + def key(self, value): + assert self._key is None + self._key = value + return self._key diff --git a/mathics/doc/documentation/1-Manual.mdoc b/mathics/doc/documentation/1-Manual.mdoc index d3357e2cd..f878f2660 100644 --- a/mathics/doc/documentation/1-Manual.mdoc +++ b/mathics/doc/documentation/1-Manual.mdoc @@ -1,14 +1,25 @@ + -\Mathics---to be pronounced like "Mathematics" without the "emat"---is a general-purpose computer algebra system (CAS). It is meant to be a free, open-source alternative to \Mathematica. It is free both as in "free beer" and as in "freedom". Mathics can be run \Mathics locally, and to facilitate installation of the vast amount of software need to run this, there is a :docker image available on dockerhub: https://hub.docker.com/r/mathicsorg/mathics. +\Mathics---to be pronounced like "Mathematics" without the "emat"---is +a :computer algebra +system:https://en.wikipedia.org/wiki/Computer_algebra_system. It +is a free, open-source alternative to \Mathematica or the \Wolfram +Language. However, \Mathics builds around and on top of the Python +ecosystem of libraries and tools. So in a sense, you can think of it +as a WMA front-end to the Python ecosystem of tools. + +\Mathics is free both as in "free beer" but also, more importantly, as in "freedom". \Mathics can be run locally. But to facilitate installation of the vast amount of software need to run this, there is a :docker image available on dockerhub: https://hub.docker.com/r/mathicsorg/mathics. + +The programming language and built-in functions of \Mathics tries to match the \Wolfram Language, which is continually evolving changing. -The programming language of \Mathics is meant to resemble the \Wolfram Language as much as possible. However, \Mathics is in no way affiliated or supported by \Wolfram. \Mathics will probably never have the power to compete with \Mathematica in industrial applications; it is an alternative though. It also invites community development at all levels. +\Mathics is in no way affiliated or supported by \Wolfram. \Mathics will probably never have the power to compete with \Mathematica in industrial applications; it is a free alternative though. It also invites community development at all levels. See the :installation instructions: https://mathics-development-guide.readthedocs.io/en/latest/installing/index.html for the most recent instructions for installing from PyPI, or the source. -For implementation details see https://mathics-development-guide.readthedocs.io/en/latest/. +For implementation details, plrease refer to the :Developers Guide:https://mathics-development-guide.readthedocs.io/en/latest/. -
    +
    \Mathematica is great, but it a couple of disadvantages.
      @@ -21,13 +32,13 @@ The second point some may find and advantage. However, even if you are willing to pay hundreds of dollars for the software, you would will not be able to see what\'s going on "inside" the program if that is your interest. That\'s what free, open-source, and community-supported software is for! -\Mathics aims at combining the best of both worlds: the beauty of \Mathematica backed by a free, extensible Python core which includes a rich set of Python tools including: +\Mathics combines the beauty of \Mathematica implemented in an open-source environment written in Python. The Python ecosystem includes libraries and toos like:
      • :mpmath: https://mpmath.org/ for floating-point arithmetic with arbitrary precision, -
      • :numpy: https://numpy.org/numpy for numeric computation, +
      • :NumPy: https://numpy.org for numeric computation,
      • :SymPy: https://sympy.org for symbolic mathematics, and -
      • optionally :SciPy: https://www.scipy.org/ for Scientific calculations. +
      • :SciPy: https://www.scipy.org/ for Scientific calculations.
      Performance of \Mathics is not, right now, practical in large-scale projects and calculations. However can be used as a tool for exploration and education. @@ -36,16 +47,16 @@ Performance of \Mathics is not, right now, practical in large-scale projects and
      -Some of the features of \Mathics tries to be compatible with Wolfram-Language kernel within the confines of the Python ecosystem. - -Given this, it is a powerful functional programming language, driven by pattern matching and rule application. +Because \Mathics is compatible with Wolfram-Language kernel within the +confines of the Python ecosystem, it is a powerful functional +programming language, driven by pattern matching and rule application. Primitive types include rationals, complex numbers, and arbitrary-precision numbers. Other primitive types such as images or graphs, or NLP come from the various Python libraries that \Mathics uses. Outside of the "core" \Mathics kernel (which has a only primitive command-line interface), in separate github projects, as add-ons, there is:
        -
      • a Django-based web server +
      • a :Django-based web server:https://pypi.org/project/Mathics-Django/
      • a command-line interface using either prompt-toolkit, or GNU Readline
      • a :Mathics3 module for Graphs:https://pypi.org/project/pymathics-graph/ (via :NetworkX:https://networkx.org/),
      • a :Mathics3 module for NLP:https://pypi.org/project/pymathics-natlang/ (via :nltk:https://www.nltk.org/, :spacy:https://spacy.io/, and others) @@ -61,7 +72,7 @@ After that, Angus Griffith took over primary leadership and rewrote the parser t A :docker image of the v.9 release: https://hub.docker.com/r/arkadi/mathics can be found on dockerhub. -Around 2017, the project was largely abandoned in its largely Python 2.7 state, with support for Python 3.2-3.5 via six. +Around 2017, the project was largely abandoned in its largely Python 2.7 state, with some support for Python 3.2-3.5 via six. Subsequently, around mid 2020, it was picked up by the current developers. A list of authors and contributors can be found in the :AUTHORS.txt: @@ -93,7 +104,9 @@ See :The Mathics3 Developer Guide:https://mathics-development-guide.readthe The following sections are introductions to the basic principles of the language of \Mathics. A few examples and functions are presented. Only their most common usages are listed; for a full description of a Symbols possible arguments, options, etc., see its entry in the Reference of Built-in Symbols. -However if you google for "Mathematica Tutorials" you will find easily dozens of other tutorials which are applicable. Be warned though that \Mathics does not yet offer the full range and features and capabilities of \Mathematica. +However if you google for "Mathematica Tutorials" you will find easily dozens of other tutorials which are applicable. For example, see :An Elementary Introduction to the Wolfram Language:https://www.wolfram.com/language/elementary-introduction/. In the :docker image that we supply:https://hub.docker.com/r/mathicsorg/mathics, you can load "workspaces" containing the examples described in the chapters of this introduction. + +Be warned though that \Mathics does not yet offer the full range and features and capabilities of \Mathematica.
        \Mathics can be used to calculate basic stuff: @@ -168,29 +181,41 @@ Of course, \Mathics has complex numbers: = 5 \Mathics can operate with pretty huge numbers: - >> 100! - = 93326215443944152681699238856266700490715968264381621468592963895217599993229915608941463976156518286253697920827223758251185210916864000000000000000000000000 + >> 55! (* Also known as Factorial[55] *) + = 12696403353658275925965100847566516959580321051449436762275840000000000000 + +We could easily increase use a number larger than 55, but the digits will just run off the page. -('!' denotes the factorial function.) The precision of numerical evaluation can be set: >> N[Pi, 30] = 3.14159265358979323846264338328 -Division by zero is forbidden: +Division by zero gives an error: >> 1 / 0 : Infinite expression 1 / 0 encountered. = ComplexInfinity -Other expressions involving 'Infinity' are evaluated: +But zero division returns value :'ComplexInfinity':/doc/reference-of-built-in-symbols/integer-and-number-theoretical-functions/mathematical-constants/complexinfinity and that can be used as a value: + + >> Cos[ComplexInfinity] + = Indeterminate + +'ComplexInfinity' is a shorthand though for 'DirectedInfinty[]'. + +Similarly, expressions using :'Infinity':/doc/reference-of-built-in-symbols/integer-and-number-theoretical-functions/mathematical-constants/complexinfinity as a value are allowed and are evaluated: >> Infinity + 2 Infinity = Infinity -In contrast to combinatorial belief, '0^0' is undefined: +There is also the value, :'Indeterminate':/doc/reference-of-built-in-symbols/integer-and-number-theoretical-functions/mathematical-constants/indeterminate: + >> 0 ^ 0 : Indeterminate expression 0 ^ 0 encountered. = Indeterminate + +
        @@ -216,7 +243,8 @@ The relative uncertainty of '3.1416`3' is 10^-3. It is numerically equivalent, i >> 3.1416`3 == 3.1413`4 = True -We can get the precision of the number by using the \Mathics Built-in function :'Precision': /doc/reference-of-built-in-symbols/atomic-elements-of-expressions/representation-of-numbers/precision/: + +We can get the precision of the number by using the \Mathics Built-in function :'Precision': /doc/reference-of-built-in-symbols/atomic-elements-of-expressions/representation-of-numbers/precision: >> Precision[3.1413`4] = 4. @@ -226,14 +254,16 @@ While 3.1419 not the closest approximation to Pi in 4 digits after the decimal p >> Pi == 3.141987654321`3 = True -The absolute accuracy of a number, is set by adding a two RawBackquotes '``' and the number digits. +The absolute accuracy of a number, is set by adding a two RawBackquotes '``' and the number digits. For example: >> 13.1416``4 = 13.142 -is a number having a absolute uncertainty of 10^-4. This number is numerically equivalent to '13.1413``4': +is a number having an absolute uncertainty of $10^-4$. + +This number is numerically equivalent to '13.1413``4': >> 13.1416``4 == 13.1413``4 = True @@ -983,6 +1013,7 @@ Colors can be added in the list of graphics primitives to change the drawing col
        'GrayLevel[$l$]'
        specifies a color using a gray level.
    • + All components range from 0 to 1. Each color function can be supplied with an additional argument specifying the desired opacity ("alpha") of the color. There are many predefined colors, such as 'Black', 'White', 'Red', 'Green', 'Blue', etc. >> Graphics[{Red, Disk[]}] @@ -1216,6 +1247,7 @@ We want to combine 'Dice' objects using the '+' operator: >> Dice[a___] + Dice[b___] ^:= Dice[Sequence @@ {a, b}] The '^:=' ('UpSetDelayed') tells \Mathics to associate this rule with 'Dice' instead of 'Plus'. + 'Plus' is protected---we would have to unprotect it first: >> Dice[a___] + Dice[b___] := Dice[Sequence @@ {a, b}] : Tag Plus in Dice[a___] + Dice[b___] is Protected. diff --git a/mathics/doc/gather.py b/mathics/doc/gather.py new file mode 100644 index 000000000..88d01e2b9 --- /dev/null +++ b/mathics/doc/gather.py @@ -0,0 +1,374 @@ +# -*- coding: utf-8 -*- +""" +Gather module information + +Functions used to build the reference sections from module information. + +""" + +import importlib +import os.path as osp +import pkgutil +from os import listdir +from types import ModuleType +from typing import Tuple, Union + +from mathics.core.builtin import Builtin, check_requires_list +from mathics.core.util import IS_PYPY +from mathics.doc.doc_entries import DocumentationEntry +from mathics.doc.structure import DocChapter, DocGuideSection, DocSection, DocSubsection + + +def check_installed(src: Union[ModuleType, Builtin]) -> bool: + """Check if the required libraries""" + required_libs = getattr(src, "requires", []) + return check_requires_list(required_libs) if required_libs else True + + +def filter_toplevel_modules(module_list): + """ + Keep just the modules at the top level. + """ + if len(module_list) == 0: + return module_list + + modules_and_levels = sorted( + ((module.__name__.count("."), module) for module in module_list), + key=lambda x: x[0], + ) + top_level = modules_and_levels[0][0] + return (entry[1] for entry in modules_and_levels if entry[0] == top_level) + + +def gather_docs_from_files(documentation, path): + """ + Load documentation from files in path + """ + # First gather data from static XML-like files. This constitutes "Part 1" of the + # documentation. + files = listdir(path) + files.sort() + + chapter_order = 0 + for file in files: + part_title = file[2:] + if part_title.endswith(".mdoc"): + part_title = part_title[: -len(".mdoc")] + # If the filename start with a number, then is a main part. Otherwise + # is an appendix. + is_appendix = not file[0].isdigit() + chapter_order = documentation.load_part_from_file( + osp.join(path, file), + part_title, + chapter_order, + is_appendix, + ) + + +def gather_reference_part(documentation, title, modules, builtins_by_module): + """ + Build a part from a title, a list of modules and information + of builtins by modules. + """ + part_class = documentation.part_class + reference_part = part_class(documentation, title, True) + modules = filter_toplevel_modules(modules) + for module in sorted_modules(modules): + if skip_module_doc(module): + continue + chapter = doc_chapter(module, reference_part, builtins_by_module) + if chapter is None: + continue + # reference_part.chapters.append(chapter) + return reference_part + + +def doc_chapter(module, part, builtins_by_module): + """ + Build documentation structure for a "Chapter" - reference section which + might be a Mathics Module. + """ + # TODO: reformulate me in a way that symbols are always translated to + # sections, and guide sections do not contain subsections. + documentation = part.documentation if part else None + chapter_class = documentation.chapter_class if documentation else DocChapter + doc_class = documentation.doc_class if documentation else DocumentationEntry + title, text = get_module_doc(module) + chapter = chapter_class(part, title, doc_class(text, title, None)) + part.chapters.append(chapter) + + assert len(chapter.sections) == 0 + + # visited = set(sec.title for sec in symbol_sections) + # If the module is a package, add the guides and symbols from the submodules + if module.__file__.endswith("__init__.py"): + guide_sections, symbol_sections = gather_guides_and_sections( + chapter, module, builtins_by_module + ) + chapter.guide_sections.extend(guide_sections) + + for sec in symbol_sections: + if sec.title in visited: + print(sec.title, "already visited. Skipped.") + else: + visited.add(sec.title) + chapter.sections.append(sec) + else: + symbol_sections = gather_sections(chapter, module, builtins_by_module) + chapter.sections.extend(symbol_sections) + + return chapter + + +def gather_sections(chapter, module, builtins_by_module, section_class=None) -> list: + """Build a list of DocSections from a "top-level" module""" + symbol_sections = [] + if skip_module_doc(module): + return [] + + part = chapter.part if chapter else None + documentation = part.documentation if part else None + if section_class is None: + section_class = documentation.section_class if documentation else DocSection + + # TODO: Check the reason why for the module + # `mathics.builtin.numbers.constants` + # `builtins_by_module` has two copies of `Khinchin`. + # By now, we avoid the repetition by + # converting the entries into `set`s. + # + visited = set() + for symbol_instance in builtins_by_module[module.__name__]: + if skip_doc(symbol_instance, module): + continue + default_contexts = ("System`", "Pymathics`") + title = symbol_instance.get_name( + short=(symbol_instance.context in default_contexts) + ) + if title in visited: + continue + visited.add(title) + text = symbol_instance.__doc__ + operator = symbol_instance.get_operator() + installed = check_installed(symbol_instance) + summary_text = symbol_instance.summary_text + section = section_class( + chapter, + title, + text, + operator, + installed, + summary_text=summary_text, + ) + assert ( + section not in symbol_sections + ), f"{section.title} already in {module.__name__}" + symbol_sections.append(section) + + return symbol_sections + + +def gather_subsections(chapter, section, module, builtins_by_module) -> list: + """Build a list of DocSubsections from a "top-level" module""" + + part = chapter.part if chapter else None + documentation = part.documentation if part else None + section_class = documentation.subsection_class if documentation else DocSubsection + + def section_function( + chapter, + title, + text, + operator=None, + installed=True, + in_guide=False, + summary_text="", + ): + return section_class( + chapter, section, title, text, operator, installed, in_guide, summary_text + ) + + return gather_sections(chapter, module, builtins_by_module, section_function) + + +def gather_guides_and_sections(chapter, module, builtins_by_module): + """ + Look at the submodules in module, and produce the guide sections + and sections. + """ + guide_sections = [] + symbol_sections = [] + if skip_module_doc(module): + return guide_sections, symbol_sections + + if not module.__file__.endswith("__init__.py"): + return guide_sections, symbol_sections + + # Determine the class for sections and guide sections + part = chapter.part if chapter else None + documentation = part.documentation if part else None + guide_class = ( + documentation.guide_section_class if documentation else DocGuideSection + ) + + # Loop over submodules + docpath = f"/doc/{chapter.part.slug}/{chapter.slug}/" + + for sub_module in submodules(module): + if skip_module_doc(sub_module): + continue + + title, text = get_module_doc(sub_module) + installed = check_installed(sub_module) + + guide_section = guide_class( + chapter=chapter, + title=title, + text=text, + submodule=sub_module, + installed=installed, + ) + + submodule_symbol_sections = gather_subsections( + chapter, guide_section, sub_module, builtins_by_module + ) + + guide_section.subsections.extend(submodule_symbol_sections) + guide_sections.append(guide_section) + + # TODO, handle recursively the submodules. + # Here there I see two options: + # if sub_module.__file__.endswith("__init__.py"): + # (deeper_guide_sections, + # deeper_symbol_sections) = gather_guides_and_sections(chapter, + # sub_module, builtins_by_module) + # symbol_sections.extend(deeper_symbol_sections) + # guide_sections.extend(deeper_guide_sections) + return guide_sections, [] + + +def get_module_doc(module: ModuleType) -> Tuple[str, str]: + """ + Determine the title and text associated to the documentation + of a module. + If the module has a module docstring, extract the information + from it. If not, pick the title from the name of the module. + """ + doc = module.__doc__ + if doc is not None: + doc = doc.strip() + if doc: + title = doc.splitlines()[0] + text = "\n".join(doc.splitlines()[1:]) + else: + title = module.__name__ + for prefix in ("mathics.builtin.", "mathics.optional.", "pymathics."): + if title.startswith(prefix): + title = title[len(prefix) :] + title = title.capitalize() + text = "" + return title, text + + +def get_submodule_names(obj) -> list: + """Many builtins are organized into modules which, from a documentation + standpoint, are like Mathematica Online Guide Docs. + + "List Functions", "Colors", or "Distance and Similarity Measures" + are some examples Guide Documents group group various Builtin Functions, + under submodules relate to that general classification. + + Here, we want to return a list of the Python modules under a "Guide Doc" + module. + + As an example of a "Guide Doc" and its submodules, consider the + module named mathics.builtin.colors. It collects code and documentation pertaining + to the builtin functions that would be found in the Guide documentation for "Colors". + + The `mathics.builtin.colors` module has a submodule + `mathics.builtin.colors.named_colors`. + + The builtin functions defined in `named_colors` then are those found in the + "Named Colors" group of the "Colors" Guide Doc. + + So in this example then, in the list the modules returned for + Python module `mathics.builtin.colors` would be the + `mathics.builtin.colors.named_colors` module which contains the + definition and docs for the "Named Colors" Mathics Builtin + Functions. + """ + modpkgs = [] + if hasattr(obj, "__path__"): + for _, modname, __ in pkgutil.iter_modules(obj.__path__): + modpkgs.append(modname) + modpkgs.sort() + return modpkgs + + +def get_doc_name_from_module(module) -> str: + """ + Get the title associated to the module. + If the module has a docstring, pick the name from + its first line (the title). Otherwise, use the + name of the module. + """ + name = "???" + if module.__doc__: + lines = module.__doc__.strip() + if not lines: + name = module.__name__ + else: + name = lines.split("\n")[0] + return name + + +def skip_doc(instance, module="") -> bool: + """Returns True if we should skip the docstring extraction.""" + if not isinstance(module, str): + module = module.__name__ if module else "" + + if type(instance).__name__.endswith("Box"): + return True + if hasattr(instance, "no_doc") and instance.no_doc: + return True + + # Just include the builtins defined in the module. + if module: + if module != instance.__class__.__module__: + return True + return False + + +def skip_module_doc(module, must_be_skipped=frozenset()) -> bool: + """True if the module should not be included in the documentation""" + if IS_PYPY and module.__name__ == "builtins": + return True + return ( + module.__doc__ is None + or module in must_be_skipped + or module.__name__.split(".")[0] not in ("mathics", "pymathics") + or hasattr(module, "no_doc") + and module.no_doc + ) + + +def sorted_modules(modules) -> list: + """Return modules sorted by the ``sort_order`` attribute if that + exists, or the module's name if not.""" + return sorted( + modules, + key=lambda module: module.sort_order + if hasattr(module, "sort_order") + else module.__name__, + ) + + +def submodules(package): + """Generator of the submodules in a package""" + package_folder = package.__file__[: -len("__init__.py")] + for _, module_name, __ in pkgutil.iter_modules([package_folder]): + try: + module = importlib.import_module(package.__name__ + "." + module_name) + except Exception: + continue + yield module diff --git a/mathics/doc/latex/1-Manual.mdoc b/mathics/doc/latex/1-Manual.mdoc new file mode 120000 index 000000000..f23c9aa64 --- /dev/null +++ b/mathics/doc/latex/1-Manual.mdoc @@ -0,0 +1 @@ +../documentation/1-Manual.mdoc \ No newline at end of file diff --git a/mathics/doc/latex/Makefile b/mathics/doc/latex/Makefile index 82552c70f..26e5ac140 100644 --- a/mathics/doc/latex/Makefile +++ b/mathics/doc/latex/Makefile @@ -37,7 +37,7 @@ logo-heptatom.pdf logo-text-nodrop.pdf: (cd .. && $(BASH) ./images.sh) #: The build of the documentation which is derived from docstrings in the Python code and doctest data -documentation.tex: $(DOCTEST_LATEX_DATA_PCL) +documentation.tex: $(DOCTEST_LATEX_DATA_PCL) 1-Manual.mdoc $(PYTHON) ./doc2latex.py $(MATHICS3_MODULE_OPTION) && $(BASH) ./sed-hack.sh #: Same as mathics.pdf @@ -48,7 +48,7 @@ clean: rm -f mathics.asy mathics.aux mathics.idx mathics.log mathics.mtc mathics.mtc* mathics.out mathics.toc || true rm -f test-mathics.aux test-mathics.idx test-mathics.log test-mathics.mtc test-mathics.mtc* test-mathics.out test-mathics.toc || true rm -f mathics.fdb_latexmk mathics.ilg mathics.ind mathics.maf mathics.pre || true - rm -f mathics_*.* || true + rm -f mathics-*.* || true rm -f mathics-test.asy mathics-test.aux mathics-test.idx mathics-test.log mathics-test.mtc mathicsest.mtc* mathics-test.out mathics-test.toc || true rm -f documentation.tex $(DOCTEST_LATEX_DATA_PCL) || true rm -f mathics.pdf mathics.dvi test-mathics.pdf test-mathics.dvi || true diff --git a/mathics/doc/latex/mathics.tex b/mathics/doc/latex/mathics.tex index 74d325e45..0462a6bc1 100644 --- a/mathics/doc/latex/mathics.tex +++ b/mathics/doc/latex/mathics.tex @@ -47,7 +47,7 @@ \usepackage[k-tight]{minitoc} \setlength{\mtcindent}{0pt} \mtcsetformat{minitoc}{tocrightmargin}{2.55em plus 1fil} -\newcommand{\multicolumnmtc}{3} +\newcommand{\multicolumnmtc}{2} \makeatletter \let\SV@mtc@verse\mtc@verse \let\SV@endmtc@verse\endmtc@verse @@ -69,10 +69,13 @@ \includegraphics[height=0.08125\linewidth]{logo-text-nodrop.pdf} \\[.5em] {\LARGE\color{subtitle}\textit{\textmd{A free, open-source alternative to Mathematica}}} - \par\textmd{\Large Mathics Core Version \MathicsCoreVersion} + \par\textmd{\Large Mathics3 Core Version \MathicsCoreVersion} } \author{The Mathics3 Team} +% Since we are using a XML input we have need to specify missed hyphenation +% in LaTeX sich as here: +\hyphenation{eco-system} % Fix unicode mappings for listings % http://tex.stackexchange.com/questions/39640/typesetting-utf8-listings-with-german-umlaute @@ -133,19 +136,21 @@ \newcommand{\chapterstart}{ } \newcommand{\chaptersections}{ - \minitoc - %\begin{multicols}{2} + \begin{sloppypar} + \minitoc + \end{sloppypar} } \newcommand{\chapterend}{ %\end{multicols} } \newcommand{\referencestart}{ -\setcounter{chapter}{0} %\def\thechapter{\Roman{chapter}} \renewcommand{\chaptersections}{ - \minitoc + \begin{sloppypar} + \minitoc %\begin{multicols*}{2} + \end{sloppypar} } \renewcommand{\chapterend}{ %\end{multicols*} @@ -247,7 +252,7 @@ \newcommand{\console}[1]{\hbadness=10000{\ttfamily #1}} \setlength{\parindent}{0mm} -\setlength{\parskip}{1pt} +\setlength{\parskip}{10pt} \setlength{\mathindent}{0em} @@ -269,6 +274,7 @@ \setcounter{tocdepth}{0} \tableofcontents + \lstset{ % inputencoding=utf8, extendedchars=true, diff --git a/mathics/doc/latex_doc.py b/mathics/doc/latex_doc.py index 2a9a5b343..61eceb818 100644 --- a/mathics/doc/latex_doc.py +++ b/mathics/doc/latex_doc.py @@ -4,15 +4,12 @@ """ import re -from os import getenv from typing import Optional -from mathics.core.evaluation import Message, Print -from mathics.doc.common_doc import ( +from mathics.doc.doc_entries import ( CONSOLE_RE, DL_ITEM_RE, DL_RE, - END_LINE_SENTINAL, HYPERTEXT_RE, IMG_PNG_RE, IMG_RE, @@ -24,30 +21,29 @@ QUOTATIONS_RE, REF_RE, SPECIAL_COMMANDS, + DocTest, + DocTests, + DocText, + DocumentationEntry, + get_results_by_test, + post_sub, + pre_sub, +) +from mathics.doc.structure import ( SUBSECTION_END_RE, SUBSECTION_RE, - TESTCASE_OUT_RE, DocChapter, DocGuideSection, DocPart, DocSection, DocSubsection, - DocTest, - DocTests, - DocText, Documentation, - DocumentationEntry, MathicsMainDocumentation, - get_results_by_test, - parse_docstring_to_DocumentationEntry_items, - post_sub, - pre_sub, sorted_chapters, ) -from mathics.doc.utils import slugify # We keep track of the number of \begin{asy}'s we see so that -# we can assocation asymptote file numbers with where they are +# we can association asymptote file numbers with where they are # in the document next_asy_number = 1 @@ -135,7 +131,7 @@ def repl(match): text = text[:-1] + r"\ " return "\\code{\\lstinline%s%s%s}" % (escape_char, text, escape_char) else: - # treat double '' literaly + # treat double '' literally return "''" text = MATHICS_RE.sub(repl, text) @@ -259,6 +255,8 @@ def repl_hypertext(match) -> str: # then is is a link to a section # in this manual, so use "\ref" rather than "\href'. if content.find("/doc/") == 0: + slug = "/".join(content.split("/")[2:]).rstrip("/") + return "%s \\ref{%s}" % (text, latex_label_safe(slug)) slug = "/".join(content.split("/")[2:]).rstrip("/") return "%s of section~\\ref{%s}" % (text, latex_label_safe(slug)) else: @@ -297,7 +295,7 @@ def repl_italic(match): # text = LATEX_BETWEEN_ASY_RE.sub(repl_asy, text) def repl_subsection(match): - return "\n\\subsection*{%s}\n" % match.group(1) + return "\n\\subsection{%s}\n" % match.group(1) text = SUBSECTION_RE.sub(repl_subsection, text) text = SUBSECTION_END_RE.sub("", text) @@ -425,7 +423,7 @@ def repl_out(match): return "\\begin{%s}%s\\end{%s}" % (tag, content, tag) def repl_inline_end(match): - """Prevent linebreaks between inline code and sentence delimeters""" + """Prevent linebreaks between inline code and sentence delimiters""" code = match.group("all") if code[-2] == "}": @@ -481,72 +479,7 @@ class LaTeXDocTest(DocTest): """ def __init__(self, index, testcase, key_prefix=None): - def strip_sentinal(line): - """Remove END_LINE_SENTINAL from the end of a line if it appears. - - Some editors like to strip blanks at the end of a line. - Since the line ends in END_LINE_SENTINAL which isn't blank, - any blanks that appear before will be preserved. - - Some tests require some lines to be blank or entry because - Mathics output can be that way - """ - if line.endswith(END_LINE_SENTINAL): - line = line[: -len(END_LINE_SENTINAL)] - - # Also remove any remaining trailing blanks since that - # seems *also* what we want to do. - return line.strip() - - self.index = index - self.result = None - self.outs = [] - - # Private test cases are executed, but NOT shown as part of the docs - self.private = testcase[0] == "#" - - # Ignored test cases are NOT executed, but shown as part of the docs - # Sandboxed test cases are NOT executed if environment SANDBOX is set - if testcase[0] == "X" or (testcase[0] == "S" and getenv("SANDBOX", False)): - self.ignore = True - # substitute '>' again so we get the correct formatting - testcase[0] = ">" - else: - self.ignore = False - - self.test = strip_sentinal(testcase[1]) - - self.key = None - if key_prefix: - self.key = tuple(key_prefix + (index,)) - outs = testcase[2].splitlines() - for line in outs: - line = strip_sentinal(line) - if line: - if line.startswith("."): - text = line[1:] - if text.startswith(" "): - text = text[1:] - text = "\n" + text - if self.result is not None: - self.result += text - elif self.outs: - self.outs[-1].text += text - continue - - match = TESTCASE_OUT_RE.match(line) - if not match: - continue - symbol, text = match.group(1), match.group(2) - text = text.strip() - if symbol == "=": - self.result = text - elif symbol == ":": - out = Message("", "", text) - self.outs.append(out) - elif symbol == "|": - out = Print(text) - self.outs.append(out) + super().__init__(index, testcase, key_prefix) def __str__(self): return self.test @@ -596,7 +529,7 @@ def latex(self, doc_data: dict) -> str: class LaTeXDocumentationEntry(DocumentationEntry): - """A class to hold our internal markdown-like format data. + """A class to hold our custom XML-like format. The `latex()` method can turn this into LaTeX. Mathics core also uses this in getting usage strings (`??`). @@ -712,16 +645,32 @@ def latex( intro, short, ) + + if self.part.is_reference: + sort_section_function = sorted + else: + sort_section_function = lambda x: x + chapter_sections = [ ("\n\n\\chapter{%(title)s}\n\\chapterstart\n\n%(intro)s") % {"title": escape_latex(self.title), "intro": intro}, "\\chaptersections\n", + # #################### "\n\n".join( section.latex(doc_data, quiet) # Here we should use self.all_sections, but for some reason # guidesections are not properly loaded, duplicating # the load of subsections. - for section in sorted(self.sections) + for section in sorted(self.guide_sections) + if not filter_sections or section.title in filter_sections + ), + # ################### + "\n\n".join( + section.latex(doc_data, quiet) + # Here we should use self.all_sections, but for some reason + # guidesections are not properly loaded, duplicating + # the load of subsections. + for section in sort_section_function(self.sections) if not filter_sections or section.title in filter_sections ), "\n\\chapterend\n", @@ -769,27 +718,9 @@ def __init__( in_guide=False, summary_text="", ): - self.chapter = chapter - self.in_guide = in_guide - self.installed = installed - self.operator = operator - self.slug = slugify(title) - self.subsections = [] - self.subsections_by_slug = {} - self.summary_text = summary_text - self.title = title - - if text.count("
    ") != text.count("
    "): - raise ValueError( - "Missing opening or closing
    tag in " - "{} documentation".format(title) - ) - - # Needs to come after self.chapter is initialized since - # DocumentationEntry uses self.chapter. - self.doc = LaTeXDocumentationEntry(text, title, self) - - chapter.sections_by_slug[self.slug] = self + super().__init__( + chapter, title, text, operator, installed, in_guide, summary_text + ) def latex(self, doc_data: dict, quiet=False) -> str: """Render this Section object as LaTeX string and return that. @@ -812,11 +743,11 @@ def latex(self, doc_data: dict, quiet=False) -> str: sections = "\n\n".join(section.latex(doc_data) for section in self.subsections) slug = f"{self.chapter.part.slug}/{self.chapter.slug}/{self.slug}" section_string = ( - "\n\n\\section*{%s}{%s}\n" % (title, index) + "\n\n\\section{%s}{%s}\n" % (title, index) + "\n\\label{%s}" % latex_label_safe(slug) + "\n\\sectionstart\n\n" + f"{content}" - + ("\\addcontentsline{toc}{section}{%s}" % title) + # + ("\\addcontentsline{toc}{section}{%s}" % title) + sections + "\\sectionend" ) @@ -839,7 +770,6 @@ def __init__( installed: bool = True, ): super().__init__(chapter, title, text, submodule, installed) - self.doc = LaTeXDocumentationEntry(text, title, self) def get_tests(self): # FIXME: The below is a little weird for Guide Sections. @@ -866,6 +796,7 @@ def latex(self, doc_data: dict, quiet=False) -> str: # The leading spaces help show chapter level. print(f" Formatting Guide Section {self.title}") intro = self.doc.latex(doc_data).strip() + slug = f"{self.chapter.part.slug}/{self.chapter.slug}/{self.slug}" if intro: short = "short" if len(intro) < 300 else "" intro = "\\begin{guidesectionintro%s}\n%s\n\n\\end{guidesectionintro%s}" % ( @@ -875,10 +806,14 @@ def latex(self, doc_data: dict, quiet=False) -> str: ) guide_sections = [ ( - "\n\n\\section{%(title)s}\n\\sectionstart\n\n%(intro)s" - "\\addcontentsline{toc}{section}{%(title)s}" + "\n\n\\section{%(title)s}\n\\label{%(label)s}\n\\sectionstart\n\n%(intro)s" + # "\\addcontentsline{toc}{section}{%(title)s}" ) - % {"title": escape_latex(self.title), "intro": intro}, + % { + "title": escape_latex(self.title), + "label": latex_label_safe(slug), + "intro": intro, + }, "\n\n".join(section.latex(doc_data) for section in self.subsections), ] return "".join(guide_sections) @@ -916,23 +851,6 @@ def __init__( super().__init__( chapter, section, title, text, operator, installed, in_guide, summary_text ) - self.doc = LaTeXDocumentationEntry(text, title, section) - - if in_guide: - # Tests haven't been picked out yet from the doc string yet. - # Gather them here. - self.items = parse_docstring_to_DocumentationEntry_items( - text, LaTeXDocTests, LaTeXDocTest, LaTeXDocText - ) - else: - self.items = [] - - if text.count("
    ") != text.count("
    "): - raise ValueError( - "Missing opening or closing
    tag in " - "{} documentation".format(title) - ) - self.section.subsections_by_slug[self.slug] = self def latex(self, doc_data: dict, quiet=False, chapters=None) -> str: """Render this Subsection object as LaTeX string and return that. @@ -956,10 +874,10 @@ def latex(self, doc_data: dict, quiet=False, chapters=None) -> str: slug = f"{self.chapter.part.slug}/{self.chapter.slug}/{self.section.slug}/{self.slug}" section_string = ( - "\n\n\\subsection*{%(title)s}%(index)s\n" + "\n\n\\subsection{%(title)s}%(index)s\n" + "\n\\label{%s}" % latex_label_safe(slug) + "\n\\subsectionstart\n\n%(content)s" - "\\addcontentsline{toc}{subsection}{%(title)s}" + # "\\addcontentsline{toc}{subsection}{%(title)s}" "%(sections)s" "\\subsectionend" ) % { diff --git a/mathics/doc/structure.py b/mathics/doc/structure.py new file mode 100644 index 000000000..9135c5702 --- /dev/null +++ b/mathics/doc/structure.py @@ -0,0 +1,707 @@ +# -*- coding: utf-8 -*- +""" +Structural elements of Mathics Documentation + +This module contains the classes representing the Mathics documentation structure, +and extended regular expressions used to parse it. + +""" +import logging +import re +from os import environ +from typing import Iterator, List, Optional + +from mathics import settings +from mathics.core.builtin import check_requires_list +from mathics.core.load_builtin import ( + builtins_by_module as global_builtins_by_module, + mathics3_builtins_modules, +) +from mathics.doc.doc_entries import DocumentationEntry, Tests, filter_comments +from mathics.doc.utils import slugify +from mathics.eval.pymathics import pymathics_builtins_by_module, pymathics_modules + +CHAPTER_RE = re.compile('(?s)(.*?)') +SECTION_RE = re.compile('(?s)(.*?)
    (.*?)
    ') +SUBSECTION_RE = re.compile('(?s)') +SUBSECTION_END_RE = re.compile("") + +# Debug flags. + +# Set to True if want to follow the process +# The first phase is building the documentation data structure +# based on docstrings: + +MATHICS_DEBUG_DOC_BUILD: bool = "MATHICS_DEBUG_DOC_BUILD" in environ + +# After building the doc structure, we extract test cases. +MATHICS_DEBUG_TEST_CREATE: bool = "MATHICS_DEBUG_TEST_CREATE" in environ + +# Name of the Mathics3 Module part of the document. +MATHICS3_MODULES_TITLE = "Mathics3 Modules" + + +# DocSection has to appear before DocGuideSection which uses it. +class DocSection: + """An object for a Documented Section. + A Section is part of a Chapter. It can contain subsections. + """ + + def __init__( + self, + chapter, + title: str, + text: str, + operator, + installed: bool = True, + in_guide: bool = False, + summary_text: str = "", + ): + self.chapter = chapter + self.in_guide = in_guide + self.installed = installed + self.items = [] # tests in section when this is under a guide section + self.operator = operator + self.slug = slugify(title) + self.subsections = [] + self.subsections_by_slug = {} + self.summary_text = summary_text + self.tests = None # tests in section when not under a guide section + self.title = title + + if text.count("
    ") != text.count("
    "): + raise ValueError( + "Missing opening or closing
    tag in " + "{} documentation".format(title) + ) + + # Needs to come after self.chapter is initialized since + # DocumentationEntry uses self.chapter. + # Notice that we need the documentation object, to have access + # to the suitable subclass of DocumentationElement. + documentation = self.chapter.part.documentation + self.doc = documentation.doc_class(text, title, None).set_parent_path(self) + + chapter.sections_by_slug[self.slug] = self + if MATHICS_DEBUG_DOC_BUILD: + print(" DEBUG Creating Section", title) + + # Add __eq__ and __lt__ so we can sort Sections. + def __eq__(self, other) -> bool: + return self.title == other.title + + def __lt__(self, other) -> bool: + return self.title < other.title + + def __str__(self) -> str: + return f" == {self.title} ==\n{self.doc}" + + @property + def parent(self): + "the container where the section is" + return self.chapter + + @parent.setter + def parent(self, value): + "the container where the section is" + raise TypeError("parent is a read-only property") + + def get_tests(self): + """yield tests""" + if self.installed: + for test in self.doc.get_tests(): + yield test + + +# DocChapter has to appear before DocGuideSection which uses it. +class DocChapter: + """An object for a Documented Chapter. + A Chapter is part of a Part[dChapter. It can contain (Guide or plain) Sections. + """ + + def __init__(self, part, title, doc=None, chapter_order: Optional[int] = None): + self.chapter_order = chapter_order + self.doc = doc + self.guide_sections = [] + self.part = part + self.title = title + self.slug = slugify(title) + self.sections = [] + self.sections_by_slug = {} + self.sort_order = None + if doc: + self.doc.set_parent_path(self) + + part.chapters_by_slug[self.slug] = self + + if MATHICS_DEBUG_DOC_BUILD: + print(" DEBUG Creating Chapter", title) + + def __str__(self) -> str: + """ + A DocChapter is represented as the index of its sections + and subsections. + """ + sections_descr = "" + for section in self.all_sections: + sec_class = "@>" if isinstance(section, DocGuideSection) else "@ " + sections_descr += f" {sec_class} " + section.title + "\n" + for subsection in section.subsections: + sections_descr += " * " + subsection.title + "\n" + + return f" = {self.part.title}: {self.title} =\n\n{sections_descr}" + + @property + def all_sections(self): + "guides and normal sections" + return sorted(self.guide_sections) + sorted(self.sections) + + @property + def parent(self): + "the container where the chapter is" + return self.part + + @parent.setter + def parent(self, value): + "the container where the chapter is" + raise TypeError("parent is a read-only property") + + +class DocGuideSection(DocSection): + """An object for a Documented Guide Section. + A Guide Section is part of a Chapter. "Colors" or "Special Functions" + are examples of Guide Sections, and each contains a number of Sections. + like NamedColors or Orthogonal Polynomials. + """ + + def __init__( + self, + chapter: DocChapter, + title: str, + text: str, + submodule, + installed: bool = True, + ): + super().__init__(chapter, title, text, None, installed, False) + self.section = submodule + + if MATHICS_DEBUG_DOC_BUILD: + print(" DEBUG Creating Guide Section", title) + + +class DocPart: + """ + Represents one of the main parts of the document. Parts + can be loaded from a mdoc file, generated automatically from + the docstrings of Builtin objects under `mathics.builtin`. + """ + + chapter_class = DocChapter + + def __init__(self, documentation, title, is_reference=False): + self.documentation = documentation + self.title = title + self.chapters = [] + self.chapters_by_slug = {} + self.is_reference = is_reference + self.is_appendix = False + self.slug = slugify(title) + documentation.parts_by_slug[self.slug] = self + if MATHICS_DEBUG_DOC_BUILD: + print("DEBUG Creating Part", title) + + def __str__(self) -> str: + return f" Part {self.title}\n\n" + "\n\n".join( + str(chapter) for chapter in sorted_chapters(self.chapters) + ) + + +class Documentation: + """ + `Documentation` describes an object containing the whole documentation system. + Documentation + | + +--------0> Parts + | + +-----0> Chapters + | + +-----0>Sections + | | + | +------0> SubSections + | + +---->0>GuideSections + | + +------0> SubSections + + (with 0>) meaning "aggregation". + + Each element contains a title, a collection of elements of the following class + in the hierarchy. Parts, Chapters, Guide Sections, Sections and SubSections contains a doc + attribute describing the content to be shown after the title, and before + the elements of the subsequent terms in the hierarchy. + """ + + def __init__(self, title: str = "Title", doc_dir: str = ""): + """ + Parameters + ---------- + title : str, optional + The title of the Documentation. The default is "Title". + doc_dir : str, optional + The path where the sources can be loaded. The default is "", + meaning that no sources must be loaded. + """ + # This is a way to load the default classes + # without defining these attributes as class + # attributes. + self._set_classes() + self.appendix = [] + self.doc_dir = doc_dir + self.parts = [] + self.parts_by_slug = {} + self.title = title + + def _set_classes(self): + """ + Set the classes of the subelements. Must be overloaded + by the subclasses. + """ + if not hasattr(self, "part_class"): + self.chapter_class = DocChapter + self.doc_class = DocumentationEntry + self.guide_section_class = DocGuideSection + self.part_class = DocPart + self.section_class = DocSection + self.subsection_class = DocSubsection + + def __str__(self): + result = self.title + "\n" + len(self.title) * "~" + "\n" + return ( + result + "\n\n".join([str(part) for part in self.parts]) + "\n" + 60 * "-" + ) + + def add_section( + self, + chapter, + section_name: str, + section_object, + operator, + is_guide: bool = False, + in_guide: bool = False, + summary_text="", + ): + """ + Adds a DocSection or DocGuideSection + object to the chapter, a DocChapter object. + "section_object" is either a Python module or a Class object instance. + """ + if section_object is not None: + required_libs = getattr(section_object, "requires", []) + installed = check_requires_list(required_libs) if required_libs else True + # FIXME add an additional mechanism in the module + # to allow a docstring and indicate it is not to go in the + # user manual + if not section_object.__doc__: + return None + + installed = True + + if is_guide: + section = self.guide_section_class( + chapter, + section_name, + section_object.__doc__, + section_object, + installed=installed, + ) + chapter.guide_sections.append(section) + else: + section = self.section_class( + chapter, + section_name, + section_object.__doc__, + operator=operator, + installed=installed, + in_guide=in_guide, + summary_text=summary_text, + ) + chapter.sections.append(section) + + return section + + def add_subsection( + self, + chapter, + section, + subsection_name: str, + instance, + operator=None, + in_guide=False, + ): + """ + Append a subsection for ``instance`` into ``section.subsections`` + """ + + required_libs = getattr(instance, "requires", []) + installed = check_requires_list(required_libs) if required_libs else True + + # FIXME add an additional mechanism in the module + # to allow a docstring and indicate it is not to go in the + # user manual + if not instance.__doc__: + return + summary_text = ( + instance.summary_text if hasattr(instance, "summary_text") else "" + ) + subsection = self.subsection_class( + chapter, + section, + subsection_name, + instance.__doc__, + operator=operator, + installed=installed, + in_guide=in_guide, + summary_text=summary_text, + ) + section.subsections.append(subsection) + + def doc_part(self, title, start): + """ + Build documentation structure for a "Part" - Reference + section or collection of Mathics3 Modules. + """ + + builtin_part = self.part_class(self, title, is_reference=start) + self.parts.append(builtin_part) + + def get_part(self, part_slug): + """return a section from part key""" + return self.parts_by_slug.get(part_slug) + + def get_chapter(self, part_slug, chapter_slug): + """return a section from part and chapter keys""" + part = self.parts_by_slug.get(part_slug) + if part: + return part.chapters_by_slug.get(chapter_slug) + return None + + def get_section(self, part_slug, chapter_slug, section_slug): + """return a section from part, chapter and section keys""" + part = self.parts_by_slug.get(part_slug) + if part: + chapter = part.chapters_by_slug.get(chapter_slug) + if chapter: + return chapter.sections_by_slug.get(section_slug) + return None + + def get_subsection(self, part_slug, chapter_slug, section_slug, subsection_slug): + """ + return a section from part, chapter, section and subsection + keys + """ + part = self.parts_by_slug.get(part_slug) + if part: + chapter = part.chapters_by_slug.get(chapter_slug) + if chapter: + section = chapter.sections_by_slug.get(section_slug) + if section: + return section.subsections_by_slug.get(subsection_slug) + + return None + + # FIXME: turn into a @property tests? + def get_tests(self) -> Iterator: + """ + Returns a generator to extracts lists test objects. + """ + for part in self.parts: + for chapter in sorted_chapters(part.chapters): + if MATHICS_DEBUG_TEST_CREATE: + print(f"DEBUG Gathering tests for Chapter {chapter.title}") + + tests = chapter.doc.get_tests() + if tests: + yield Tests(part.title, chapter.title, "", tests) + + for section in chapter.all_sections: + if section.installed: + if MATHICS_DEBUG_TEST_CREATE: + if isinstance(section, DocGuideSection): + print( + f"DEBUG Gathering tests for Guide Section {section.title}" + ) + else: + print( + f"DEBUG Gathering tests for Section {section.title}" + ) + + tests = section.doc.get_tests() + if tests: + yield Tests( + part.title, + chapter.title, + section.title, + tests, + ) + + def load_documentation_sources(self): + """ + Extract doctest data from various static XML-like doc files, Mathics3 Built-in functions + (inside mathics.builtin), and external Mathics3 Modules. + + The extracted structure is stored in ``self``. + """ + from mathics.doc.gather import gather_docs_from_files, gather_reference_part + + assert ( + len(self.parts) == 0 + ), "The documentation must be empty to call this function." + + gather_docs_from_files(self, self.doc_dir) + # Next extract data that has been loaded into Mathics3 when it runs. + # This is information from `mathics.builtin`. + # This is Part 2 of the documentation. + + # Notice that in order to generate the documentation + # from the builtin classes, it is needed to call first to + # import_and_load_builtins() + + for title, modules, builtins_by_module in [ + ( + "Reference of Built-in Symbols", + mathics3_builtins_modules, + global_builtins_by_module, + ), + ( + MATHICS3_MODULES_TITLE, + pymathics_modules, + pymathics_builtins_by_module, + ), + ]: + self.parts.append( + gather_reference_part(self, title, modules, builtins_by_module) + ) + + # Finally, extract Appendix information. This include License text + # This is the final Part of the documentation. + + for part in self.appendix: + self.parts.append(part) + + def load_part_from_file( + self, + filename: str, + part_title: str, + chapter_order: int, + is_appendix: bool = False, + ) -> int: + """Load a document file (tagged XML-like in custom format) as + a part of the documentation""" + part = self.part_class(self, part_title) + with open(filename, "rb") as src_file: + text = src_file.read().decode("utf8") + + text = filter_comments(text) + chapters = CHAPTER_RE.findall(text) + for chapter_title, text in chapters: + chapter = self.chapter_class( + part, chapter_title, chapter_order=chapter_order + ) + chapter_order += 1 + text += '
    ' + section_texts = SECTION_RE.findall(text) + for pre_text, title, text in section_texts: + if title: + section = self.section_class( + chapter, title, text, operator=None, installed=True + ) + chapter.sections.append(section) + subsections = SUBSECTION_RE.findall(text) + for subsection_title in subsections: + subsection = self.subsection_class( + chapter, + section, + subsection_title, + text, + ) + section.subsections.append(subsection) + else: + section = None + if not chapter.doc: + chapter.doc = self.doc_class(pre_text, title, section) + pass + + part.chapters.append(chapter) + if is_appendix: + part.is_appendix = True + self.appendix.append(part) + else: + self.parts.append(part) + return chapter_order + + +class DocSubsection: + """An object for a Documented Subsection. + A Subsection is part of a Section. + """ + + def __init__( + self, + chapter, + section, + title, + text, + operator=None, + installed=True, + in_guide=False, + summary_text="", + ): + """ + Information that goes into a subsection object. This can be a written text, or + text extracted from the docstring of a builtin module or class. + + About some of the parameters... + + Some subsections are contained in a grouping module and need special work to + get the grouping module name correct. + + For example the Chapter "Colors" is a module so the docstring text for it is in + mathics/builtin/colors/__init__.py . In mathics/builtin/colors/named-colors.py we have + the "section" name for the class Red (the subsection) inside it. + """ + title_summary_text = re.split(" -- ", title) + len_title = len(title_summary_text) + # We need the documentation object, to have access + # to the suitable subclass of DocumentationElement. + documentation = chapter.part.documentation + + self.title = title_summary_text[0] if len_title > 0 else "" + self.summary_text = title_summary_text[1] if len_title > 1 else summary_text + self.doc = documentation.doc_class(text, title, None) + self.chapter = chapter + self.in_guide = in_guide + self.installed = installed + self.operator = operator + + self.section = section + self.slug = slugify(title) + self.subsections = [] + self.title = title + self.doc.set_parent_path(self) + + # This smells wrong: Here a DocSection (a level in the documentation system) + # is mixed with a DocumentationEntry. `items` is an attribute of the + # `DocumentationEntry`, not of a Part / Chapter/ Section. + # The content of a subsection should be stored in self.doc, + # and the tests should set the route (key) through self.doc.set_parent_doc + if in_guide: + # Tests haven't been picked out yet from the doc string yet. + # Gather them here. + self.items = self.doc.items + + for item in self.items: + for test in item.get_tests(): + assert test.key is not None + else: + self.items = [] + + if text.count("
    ") != text.count("
    "): + raise ValueError( + "Missing opening or closing
    tag in " + "{} documentation".format(title) + ) + self.section.subsections_by_slug[self.slug] = self + + if MATHICS_DEBUG_DOC_BUILD: + print(" DEBUG Creating Subsection", title) + + def __str__(self) -> str: + return f"=== {self.title} ===\n{self.doc}" + + @property + def parent(self): + """the chapter where the section is""" + return self.section + + @parent.setter + def parent(self, value): + raise TypeError("parent is a read-only property") + + def get_tests(self): + """yield tests""" + if self.installed: + for test in self.doc.get_tests(): + yield test + + +class MathicsMainDocumentation(Documentation): + """MathicsMainDocumentation specializes ``Documentation`` by providing the attributes + and methods needed to generate the documentation from the Mathics library. + + The parts of the documentation are loaded from the Markdown files contained + in the path specified by ``self.doc_dir``. Files with names starting in numbers + are considered parts of the main text, while those that starts with other characters + are considered as appendix parts. + + In addition to the parts loaded from our custom-marked XML + document file, a ``Reference of Builtin-Symbols`` part and a part + for the loaded Pymathics modules are automatically generated. + + In the ``Reference of Built-in Symbols`` tom-level modules and files in ``mathics.builtin`` + are associated to Chapters. For single file submodules (like ``mathics.builtin.procedure``) + The chapter contains a Section for each Symbol in the module. For sub-packages + (like ``mathics.builtin.arithmetic``) sections are given by the sub-module files, + and the symbols in these sub-packages defines the Subsections. ``__init__.py`` in + subpackages are associated to GuideSections. + + In a similar way, in the ``Mathics3 Modules`` part, each ``Mathics3`` module defines a Chapter, + files in the module defines Sections, and Symbols defines Subsections. + + + ``MathicsMainDocumentation`` is also used for creating test data and saving it to a + Python Pickle file and running tests that appear in the documentation (doctests). + + There are other classes DjangoMathicsDocumentation and LaTeXMathicsDocumentation + that format the data accumulated here. In fact I think those can sort of serve + instead of this. + + """ + + def __init__(self): + super().__init__(title="Mathics Main Documentation", doc_dir=settings.DOC_DIR) + self.doctest_latex_pcl_path = settings.DOCTEST_LATEX_DATA_PCL + self.pymathics_doc_loaded = False + self.doc_data_file = settings.get_doctest_latex_data_path( + should_be_readable=True + ) + + def gather_doctest_data(self): + """ + Populates the documentatation. + (deprecated) + """ + logging.warning( + "gather_doctest_data is deprecated. Use load_documentation_sources" + ) + return self.load_documentation_sources() + + +def sorted_chapters(chapters: List[DocChapter]) -> List[DocChapter]: + """Return chapters sorted by title""" + return sorted( + chapters, + key=lambda chapter: str(chapter.sort_order) + if chapter.sort_order is not None + else chapter.title, + ) + + +def sorted_modules(modules) -> list: + """Return modules sorted by the ``sort_order`` attribute if that + exists, or the module's name if not.""" + return sorted( + modules, + key=lambda module: module.sort_order + if hasattr(module, "sort_order") + else module.__name__, + ) diff --git a/mathics/docpipeline.py b/mathics/docpipeline.py old mode 100755 new mode 100644 index 54fab4034..58c9b776b --- a/mathics/docpipeline.py +++ b/mathics/docpipeline.py @@ -13,193 +13,262 @@ import os import os.path as osp import pickle -import re import sys from argparse import ArgumentParser +from collections import namedtuple from datetime import datetime -from typing import Dict, Optional, Set, Tuple, Union +from typing import Callable, Dict, Optional, Set, Union import mathics from mathics import settings, version_string -from mathics.core.definitions import Definitions -from mathics.core.evaluation import Evaluation, Output +from mathics.core.evaluation import Output from mathics.core.load_builtin import _builtins, import_and_load_builtins -from mathics.core.parser import MathicsSingleLineFeeder -from mathics.doc.common_doc import ( - DocGuideSection, - DocSection, - DocTest, - DocTests, - MathicsMainDocumentation, -) -from mathics.doc.utils import load_doctest_data, print_and_log +from mathics.doc.common_doc import DocGuideSection, DocSection, MathicsMainDocumentation +from mathics.doc.doc_entries import DocTest, DocTests +from mathics.doc.utils import load_doctest_data, print_and_log, slugify from mathics.eval.pymathics import PyMathicsLoadException, eval_LoadModule +from mathics.session import MathicsSession +from mathics.settings import get_doctest_latex_data_path from mathics.timing import show_lru_cache_statistics - -class TestOutput(Output): - def max_stored_size(self, _): - return None - - # Global variables # FIXME: After 3.8 is the minimum Python we can turn "str" into a Literal SEP: str = "-" * 70 + "\n" STARS: str = "*" * 10 +MAX_TESTS = 100000 # A number greater than the total number of tests. +# When 3.8 is base, the below can be a Literal type. +INVALID_TEST_GROUP_SETUP = (None, None) -DEFINITIONS = None -DOCUMENTATION = None -CHECK_PARTIAL_ELAPSED_TIME = False -LOGFILE = None +TestParameters = namedtuple( + "TestParameters", + [ + "check_partial_elapsed_time", + "data_path", + "keep_going", + "max_tests", + "quiet", + "output_format", + "reload", + "start_at", + ], +) -MAX_TESTS = 100000 # A number greater than the total number of tests. +class TestOutput(Output): + """Output class for tests""" + + def max_stored_size(self, _): + return None -def doctest_compare(result: Optional[str], wanted: Optional[str]) -> bool: +class DocTestPipeline: """ - Performs a doctest comparison between ``result`` and ``wanted`` and returns - True if the test should be considered a success. + This class gathers all the information required to process + the doctests and generate the data for the documentation. """ - if wanted in ("...", result): - return True - if result is None or wanted is None: - return False + def __init__(self, args, output_format="latex", data_path: Optional[str] = None): + self.session = MathicsSession() + self.output_data = {} + + # LoadModule Mathics3 modules + if args.pymathics: + required_modules = set(args.pymathics.split(",")) + load_pymathics_modules(required_modules, self.session.definitions) + + self.builtin_total = len(_builtins) + self.documentation = MathicsMainDocumentation() + self.documentation.load_documentation_sources() + self.logfile = open(args.logfilename, "wt") if args.logfilename else None + + self.parameters = TestParameters( + check_partial_elapsed_time=args.elapsed_times, + data_path=data_path, + keep_going=args.keep_going and not args.stop_on_failure, + max_tests=args.count + args.skip, + quiet=args.quiet, + output_format=output_format, + reload=args.reload and not (args.chapters or args.sections), + start_at=args.skip + 1, + ) + self.status = TestStatus(data_path, self.parameters.quiet) + + def reset_user_definitions(self): + """Reset the user definitions""" + return self.session.definitions.reset_user_definitions() + + def print_and_log(self, message): + """Print and log a message in the logfile""" + if not self.parameters.quiet: + print(message) + if self.logfile: + print_and_log(self.logfile, message.encode("utf-8")) + + def validate_group_setup( + self, + include_set: set, + entity_name: Optional[str], + ): + """ + Common things that need to be done before running a group of doctests. + """ + test_parameters = self.parameters + + if self.documentation is None: + self.print_and_log("Documentation is not initialized.") + return INVALID_TEST_GROUP_SETUP + + if entity_name is not None: + include_names = ", ".join(include_set) + print(f"Testing {entity_name}(s): {include_names}") + else: + include_names = None - result_list = result.splitlines() - wanted_list = wanted.splitlines() - if result_list == [] and wanted_list == ["#<--#"]: - return True + if test_parameters.reload: + doctest_latex_data_path = get_doctest_latex_data_path( + should_be_readable=True + ) + self.output_data = load_doctest_data(doctest_latex_data_path) + else: + self.output_data = {} - if len(result_list) != len(wanted_list): - return False + # For consistency set the character encoding ASCII which is + # the lowest common denominator available on all systems. + settings.SYSTEM_CHARACTER_ENCODING = "ASCII" - for res, want in zip(result_list, wanted_list): - wanted_re = re.escape(want.strip()) - wanted_re = wanted_re.replace("\\.\\.\\.", ".*?") - wanted_re = f"^{wanted_re}$" - if not re.match(wanted_re, res.strip()): - return False - return True + if self.session.definitions is None: + self.print_and_log("Definitions are not initialized.") + return INVALID_TEST_GROUP_SETUP + + # Start with a clean variables state from whatever came before. + # In the test suite however, we may set new variables. + self.reset_user_definitions() + return self.output_data, include_names + + +class TestStatus: + """ + Status parameters of the tests + """ + + def __init__(self, data_path: Optional[str] = None, quiet=False): + self.texdatafolder = osp.dirname(data_path) if data_path is not None else None + self.total = 0 + self.failed = 0 + self.skipped = 0 + self.failed_sections = set() + self.prev_key = [] + self.quiet = quiet + + def find_texdata_folder(self): + """Generate a folder for texdata""" + return self.textdatafolder + + def mark_as_failed(self, key): + """Mark a key as failed""" + self.failed_sections.add(key) + self.failed += 1 + + def section_name_for_print(self, test) -> str: + """ + If the test has a different key, + returns a printable version of the section name. + Otherwise, return the empty string. + """ + key = list(test.key)[1:-1] + if key != self.prev_key: + return " / ".join(key) + return "" + + def show_section(self, test): + """Show information about the current test case""" + section_name_for_print = self.section_name_for_print(test) + if section_name_for_print: + if self.quiet: + print(f"Testing section: {section_name_for_print}") + else: + print(f"{STARS} {section_name_for_print} {STARS}") + + def show_test(self, test, index, subindex): + """Show the current test""" + test_str = test.test + if not self.quiet: + print(f"{index:4d} ({subindex:2d}): TEST {test_str}") def test_case( test: DocTest, - index: int = 0, - subindex: int = 0, - quiet: bool = False, - section_name: str = "", - section_for_print="", - chapter_name: str = "", - part: str = "", + test_pipeline: DocTestPipeline, + fail: Optional[Callable] = lambda x: False, ) -> bool: """ Run a single test cases ``test``. Return True if test succeeds and False if it fails. ``index``gives the global test number count, while ``subindex`` counts from the beginning of the section or subsection. - The test results are assumed to be foramtted to ASCII text. + The test results are assumed to be formatted to ASCII text. """ - - global CHECK_PARTIAL_ELAPSED_TIME - test_str, wanted_out, wanted = test.test, test.outs, test.result - - def fail(why): - print_and_log( - LOGFILE, - f"""{SEP}Test failed: in {part} / {chapter_name} / {section_name} -{part} -{why} -""".encode( - "utf-8" - ), - ) - return False - - if not quiet: - if section_for_print: - print(f"{STARS} {section_for_print} {STARS}") - print(f"{index:4d} ({subindex:2d}): TEST {test_str}") - - feeder = MathicsSingleLineFeeder(test_str, filename="") - evaluation = Evaluation( - DEFINITIONS, catch_interrupt=False, output=TestOutput(), format="text" - ) + test_parameters = test_pipeline.parameters try: - time_parsing = datetime.now() - query = evaluation.parse_feeder(feeder) - if CHECK_PARTIAL_ELAPSED_TIME: - print(" parsing took", datetime.now() - time_parsing) - if query is None: - # parsed expression is None - result = None - out = evaluation.out - else: - result = evaluation.evaluate(query) - if CHECK_PARTIAL_ELAPSED_TIME: - print(" evaluation took", datetime.now() - time_parsing) - out = result.out - result = result.result + time_start = datetime.now() + result = test_pipeline.session.evaluate_as_in_cli(test.test, src_name="") + out = result.out + result = result.result except Exception as exc: fail(f"Exception {exc}") info = sys.exc_info() sys.excepthook(*info) return False - time_comparing = datetime.now() - comparison_result = doctest_compare(result, wanted) - - if CHECK_PARTIAL_ELAPSED_TIME: - print(" comparison took ", datetime.now() - time_comparing) + time_start = datetime.now() + comparison_result = test.compare_result(result) + if test_parameters.check_partial_elapsed_time: + print(" comparison took ", datetime.now() - time_start) if not comparison_result: print("result != wanted") - fail_msg = f"Result: {result}\nWanted: {wanted}" + fail_msg = f"Result: {result}\nWanted: {test.result}" if out: fail_msg += "\nAdditional output:\n" fail_msg += "\n".join(str(o) for o in out) return fail(fail_msg) - output_ok = True - time_comparing = datetime.now() - if len(wanted_out) == 1 and wanted_out[0].text == "...": - # If we have ... don't check - pass - elif len(out) != len(wanted_out): - # Mismatched number of output lines, and we don't have "..." - output_ok = False - else: - # Need to check all output line by line - for got, wanted in zip(out, wanted_out): - if not got == wanted and wanted.text != "...": - output_ok = False - break - if CHECK_PARTIAL_ELAPSED_TIME: - print(" comparing messages took ", datetime.now() - time_comparing) + + time_start = datetime.now() + output_ok = test.compare_out(out) + if test_parameters.check_partial_elapsed_time: + print(" comparing messages took ", datetime.now() - time_start) if not output_ok: return fail( "Output:\n%s\nWanted:\n%s" - % ("\n".join(str(o) for o in out), "\n".join(str(o) for o in wanted_out)) + % ( + "\n".join(str(o) for o in out), + "\n".join(str(o) for o in test.outs), + ) ) return True -def create_output(tests, doctest_data, output_format="latex"): - if DEFINITIONS is None: - print_and_log(LOGFILE, "Definitions are not initialized.") +def create_output(test_pipeline, tests): + """ + Populate ``doctest_data`` with the results of the + ``tests`` in the format ``output_format`` + """ + output_format = test_pipeline.parameters.output_format + if test_pipeline.session.definitions is None: + test_pipeline.print_and_log("Definitions are not initialized.") return - DEFINITIONS.reset_user_definitions() + doctest_data = test_pipeline.output_data + test_pipeline.reset_user_definitions() + session = test_pipeline.session + for test in tests: if test.private: continue key = test.key - evaluation = Evaluation( - DEFINITIONS, format=output_format, catch_interrupt=True, output=TestOutput() - ) try: - result = evaluation.parse_evaluate(test.test) + result = session.evaluate_as_in_cli(test.test, form=output_format) except Exception: # noqa result = None if result is None: @@ -215,448 +284,346 @@ def create_output(tests, doctest_data, output_format="latex"): } +def load_pymathics_modules(module_names: set, definitions): + """ + Load pymathics modules + + PARAMETERS + ========== + + module_names: set + a set of modules to be loaded. + + Return + ====== + loaded_modules : set + the set of successfully loaded modules. + """ + loaded_modules = [] + for module_name in module_names: + try: + eval_LoadModule(module_name, definitions) + except PyMathicsLoadException: + print(f"Python module {module_name} is not a Mathics3 module.") + + except Exception as exc: + print(f"Python import errors with: {exc}.") + else: + print(f"Mathics3 Module {module_name} loaded") + loaded_modules.append(module_name) + + return set(loaded_modules) + + def show_test_summary( - total: int, - failed: int, + test_pipeline: DocTestPipeline, entity_name: str, entities_searched: str, - keep_going: bool, - generate_output: bool, - output_data, ): """ Print and log test summary results. - If ``generate_output`` is True, we will also generate output data + If ``data_path`` is not ``None``, we will also generate output data to ``output_data``. """ + test_parameters: TestParameters = test_pipeline.parameters + test_status: TestStatus = test_pipeline.status + failed = test_status.failed print() - if total == 0: - print_and_log( - LOGFILE, f"No {entity_name} found with a name in: {entities_searched}." + if test_status.total == 0: + test_parameters.print_and_log( + f"No {entity_name} found with a name in: {entities_searched}.", ) if "MATHICS_DEBUG_TEST_CREATE" not in os.environ: print(f"Set environment MATHICS_DEBUG_TEST_CREATE to see {entity_name}.") elif failed > 0: print(SEP) - if not generate_output: - print_and_log( - LOGFILE, f"""{failed} test{'s' if failed != 1 else ''} failed.""" + if test_parameters.data_path is None: + test_pipeline.print_and_log( + f"""{failed} test{'s' if failed != 1 else ''} failed.""", ) else: - print_and_log(LOGFILE, "All tests passed.") + test_pipeline.print_and_log("All tests passed.") - if generate_output and (failed == 0 or keep_going): - save_doctest_data(output_data) - return + if test_parameters.data_path and (failed == 0 or test_parameters.keep_going): + save_doctest_data(test_pipeline) + + +def section_tests_iterator( + section, test_pipeline, include_subsections=None, exclude_sections=None +): + """ + Iterator over tests in a section. + A section contains tests in its documentation entry, + in the head of the chapter and in its subsections. + This function is a generator of all these tests. + + Before yielding a test from a documentation entry, + the user definitions are reset. + """ + chapter = section.chapter + subsections = [section] + if chapter.doc: + subsections = [chapter.doc] + subsections + if section.subsections: + subsections = subsections + section.subsections + + for subsection in subsections: + if ( + include_subsections is not None + and subsection.title not in include_subsections + ): + continue + if exclude_sections and subsection.title in exclude_sections: + continue + test_pipeline.reset_user_definitions() + + for tests in subsection.get_tests(): + if isinstance(tests, DocTests): + for test in tests: + yield test + else: + yield tests def test_section_in_chapter( + test_pipeline: DocTestPipeline, section: Union[DocSection, DocGuideSection], - total: int, - failed: int, - quiet, - stop_on_failure, - prev_key: list, include_sections: Optional[Set[str]] = None, - start_at: int = 0, - skipped: int = 0, - max_tests: int = MAX_TESTS, -) -> Tuple[int, int, list]: + exclude_sections: Optional[Set[str]] = None, +): """ Runs a tests for section ``section`` under a chapter or guide section. Note that both of these contain a collection of section tests underneath. - - ``total`` and ``failed`` give running tallies on the number of tests run and - the number of tests respectively. - - If ``quiet`` is True, the progress and results of the tests are shown. - If ``stop_on_failure`` is true then the remaining tests in a section are skipped when a test - fails. """ - section_name = section.title + test_parameters: TestParameters = test_pipeline.parameters + test_status: TestStatus = test_pipeline.status # Start out assuming all subsections will be tested include_subsections = None - - if include_sections is not None and section_name not in include_sections: + if include_sections is not None and section.title not in include_sections: # use include_section to filter subsections include_subsections = include_sections chapter = section.chapter - chapter_name = chapter.title - part_name = chapter.part.title index = 0 - if len(section.subsections) > 0: - for subsection in section.subsections: - if ( - include_subsections is not None - and subsection.title not in include_subsections - ): - continue - - DEFINITIONS.reset_user_definitions() - for test in subsection.get_tests(): - # Get key dropping off test index number - key = list(test.key)[1:-1] - if prev_key != key: - prev_key = key - section_name_for_print = " / ".join(key) - if quiet: - # We don't print with stars inside in test_case(), so print here. - print(f"Testing section: {section_name_for_print}") - index = 0 - else: - # Null out section name, so that on the next iteration we do not print a section header - # in test_case(). - section_name_for_print = "" - - if isinstance(test, DocTests): - for doctest in test.tests: - index += 1 - total += 1 - if not test_case( - doctest, - total, - index, - quiet=quiet, - section_name=section_name, - section_for_print=section_name_for_print, - chapter_name=chapter_name, - part=part_name, - ): - failed += 1 - if stop_on_failure: - break - elif test.ignore: - continue - - else: - index += 1 - - if index < start_at: - skipped += 1 - continue - - total += 1 - if not test_case( - test, - total, - index, - quiet=quiet, - section_name=section_name, - section_for_print=section_name_for_print, - chapter_name=chapter_name, - part=part_name, - ): - failed += 1 - if stop_on_failure: - break - pass - pass - pass - pass - else: - if include_subsections is None or section.title in include_subsections: - DEFINITIONS.reset_user_definitions() - for test in section.get_tests(): - # Get key dropping off test index number - key = list(test.key)[1:-1] - if prev_key != key: - prev_key = key - section_name_for_print = " / ".join(key) - if quiet: - print(f"Testing section: {section_name_for_print}") - index = 0 - else: - # Null out section name, so that on the next iteration we do not print a section header. - section_name_for_print = "" - - if test.ignore: - continue - - else: - index += 1 - - if index < start_at: - skipped += 1 - continue - - total += 1 - if total >= max_tests: - break - - if not test_case( - test, - total, - index, - quiet=quiet, - section_name=section.title, - section_for_print=section_name_for_print, - chapter_name=chapter.title, - part=part_name, - ): - failed += 1 - if stop_on_failure: - break - pass - pass - - pass - return total, failed, prev_key - - -# When 3.8 is base, the below can be a Literal type. -INVALID_TEST_GROUP_SETUP = (None, None) - - -def validate_group_setup( - include_set: set, - entity_name: Optional[str], - reload: bool, -) -> tuple: - """ - Common things that need to be done before running a group of doctests. - """ + subsections = [section] + if chapter.doc: + subsections = [chapter.doc] + subsections + if section.subsections: + subsections = subsections + section.subsections + + section_name_for_print = "" + for doctest in section_tests_iterator( + section, test_pipeline, include_subsections, exclude_sections + ): + if doctest.ignore: + continue + section_name_for_print = test_status.section_name_for_print(doctest) + test_status.show_section(doctest) + key = list(doctest.key)[1:-1] + if key != test_status.prev_key: + index = 1 + else: + index += 1 + test_status.prev_key = key + test_status.total += 1 + if test_status.total > test_parameters.max_tests: + return + if test_status.total < test_parameters.start_at: + test_status.skipped += 1 + continue - if DOCUMENTATION is None: - print_and_log(LOGFILE, "Documentation is not initialized.") - return INVALID_TEST_GROUP_SETUP + def fail_message(why): + test_pipeline.print_and_log( + (f"""{SEP}Test failed: in {section_name_for_print}\n""" f"""{why}"""), + ) + return False - if entity_name is not None: - include_names = ", ".join(include_set) - print(f"Testing {entity_name}(s): {include_names}") - else: - include_names = None + test_status.show_test(doctest, test_status.total, index) - if reload: - doctest_latex_data_path = settings.get_doctest_latex_data_path( - should_be_readable=True + success = test_case( + doctest, + test_pipeline, + fail=fail_message, ) - output_data = load_doctest_data(doctest_latex_data_path) - else: - output_data = {} - - # For consistency set the character encoding ASCII which is - # the lowest common denominator available on all systems. - settings.SYSTEM_CHARACTER_ENCODING = "ASCII" + if not success: + test_status.mark_as_failed(doctest.key[:-1]) + if not test_pipeline.parameters.keep_going: + return - if DEFINITIONS is None: - print_and_log(LOGFILE, "Definitions are not initialized.") - return INVALID_TEST_GROUP_SETUP - - # Start with a clean variables state from whatever came before. - # In the test suite however, we may set new variables. - DEFINITIONS.reset_user_definitions() - return output_data, include_names + return def test_tests( - index: int, - quiet: bool = False, - stop_on_failure: bool = False, - start_at: int = 0, - max_tests: int = MAX_TESTS, - excludes: Set[str] = set(), - generate_output: bool = False, - reload: bool = False, - keep_going: bool = False, -) -> Tuple[int, int, int, set, int]: + test_pipeline: DocTestPipeline, + excludes: Optional[Set[str]] = None, +): """ - Runs a group of related tests, ``Tests`` provided that the section is not listed in ``excludes`` and - the global test count given in ``index`` is not before ``start_at``. + Runs a group of related tests, ``Tests`` provided that the section is not + listed in ``excludes`` and the global test count given in ``index`` is not + before ``start_at``. - Tests are from a section or subsection (when the section is a guide section), + Tests are from a section or subsection (when the section is a guide + section). If ``quiet`` is True, the progress and results of the tests + are shown. - If ``quiet`` is True, the progress and results of the tests are shown. - - ``index`` has the current count. We will stop on the first failure if ``stop_on_failure`` is true. + ``index`` has the current count. We will stop on the first failure + if ``keep_going`` is false. """ + test_status: TestStatus = test_pipeline.status + test_parameters: TestParameters = test_pipeline.parameters + # For consistency set the character encoding ASCII which is + # the lowest common denominator available on all systems. - total = index = failed = skipped = 0 - prev_key = [] - failed_symbols = set() + settings.SYSTEM_CHARACTER_ENCODING = "ASCII" + test_pipeline.reset_user_definitions() - output_data, names = validate_group_setup( + output_data, names = test_pipeline.validate_group_setup( set(), None, - reload, ) if (output_data, names) == INVALID_TEST_GROUP_SETUP: - return total, failed, skipped, failed_symbols, index + return - for part in DOCUMENTATION.parts: + # Loop over the whole documentation. + for part in test_pipeline.documentation.parts: for chapter in part.chapters: for section in chapter.all_sections: section_name = section.title - if section_name in excludes: + if excludes and section_name in excludes: continue - if total >= max_tests: - break - ( - total, - failed, - prev_key, - ) = test_section_in_chapter( + if test_status.total >= test_parameters.max_tests: + show_test_summary( + test_pipeline, + "chapters", + "", + ) + return + test_section_in_chapter( + test_pipeline, section, - total, - failed, - quiet, - stop_on_failure, - prev_key, - start_at=start_at, - max_tests=max_tests, + exclude_sections=excludes, ) - if generate_output and failed == 0: - create_output(section.doc.get_tests(), output_data) - pass - pass - + if test_status.failed_sections: + if not test_parameters.keep_going: + show_test_summary( + test_pipeline, + "chapters", + "", + ) + return + else: + if test_parameters.data_path: + create_output( + test_pipeline, + section_tests_iterator( + section, + test_pipeline, + exclude_sections=excludes, + ), + ) show_test_summary( - total, - failed, + test_pipeline, "chapters", - names, - keep_going, - generate_output, - output_data, + "", ) - if generate_output and (failed == 0 or keep_going): - save_doctest_data(output_data) - return total, failed, skipped, failed_symbols, index + return def test_chapters( + test_pipeline: DocTestPipeline, include_chapters: set, - quiet=False, - stop_on_failure=False, - generate_output=False, - reload=False, - keep_going=False, - start_at: int = 0, - max_tests: int = MAX_TESTS, -) -> int: + exclude_sections: set, +): """ Runs a group of related tests for the set specified in ``chapters``. If ``quiet`` is True, the progress and results of the tests are shown. - - If ``stop_on_failure`` is true then the remaining tests in a section are skipped when a test - fails. """ - failed = total = 0 + test_status = test_pipeline.status + test_parameters = test_pipeline.parameters - output_data, chapter_names = validate_group_setup( - include_chapters, "chapters", reload + output_data, chapter_names = test_pipeline.validate_group_setup( + include_chapters, "chapters" ) if (output_data, chapter_names) == INVALID_TEST_GROUP_SETUP: - return total - - prev_key = [] - seen_chapters = set() + return - for part in DOCUMENTATION.parts: - for chapter in part.chapters: - chapter_name = chapter.title - if chapter_name not in include_chapters: + for chapter_name in include_chapters: + chapter_slug = slugify(chapter_name) + for part in test_pipeline.documentation.parts: + chapter = part.chapters_by_slug.get(chapter_slug, None) + if chapter is None: continue - seen_chapters.add(chapter_name) - for section in chapter.all_sections: - ( - total, - failed, - prev_key, - ) = test_section_in_chapter( + test_section_in_chapter( + test_pipeline, section, - total, - failed, - quiet, - stop_on_failure, - prev_key, - start_at=start_at, - max_tests=max_tests, + exclude_sections=exclude_sections, ) - if generate_output and failed == 0: - create_output(section.doc.get_tests(), output_data) - pass - pass - - if seen_chapters == include_chapters: - break - if chapter_name in include_chapters: - seen_chapters.add(chapter_name) - pass + if test_parameters.data_path is not None and test_status.failed == 0: + create_output( + test_pipeline, + section.doc.get_tests(), + ) show_test_summary( - total, - failed, + test_pipeline, "chapters", chapter_names, - keep_going, - generate_output, - output_data, ) - return total + + return def test_sections( + test_pipeline: DocTestPipeline, include_sections: set, - quiet=False, - stop_on_failure=False, - generate_output=False, - reload=False, - keep_going=False, -) -> int: + exclude_subsections: set, +): """Runs a group of related tests for the set specified in ``sections``. If ``quiet`` is True, the progress and results of the tests are shown. - ``index`` has the current count. If ``stop_on_failure`` is true + ``index`` has the current count. If ``keep_going`` is false then the remaining tests in a section are skipped when a test fails. If ``keep_going`` is True and there is a failure, the next section is continued after failure occurs. """ + test_status = test_pipeline.status + test_parameters = test_pipeline.parameters - total = failed = 0 - prev_key = [] - - output_data, section_names = validate_group_setup( - include_sections, "section", reload + output_data, section_names = test_pipeline.validate_group_setup( + include_sections, "section" ) if (output_data, section_names) == INVALID_TEST_GROUP_SETUP: - return total + return seen_sections = set() seen_last_section = False last_section_name = None section_name_for_finish = None - prev_key = [] - for part in DOCUMENTATION.parts: + for part in test_pipeline.documentation.parts: for chapter in part.chapters: for section in chapter.all_sections: - ( - total, - failed, - prev_key, - ) = test_section_in_chapter( + test_section_in_chapter( + test_pipeline, section=section, - total=total, - quiet=quiet, - failed=failed, - stop_on_failure=stop_on_failure, - prev_key=prev_key, include_sections=include_sections, + exclude_sections=exclude_subsections, ) - if generate_output and failed == 0: - create_output(section.doc.get_tests(), output_data) - pass + if test_parameters.data_path is not None and test_status.failed == 0: + create_output( + test_pipeline, + section.doc.get_tests(), + ) if last_section_name != section_name_for_finish: if seen_sections == include_sections: @@ -665,107 +632,76 @@ def test_sections( if section_name_for_finish in include_sections: seen_sections.add(section_name_for_finish) last_section_name = section_name_for_finish - pass if seen_last_section: - break - pass + show_test_summary(test_pipeline, "sections", section_names) + return - show_test_summary( - total, - failed, - "sections", - section_names, - keep_going, - generate_output, - output_data, - ) - return total + show_test_summary(test_pipeline, "sections", section_names) + return + + +def show_report(test_pipeline): + """Print a report with the results of the tests""" + test_status = test_pipeline.status + test_parameters = test_pipeline.parameters + total, failed = test_status.total, test_status.failed + builtin_total = test_pipeline.builtin_total + skipped = test_status.skipped + if test_parameters.max_tests == MAX_TESTS: + test_pipeline.print_and_log( + f"{total} Tests for {builtin_total} built-in symbols, {total-failed} " + f"passed, {failed} failed, {skipped} skipped.", + ) + else: + test_pipeline.print_and_log( + f"{total} Tests, {total - failed} passed, {failed} failed, {skipped} " + "skipped.", + ) + if test_status.failed_sections: + if not test_pipeline.parameters.keep_going: + test_pipeline.print_and_log( + "(not all tests are accounted for due to --)", + ) + test_pipeline.print_and_log("Failed:") + for part, chapter, section in sorted(test_status.failed_sections): + test_pipeline.print_and_log(f" - {section} in {part} / {chapter}") + + if test_parameters.data_path is not None and ( + test_status.failed == 0 or test_parameters.doc_even_if_error + ): + save_doctest_data(test_pipeline) + return def test_all( - quiet=False, - generate_output=True, - stop_on_failure=False, - start_at=0, - max_tests: int = MAX_TESTS, - texdatafolder=None, - doc_even_if_error=False, - excludes: set = set(), -) -> int: - if not quiet: + test_pipeline: DocTestPipeline, + excludes: Optional[Set[str]] = None, +): + """ + Run all the tests in the documentation. + """ + test_parameters = test_pipeline.parameters + test_status = test_pipeline.status + if not test_parameters.quiet: print(f"Testing {version_string}") - if generate_output: - if texdatafolder is None: - texdatafolder = osp.dirname( - settings.get_doctest_latex_data_path( - should_be_readable=False, create_parent=True - ) - ) - - total = failed = skipped = 0 try: - index = 0 - failed_symbols = set() - output_data = {} - sub_total, sub_failed, sub_skipped, symbols, index = test_tests( - index, - quiet=quiet, - stop_on_failure=stop_on_failure, - start_at=start_at, - max_tests=max_tests, + test_tests( + test_pipeline, excludes=excludes, - generate_output=generate_output, - reload=False, - keep_going=not stop_on_failure, ) - - total += sub_total - failed += sub_failed - skipped += sub_skipped - failed_symbols.update(symbols) - builtin_total = len(_builtins) except KeyboardInterrupt: print("\nAborted.\n") - return total + return - if failed > 0: + if test_status.failed > 0: print(SEP) - if max_tests == MAX_TESTS: - print_and_log( - LOGFILE, - f"{total} Tests for {builtin_total} built-in symbols, {total-failed} " - f"passed, {failed} failed, {skipped} skipped.", - ) - else: - print_and_log( - LOGFILE, - f"{total} Tests, {total - failed} passed, {failed} failed, {skipped} " - "skipped.", - ) - if failed_symbols: - if stop_on_failure: - print_and_log( - LOGFILE, "(not all tests are accounted for due to --stop-on-failure)" - ) - print_and_log(LOGFILE, "Failed:") - for part, chapter, section in sorted(failed_symbols): - print_and_log(LOGFILE, f" - {section} in {part} / {chapter}") - - if generate_output and (failed == 0 or doc_even_if_error): - save_doctest_data(output_data) - return total - if failed == 0: - print("\nOK") - else: - print("\nFAILED") - sys.exit(1) # Travis-CI knows the tests have failed - return total + show_report(test_pipeline) -def save_doctest_data(output_data: Dict[tuple, dict]): +def save_doctest_data(doctest_pipeline: DocTestPipeline): """ Save doctest tests and test results to a Python PCL file. @@ -779,9 +715,14 @@ def save_doctest_data(output_data: Dict[tuple, dict]): * test number and the value is a dictionary of a Result.getdata() dictionary. """ - doctest_latex_data_path = settings.get_doctest_latex_data_path( - should_be_readable=False, create_parent=True - ) + output_data: Dict[tuple, dict] = doctest_pipeline.output_data + + if len(output_data) == 0: + print("output data is empty") + return + print("saving", len(output_data), "entries") + print(output_data.keys()) + doctest_latex_data_path = doctest_pipeline.parameters.data_path print(f"Writing internal document data to {doctest_latex_data_path}") i = 0 for key in output_data: @@ -793,41 +734,47 @@ def save_doctest_data(output_data: Dict[tuple, dict]): pickle.dump(output_data, output_file, 4) -def write_doctest_data(quiet=False, reload=False): +def write_doctest_data(doctest_pipeline: DocTestPipeline): """ Get doctest information, which involves running the tests to obtain test results and write out both the tests and the test results. """ - if not quiet: + test_parameters = doctest_pipeline.parameters + if not test_parameters.quiet: print(f"Extracting internal doc data for {version_string}") print("This may take a while...") try: - output_data = load_doctest_data() if reload else {} - for tests in DOCUMENTATION.get_tests(): - create_output(tests, output_data) + doctest_pipeline.output_data = ( + load_doctest_data(test_parameters.data_path) + if test_parameters.reload + else {} + ) + for tests in doctest_pipeline.documentation.get_tests(): + create_output( + doctest_pipeline, + tests, + ) except KeyboardInterrupt: print("\nAborted.\n") return print("done.\n") - save_doctest_data(output_data) + save_doctest_data(doctest_pipeline) -def main(): - global DEFINITIONS - global LOGFILE - global CHECK_PARTIAL_ELAPSED_TIME - - import_and_load_builtins() - DEFINITIONS = Definitions(add_builtin=True) +def build_arg_parser(): + """Build the argument parser""" parser = ArgumentParser(description="Mathics test suite.", add_help=False) parser.add_argument( "--help", "-h", help="show this help message and exit", action="help" ) parser.add_argument( - "--version", "-v", action="version", version="%(prog)s " + mathics.__version__ + "--version", + "-v", + action="version", + version="%(prog)s " + mathics.__version__, ) parser.add_argument( "--chapters", @@ -889,7 +836,10 @@ def main(): "--doc-only", dest="doc_only", action="store_true", - help="generate pickled internal document data without running tests; Can't be used with --section or --reload.", + help=( + "generate pickled internal document data without running tests; " + "Can't be used with --section or --reload." + ), ) parser.add_argument( "--reload", @@ -899,7 +849,11 @@ def main(): help="reload pickled internal document data, before possibly adding to it", ) parser.add_argument( - "--quiet", "-q", dest="quiet", action="store_true", help="hide passed tests" + "--quiet", + "-q", + dest="quiet", + action="store_true", + help="hide passed tests", ) parser.add_argument( "--keep-going", @@ -909,7 +863,11 @@ def main(): help="create documentation even if there is a test failure", ) parser.add_argument( - "--stop-on-failure", "-x", action="store_true", help="stop on failure" + "--stop-on-failure", + "-x", + dest="stop_on_failure", + action="store_true", + help="stop on failure", ) parser.add_argument( "--skip", @@ -932,93 +890,54 @@ def main(): action="store_true", help="print cache statistics", ) - global LOGFILE - - args = parser.parse_args() - - if args.elapsed_times: - CHECK_PARTIAL_ELAPSED_TIME = True - # If a test for a specific section is called - # just test it - if args.logfilename: - LOGFILE = open(args.logfilename, "wt") - - global DOCUMENTATION - DOCUMENTATION = MathicsMainDocumentation() - - # LoadModule Mathics3 modules - if args.pymathics: - for module_name in args.pymathics.split(","): - try: - eval_LoadModule(module_name, DEFINITIONS) - except PyMathicsLoadException: - print(f"Python module {module_name} is not a Mathics3 module.") - - except Exception as e: - print(f"Python import errors with: {e}.") - else: - print(f"Mathics3 Module {module_name} loaded") + return parser.parse_args() - DOCUMENTATION.load_documentation_sources() - start_time = None - total = 0 +def main(): + """main""" + args = build_arg_parser() + data_path = ( + get_doctest_latex_data_path(should_be_readable=False, create_parent=True) + if args.output + else None + ) + + test_pipeline = DocTestPipeline(args, output_format="latex", data_path=data_path) + test_status = test_pipeline.status if args.sections: include_sections = set(args.sections.split(",")) - + exclude_subsections = set(args.exclude.split(",")) start_time = datetime.now() - total = test_sections( - include_sections, - stop_on_failure=args.stop_on_failure, - generate_output=args.output, - reload=args.reload, - keep_going=args.keep_going, - ) + test_sections(test_pipeline, include_sections, exclude_subsections) elif args.chapters: start_time = datetime.now() - start_at = args.skip + 1 include_chapters = set(args.chapters.split(",")) - - total = test_chapters( - include_chapters, - stop_on_failure=args.stop_on_failure, - generate_output=args.output, - reload=args.reload, - start_at=start_at, - max_tests=args.count, - ) + exclude_sections = set(args.exclude.split(",")) + test_chapters(test_pipeline, include_chapters, exclude_sections) else: if args.doc_only: - write_doctest_data( - quiet=args.quiet, - reload=args.reload, - ) + write_doctest_data(test_pipeline) else: excludes = set(args.exclude.split(",")) - start_at = args.skip + 1 start_time = datetime.now() - total = test_all( - quiet=args.quiet, - generate_output=args.output, - stop_on_failure=args.stop_on_failure, - start_at=start_at, - max_tests=args.count, - doc_even_if_error=args.keep_going, - excludes=excludes, - ) - pass - pass + test_all(test_pipeline, excludes=excludes) - if total > 0 and start_time is not None: - end_time = datetime.now() - print("Test evalation took ", end_time - start_time) + if test_status.total > 0 and start_time is not None: + print("Test evaluation took ", datetime.now() - start_time) - if LOGFILE: - LOGFILE.close() + if test_pipeline.logfile: + test_pipeline.logfile.close() if args.show_statistics: show_lru_cache_statistics() + if test_status.failed == 0: + print("\nOK") + else: + print("\nFAILED") + sys.exit(1) # Travis-CI knows the tests have failed + if __name__ == "__main__": + import_and_load_builtins() main() diff --git a/mathics/eval/__init__.py b/mathics/eval/__init__.py index e66f87c70..a883eda56 100644 --- a/mathics/eval/__init__.py +++ b/mathics/eval/__init__.py @@ -5,7 +5,7 @@ evaluation. If there were an instruction interpreter, these functions that start "eval_" would be the interpreter instructions. -These operatations then should include the most commonly-used Builtin-functions like +These operations then should include the most commonly-used Builtin-functions like ``N[]`` and routines in support of performing those evaluation operations/instructions. Performance of the operations here can be important for overall interpreter performance. diff --git a/mathics/eval/arithmetic.py b/mathics/eval/arithmetic.py index 035dff801..3a219c03d 100644 --- a/mathics/eval/arithmetic.py +++ b/mathics/eval/arithmetic.py @@ -339,12 +339,10 @@ def eval_mpmath_function( return call_mpmath(mpmath_function, tuple(float_args), FP_MANTISA_BINARY_DIGITS) else: - with mpmath.workprec(prec): - # to_mpmath seems to require that the precision is set from outside - mpmath_args = [x.to_mpmath() for x in args] - if None in mpmath_args: - return - return call_mpmath(mpmath_function, tuple(mpmath_args), prec) + mpmath_args = [x.to_mpmath(prec) for x in args] + if None in mpmath_args: + return + return call_mpmath(mpmath_function, tuple(mpmath_args), prec) def eval_Plus(*items: BaseElement) -> BaseElement: diff --git a/mathics/eval/files_io/files.py b/mathics/eval/files_io/files.py index e7163f703..3678f6de1 100644 --- a/mathics/eval/files_io/files.py +++ b/mathics/eval/files_io/files.py @@ -17,7 +17,7 @@ from mathics.core.util import canonic_filename # Python representation of $InputFileName. On Windows platforms, we -# canonicalize this to its Posix equvivalent name. +# canonicalize this to its Posix equivalent name. # FIXME: Remove this as a module-level variable and instead # define it in a session definitions object. # With this, multiple sessions will have separate diff --git a/mathics/eval/image.py b/mathics/eval/image.py index c06a7e8d8..acf4874ae 100644 --- a/mathics/eval/image.py +++ b/mathics/eval/image.py @@ -114,7 +114,7 @@ def extract_exif(image, evaluation: Evaluation) -> Optional[Expression]: # EXIF has the following types: Short, Long, Rational, Ascii, Byte # (see http://www.exiv2.org/tags.html). we detect the type from the - # Python type Pillow gives us and do the appropiate MMA handling. + # Python type Pillow gives us and do the appropriate MMA handling. if isinstance(v, tuple) and len(v) == 2: # Rational value = Rational(v[0], v[1]) @@ -298,7 +298,7 @@ def resize_width_height( return image.filter(lambda im: im.resize((width, height), resample=resample)) - # The Below code is hand-crapted Guassian resampling code, which is what + # The Below code is hand-crapted Gaussian resampling code, which is what # WMA does. For now, are going to punt on this, and we use PIL methods only. # Gaussian need sto unrounded values to compute scaling ratios. @@ -330,7 +330,7 @@ def resize_width_height( # kwargs = {"downscale": (1.0 / s)} # # scikit_image in version 0.19 changes the resize parameter deprecating # # "multichannel". scikit_image also doesn't support older Pythons like 3.6.15. - # # If we drop suport for 3.6 we can probably remove + # # If we drop support for 3.6 we can probably remove # if skimage_version >= "0.19": # # Not totally sure that we want channel_axis=1, but it makes the # # test work. multichannel is deprecated in scikit-image-19.2 diff --git a/mathics/eval/makeboxes.py b/mathics/eval/makeboxes.py index 1ed651c34..770d53d7b 100644 --- a/mathics/eval/makeboxes.py +++ b/mathics/eval/makeboxes.py @@ -68,7 +68,7 @@ def _boxed_string(string: str, **options): # 640 = sys.int_info.str_digits_check_threshold. -# Someday when 3.11 is the minumum version of Python supported, +# Someday when 3.11 is the minimum version of Python supported, # we can replace the magic value 640 below with sys.int.str_digits_check_threshold. def int_to_string_shorter_repr(value: Integer, form: Symbol, max_digits=640): """Convert value to a String, restricted to max_digits characters. @@ -94,7 +94,7 @@ def int_to_string_shorter_repr(value: Integer, form: Symbol, max_digits=640): # Estimate the number of decimal digits num_digits = int(value.bit_length() * 0.3) - # If the estimated number is bellow the threshold, + # If the estimated number is below the threshold, # return it as it is. if num_digits <= max_digits: if is_negative: @@ -103,7 +103,7 @@ def int_to_string_shorter_repr(value: Integer, form: Symbol, max_digits=640): # estimate the size of the placeholder size_placeholder = len(str(num_digits)) + 6 - # Estimate the number of avaliable decimal places + # Estimate the number of available decimal places avaliable_digits = max(max_digits - size_placeholder, 0) # how many most significative digits include len_msd = (avaliable_digits + 1) // 2 diff --git a/mathics/eval/nevaluator.py b/mathics/eval/nevaluator.py index 79d872507..c662f8b30 100644 --- a/mathics/eval/nevaluator.py +++ b/mathics/eval/nevaluator.py @@ -90,7 +90,7 @@ def eval_NValues( # Here we look for the NValues associated to the # lookup_name of the expression. - # If a rule is found and successfuly applied, + # If a rule is found and successfully applied, # reevaluate the result and apply `eval_NValues` again. # This should be implemented as a loop instead of # recursively. diff --git a/mathics/eval/numbers/calculus/optimizers.py b/mathics/eval/numbers/calculus/optimizers.py index cfdba2b5a..4798f1362 100644 --- a/mathics/eval/numbers/calculus/optimizers.py +++ b/mathics/eval/numbers/calculus/optimizers.py @@ -391,9 +391,9 @@ def is_zero( eps_expr: BaseElement = Integer10 ** (-prec_goal) if prec_goal else Integer0 if acc_goal: eps_expr = eps_expr + Integer10 ** (-acc_goal) / abs(val) - threeshold_expr = Expression(SymbolLog, eps_expr) - threeshold: Real = eval_N(threeshold_expr, evaluation) - return threeshold.to_python() > 0 + threshold_expr = Expression(SymbolLog, eps_expr) + threshold: Real = eval_N(threshold_expr, evaluation) + return threshold.to_python() > 0 def determine_epsilon(x0: Real, options: dict, evaluation: Evaluation) -> Real: diff --git a/mathics/eval/numbers/calculus/series.py b/mathics/eval/numbers/calculus/series.py index abdff9da6..a998a0c3d 100644 --- a/mathics/eval/numbers/calculus/series.py +++ b/mathics/eval/numbers/calculus/series.py @@ -81,7 +81,7 @@ def same_monomial(expr, x, x0): # coeffs_powers = [] # coeffs_x = [] # for element in elements: -# if x.sameQ(elemnt): +# if x.sameQ(element): # coeffs_x.append(x) # elif isinstance(element, Atom): # coeffs_free.append(element) diff --git a/mathics/eval/parts.py b/mathics/eval/parts.py index 61a1adf33..910f96ed5 100644 --- a/mathics/eval/parts.py +++ b/mathics/eval/parts.py @@ -59,7 +59,7 @@ def get_subpart(sub_expression: BaseElement, sub_indices: List[int]) -> BaseElem def set_part(expression, indices: List[int], new_atom: Atom) -> BaseElement: - """Replace all parts of ``expression`` specified by ``indicies`` with + """Replace all parts of ``expression`` specified by ``indices`` with ``new_atom`. Return the modified compound expression. """ @@ -435,7 +435,7 @@ def python_seq(start, stop, step, length): if start == 0 or stop == 0: return None - # wrap negative values to postive and convert from 1-based to 0-based + # wrap negative values to positive and convert from 1-based to 0-based if start < 0: start += length else: @@ -547,7 +547,7 @@ def sliced(x, s): def deletecases_with_levelspec(expr, pattern, evaluation, levelspec=1, n=-1): """ - This function walks the expression `expr` and deleting occurrencies of `pattern` + This function walks the expression `expr` and deleting occurrences of `pattern` If levelspec specifies a number, only those positions with `levelspec` "coordinates" are return. By default, it just return diff --git a/mathics/eval/plot.py b/mathics/eval/plot.py index 691616bbc..a7edb5a0d 100644 --- a/mathics/eval/plot.py +++ b/mathics/eval/plot.py @@ -248,7 +248,7 @@ def eval_ListPlot( is_axis_filling = is_discrete_plot if filling == "System`Axis": - # TODO: Handle arbitary axis intercepts + # TODO: Handle arbitrary axis intercepts filling = 0.0 is_axis_filling = True elif filling == "System`Bottom": diff --git a/mathics/eval/testing_expressions.py b/mathics/eval/testing_expressions.py index 2bd751944..503e20a60 100644 --- a/mathics/eval/testing_expressions.py +++ b/mathics/eval/testing_expressions.py @@ -36,11 +36,13 @@ def do_cmp(x1, x2) -> Optional[int]: # we don't want to compare anything that # cannot be represented as a numeric value if s1.is_number and s2.is_number: - if s1 == s2: + delta = s1 - s2 + if delta.is_zero: return 0 - if s1 < s2: + if delta.is_extended_negative: return -1 - return 1 + if delta.is_extended_positive: + return 1 return None diff --git a/mathics/format/__init__.py b/mathics/format/__init__.py index 3c35e0041..62318a824 100644 --- a/mathics/format/__init__.py +++ b/mathics/format/__init__.py @@ -16,7 +16,7 @@ For example, in graphics we may be several different kinds of renderers, SVG, or Asymptote for a particular kind of graphics Box. -The front-end nees to decides which format it better suited for it. +The front-end needs to decides which format it better suited for it. The Box, however, is created via a particular high-level Form. As another example, front-end may decide to use MathJaX to render diff --git a/mathics/format/latex.py b/mathics/format/latex.py index 6f83062d3..6aee6d034 100644 --- a/mathics/format/latex.py +++ b/mathics/format/latex.py @@ -408,7 +408,7 @@ def graphics3dbox(self, elements=None, **options) -> str: # TODO: Intelligently place the axes on the longest non-middle edge. # See algorithm used by web graphics in mathics/web/media/graphics.js - # for details of this. (Projection to sceen etc). + # for details of this. (Projection to screen etc). # Choose axes placement (boundbox edge vertices) axes_indices = [] diff --git a/mathics/format/svg.py b/mathics/format/svg.py index df66c5958..302ceecfd 100644 --- a/mathics/format/svg.py +++ b/mathics/format/svg.py @@ -199,7 +199,7 @@ def density_plot_box(self, **options): # since it is a cute idea, it is worthy of comment space... Put # two triangles together to get a parallelogram. Compute the # midpoint color in the enter and along all four sides. Then use - # two overlayed rectangular gradients each at opacity 0.5 + # two overlaid rectangular gradients each at opacity 0.5 # to go from the center to each of the (square) sides. svg_data = ["<--DensityPlot-->"] @@ -258,10 +258,10 @@ def graphics_box(self, elements=None, **options: dict) -> str: ``elements`` could be a ``GraphicsElements`` object, a tuple or a list. - Options is a dictionary of Graphics options dictionary. Intersting Graphics options keys: + Options is a dictionary of Graphics options dictionary. Interesting Graphics options keys: ``data``: a tuple bounding box information as well as a copy of ``elements``. If given - this supercedes the information in the ``elements`` parameter. + this supersedes the information in the ``elements`` parameter. ``evaluation``: an ``Evaluation`` object that can be used when further evaluation is needed. """ @@ -310,7 +310,7 @@ def graphics_box(self, elements=None, **options: dict) -> str: tooltip_text = self.tooltip_text if hasattr(self, "tooltip_text") else "" if self.background_color is not None: - # FIXME: tests don't seem to cover this secton of code. + # FIXME: tests don't seem to cover this section of code. # Wrap svg_elements in a rectangle background = "rgba(100%,100%,100%,100%)" diff --git a/mathics/format/text.py b/mathics/format/text.py index 3d4be51e4..422ce940a 100644 --- a/mathics/format/text.py +++ b/mathics/format/text.py @@ -73,7 +73,7 @@ def gridbox(self, elements=None, **box_options) -> str: cells = [ [ - # TODO: check if this evaluation is necesary. + # TODO: check if this evaluation is necessary. boxes_to_text(item, **box_options).splitlines() for item in row ] diff --git a/mathics/main.py b/mathics/main.py index 18a5a139f..1998291d2 100755 --- a/mathics/main.py +++ b/mathics/main.py @@ -355,7 +355,7 @@ def main() -> int: argparser.add_argument( "--strict-wl-output", - help="Most WL-output compatible (at the expense of useability).", + help="Most WL-output compatible (at the expense of usability).", action="store_true", ) diff --git a/mathics/packages/DiscreteMath/CombinatoricaV0.9.m b/mathics/packages/DiscreteMath/CombinatoricaV0.9.m index f0e5fe3db..cbcb92834 100644 --- a/mathics/packages/DiscreteMath/CombinatoricaV0.9.m +++ b/mathics/packages/DiscreteMath/CombinatoricaV0.9.m @@ -596,7 +596,7 @@ Permute[l_List,p_?PermutationQ] := l [[ p ]] Permute[l_List,p_List] := Map[ (Permute[l,#])&, p] /; (Apply[And, Map[PermutationQ, p]]) -(* Section 1.1.1 Lexicographically Ordered Permutions, Pages 3-4 *) +(* Section 1.1.1 Lexicographically Ordered Permutations, Pages 3-4 *) LexicographicPermutations[{}] := {{}} @@ -683,7 +683,7 @@ ] ] -(* Section 1.1.5 Backtracking and Distict Permutations, Page 12-13 *) +(* Section 1.1.5 Backtracking and Distinct Permutations, Page 12-13 *) Backtrack[space_List,partialQ_,solutionQ_,flag_:One] := Module[{n=Length[space],all={},done,index,v=2,solution}, index=Prepend[ Table[0,{n-1}],1]; diff --git a/mathics/packages/Utilities/CleanSlate.m b/mathics/packages/Utilities/CleanSlate.m index ecf21fda1..fbff34895 100644 --- a/mathics/packages/Utilities/CleanSlate.m +++ b/mathics/packages/Utilities/CleanSlate.m @@ -202,7 +202,7 @@ the context search path ($ContextPath) since the CleanSlate package was \ incorrectly specified, or is not on $ContextPath."; CleanSlate::nopurge = "The context `1` cannot be purged, because it was \ -present when the CleanSlate package was initally read in."; +present when the CleanSlate package was initially read in."; CleanSlate::noself = "CleanSlate cannot purge its own context."; diff --git a/mathics/packages/VectorAnalysis/VectorAnalysis.m b/mathics/packages/VectorAnalysis/VectorAnalysis.m index bac1eee2d..6efd750fc 100644 --- a/mathics/packages/VectorAnalysis/VectorAnalysis.m +++ b/mathics/packages/VectorAnalysis/VectorAnalysis.m @@ -26,7 +26,7 @@ DotProduct::usage = "DotProduct[v1, v2] gives the dot product between v1 and v2 in three spatial dimensions. DotProduct[v1, v2, coordsys] gives the dot product of vectors v1 -and v2 in the specified coodrinate system, coordsys."; +and v2 in the specified coordinate system, coordsys."; DotProduct[v1_?$IsVecQ, v2_?$IsVecQ, coordsys_:CoordinateSystem] := Module[{c1, c2}, @@ -42,7 +42,7 @@ CrossProduct::usage = "CrossProduct[v1, v2] gives the cross product between v1 and v2 in three spatial dimensions. DotProduct[v1, v2, coordsys] gives the cross product of -vectors v1 and v2 in the specified coodrinate system, coordsys."; +vectors v1 and v2 in the specified coordinate system, coordsys."; CrossProduct[v1_?$IsVecQ, v2_?$IsVecQ, coordsys_:CoordinateSystem] := Module[{c1, c2}, @@ -59,7 +59,7 @@ "ScalarTripleProduct[v1, v2, v3] gives the scalar triple product product between v1, v2 and v3 in three spatial dimensions. ScalarTripleProduct[v1, v2, v3, coordsys] gives the scalar triple product of -vectors v1, v2 and v3 in the specified coodrinate system, coordsys."; +vectors v1, v2 and v3 in the specified coordinate system, coordsys."; ScalarTripleProduct[v1_?$IsVecQ, v2_?$IsVecQ, v3_?$IsVecQ, coordsys_:CoordinateSystem] := @@ -116,7 +116,7 @@ (* ============================ Coordinates ============================ *) Coordinates::usage = -"Coordinates[] gives the default cordinate variables of the current coordinate +"Coordinates[] gives the default coordinate variables of the current coordinate system. Coordinates[coordsys] gives the default coordinate variables of the specified coordinate system, coordsys."; @@ -133,8 +133,8 @@ (* ============================= Parameters ============================ *) Parameters::usage = -"Parameters[] gives the default paramater variables of the current coordinate -system. Parameters[coordsys] gives the default paramater variables for the +"Parameters[] gives the default parameter variables of the current coordinate +system. Parameters[coordsys] gives the default parameter variables for the specified coordinate system, coordsys."; Parameters[] := Parameters[CoordinateSystem]; diff --git a/mathics/session.py b/mathics/session.py index 874b61a2a..ccb8dd801 100644 --- a/mathics/session.py +++ b/mathics/session.py @@ -13,7 +13,7 @@ from typing import Optional from mathics.core.definitions import Definitions, autoload_files -from mathics.core.evaluation import Evaluation +from mathics.core.evaluation import Evaluation, Result from mathics.core.parser import MathicsSingleLineFeeder, parse @@ -79,7 +79,14 @@ def reset(self, add_builtin=True, catch_interrupt=False): """ reset the definitions and the evaluation objects. """ - self.definitions = Definitions(add_builtin) + try: + self.definitions = Definitions(add_builtin) + except KeyError: + from mathics.core.load_builtin import import_and_load_builtins + + import_and_load_builtins() + self.definitions = Definitions(add_builtin) + self.evaluation = Evaluation( definitions=self.definitions, catch_interrupt=catch_interrupt ) @@ -94,10 +101,20 @@ def evaluate(self, str_expression, timeout=None, form=None): self.last_result = expr.evaluate(self.evaluation) return self.last_result - def evaluate_as_in_cli(self, str_expression, timeout=None, form=None): + def evaluate_as_in_cli(self, str_expression, timeout=None, form=None, src_name=""): """This method parse and evaluate the expression using the session.evaluation.evaluate method""" - query = self.evaluation.parse(str_expression) - res = self.evaluation.evaluate(query) + query = self.evaluation.parse(str_expression, src_name) + if query is not None: + res = self.evaluation.evaluate(query, timeout=timeout, format=form) + else: + res = Result( + self.evaluation.out, + None, + self.evaluation.definitions.get_line_no(), + None, + form, + ) + self.evaluation.out = [] self.evaluation.stopped = False return res @@ -110,8 +127,10 @@ def format_result(self, str_expression=None, timeout=None, form=None): form = self.form return res.do_format(self.evaluation, form) - def parse(self, str_expression): + def parse(self, str_expression, src_name=""): """ Just parse the expression """ - return parse(self.definitions, MathicsSingleLineFeeder(str_expression)) + return parse( + self.definitions, MathicsSingleLineFeeder(str_expression, src_name) + ) diff --git a/mathics/settings.py b/mathics/settings.py index 4fe8ab5a9..12e3df8c6 100644 --- a/mathics/settings.py +++ b/mathics/settings.py @@ -60,11 +60,11 @@ def get_srcdir(): # In contrast to ROOT_DIR, LOCAL_ROOT_DIR is used in building # LaTeX documentation. When Mathics is installed, we don't want LaTeX file documentation.tex -# to get put in the installation directory, but instead we build documentaiton +# to get put in the installation directory, but instead we build documentation # from checked-out source and that is where this should be put. LOCAL_ROOT_DIR = get_srcdir() -# Location of doctests and test results formated for LaTeX. This data +# Location of doctests and test results formatted for LaTeX. This data # is stoared as a Python Pickle format, but storing this in JSON if # possible would be preferable and faster diff --git a/mathics/timing.py b/mathics/timing.py index 2f9538a61..f33bc8528 100644 --- a/mathics/timing.py +++ b/mathics/timing.py @@ -22,10 +22,10 @@ def long_running_function(): def timed(*args, **kw): method_name = method.__name__ # print(f"{date.today()} {method_name} starts") - ts = time.time() + t_start = time.time() result = method(*args, **kw) - te = time.time() - elapsed = (te - ts) * 1000 + t_end = time.time() + elapsed = (t_end - t_start) * 1000 if elapsed > MIN_ELAPSE_REPORT: if "log_time" in kw: name = kw.get("log_name", method.__name__.upper()) @@ -52,11 +52,11 @@ def __init__(self, name: str): def __enter__(self): # print(f"{date.today()} {method_name} starts") - self.ts = time.time() + self.t_start = time.time() def __exit__(self, exc_type, exc_value, exc_tb): - te = time.time() - elapsed = (te - self.ts) * 1000 + t_end = time.time() + elapsed = (t_end - self.t_start) * 1000 if elapsed > MIN_ELAPSE_REPORT: print("%r %2.2f ms" % (self.name, elapsed)) diff --git a/pyproject.toml b/pyproject.toml index 92b5262ac..105d39603 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,8 +1,11 @@ [build-system] requires = [ - "setuptools>=61.2", - "cython>=0.15.1; implementation_name!='pypy'" + "setuptools>=70.0.0", # CVE-2024-38335 recommends this + "cython>=0.15.1; implementation_name!='pypy'", + # For mathics-generate-json-table + "Mathics-Scanner >= 1.3.0", ] +build-backend = "setuptools.build_meta" [project] name = "Mathics3" @@ -21,9 +24,9 @@ dependencies = [ "python-dateutil", "requests", "setuptools", - "sympy>=1.8", + "sympy>=1.11,<1.13", ] -requires-python = ">=3.7" +requires-python = ">=3.8" # Sympy 1.11 is supported only down to 3.8 readme = "README.rst" license = {text = "GPL"} keywords = ["Mathematica", "Wolfram", "Interpreter", "Shell", "Math", "CAS"] @@ -35,7 +38,6 @@ classifiers = [ "Intended Audience :: Science/Research", "License :: OSI Approved :: GNU General Public License v3 (GPLv3)", "Programming Language :: Python", - "Programming Language :: Python :: 3.7", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", @@ -76,46 +78,9 @@ mathics = "mathics.main:main" [tool.setuptools] include-package-data = false -packages = [ - "mathics", - "mathics.algorithm", - "mathics.compile", - "mathics.core", - "mathics.core.convert", - "mathics.core.parser", - "mathics.builtin", - "mathics.builtin.arithfns", - "mathics.builtin.assignments", - "mathics.builtin.atomic", - "mathics.builtin.binary", - "mathics.builtin.box", - "mathics.builtin.colors", - "mathics.builtin.distance", - "mathics.builtin.exp_structure", - "mathics.builtin.drawing", - "mathics.builtin.fileformats", - "mathics.builtin.files_io", - "mathics.builtin.forms", - "mathics.builtin.functional", - "mathics.builtin.image", - "mathics.builtin.intfns", - "mathics.builtin.list", - "mathics.builtin.matrices", - "mathics.builtin.numbers", - "mathics.builtin.numpy_utils", - "mathics.builtin.pymimesniffer", - "mathics.builtin.pympler", - "mathics.builtin.quantum_mechanics", - "mathics.builtin.scipy_utils", - "mathics.builtin.specialfns", - "mathics.builtin.statistics", - "mathics.builtin.string", - "mathics.builtin.testing_expressions", - "mathics.builtin.vectors", - "mathics.eval", - "mathics.doc", - "mathics.format", -] + +[tool.setuptools.packages.find] +include = ["mathics*"] [tool.setuptools.package-data] "mathics" = [ diff --git a/setup.py b/setup.py index c57d0b45f..d2d61b7d2 100644 --- a/setup.py +++ b/setup.py @@ -21,7 +21,7 @@ python setup.py clean -> will clean all trash (*.pyc and stuff) -To get a full list of avaiable commands, read the output of: +To get a full list of available commands, read the output of: python setup.py --help-commands @@ -34,6 +34,7 @@ import sys from setuptools import Extension, setup +from setuptools.command.build_py import build_py as setuptools_build_py log = logging.getLogger(__name__) @@ -97,6 +98,25 @@ def get_srcdir(): CMDCLASS = {"build_ext": build_ext} +class build_py(setuptools_build_py): + def run(self): + if not os.path.exists("mathics/data/op-tables.json"): + os.system( + "mathics-generate-json-table" + " --field=ascii-operator-to-symbol" + " --field=ascii-operator-to-unicode" + " --field=ascii-operator-to-wl-unicode" + " --field=operator-to-ascii" + " --field=operator-to-unicode" + " -o mathics/data/op-tables.json" + ) + self.distribution.package_data["mathics"].append("data/op-tables.json") + setuptools_build_py.run(self) + + +CMDCLASS["build_py"] = build_py + + setup( cmdclass=CMDCLASS, ext_modules=EXTENSIONS, diff --git a/test/builtin/test_makeboxes.py b/test/builtin/test_makeboxes.py index 3650f9681..fdf31e420 100644 --- a/test/builtin/test_makeboxes.py +++ b/test/builtin/test_makeboxes.py @@ -1,9 +1,8 @@ # -*- coding: utf-8 -*- import os -from test.helper import check_evaluation, session +from test.helper import check_evaluation import pytest -from mathics_scanner.errors import IncompleteSyntaxError # To check the progress in the improvement of formatting routines, set this variable to 1. # Otherwise, the tests are going to be skipped. diff --git a/test/builtin/test_number_form.py b/test/builtin/test_number_form.py new file mode 100644 index 000000000..74b8fddf6 --- /dev/null +++ b/test/builtin/test_number_form.py @@ -0,0 +1,49 @@ +import pytest +import sympy + +from mathics.builtin.makeboxes import int_to_tuple_info, real_to_tuple_info +from mathics.core.atoms import Integer, Integer0, Integer1, IntegerM1, Real + +# from packaging.version import Version + + +@pytest.mark.parametrize( + ("integer", "expected", "exponent", "is_nonnegative"), + [ + (Integer0, "0", 0, True), + (Integer1, "1", 0, True), + (IntegerM1, "1", 0, False), + (Integer(999), "999", 2, True), + (Integer(1000), "1000", 3, True), + (Integer(-9999), "9999", 3, False), + (Integer(-10000), "10000", 4, False), + ], +) +def test_int_to_tuple_info( + integer: Integer, expected: str, exponent: int, is_nonnegative: bool +): + assert int_to_tuple_info(integer) == (expected, exponent, is_nonnegative) + + +@pytest.mark.parametrize( + ("real", "digits", "expected", "exponent", "is_nonnegative"), + [ + # Using older uncorrected version of Real() + # ( + # (Real(sympy.Float(0.0, 10)), 10, "0", -10, True) + # if Version(sympy.__version__) < Version("1.13.0") + # else (Real(sympy.Float(0.0, 10)), 10, "0000000000", -1, True) + # ), + (Real(sympy.Float(0.0, 10)), 10, "0", -10, True), + (Real(0), 1, "0", 0, True), + (Real(0), 2, "0", 0, True), + (Real(0.1), 2, "1", -1, True), + (Real(0.12), 2, "12", -1, True), + (Real(-0.12), 2, "12", -1, False), + (Real(3.141593), 10, "3141593", 0, True), + ], +) +def test_real_to_tuple_info( + real: Real, digits: int, expected: str, exponent: int, is_nonnegative: bool +): + assert real_to_tuple_info(real, digits) == (expected, exponent, is_nonnegative) diff --git a/test/consistency-and-style/test_duplicate_builtins.py b/test/consistency-and-style/test_duplicate_builtins.py index 3e1914119..c8dc3c6e5 100644 --- a/test/consistency-and-style/test_duplicate_builtins.py +++ b/test/consistency-and-style/test_duplicate_builtins.py @@ -2,7 +2,7 @@ Checks that builtin functions do not get redefined. In the past when reorganizing builtin functions we sometimes -had missing or duplicate build-in functions definitions. +had missing or duplicate built-in functions definitions. """ import os diff --git a/test/consistency-and-style/test_summary_text.py b/test/consistency-and-style/test_summary_text.py index e27d4c79c..6fafd92f5 100644 --- a/test/consistency-and-style/test_summary_text.py +++ b/test/consistency-and-style/test_summary_text.py @@ -9,7 +9,7 @@ from mathics import __file__ as mathics_initfile_path from mathics.core.builtin import Builtin from mathics.core.load_builtin import name_is_builtin_symbol -from mathics.doc.common_doc import skip_doc +from mathics.doc.gather import skip_doc # Get file system path name for mathics.builtin mathics_path = osp.dirname(mathics_initfile_path) @@ -157,10 +157,10 @@ def check_well_formatted_docstring(docstr: str, instance: Builtin, module_name: ), f"missing
    field {instance.get_name()} from {module_name}" assert ( docstr.count("
    • ") == 0 - ), f"unnecesary
    {instance.get_name()} from {module_name}" + ), f"unnecessary {instance.get_name()} from {module_name}" assert ( docstr.count("") == 0 - ), f"unnecesary field {instance.get_name()} from {module_name}" + ), f"unnecessary field {instance.get_name()} from {module_name}" assert ( docstr.count("") > 0 diff --git a/test/core/test_atoms.py b/test/core/test_atoms.py index 7dbd155d0..66f68d96a 100644 --- a/test/core/test_atoms.py +++ b/test/core/test_atoms.py @@ -134,7 +134,7 @@ def test_Integer(): def test_MachineReal(): check_group(MachineReal(5), MachineReal(3.5), Integer(1.00001)) # MachineReal0 should be predefined; `int` and float arguments are allowed - # `int` arguemnts are converted to float. + # `int` arguments are converted to float. check_object_uniqueness( MachineReal, [0.0], MachineReal0, MachineReal(0), MachineReal(0.0) ) diff --git a/test/core/test_rules.py b/test/core/test_rules.py index 4c457c12f..280b5849d 100644 --- a/test/core/test_rules.py +++ b/test/core/test_rules.py @@ -28,7 +28,7 @@ because it ignores that the attribute is clean at the time in which the rule is applied. -In Mathics, on the other hand, attributes are taken into accout just +In Mathics, on the other hand, attributes are taken into account just at the moment of the replacement, so the output of both expressions are the opposite. diff --git a/test/doc/__init__.py b/test/doc/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/test/doc/test_common.py b/test/doc/test_common.py index d8dd5b19f..8d7ff17e7 100644 --- a/test/doc/test_common.py +++ b/test/doc/test_common.py @@ -9,12 +9,14 @@ DocChapter, DocPart, DocSection, + Documentation, + MathicsMainDocumentation, +) +from mathics.doc.doc_entries import ( DocTest, DocTests, DocText, - Documentation, DocumentationEntry, - MathicsMainDocumentation, parse_docstring_to_DocumentationEntry_items, ) from mathics.settings import DOC_DIR diff --git a/test/doc/test_doctests.py b/test/doc/test_doctests.py new file mode 100644 index 000000000..cee480a85 --- /dev/null +++ b/test/doc/test_doctests.py @@ -0,0 +1,112 @@ +""" +Pytests for the documentation system. Basic functions and classes. +""" +import os.path as osp + +from mathics.core.evaluation import Message, Print +from mathics.core.load_builtin import import_and_load_builtins +from mathics.doc.common_doc import ( + DocChapter, + DocPart, + DocSection, + Documentation, + MathicsMainDocumentation, +) +from mathics.doc.doc_entries import ( + DocTest, + DocTests, + DocText, + DocumentationEntry, + parse_docstring_to_DocumentationEntry_items, +) +from mathics.settings import DOC_DIR + +import_and_load_builtins() +DOCUMENTATION = MathicsMainDocumentation() +DOCUMENTATION.load_documentation_sources() + + +def test_load_doctests(): + # there are in master 3959 tests... + all_the_tests = tuple((tests for tests in DOCUMENTATION.get_tests())) + visited_positions = set() + # Check that there are not dupliceted entries + for tests in all_the_tests: + position = (tests.part, tests.chapter, tests.section) + print(position) + assert position not in visited_positions + visited_positions.add(position) + + +def test_create_doctest(): + """initializing DocTest""" + + key = ( + "Part title", + "Chapter Title", + "Section Title", + ) + test_cases = [ + { + "test": [">", "2+2", "\n = 4"], + "properties": { + "private": False, + "ignore": False, + "result": "4", + "outs": [], + "key": key + (1,), + }, + }, + { + "test": ["#", "2+2", "\n = 4"], + "properties": { + "private": True, + "ignore": False, + "result": "4", + "outs": [], + "key": key + (1,), + }, + }, + { + "test": ["S", "2+2", "\n = 4"], + "properties": { + "private": False, + "ignore": False, + "result": "4", + "outs": [], + "key": key + (1,), + }, + }, + { + "test": ["X", 'Print["Hola"]', "| Hola"], + "properties": { + "private": False, + "ignore": True, + "result": None, + "outs": [Print("Hola")], + "key": key + (1,), + }, + }, + { + "test": [ + ">", + "1 / 0", + "\n : Infinite expression 1 / 0 encountered.\n ComplexInfinity", + ], + "properties": { + "private": False, + "ignore": False, + "result": None, + "outs": [ + Message( + symbol="", text="Infinite expression 1 / 0 encountered.", tag="" + ) + ], + "key": key + (1,), + }, + }, + ] + for index, test_case in enumerate(test_cases): + doctest = DocTest(1, test_case["test"], key) + for property_key, value in test_case["properties"].items(): + assert getattr(doctest, property_key) == value diff --git a/test/doc/test_latex.py b/test/doc/test_latex.py index ddc37bae5..2645421f7 100644 --- a/test/doc/test_latex.py +++ b/test/doc/test_latex.py @@ -5,6 +5,7 @@ from mathics.core.evaluation import Message, Print from mathics.core.load_builtin import import_and_load_builtins +from mathics.doc.doc_entries import parse_docstring_to_DocumentationEntry_items from mathics.doc.latex_doc import ( LaTeXDocChapter, LaTeXDocPart, @@ -14,7 +15,6 @@ LaTeXDocText, LaTeXDocumentationEntry, LaTeXMathicsDocumentation, - parse_docstring_to_DocumentationEntry_items, ) from mathics.settings import DOC_DIR @@ -90,7 +90,7 @@ def test_load_latex_documentation(): ).strip() == "Let's sketch the function\n\\begin{tests}" assert ( first_section.latex(doc_data)[:30] - ).strip() == "\\section*{Curve Sketching}{}" + ).strip() == "\\section{Curve Sketching}{}" assert ( third_chapter.latex(doc_data)[:38] ).strip() == "\\chapter{Further Tutorial Examples}" @@ -102,11 +102,15 @@ def test_chapter(): chapter = part.chapters_by_slug["testing-expressions"] print(chapter.sections_by_slug.keys()) section = chapter.sections_by_slug["numerical-properties"] - latex_section_head = section.latex({})[:63].strip() - assert ( - latex_section_head - == "\section*{Numerical Properties}{\index{Numerical Properties}}" + expected_latex_section_head = ( + "\\section{Numerical Properties}\n" + "\\label{reference-of-built-in-symbols/testing-expressions/numerical-properties}\n" + "\\sectionstart\n\n\n\n" + "\\subsection{CoprimeQ}\index{CoprimeQ}" ) + latex_section_head = section.latex({}).strip()[: len(expected_latex_section_head)] + + assert latex_section_head == expected_latex_section_head print(60 * "@") latex_chapter = chapter.latex({}, quiet=False) diff --git a/test/eval/test_tensors.py b/test/eval/test_tensors.py index 1c151b57d..b06779a4e 100644 --- a/test/eval/test_tensors.py +++ b/test/eval/test_tensors.py @@ -40,7 +40,7 @@ def testCartesianProduct(self): (lambda item, level: level > 1), # True to unpack the next list, False to unpack the current list at the next level (lambda item: item), - # get elements from Expression, for iteratable objects (tuple, list, etc.) it's just identity + # get elements from Expression, for iterable objects (tuple, list, etc.) it's just identity list, # apply_head: each level of result would be in form of apply_head(...) tuple, diff --git a/test/format/test_format.py b/test/format/test_format.py index 6347389d9..c8eadc26e 100644 --- a/test/format/test_format.py +++ b/test/format/test_format.py @@ -303,7 +303,7 @@ }, # Notice that differetly from "text", where InputForm # preserves the quotes in strings, MathTeXForm just - # sorrounds the string in a ``\text{...}`` command, + # surrounds the string in a ``\text{...}`` command, # in the same way that all the other forms. This choice # follows the behavior in WMA. "latex": { diff --git a/test/helper.py b/test/helper.py index 16c7eca51..cff356df5 100644 --- a/test/helper.py +++ b/test/helper.py @@ -41,7 +41,7 @@ def check_evaluation( its results Compares the expressions represented by ``str_expr`` and ``str_expected`` by - evaluating the first, and optionally, the second. If ommited, `str_expected` + evaluating the first, and optionally, the second. If omitted, `str_expected` is assumed to be `"Null"`. to_string_expr: If ``True`` (default value) the result of the evaluation is diff --git a/test/package/test_combinatorica.py b/test/package/test_combinatorica.py index 98f2b5e43..631b1ac73 100644 --- a/test/package/test_combinatorica.py +++ b/test/package/test_combinatorica.py @@ -322,7 +322,7 @@ def test_inversions_and_inversion_vectors_1_3(): ( "Inversions[Reverse[Range[8]]]", "Binomial[8, 2]", - "# permutions is [0 .. Binomial(n 2)]; largest is reverse 1.3.2, Page 29", + "# permutations is [0 .. Binomial(n 2)]; largest is reverse 1.3.2, Page 29", ), ( "Union [ Map[Inversions, Permutations[Range[4]]] ]", diff --git a/test/test_context.py b/test/test_context.py index 0ecc659dc..c300cc1e0 100644 --- a/test/test_context.py +++ b/test/test_context.py @@ -83,7 +83,7 @@ def test_context1(): "Start a context. Add it to the context path", ), ( - """Minus::usage=" usage string setted in the package for Minus";""", + """Minus::usage=" usage string set in the package for Minus";""", None, None, "set the usage string for a protected symbol ->no error", @@ -158,7 +158,7 @@ def test_context1(): "try to set a value for a protected symbol ->error", ), ( - """Plus::usage=" usage string setted in the package for Plus";""", + """Plus::usage=" usage string set in the package for Plus";""", None, None, "set the usage string for a protected symbol ->no error", @@ -236,13 +236,13 @@ def test_context1(): ("""apackage`B""", "6", None, "get B using its fully qualified name"), ( """Plus::usage""", - ' " usage string setted in the package for Plus" ', + ' " usage string set in the package for Plus" ', None, "custom usage for Plus", ), ( """Minus::usage""", - '" usage string setted in the package for Minus"', + '" usage string set in the package for Minus"', None, "custom usage for Minus", ), diff --git a/test/test_evaluation.py b/test/test_evaluation.py index daaba75ba..14a6c4e10 100644 --- a/test/test_evaluation.py +++ b/test/test_evaluation.py @@ -249,7 +249,7 @@ def test_system_specific_long_integer(): ) for i, (str_expr, message) in enumerate(test_input_and_name): - # This works but the $Precision is coming out UnsignedInt128 rather tha + # This works but the $Precision is coming out UnsignedInt128 rather than # UnsignedInt32 # ( # 'Eigenvalues[{{-8, 12, 4}, {12, -20, 0}, {4, 0, -2}}, Method->"mpmath"]', diff --git a/test/test_numericq.py b/test/test_numericq.py index 526f2f176..de37a4174 100644 --- a/test/test_numericq.py +++ b/test/test_numericq.py @@ -110,7 +110,7 @@ def test_atomic_numericq(str_expr, str_expected): """F[1,l->2]""", "False", ), - # NumericQ returs True for expressions that + # NumericQ returns True for expressions that # cannot be evaluated to a number: ("1/(Sin[1]^2+Cos[1]^2-1)", "True"), ("Simplify[1/(Sin[1]^2+Cos[1]^2-1)]", "False"),