TODO list towards Release 8.0

[mmatera]

• Double-check the documentation.
• Completing the amslatex field in mathics-scanner
• Some level of support for Paclet? (which level?)
• Handle MathML in Mathics-Django doc.
• Remove the ContextPath on TraceEvaluation, or at least have an option for turning this off
• Improve consistency of graphics in their different formats (asymptote, SVG, etc and with WMA)
• move 3D eval code out of mathics.builtin.plot and into mathics.eval.plot
• Remove the ContextPath on TraceEvaluation, or at least have an option for turning this off.
• PR's Rewrite Trace* routines in a way that is accessible and printable inside Mathics3 #1241
• Implement Export[filename, graph] #1242
• mathicsscript: Starting mathicsscript in vi editing mode mathicsscript#85
  ... ?

[rocky]

Here are my thoughts... Feel free to comment on...

What is this release about?

This release is to get out some of the major changes that have gone on already in advance of redoing Boxing and Formatting.
In my view, the major changes have been to start to get (but not fully finish) operators controlled from a new operators YAML table . A small pass was made over the Mathics parser to handle box operators more properly. And finally, we started adding (but again not fully finished) debugging.

Also important is the work by David A. Roberts for pyodide and Aravindh Krishnamoorthy on Rubi.

API Breakage

In doing this, the JSON tables, in particular, had to change and "break". And so we started on this journey of other breaking API changes.

I have mixed feelings about this, improving the API is a good thing and probably inevitable, and if we break the API, we may as well break it big time. On the other hand, if this goes on too long, then we spend weeks fixing the newly created API breakage, and the release gets delayed further.

Concerning this point, in particular, there was this one PR where the 3 classes of variables definitions, "ownvalues", "upvalues", and "downvalues" were grouped into a new dictionary of the Definitions class. While I guess this will eventually get merged in, the timing for thinking about this is ill-coordinated and ill-timed, in my opinion. A better use of resources might have been to focus on the release and discuss this later.

But here is my dilemma with this now. If this is going to get merged in right after release, well we are back in the same boat of we will be working with a changed API and we'll have to do as we do now and any developers will have to work off of master git branches instead of pulling from PyPI. I had hoped we might be able to avoid that.

In making the Definitions object change, the Mathics3 Debugger in particular, (which hasn't had a first release yet), will also have to change because it uses the current interface for getting these values.

So here, I have a slight inclination to put this in before release, suffer the round of breakage yet again, and hope that we won't decide well, actually, there is something even better...

(Remember the weird proposal once where there was this push to organize all operators by a Unicode symbol in mathics core. That idea was ill-advised. This isn't, so it is better. But the point is that coming up with well-defined solid interfaces and API's has not been this project's strong point. At a minimum we should learn from the experiences and design APIs with better forethought and communication, instead of the current practice of thinking about this and shoving it at the stage where we are focused on getting out a release)

What is currently going on to get a Release finished?

---

Docker image update

Right now, one of my main tasks is a preliminary update of the docker container that has everything. This has been very helpful to me in finding cross-repository breakage (of which there has been a bit already).

It also allows users to try things out in advance of committing to a release. I am not sure we get a lot of feedback this way, but if we don't provide the opportunity we definitely won't get any feedback until after a release which is not good.

To speed up the process of fixing small things, I have been merging languishing PRs and, in some cases, even committing them to master. I hope to contain this kind of thing only around release time when people should not be making big changes unrelated to the release. But if this is a problem, let me know.

Documentation update

In conjunction with getting the docker image created, I have also been going over the documentation. All sorts of stupid and tedious stuff that we typically don't do during development. Like:

• looking at the docs, both in Django and in PDF
• checking summaries to make sure there is an active verb summary except in rare cases where this is not possible or appropriate
• giving meaningful examples (Mathematica tends to try to impress with complicated examples with large numbers that doesn't help comprehension when someone is just trying to figure out what a function does and why one would use it). Here I generally start with the list of new and changed builtin functions listed in CHANGES.rst.

Specific TODO items and optional features

MathML in Django

So now let me come back to handling MathML in Django. Is it totally necessary,? No.

It would be nice though. It was offered as a reason for why it was okay to change things on the LaTeX side and break things elsewhere. I have come to learn that a description that says "fix" really means "change" here and possibly break elsewhere. A little while back in a discussion related to Rubi the issue came up with whether we wanted to finish now or finish later. And someone else opined that it would be better to finish now because "later" may not come.

No doubt there was an honest misunderstanding of the difficulty needed to finish. But have a tendency to underestimate the difficulty of tasks. I hope we can learn from that.

Completing amslatex fields

This feels like a no-brainer and I wonder why this isn't done already.

As laid out above, this was about fleshing out the YAML tables to the extent we can. There probably will be another pass at named-characters.yml to remove operator function information now that it is in its own table. But that is too disruptive for now. And it will also proceed in conjunction with a pass over the scanner to remove the "prescan" loop which handles comments and special characters improperly. And could probably use more of the YAML table as well.

But finishing amslatex equivalences for operators has none of this complexity.

Paclet support

With this release where things have been cleaned up sufficiently, and we have a debugging framework in place, we are now in a better position to support more Mathematica packages off the shelf. When they fail I have the tools to pinpoint what's wrong and fix. There is still a lot to do on the debugging side, but that can come later and should not delay a release.

So now that we have the possibility of handling packages we should make sure that those people who have access to Paclet code can be able to run that on Mathics3. That's why we added a Packages directory in the user space. That's why Between was added (which can be used to check version numbers).

There is no hard requirement of what version of Paclet should be supported before release. But if there is some version of Paclet that works that would be nice to know. Based on Combinatorica experience, the oldest version of Paclet will probably be the smallest and easiest to get working. (It probably has the fewest features too). But from there we can expand more incrementally and easily.

Ensure custom pyodide code works

We should make sure that the pyodide work that @davidar has done still works with the API changes that have been made.

Make PyPI releases of Mathics3 Trepan module and trepan3k plugin for Mathics3

Not much to say here, I just gotta do it.

Hopefully low-hanging stuff

Here is a personal list of things I had in the back of my mind as being simple user-visible items -- low-hanging-fruit with little chance of major disruption:

• Go over graphics Plot line widths and dot points to make these match what we see from WMA better. The lines might be a little too thin and the dots might be a little too small
• Try to match closer the layout style colors that WMA uses
• Try to get asymptote, matplotlib and SVG output matching better. (I assume they do in WMA)
• Remove the ContextPath on TraceEvaluation, or at least have an option for turning this off
• Finish Add Stack[] #1259 It can go in as is, but if we could get it to better match WMA that would be nice.
• PR's Rewrite Trace* routines in a way that is accessible and printable inside Mathics3 #1241 and Implement Export[filename, graph] #1242
• mathicsscript: Starting mathicsscript in vi editing mode mathicsscript#85

Personal pet peeves:
Personal pet peeves:

• move 3D eval code out of mathics.builtin.plot and into mathics.eval.plot
• Move "tech note structure "String" mathics.strings and place functions in their corresponding Guide Section

[mmatera]

@rocky, thank you for the detailed presentation of aims, reasons, and goals. I think at this point it is OK to proceed with the no-breaking, non-polemic changes.
From the list of things to do, the next thing I can do is pass over the yaml tables and complete the amslatex representation of non-standard operators as much as possible. Then I could go over to move the eval code in mathics.builtin.plot to the mathics.eval.plot.

In the meantime, I will try to discover how to make the MathJax code work to process the LaTeX commands in the online documentation. Right now, I am aware of just one symbol (DivisorSigma) that shows a LaTeX string instead of a UNICODE symbol. This is because it has an underscore, which was not well represented either in the PDF or in the online help.

Regarding the proposed changes in Definition (collecting several lists in a single dictionary) I would think that is not an API breaking change but just an intermediate change in the way that information is stored internally: as I understand it, the API of the mathics.core.definition module consists of the specific access methods (Definitions.get_*values). In any case, this is for a further discussion. Something that would be an API-breaking change is related to the formatting code. During the last year, I have been working in a proposal that is more compliant with the WMA interface. Now I am trying to organize the changes in a way that can be easier to discuss step by step. For me, the moment in the year that I have to do that work is this month, but once I have it, we can discuss it in any other moment. However, I think that it worth to include some small changes like the get_function change, or the inclusion of FormatValues without a big impact in the API, but this is already done.

[rocky]

Regarding the proposed changes in Definition (collecting several lists in a single dictionary) I would think that is not an API breaking change but just an intermediate change in the way that information is stored internally: as I understand it, the API of the mathics.core.definition module consists of the specific access methods (Definitions.get_*values). In any case, this is for a further discussion. Something that would be an API-breaking change is related to the formatting code.

"API" stands for Application Programming Interface". Think of the Mathics3 modules and the front-ends as "applications". The "Programming Interface" consists of the function calls that these applications make in order to get their work done. In particular, the methods that get called on a definition object. When the methods these applications need to use change, that , by definition, is an API change.

During the last year, I have been working in a proposal that is more compliant with the WMA interface.

Great! It would be helpful if you could just describe how the WMA interface works and how Mathics3 is not compliant.

Now I am trying to organize the changes in a way that can be easier to discuss step by step.

Let's take this one step at a time: describe share and understand and discuss the problem. Then come up with a plan.

For me, the moment in the year that I have to do that work is this month, but once I have it, we can discuss it in any other moment.

What has happened in the past is that you think you have an understanding, most likely a better one than previously, but not a full understanding or one as complete as others might have.

As a result, there is some floundering around and major disruption that I think can be reduced by describing and discussing first. The unicode operator refactor proposal was only one example,; Hacks to the iwolfram interface another, but there have been other instances like this.

However, I think that it worth to include some small changes like the get_function change, or the inclusion of FormatValues without a big impact in the API, but this is already done.

If FormatValues helps out, that's okay. And if adding the dictionary before the release will increase the likelihood that people can use released PyPI for a longer period of time, then let's get those merged in.

[mmatera]

"API" stands for Application Programming Interface". Think of the Mathics3 modules and the front-ends as "applications". The "Programming Interface" consists of the function calls that these applications make in order to get their work done. In particular, the methods that get called on a definition object. When the methods these applications need to use change, that , by definition, is an API change.

Indeed. But what I tried to show in that branch is that except in very few places, the list of rules are internal attributes of the class, so their name and organization are not a part of the API. Accessing these list attributes directly from the modules would be then a kind of hack, used for speeding up certain low-level functions in Mathics-core, or just as a way to deal with an incomplete API. So, if we use the Definitions.get_* interface, a further redesign of the Definition class would not break the API.

During the last year, I have been working in a proposal that is more compliant with the WMA interface.
Great! It would be helpful if you could just describe how the WMA interface works and how Mathics3 is not compliant.
Now I am trying to organize the changes in a way that can be easier to discuss step by step.
Let's take this one step at a time: describe share and understand and discuss the problem. Then come up with a plan.

Yes, but that takes time. Also, for me is harder to present the case in abstract terms.

For me, the moment in the year that I have to do that work is this month, but once I have it, we can discuss it in any other moment.

What has happened in the past is that you think you have an understanding, most likely a better one than previously, but not a full understanding or one as complete as others might have.

Due to my formation, I do not think that a "full understading" is something achievable for anyone: I am a scientist, not a priest. And yes, I think that there are other people that understands things better than me. For example, it is quite possible that Stephen Wolfram understand much better how the Boxing evaluation works, but I do not have access to that knowledge. So, the best we can do and use is always incremental.

As a result, there is some floundering around and major disruption that I think can be reduced by describing and discussing first. The unicode operator refactor proposal was only one example,; Hacks to the iwolfram interface another, but there have been other instances like this.

I agree with this, and sorry. On the other hand, if I didn't go on with these hacky solutions, the jupyter wolframkernel would not exist, and at some point it was useful for me and I guess for others, even if it is quite incomplete and not fully working. The unicode refactor took a lot of work, and we could not imagine how much work required until we start with it. It was a proposal, and that proposal helped us to improve a lot many aspects of the code, not only that specific feature.

However, I think that it worth to include some small changes like the get_function change, or the inclusion of FormatValues without a big impact in the API, but this is already done.

If FormatValues helps out, that's okay. And if adding the dictionary before the release will increase the likelihood that people can use released PyPI for a longer period of time, then let's get those merged in.

FormatValues is already included. The Definition attribute reorganization was left for a future discussion.