Unit tests for macros (checking compatiblity with current build)

[luzpaz]

FreeCAD Macros (either within this repo or pulled from the wiki) can break as FreeCAD evolves. Does it make sense that with every macro that exists and added in the future - a test is attached to it in order to see that it's functionality still works correctly. If not then the script (something like the Test Workbench) notifies the tester and can recommend creating a ticket to address the issue?

[galou]

It makes sense but the issue is that most macros are a single file and are not importable for now. We'd first need to hack into the import system to be able to import FCMacro files.
Is there already a possibility to test GUIs within FreeCAD?

[onekk]

Interesting, but probably not feasible.

I've seen other approaches, like a "score system", not very automatic, but probably it will direct users to "well maintained and working macros".

As example there are macros in the wiki that are not maintained and will work on 0.16 (as example), it has a sense to maintain them on the wiki for "historical purpose" and even for "long term history", in the sense of documenting and not wasting precious work, even if it is not functional anymore. (There is an entire discussion about "Vanishing of knowledge" due to the disappearing of github sites and so on, as an example microjelly2 code has mostly vanished, but there are bunch of interesting forum posts that have link to his code.)

From the functional point of view it is a waste of space, but we read books, that were written 5000 years ago.

to be on Topic.

A macro will have a score that will tell user that is maintained and his code could be retrieved, with maybe an additional info if the macro has been tested and found to make the work for which it has been made.

Maybe some easy info like 3.5/5.0 score will tell that is a decent macro, maybe only a little old but functional, 1.0/5.0 will signal that maybe is not working on recent (0.20 onward) version of FreeCAD.

As a rule of thumb, if a Macro will work on the actual stable version and the "old stable" will have a > 3.5 score (mean is 2.5) a perfectly maintained macro wil have 4.5/5.0 or more score.

A macro that works maybe only on 0.19 and earlier version wil have a 2.5/5.0 score, while a macro that is working on 0.16 wil have a 1.0/5.0 score, older coudl have a 0.5/5.0 score to signal that they are kepp for "informative purpose only".

Or other score scheme.

and Maybe the Wiki page of Macro will list only the > 2.5 score macros, the other will be placed in "Historic Macros" page, frkm time to time updated (if Wiki will not be superseeded in short time by some other documentation system).

Regards

Carlo D.

[galou]

Thanks Carlo for your thoughts. To clarify things we (at least myself) were taking about macros in this repository only. Macros on the Wiki are another story.
The scoring is certainly a good idea, here are my thoughts/questions about it:

• I would score "is it working" and "it is useful" as two different scores,
• There will be a need for a score for each stable FreeCAD version (only the "is it working" score),
• How would you implement it? I personally don't have the capacity to implement such a scoring system.
• A scoring system doesn't replace unit-tests that are the original topic of this conversation.

[onekk]

Probably I've not seen the place of this proposal, so the wrong assumptions.

After a quick glance at the repository, from what I see there are macros quite old.

To add a Unit test, probably macros should be created with this test in mind.

So the only things you could do is to test by hand, and this will require some manpower and probably even some correct files or document "with appropriate content to test".

I'm not a programmer I write Python scripts, but I have done some work so I have some ideas.

I don't know what "peer review" is meaning, if this means that someone has to approve the insertion of the macro he could attribute a score that will be eventually shown in a page, (maybe a GH Wiki page), or eventually directly in the README.md that will link the actual README.md placed in another page where a "macro developer" will find all requirement top have his macro inserted here.

A transition phase should be needed, so maybe an archived directory will contains all the macros, and only those that had a score and are correctly maintained will be in the "front page".

Obviously if an automated test system could be made it will ease the work of the "reviewer" but IMHO a "macro developer" should maintain his work, maybe only to check if his work could be run on actual version.

Actually there is this "required" field: (even if the note is not make it mandatory)

__Requires__ - e.g. FreeCAD >= v0.17, there is no programmatic use of this for now

__Tested_on__ - 0.17, 0.18, 0.20, 0.20.1 and so on

Probably if authors are active a simple standardized email that will advise that a repository revision has to be made and if nothing is done is work will be "archived" will suffice, if I'm a "macro developer" is my interest to maintain it.

The score could be made eventually even using some user feedback.

As example in Arch Linux there the AUR system where every package has his proper page and "registered users" could mark as obsolete a package when a new revision of the underlying software is released and the package is not updated.

But this will need many work to be done, probably chennes (on FC forum) wil have better ideas as he is maintaining the AddOn Manager that is similar to this repo but is for Workbenches.

Sorry for the long post and the lack of proper solutions.

Regards

Carlo D.