[REVIEW]: Algorithms for SEM-EDS Mineral Dust Classification

[editorialbot]

Submitting author: @weber1158 (Austin M. Weber)
Repository: https://github.com/weber1158/eds-classification
Branch with paper.md (empty if default branch):
Version: v1.3.0
Editor: @rwegener2
Reviewers: @ThFriedrich, @kstenio
Archive: Pending

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/c2564d4c44b4ee77c24ac32f7431a6b2"><img src="https://joss.theoj.org/papers/c2564d4c44b4ee77c24ac32f7431a6b2/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/c2564d4c44b4ee77c24ac32f7431a6b2/status.svg)](https://joss.theoj.org/papers/c2564d4c44b4ee77c24ac32f7431a6b2)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@ThFriedrich & @dxm447, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review.
First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @rwegener2 know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @ThFriedrich

📝 Checklist for @kstenio

[editorialbot]

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.90  T=0.05 s (532.8 files/s, 161292.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Julia                            3            145            492           2475
MATLAB                          18            195           1231           2193
Markdown                         5            270              0            505
TeX                              1              8              0             92
Jupyter Notebook                 1              0           1066             79
YAML                             1              1              4             23
-------------------------------------------------------------------------------
SUM:                            29            619           2793           5367
-------------------------------------------------------------------------------

Commit count by author:

    78	Austin M. Weber

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

✅ OK DOIs

- 10.1016/j.gloplacha.2022.103837 is OK
- 10.1029/2002GL016641 is OK
- 10.1111/j.1600-0889.2011.00550.x is OK
- 10.5194/acp-23-3861-2023 is OK
- 10.1017/jog.2016.8 is OK
- 10.5194/cp-17-1341-2021 is OK
- 10.1017/S143192762200023X is OK
- 10.1007/978-1-4020-2841-0 is OK

🟡 SKIP DOIs

- No DOI given, and none found for title: Elements

❌ MISSING DOIs

- None

❌ INVALID DOIs

- None

[editorialbot]

Paper file info:

📄 Wordcount for paper.md is 1027

✅ The paper includes a Statement of need section

[editorialbot]

License info:

✅ License found: MIT License (Valid open source OSI approved license)

[rwegener2]

👋🏼 @weber1158, @ThFriedrich, and @dxm447 this is the review thread for the paper. All of our communications will happen here from now on.

As a reviewer, the first step is to create a checklist for your review by entering

@editorialbot generate my checklist

as the top of a new comment in this thread.

These checklists contain the JOSS requirements. As you go over the submission, please check any items that you feel have been satisfied. The first comment in this thread also contains links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention openjournals/joss-reviews#REVIEW_NUMBER so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for reviews to be completed within about 2-4 weeks. Please let me know if any of you require some more time. We can also use EditorialBot (our bot) to set automatic reminders if you know you'll be away for a known period of time.

Please feel free to ping me (@rwegener2) if you have any questions/concerns.

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[ThFriedrich]

Review checklist for @ThFriedrich

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the https://github.com/weber1158/eds-classification?
• License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@weber1158) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1. Contribute to the software 2. Report issues or problems with the software 3. Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[rwegener2]

Hi @ThFriedrich & @dxm447, just a friendly reminder about this review. @dxm447 the first step is to generate your checklist using @editorialbot generate my checklist.

If you have any questions please don't hesitate to reach out!

[ThFriedrich]

The paper is well written and the repository clean, structured and well documented. I recommend accepting the article, though I may add a few minor comments:

Formalities

• Community Guidelines are missing
• Some features don't seem to have a relevant context and I wonder why they were added, i.e.: get_sem_metadata(), sem_pixel_size() and convergence_angle()

Concept

• The approach overall seems very solid
• About the usage of SMOTE for the generation of synthetic data: The empirical nature of SMOTE seems fine in principle, but it generates uncorrelated random values in high-dimensional spaces, which could produce inconsistencies to real samples. The results suggest, that this is not an issue here, but perhaps one could check whether the validation accuracy drops if the validation set contains only real samples.

Implementations

• xray_peak_label function could be more sophisticated:
  • multiple assigned labels per peak may be avoided by considering intensity ratios between e.g. Ka1 and Kb1 or ideally some kind of deconvolution, though this may be a bit much to ask for. I'd consider this optional, since it is clearly not a core functionality of the software
  • tolerance in X is hardcoded and somewhat arbitrary. For example in the Albite example it causes to C-peak to not be identified

---

These are all quite minor suggestions. Overall I think this is very nice work, that is very accessible and easy to use. I'd be happy accepting it as it is.

[weber1158]

@ThFriedrich Thank you for taking the time to review my submission. I am glad you found that the functions in the repository are working properly, and I am especially happy that you find them easy to use. Making the functions as accessible as possible was a top priority of mine.

Formalities

• I can add community guidelines to the documentation.
• I agree that the get_sem_metadata, sem_pixel_size, and convergence_angle functions aren't necessarily relevant to EDS research. I only included them because they are useful for other SEM applications. A few people in my department have been using them, so I decided that they were worth adding to the repository.

Concept

• You make a good point about how SMOTE synthesizes uncorrelated data points. Luckily, it does not seem to affect the performance of the machine learning model: the accuracy when using only the measured data is 99.5%.

Implementations

I really appreciate the feedback here. The xray_peak_label function has been a fun programming challenge for me, and I would love to improve the algorithm further.

• I have been trying to do exactly what you mentioned about using ratios to restrict the labeling to one element per peak, though this is still a work in progress. The original xray_peak_label function (in previous versions) only assigns one element to each peak, but this occasionally results in misclassifications because labels are preferentially assigned to the lowest atomic number element that emits a characteristic x-ray in the relevant energy range. My solution in the latest version was to label each peak with every possible element, which gives the user the ability to make an educated decision about which element is most likely. But like you say, it would be ideal if xray_peak_label was able to deduce the correct element automatically.

• I will definitely consider adding an optional input argument to set the tolerance in X. It should be easy to do. The only reason I did not do so before is because I wanted the algorithm to be as simplistic as possible for the user. Alternatively, I can see about making the function recursive so that if there are any major peaks that remain unlabeled the algorithm will run again using a higher tolerance.

---

Thank you again for your review and feedback!

[rwegener2]

@ThFriedrich thanks so much for your thoughtful feedback!

@dxm447 do you have a timeline for when you anticipate being able to complete your review?

@weber1158 Thanks for responding to the review comments! At JOSS the review process is iterative and the aim is to help you improve your software through community feedback. Because of this we ask that you take the time now to implement the feedback that the reviewers suggest. You are welcome to start an issue in your github repo if there is more detailed discussion to be had, just make sure to mention this review issue so I can track the progress.

Please comment again when you have been able to address the changes that @ThFriedrich has proposed (ex. adding community guidelines, optional tolerance argument), so he can re-evaluate the JOSS review criteria. Thanks, and please reach out if you have any questions!

[weber1158]

@rwegener2 Thank you for the reminder!

@ThFriedrich

• I have added a CONTRIBUTING.md file to the main branch that contains the community guidelines (here).
• I have also accepted your suggestion and added an optional input to the xray_peak_label function that allows the user to adjust the tolerance in x.

[dxm447]

@ThFriedrich thanks so much for your thoughtful feedback!

@dxm447 do you have a timeline for when you anticipate being able to complete your review?

@weber1158 Thanks for responding to the review comments! At JOSS the review process is iterative and the aim is to help you improve your software through community feedback. Because of this we ask that you take the time now to implement the feedback that the reviewers suggest. You are welcome to start an issue in your github repo if there is more detailed discussion to be had, just make sure to mention this review issue so I can track the progress.

Please comment again when you have been able to address the changes that @ThFriedrich has proposed (ex. adding community guidelines, optional tolerance argument), so he can re-evaluate the JOSS review criteria. Thanks, and please reach out if you have any questions!

@rwegener2 I will be done by the middle of the week.

[weber1158]

@ThFriedrich and @dxm447

I have updated the xray_peak_label function again with a new improvement: I added an optional input that allows the user to choose a labeling method. For example:

• xray_peak_label(axHandle,"best") will label each peak in an EDS spectrum using a best-guess method based on whether multiple K- and/or L-series lines for an element are present.

• xray_peak_label(axHandle,"first") will label each peak with the first possible element that the peak may represent according to atomic number.

• xray_peak_label(axHandle,"all") will label each peak with all of the possible elements that the peak could represent.

The function documentation has been updated accordingly here.

[rwegener2]

@editorialbot remove @dxm447 from reviewers

[editorialbot]

@dxm447 removed from the reviewers list!

[rwegener2]

@weber1158, dxm447 has not responded to my communications, so I will begin the process of searching for a new second reviewer. I apologize for the delay this has caused in the submission process. Thank you for your patience.

[rwegener2]

👋🏻 @kstenio, would you be willing to review this submission for JOSS? We carry out our checklist-driven reviews here in GitHub issues and follow these guidelines: https://joss.readthedocs.io/en/latest/review_criteria.html

[kstenio]

Hello @rwegener2, thanks for reaching out. It's been ages since the last time I worked with MATLAB, so I don't have a proper license right now. I'll check if my old institutional email will be eligible for a licence.

@weber1158 any chance this algorithm would work with GNU Octave?

[weber1158]

@kstenio Unfortunately, I do not think the software will be compatible with GNU Octave.

However, the software was designed in MATLAB Online, which is available to anyone for up to 20 hours a month when they create a MathWorks account. No license required.

[kstenio]

@kstenio Unfortunately, I do not think the software will be compatible with GNU Octave.

However, the software was designed in MATLAB Online, which is available to anyone for up to 20 hours a month when they create a MathWorks account. No license required.

I see. I did use my old institutional email for setting up an account, but seems like UFSCar does not have a global license agreement with MathWorks.

Either way, I'm somewhat available to do a review @rwegener2.

[rwegener2]

Hi @kstenio, thanks for willingness to consider this review! We request that our reviewers complete their initial review within about 2 weeks. Will that work for you?

[kstenio]

Hi @kstenio, thanks for willingness to consider this review! We request that our reviewers complete their initial review within about 2 weeks. Will that work for you?

Yes. I'll be able to do it in this timeframe

[rwegener2]

Great, thanks for your time and expertise, @kstenio!

[rwegener2]

@editorialbot add @kstenio as reviewer

[editorialbot]

@kstenio added to the reviewers list!

[rwegener2]

Welcome to the review, @kstenio! This is the review thread for the paper.

As a reviewer, the first step is to create a checklist for your review by entering

@editorialbot generate my checklist

as the top of a new comment in this thread.

These checklists contain the JOSS requirements. As you go over the submission, please check any items that you feel have been satisfied. The first comment in this thread also contains links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention openjournals/joss-reviews#REVIEW_NUMBER so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

Please feel free to ping me (@rwegener2) if you have any questions/concerns.

[kstenio]

Review checklist for @kstenio

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the https://github.com/weber1158/eds-classification?
• License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@weber1158) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1. Contribute to the software 2. Report issues or problems with the software 3. Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[kstenio]

Hi @rwegener2 and @weber1158,

From my understanding, this paper meets many of the JOSS requirements, but a few areas could benefit from clarification or expansion. Specifically, there are points regarding authorship acknowledgment, references to previous work, and testing instructions that I believe should be addressed.

I’ve outlined my inquiries in the following issues:

• JOSS Review - Issue #1 - Only a single author? weber1158/eds-classification#1
• JOSS Review - Issue #2 - Links should use permalinks weber1158/eds-classification#2
• JOSS Review - Issue #3 - Is the supplemental information an entirely new paper? weber1158/eds-classification#3
• JOSS Review - Issue #4 - Lack of instructions for running tests weber1158/eds-classification#4

That said, I would recommend this paper for publication.

[weber1158]

@editorialbot generate pdf

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[weber1158]

@kstenio I appreciate you taking the time to review my software submission.

You will find my responses to your feedback on the main repository's Issues page. My revisions to paper.md seem to be compiling correctly (see my @ editorialbot generate pdf command above or click here to download a PDF copy).

Please let me know if I have addressed each issue to your satisfaction so that I may mark the issues as "closed". Thank you again for your time!

[kstenio]

The replies and improvements to the repository and paper addressed all the issues I did open. To my understanding, the paper is ready to go!