[REVIEW]: AlignSAR: An open-source toolbox of SAR benchmark dataset creation for machine learning applications

[editorialbot]

Submitting author: @LC-SAR (Ling Chang)
Repository: https://github.com/AlignSAR/alignSAR
Branch with paper.md (empty if default branch): main
Version: v1.1
Editor: @rwegener2
Reviewers: @philippemiron, @shahchiragh
Archive: Pending

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/0ca20174f004319d43ed8756f8e48f8f"><img src="https://joss.theoj.org/papers/0ca20174f004319d43ed8756f8e48f8f/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/0ca20174f004319d43ed8756f8e48f8f/status.svg)](https://joss.theoj.org/papers/0ca20174f004319d43ed8756f8e48f8f)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@philippemiron & @shahchiragh, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review.
First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @rwegener2 know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @philippemiron

📝 Checklist for @shahchiragh

[editorialbot]

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

✅ OK DOIs

- 10.1109/LGRS.2024.3376992 is OK
- 10.1109/36.964973 is OK
- 10.1109/36.898661 is OK
- 10.1088/0266-5611/14/4/001 is OK
- 10.1201/9781420054989 is OK
- 10.48550/arXiv.2006.12485 is OK
- 10.1109/MGRS.2020.3046356 is OK
- 10.1109/TGRS.2004.828196 is OK
- 10.1109/TGRS.2024.3419083 is OK
- 10.1109/JSTARS.2022.3180994 is OK

🟡 SKIP DOIs

- No DOI given, and none found for title: Synthetic aperture radar

❌ MISSING DOIs

- None

❌ INVALID DOIs

- None

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.90  T=0.34 s (658.3 files/s, 381601.2 lines/s)
---------------------------------------------------------------------------------------
Language                             files          blank        comment           code
---------------------------------------------------------------------------------------
C++                                     32           5012          10190          40132
C                                       22           2460           5027          22001
Python                                  73           2767           2278          10051
C/C++ Header                            34           1325           4135           4386
C Shell                                 20            824           1737           3655
Bourne Shell                            11            612            835           2997
Jupyter Notebook                         8              0           5975            743
XML                                      8              6              3            463
make                                     4             74            195            311
Markdown                                 4             48              0            129
TeX                                      1              6              0            107
Windows Module Definition                1              1              2             52
JSON                                     1              0              0             39
Bourne Again Shell                       1              5              0             38
Dockerfile                               1              6              4             27
YAML                                     1              2              4             18
---------------------------------------------------------------------------------------
SUM:                                   222          13148          30385          85149
---------------------------------------------------------------------------------------

Commit count by author:

    77	Ling Chang
    62	espiritocz
    45	LC-SAR
    16	XuZhangUT
    13	Milan Lazecky
     6	AlignSAR
     2	Anurag
     1	Jose Manuel Delgado Blasco
     1	unknown

[editorialbot]

Paper file info:

📄 Wordcount for paper.md is 1088

✅ The paper includes a Statement of need section

[editorialbot]

License info:

✅ License found: Apache License 2.0 (Valid open source OSI approved license)

[editorialbot]

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

[rwegener2]

👋🏼 @LC-SAR, @philippemiron, and @shahchiragh this is the review thread for the paper. All of our communications will happen here from now on.

As a reviewer, the first step is to create a checklist for your review by entering

@editorialbot generate my checklist

as the top of a new comment in this thread.

These checklists contain the JOSS requirements. As you go over the submission, please check any items that you feel have been satisfied. The first comment in this thread also contains links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention openjournals/joss-reviews#REVIEW_NUMBER so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for reviews to be completed within about 2-4 weeks. Please let me know if any of you require some more time. We can also use EditorialBot (our bot) to set automatic reminders if you know you'll be away for a known period of time.

Please feel free to ping me (@rwegener2) if you have any questions/concerns.

[philippemiron]

Review checklist for @philippemiron

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the https://github.com/AlignSAR/alignSAR?
• License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@LC-SAR) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1. Contribute to the software 2. Report issues or problems with the software 3. Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[rwegener2]

Hey @philippemiron and @shahchiragh, just a friendly ping about this review. @shahchiragh, the first step is to generate your checklist using @editorialbot generate my checklist.

If you have any questions don't hesitate to reach out!

[shahchiragh]

Review checklist for @shahchiragh

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the https://github.com/AlignSAR/alignSAR?
• License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@LC-SAR) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1. Contribute to the software 2. Report issues or problems with the software 3. Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[shahchiragh]

@editorialbot check references

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

✅ OK DOIs

- 10.1109/LGRS.2024.3376992 is OK
- 10.1109/36.964973 is OK
- 10.1109/36.898661 is OK
- 10.1088/0266-5611/14/4/001 is OK
- 10.1201/9781420054989 is OK
- 10.48550/arXiv.2006.12485 is OK
- 10.1109/MGRS.2020.3046356 is OK
- 10.1109/TGRS.2004.828196 is OK
- 10.1109/TGRS.2024.3419083 is OK
- 10.1109/JSTARS.2022.3180994 is OK

🟡 SKIP DOIs

- No DOI given, and none found for title: Synthetic aperture radar

❌ MISSING DOIs

- None

❌ INVALID DOIs

- None

[shahchiragh]

Hi @rwegener2 , I’ve completed an initial round of the review. However, a few checks, specifically related to installation and documentation, are still pending as they depend on the software installation process. I will revisit these once the authors address the comments I’ve summarized in the repo issue.

Additionally, there are a few aspects related to the software paper, such as the "Quality of Writing" and "References," that I would like to bring to the authors' attention for further refinement of the paper. Please feel free to reach out for any questions or comments. Thanks!

[rwegener2]

Thanks for the thoughtful comments @shahchiragh! Sounds good for next steps.

[LC-SAR]

Thank you @rwegener2, @philippemiron and @shahchiragh, we have updated the AlignSAR repo based on your valuable suggestions. Please check my response to your comments in this channel, and our updated repo.

[rwegener2]

Hi there everyone,

In reading over the status of this paper review I'm noticing some themes from both reviewers. It seems that installation and the ability to find and run tests have been difficult for both reviewers. It appears that some of this information is available, but was perhaps hard to find. @philippemiron and @shahchiragh, if either of you have suggestions for how the authors could reorganize to make the tests easier to run or locate those could be useful pieces of feedback for the authors.

As for installation, that is an important point for the authors to address. It also hinders the ability of a reviewer to run evaluate the testing and functionality aspects of the review. @LC-SAR if there is any information (Ex. operating systems and versions) that the reviewers can provide to help you debug these installation problems please feel free to ask follow up questions.

[LC-SAR]

Thank you @rwegener2 for your comments and suggestions. We have updated the repo based on two reviewers' suggestions. For theirlast comments on docker installation, we tested internally again with three different operating systems/software, all work from our side. We await two reviewers' further responses, and then we will update the information related to e.g. operating systems and versions on the github repo. Thank you

[rwegener2]

Hi @LC-SAR. I understand that it can be difficult to test and account for every possible architecture. Still, it is important to remember that our reviewers are volunteers and debugging an installation can be quite time intensive, especially when they are new to a software. I also went through and tested the installation. It was successful when I installed on my Mac (Sonoma 14.6.1). I provided some suggestions for improving the installation instructions in a repository issue.

Hopefully between myself and the reviewers we can help ensure the AlignSAR installation and the instructions are as clear and accessible to as many people as possible. It seems like perhaps the Mac installation is operating well, but @philippemiron and @shahchirag are operating on Linux systems (is that correct?). Can you both please provide your linux flavors and versions?

I see a few possible routes forward:

1. @LC-SAR from this comment it sounds like you already have a system for testing multiple infrastructures. You could try the installation using the same linux versions as the reviewers in hopes of reproducing their errors.
2. It is certainly fair if you are not interested in supporting every operating system/version. Another option would be to include in your documentation that only certain operating systems and linux versions are supported. In that case I could begin a search for another set of reviewers that are able to meet the operating system requirements of this library.

I think this is a great software for the community and I'd love to see it keep moving forward. Let me know what you think the best next step is.

[shahchiragh]

Hi @rwegener2, apologies for the delay in getting back and troubleshooting this issue, as I understand it may have added extra work for the authors. I was able to take out some time to thoroughly troubleshoot the installation steps and identified the necessary modifications to the Dockerfile that resolved the issues. I’m using macOS Sequoia 15.2 and I had to adjust my Python version to 2.7.18. Even after the Python version change, I ran into several installation errors, but with some additional tweaks, I was able to get everything working. I've shared these changes with @LC-SAR in Issue 3 as a reference for troubleshooting, especially for users with different setups. I'll now proceed with the remaining items in the review, but if there's anything else you’d like me to check, please don’t hesitate to reach out.

[shahchiragh]

Review checklist for @shahchiragh

Conflict of interest

• I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• Repository: Is the source code for this software available at the https://github.com/AlignSAR/alignSAR?
• License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
• Contribution and authorship: Has the submitting author (@LC-SAR) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• Installation: Does installation proceed as outlined in the documentation?
• Functionality: Have the functional claims of the software been confirmed?
• Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• Community guidelines: Are there clear guidelines for third parties wishing to 1. Contribute to the software 2. Report issues or problems with the software 3. Seek support

Software paper

• Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• State of the field: Do the authors describe how this software compares to other commonly-used packages?
• Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[shahchiragh]

@editorialbot check references

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

✅ OK DOIs

- 10.1109/LGRS.2024.3376992 is OK
- 10.1109/36.964973 is OK
- 10.1109/36.898661 is OK
- 10.1088/0266-5611/14/4/001 is OK
- 10.1201/9781420054989 is OK
- 10.48550/arXiv.2006.12485 is OK
- 10.1109/MGRS.2020.3046356 is OK
- 10.1109/TGRS.2004.828196 is OK
- 10.1109/TGRS.2024.3419083 is OK
- 10.1109/JSTARS.2022.3180994 is OK
- 10.1109/MGRS.2013.2248301 is OK
- 10.3390/rs12152430 is OK

🟡 SKIP DOIs

- No DOI given, and none found for title: Doris: The delft object-oriented radar interferome...
- No DOI given, and none found for title: DORIS: The delft object-oriented radar interferome...
- No DOI given, and none found for title: Sentinel Application Platform (SNAP)
- No DOI given, and none found for title: GAMMA software
- No DOI given, and none found for title: Alaska Satellite Facility (ASF)

❌ MISSING DOIs

- 10.1016/0165-1684(92)90103-4 may be a valid DOI for title: Synthetic aperture radar: systems and signal proce...

❌ INVALID DOIs

- None

[rwegener2]

I was able to take out some time to thoroughly troubleshoot the installation steps and identified the necessary modifications to the Dockerfile that resolved the issues. ... I'll now proceed with the remaining items in the review, but if there's anything else you’d like me to check, please don’t hesitate to reach out.

Sounds good, @shahchiragh. It is very kind of you to take the time to dig into the errors. Thanks so much for your thoroughness in this review!

[LC-SAR]

Dear @shahchiragh, thanks for taking the time to test and offer suggestions on improving our dockerfile. We tested the dockerfile you suggested on mac OS successfully, and included it in our github repo, see https://github.com/AlignSAR/alignSAR/blob/main/misc/Dockerfile1. In Dockerfile1, we only modified the version of fftw from 3.3.10 to 3.2.2, as the higher version of fftw is not compatible with Doris software. We also supplemented the instruction in https://github.com/AlignSAR/alignSAR/blob/main/README.md, explaining the 'Dockerfile works well with three different operating systems/docker versions: macOS Ventura 13.7.3 (docker version: version 4.37.2), ubuntu 20.04.6 LTS (docker version: 24.0.5) and Ubuntu 20.04.1 (docker version: 24.0.7). If you encounter version-related errors or missing Python packages while running this Dockerfile on macOS Sequoia 15.2, please use Dockerfile1 in the folder 'misc' instead.'

Looking forward to your further suggestions.