Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error with metric upload to API #1319

Closed
mckenziephagen opened this issue Aug 14, 2024 · 6 comments
Closed

Error with metric upload to API #1319

mckenziephagen opened this issue Aug 14, 2024 · 6 comments
Labels
bug

Comments

@mckenziephagen
Copy link
Collaborator

mckenziephagen commented Aug 14, 2024 •
edited
Loading

What happened?

MRIQC fails when uploading IQMs to API.

Error log: 

/opt/conda/lib/python3.11/site-packages/dipy/denoise/noise_estimate.py:252: RuntimeWarning: divide by zero encountered in divide
  s = sum_m2 / (2 * K * sigma**2)
/opt/conda/lib/python3.11/site-packages/dipy/denoise/noise_estimate.py:252: RuntimeWarning: invalid value encountered in divide
  s = sum_m2 / (2 * K * sigma**2)
/opt/conda/lib/python3.11/site-packages/nireports/interfaces/reporting/base.py:95: UserWarning: The given float value must not exceed 0.0. But, you have given threshold=0.001.
  mask_nii = threshold_img(fixed_image_nii, 1e-3)
/opt/conda/lib/python3.11/site-packages/mriqc/interfaces/anatomical.py:490: RuntimeWarning: divide by zero encountered in divide
  bg_data[bg_data > 0] = bg_data[bg_data > 0] / bg_spread
/opt/conda/lib/python3.11/site-packages/nilearn/plotting/displays/_slicers.py:420: UserWarning: empty mask
  xmin_, xmax_, ymin_, ymax_, zmin_, zmax_ = get_mask_bounds(
Traceback (most recent call last):
  File "/opt/conda/bin/mriqc", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/mriqc/cli/run.py", line 178, in main
    mriqc_wf.run(**_plugin)
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/workflows.py", line 638, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 196, in run
    notrun.append(self._clean_queue(jobid, graph, result=result))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 259, in _clean_queue
    raise RuntimeError(''.join(result['traceback']))
RuntimeError: Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 64, in run_node
    result['result'] = node.run(updatehash=updatehash)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 1380, in _run_interface
    result = self._collate_results(
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 1249, in _collate_results
    for i, nresult, err in nodes:
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/utils.py", line 94, in nodelist_runner
    result = node.run(updatehash=updatehash)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node _UploadMetrics0.

Traceback:
        Traceback (most recent call last):
          File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 537, in _make_request
            response = conn.getresponse()
                       ^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/urllib3/connection.py", line 466, in getresponse
            httplib_response = super().getresponse()
                               ^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/http/client.py", line 1390, in getresponse
            response.begin()
          File "/opt/conda/lib/python3.11/http/client.py", line 325, in begin
            version, status, reason = self._read_status()
                                      ^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/http/client.py", line 286, in _read_status
            line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/socket.py", line 706, in readinto
            return self._sock.recv_into(b)
                   ^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/ssl.py", line 1314, in recv_into
            return self.read(nbytes, buffer)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/ssl.py", line 1166, in read
            return self._sslobj.read(len, buffer)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        TimeoutError: The read operation timed out

        The above exception was the direct cause of the following exception:

        Traceback (most recent call last):
          File "/opt/conda/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
            resp = conn.urlopen(
                   ^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 847, in urlopen
            retries = retries.increment(
                      ^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/urllib3/util/retry.py", line 470, in increment
            raise reraise(type(error), error, _stacktrace)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/urllib3/util/util.py", line 39, in reraise
            raise value
          File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
            response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 539, in _make_request
            self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
          File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 370, in _raise_timeout
            raise ReadTimeoutError(
        urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='mriqc.nimh.nih.gov', port=443): Read timed out. (read timeout=15)

        During handling of the above exception, another exception occurred:

        Traceback (most recent call last):
          File "/opt/conda/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run
            runtime = self._run_interface(runtime)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/mriqc/interfaces/webapi.py", line 141, in _run_interface
            response = upload_qc_metrics(
                       ^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/mriqc/interfaces/webapi.py", line 244, in upload_qc_metrics
            response = requests.post(
                       ^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/requests/api.py", line 115, in post
            return request("post", url, data=data, json=json, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/requests/api.py", line 59, in request
            return session.request(method=method, url=url, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
            resp = self.send(prep, **send_kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
            r = adapter.send(request, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/requests/adapters.py", line 532, in send
            raise ReadTimeout(e, request=request)
        requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='mriqc.nimh.nih.gov', port=443): Read timed out. (read timeout=15)

Running with --no-sub fixes this.

What command did you use?

singularity run -e mriqc-24.0.0.simg ds002785 ds002785/derivatives/mriqc-24.0.0 participant --participant-label sub-0029 -w /lscratch/mphagen/work/ --omp-nthreads 8 --mem 10 --verbose-reports

What version of the software are you running?

24

How are you running this software?

Singularity

Is your data BIDS valid?

Yes

Are you reusing any previously computed results?

No

Please copy and paste any relevant log output.

Full output log 
Subject sub-0029
Running task 30
Commandline: singularity run -e mriqc-24.0.0.simg ds002785 ds002785/derivatives/mriqc-24.0.0 participant --participant-label sub-0029 -w /lscratch/mphagen/work/ -
-omp-nthreads 8 --mem 10 --verbose-reports
------------------------------------------------------------------
  Running MRIQC version 24.1.0.dev0+g3fe90466.d20240417
  ----------------------------------------------------------------
  * BIDS dataset path: /scratch/users/mphagen/mriqc-protocol/ds002785.
  * Output folder: ds002785/derivatives/mriqc-24.0.0.
  * Analysis levels: ['participant'].
------------------------------------------------------------------
2024-08-14 11:00:31 | IMPORTANT | mriqc            | Building MRIQC's workflows...
2024-08-14 11:00:32 | IMPORTANT | mriqc            | DataLad dataset identified, attempting to `datalad get` unavailable files.
action summary:
  get (notneeded: 6)
2024-08-14 11:00:34 | IMPORTANT | mriqc            | DataLad dataset identified, attempting to `datalad get` unavailable files.
2024-08-14 11:00:35 | IMPORTANT | mriqc            | DataLad dataset identified, attempting to `datalad get` unavailable files.
2024-08-14 11:00:41 | IMPORTANT | mriqc            | Workflow building finished (exit code 0).
2024-08-14 11:00:47 | WARNING  | mriqc            | IMPORTANT: Anonymized quality metrics (IQMs) will be submitted to MRIQC's metrics repository. Submission of IQ
Ms can be disabled using the ``--no-sub`` argument. Please visit https://mriqc.readthedocs.io/en/latest/dsa.html to revise MRIQC's Data Sharing Agreement.
2024-08-14 11:01:36 | WARNING  | nipype.interface | Insufficient number of low-b orientations (1) to safely calculate signal drift.
2024-08-14 11:01:38 | INFO     | niworkflows      | Estimating initial transform using AffineInitializer
2024-08-14 11:01:45 | INFO     | niworkflows      | Terminal outputs of initialization saved (/lscratch/mphagen/work/mriqc_wf/dwiMRIQC/SpatialNormalization/_in_fi
le_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..dwi..sub-0029_dwi.nii.gz/EPI2MNI/stderr.nipype-init).
2024-08-14 11:01:45 | INFO     | niworkflows      | Loading settings from file /opt/conda/lib/python3.11/site-packages/niworkflows/data/boldref-mni_registration_p
recise_000.json.
2024-08-14 11:01:45 | INFO     | niworkflows      | Retry #1, commandline: 
antsRegistration --collapse-output-transforms 1 --dimensionality 3 --float 0 --initial-moving-transform [ /lscratch/mphagen/work/mriqc_wf/dwiMRIQC/SpatialNormaliz
ation/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..dwi..sub-0029_dwi.nii.gz/EPI2MNI/transform.mat, 0 ] --initialize-transforms-per-stag
e 0 --interpolation LanczosWindowedSinc --output [ epi_to_mni, epi_to_mni_Warped.nii.gz ] --transform Rigid[ 0.05 ] --metric Mattes[ /templateflow/tpl-MNI152NLin2
009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-fMRIPrep_boldref.nii.gz, /lscratch/mphagen/work/mriqc_wf/dwiMRIQC/SpatialNormalization/_in_file_..scratch..users..mph
agen..mriqc-protocol..ds002785..sub-0029..dwi..sub-0029_dwi.nii.gz/SharpenEPI/sub-0029_dwi_valid_average_corrected.nii.gz, 1, 56, Regular, 0.25 ] --convergence [ 
10000x1000x100, 1e-06, 20 ] --smoothing-sigmas 4.0x2.0x1.0vox --shrink-factors 4x2x1 --use-histogram-matching 1 --masks [ /templateflow/tpl-MNI152NLin2009cAsym/tp
l-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz, /lscratch/mphagen/work/mriqc_wf/dwiMRIQC/synthstrip_wf/_in_file_..scratch..users..mphagen..mriqc-protocol..ds
002785..sub-0029..dwi..sub-0029_dwi.nii.gz/synthstrip/clipped_corrected_desc-brain_mask.nii.gz ] --transform Affine[ 0.08 ] --metric Mattes[ /templateflow/tpl-MNI
152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-fMRIPrep_boldref.nii.gz, /lscratch/mphagen/work/mriqc_wf/dwiMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..dwi..sub-0029_dwi.nii.gz/SharpenEPI/sub-0029_dwi_valid_average_corrected.nii.gz, 1, 56, Regular, 0.25 ] --convergence [ 500x250x100, 1e-06, 20 ] --smoothing-sigmas 4.0x2.0x1.0vox --shrink-factors 8x4x2 --use-histogram-matching 1 --masks [ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz, /lscratch/mphagen/work/mriqc_wf/dwiMRIQC/synthstrip_wf/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..dwi..sub-0029_dwi.nii.gz/synthstrip/clipped_corrected_desc-brain_mask.nii.gz ] --transform SyN[ 0.1, 3.0, 0.0 ] --metric CC[ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-fMRIPrep_boldref.nii.gz, /lscratch/mphagen/work/mriqc_wf/dwiMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..dwi..sub-0029_dwi.nii.gz/SharpenEPI/sub-0029_dwi_valid_average_corrected.nii.gz, 1, 4, None, 1 ] --convergence [ 100x30x20, 1e-06, 10 ] --smoothing-sigmas 3.0x2.0x1.0vox --shrink-factors 8x4x2 --use-histogram-matching 1 --masks [ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz, /lscratch/mphagen/work/mriqc_wf/dwiMRIQC/synthstrip_wf/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..dwi..sub-0029_dwi.nii.gz/synthstrip/clipped_corrected_desc-brain_mask.nii.gz ] --winsorize-image-intensities [ 0.005, 0.995 ]  --write-composite-transform 1
2024-08-14 11:02:04 | INFO     | niworkflows      | Successful spatial normalization (retry #1).
2024-08-14 11:02:04 | INFO     | niworkflows      | Report - setting fixed (/templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-fMRIPrep_boldref.nii.gz) and moving (/lscratch/mphagen/work/mriqc_wf/dwiMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..dwi..sub-0029_dwi.nii.gz/EPI2MNI/epi_to_mni_Warped.nii.gz) images
2024-08-14 11:02:04 | INFO     | niworkflows      | Generating visual report
2024-08-14 11:02:51 | INFO     | niworkflows      | Estimating initial transform using AffineInitializer
2024-08-14 11:03:07 | INFO     | niworkflows      | Terminal outputs of initialization saved (/lscratch/mphagen/work/mriqc_wf/anatMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..anat..sub-0029_T1w.nii.gz/SpatialNormalization/stderr.nipype-init).
2024-08-14 11:03:07 | INFO     | niworkflows      | Loading settings from file /opt/conda/lib/python3.11/site-packages/niworkflows/data/t1w-mni_registration_testing_000.json.
2024-08-14 11:03:07 | INFO     | niworkflows      | Retry #1, commandline: 
antsRegistration --collapse-output-transforms 1 --dimensionality 3 --float 0 --initial-moving-transform [ /lscratch/mphagen/work/mriqc_wf/anatMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..anat..sub-0029_T1w.nii.gz/SpatialNormalization/transform.mat, 0 ] --initialize-transforms-per-stage 0 --interpolation LanczosWindowedSinc --output [ ants_t1_to_mni, ants_t1_to_mni_Warped.nii.gz ] --transform Rigid[ 1.0 ] --metric Mattes[ /lscratch/mphagen/work/mriqc_wf/anatMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..anat..sub-0029_T1w.nii.gz/SpatialNormalization/fixed_masked.nii.gz, /lscratch/mphagen/work/mriqc_wf/anatMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..anat..sub-0029_T1w.nii.gz/SpatialNormalization/moving_masked.nii.gz, 1, 56, Random, 0.2 ] --convergence [ 20, 1e-07, 15 ] --smoothing-sigmas 4.0vox --shrink-factors 2 --use-histogram-matching 0 --transform Affine[ 1.0 ] --metric Mattes[ /lscratch/mphagen/work/mriqc_wf/anatMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..anat..sub-0029_T1w.nii.gz/SpatialNormalization/fixed_masked.nii.gz, /lscratch/mphagen/work/mriqc_wf/anatMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..anat..sub-0029_T1w.nii.gz/SpatialNormalization/moving_masked.nii.gz, 1, 56, Random, 0.1 ] --convergence [ 15, 1e-08, 5 ] --smoothing-sigmas 2.0vox --shrink-factors 1 --use-histogram-matching 1 --winsorize-image-intensities [ 0.005, 0.995 ]  --write-composite-transform 1
2024-08-14 11:03:12 | INFO     | niworkflows      | Successful spatial normalization (retry #1).
2024-08-14 11:03:14 | INFO     | niworkflows      | Report - setting fixed (/lscratch/mphagen/work/mriqc_wf/anatMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..anat..sub-0029_T1w.nii.gz/SpatialNormalization/fixed_masked.nii.gz) and moving (/lscratch/mphagen/work/mriqc_wf/anatMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..anat..sub-0029_T1w.nii.gz/SpatialNormalization/ants_t1_to_mni_Warped.nii.gz) images
2024-08-14 11:03:14 | INFO     | niworkflows      | Generating visual report
2024-08-14 11:05:26 | WARNING  | nipype.interface | Estimated signal variation in the background was too small (MAD=0.0, sigma=0.5239)
2024-08-14 11:05:26 | WARNING  | nipype.interface | Estimated signal variation in the background was too small (MAD=0.0, sigma=0.5239)
2024-08-14 11:05:26 | WARNING  | nipype.interface | Estimated signal variation in the background was too small (MAD=0.0, sigma=0.5239)
2024-08-14 11:06:12 | INFO     | niworkflows      | Estimating initial transform using AffineInitializer
2024-08-14 11:06:16 | INFO     | niworkflows      | Estimating initial transform using AffineInitializer
2024-08-14 11:06:17 | WARNING  | nipype.workflow  | Storing result file without outputs
2024-08-14 11:06:17 | WARNING  | nipype.workflow  | [Node] Error on "_UploadMetrics0" (/lscratch/mphagen/work/mriqc_wf/funcMRIQC/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-emomatching_acq-seq_bold.nii.gz/UploadMetrics/mapflow/_UploadMetrics0)
2024-08-14 11:06:17 | WARNING  | nipype.workflow  | Storing result file without outputs
2024-08-14 11:06:17 | WARNING  | nipype.workflow  | [Node] Error on "mriqc_wf.funcMRIQC.UploadMetrics" (/lscratch/mphagen/work/mriqc_wf/funcMRIQC/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-emomatching_acq-seq_bold.nii.gz/UploadMetrics)
2024-08-14 11:06:17 | ERROR    | nipype.workflow  | Node UploadMetrics.a1 (taskid=150) crashed: /scratch/users/mphagen/mriqc-protocol/ds002785/derivatives/mriqc-24.0.0/logs/crash-20240814-110617-mphagen-UploadMetrics.a1-9b22ba04-6602-4b06-abe0-9ba3f380cc26.txt
2024-08-14 11:06:20 | INFO     | niworkflows      | Loading settings from file /opt/conda/lib/python3.11/site-packages/niworkflows/data/boldref-mni_registration_precise_000.json.
2024-08-14 11:06:20 | INFO     | niworkflows      | Retry #1, commandline: 
antsRegistration --collapse-output-transforms 1 --dimensionality 3 --float 0 --initial-moving-transform [ /lscratch/mphagen/work/mriqc_wf/funcMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-emomatching_acq-seq_bold.nii.gz/EPI2MNI/transform.mat, 0 ] --initialize-transforms-per-stage 0 --interpolation LanczosWindowedSinc --output [ epi_to_mni, epi_to_mni_Warped.nii.gz ] --transform Rigid[ 0.05 ] --metric Mattes[ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-fMRIPrep_boldref.nii.gz, /lscratch/mphagen/work/mriqc_wf/funcMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-emomatching_acq-seq_bold.nii.gz/SharpenEPI/sub-0029_task-emomatching_acq-seq_bold_desc-realigned_valid_tstat_corrected.nii.gz, 1, 56, Regular, 0.25 ] --convergence [ 10000x1000x100, 1e-06, 20 ] --smoothing-sigmas 4.0x2.0x1.0vox --shrink-factors 4x2x1 --use-histogram-matching 1 --masks [ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz, /lscratch/mphagen/work/mriqc_wf/funcMRIQC/synthstrip_wf/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-emomatching_acq-seq_bold.nii.gz/synthstrip/clipped_corrected_desc-brain_mask.nii.gz ] --transform Affine[ 0.08 ] --metric Mattes[ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-fMRIPrep_boldref.nii.gz, /lscratch/mphagen/work/mriqc_wf/funcMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-emomatching_acq-seq_bold.nii.gz/SharpenEPI/sub-0029_task-emomatching_acq-seq_bold_desc-realigned_valid_tstat_corrected.nii.gz, 1, 56, Regular, 0.25 ] --convergence [ 500x250x100, 1e-06, 20 ] --smoothing-sigmas 4.0x2.0x1.0vox --shrink-factors 8x4x2 --use-histogram-matching 1 --masks [ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz, /lscratch/mphagen/work/mriqc_wf/funcMRIQC/synthstrip_wf/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-emomatching_acq-seq_bold.nii.gz/synthstrip/clipped_corrected_desc-brain_mask.nii.gz ] --transform SyN[ 0.1, 3.0, 0.0 ] --metric CC[ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-fMRIPrep_boldref.nii.gz, /lscratch/mphagen/work/mriqc_wf/funcMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-emomatching_acq-seq_bold.nii.gz/SharpenEPI/sub-0029_task-emomatching_acq-seq_bold_desc-realigned_valid_tstat_corrected.nii.gz, 1, 4, None, 1 ] --convergence [ 100x30x20, 1e-06, 10 ] --smoothing-sigmas 3.0x2.0x1.0vox --shrink-factors 8x4x2 --use-histogram-matching 1 --masks [ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz, /lscratch/mphagen/work/mriqc_wf/funcMRIQC/synthstrip_wf/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-emomatching_acq-seq_bold.nii.gz/synthstrip/clipped_corrected_desc-brain_mask.nii.gz ] --winsorize-image-intensities [ 0.005, 0.995 ]  --write-composite-transform 1
2024-08-14 11:06:25 | INFO     | niworkflows      | Terminal outputs of initialization saved (/lscratch/mphagen/work/mriqc_wf/funcMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-workingmemory_acq-seq_bold.nii.gz/EPI2MNI/stderr.nipype-init).
2024-08-14 11:06:25 | INFO     | niworkflows      | Loading settings from file /opt/conda/lib/python3.11/site-packages/niworkflows/data/boldref-mni_registration_precise_000.json.
2024-08-14 11:06:25 | INFO     | niworkflows      | Retry #1, commandline: 
antsRegistration --collapse-output-transforms 1 --dimensionality 3 --float 0 --initial-moving-transform [ /lscratch/mphagen/work/mriqc_wf/funcMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-workingmemory_acq-seq_bold.nii.gz/EPI2MNI/transform.mat, 0 ] --initialize-transforms-per-stage 0 --interpolation LanczosWindowedSinc --output [ epi_to_mni, epi_to_mni_Warped.nii.gz ] --transform Rigid[ 0.05 ] --metric Mattes[ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-fMRIPrep_boldref.nii.gz, /lscratch/mphagen/work/mriqc_wf/funcMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-workingmemory_acq-seq_bold.nii.gz/SharpenEPI/sub-0029_task-workingmemory_acq-seq_bold_desc-realigned_valid_tstat_corrected.nii.gz, 1, 56, Regular, 0.25 ] --convergence [ 10000x1000x100, 1e-06, 20 ] --smoothing-sigmas 4.0x2.0x1.0vox --shrink-factors 4x2x1 --use-histogram-matching 1 --masks [ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz, /lscratch/mphagen/work/mriqc_wf/funcMRIQC/synthstrip_wf/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-workingmemory_acq-seq_bold.nii.gz/synthstrip/clipped_corrected_desc-brain_mask.nii.gz ] --transform Affine[ 0.08 ] --metric Mattes[ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-fMRIPrep_boldref.nii.gz, /lscratch/mphagen/work/mriqc_wf/funcMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-workingmemory_acq-seq_bold.nii.gz/SharpenEPI/sub-0029_task-workingmemory_acq-seq_bold_desc-realigned_valid_tstat_corrected.nii.gz, 1, 56, Regular, 0.25 ] --convergence [ 500x250x100, 1e-06, 20 ] --smoothing-sigmas 4.0x2.0x1.0vox --shrink-factors 8x4x2 --use-histogram-matching 1 --masks [ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz, /lscratch/mphagen/work/mriqc_wf/funcMRIQC/synthstrip_wf/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-workingmemory_acq-seq_bold.nii.gz/synthstrip/clipped_corrected_desc-brain_mask.nii.gz ] --transform SyN[ 0.1, 3.0, 0.0 ] --metric CC[ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-fMRIPrep_boldref.nii.gz, /lscratch/mphagen/work/mriqc_wf/funcMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-workingmemory_acq-seq_bold.nii.gz/SharpenEPI/sub-0029_task-workingmemory_acq-seq_bold_desc-realigned_valid_tstat_corrected.nii.gz, 1, 4, None, 1 ] --convergence [ 100x30x20, 1e-06, 10 ] --smoothing-sigmas 3.0x2.0x1.0vox --shrink-factors 8x4x2 --use-histogram-matching 1 --masks [ /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz, /lscratch/mphagen/work/mriqc_wf/funcMRIQC/synthstrip_wf/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-workingmemory_acq-seq_bold.nii.gz/synthstrip/clipped_corrected_desc-brain_mask.nii.gz ] --winsorize-image-intensities [ 0.005, 0.995 ]  --write-composite-transform 1
2024-08-14 11:06:40 | INFO     | niworkflows      | Successful spatial normalization (retry #1).
2024-08-14 11:06:40 | INFO     | niworkflows      | Report - setting fixed (/templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-fMRIPrep_boldref.nii.gz) and moving (/lscratch/mphagen/work/mriqc_wf/funcMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-emomatching_acq-seq_bold.nii.gz/EPI2MNI/epi_to_mni_Warped.nii.gz) images
2024-08-14 11:06:40 | INFO     | niworkflows      | Generating visual report
2024-08-14 11:06:45 | INFO     | niworkflows      | Successful spatial normalization (retry #1).
2024-08-14 11:06:45 | INFO     | niworkflows      | Report - setting fixed (/templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-fMRIPrep_boldref.nii.gz) and moving (/lscratch/mphagen/work/mriqc_wf/funcMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-workingmemory_acq-seq_bold.nii.gz/EPI2MNI/epi_to_mni_Warped.nii.gz) images
2024-08-14 11:06:45 | INFO     | niworkflows      | Generating visual report
Finished tasks 30 with exit code 1

Additional information / screenshots

No response
@oesteban
Copy link
Member

Thanks!

Can you paste the contents of /scratch/users/mphagen/mriqc-protocol/ds002785/derivatives/mriqc-24.0.0/logs/crash-20240814-110617-mphagen-UploadMetrics.a1-9b22ba04-6602-4b06-abe0-9ba3f380cc26.txt? This is the 'crashfile' corresponding to the particular error.

However, the workflow crashed for another reason -- the metrics were not uploaded, that's clear, but the workflow will go on unless you are running on 'strict' mode (and you are not). So there must be another error under /scratch/users/mphagen/mriqc-protocol/ds002785/derivatives/mriqc-24.0.0/logs/

@jbwexler
Copy link

jbwexler commented Sep 5, 2024

I just ran a bunch of datasets with 24.0.1 and got the above error for all of them. Perhaps the internet was down at the time? From what I can tell, there were no other errors and I'm not running in strict mode:

(main) login4.frontera(1051)$ ls /scratch1/03201/jbwexler/openneuro_derivatives/derivatives/mriqc/ds004478-mriqc/logs/
crash-20240904-092631-jbwexler-UploadMetrics.a0-e9a2ac19-0251-4d7b-acba-3de08a1dd9ae.txt
crash-20240904-092634-jbwexler-UploadMetrics.a0-7ab1dd29-225d-4af7-83bd-dbaf8c8fb2a1.txt
crash-20240904-092634-jbwexler-UploadMetrics.a0-c8c9e83e-560b-4c94-b815-ee997cb78c4b.txt
crash-20240904-092644-jbwexler-UploadMetrics.a0-2d3d5279-dfd1-44f8-9311-9d86e8c55f21.txt
crash-20240904-092714-jbwexler-UploadMetrics.a0-193566fd-bdf4-4ef1-a4b4-3c460449d393.txt
crash-20240904-092718-jbwexler-UploadMetrics.a0-e79a29d4-8979-4af0-b91d-9f011cbed8a3.txt
mriqc-20240904-092224_a57ba336-81bb-45bb-b2d9-3858aa74ca17.log
mriqc-20240904-092225_66f5102a-eb8c-4c2e-b3f3-f143ea15ca3a.log
mriqc-20240904-092228_f8df30d4-4022-43e9-81b6-d6a7dea20837.log
mriqc-20240904-092230_b7475734-3be8-46f8-8721-1c200776456e.log
mriqc-20240904-092231_8f8ac21a-85f6-4e07-b3fb-98d74d71cbf3.log
mriqc-20240904-092236_342684f6-b222-48fb-bee7-3135c537be65.log

Here's an example command:

code/containers/scripts/singularity_cmd run --bind /tmp:/node_tmp --bind /scratch1/03201/jbwexler/mriqc_home:/home/mriqc code/containers/images/bids/bids-mriqc--24.0.1.sing sourcedata/raw /scratch1/03201/jbwexler/openneuro_derivatives/derivatives/mriqc/ds004478-mriqc participant --participant-label '01' -w '/node_tmp/work_dir/mriqc/ds004478_sub-01' -vv --nprocs 11 --ants-nthreads 8 --verbose-reports --dsname ds004478 --mem_gb 30

@oesteban
Copy link
Member

oesteban commented Sep 5, 2024

Can you paste the content of one of those crash files?

@jbwexler
Copy link

jbwexler commented Sep 5, 2024

Yep, here's an example:

Node: mriqc_wf.anatMRIQC.UploadMetrics
Working directory: /node_tmp/work_dir/mriqc/ds004478_sub-01/mriqc_wf/anatMRIQC/a63125fa58866c73d5b1e2e2859283294faa562b/UploadMetrics

Node inputs:

auth_token = 8sSYVI0XjFqacEMZ8wF4
email = <undefined>
endpoint = https://mriqc.nimh.nih.gov:443/api/v1
in_iqms = /scratch1/03201/jbwexler/openneuro_derivatives/derivatives/mriqc/ds004478-mriqc/sub-01/anat/sub-01_T1w.json
modality = undefined
strict = False

Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 64, in run_node
    result['result'] = node.run(updatehash=updatehash)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node UploadMetrics.

Traceback:
        Traceback (most recent call last):
          File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 537, in _make_request
            response = conn.getresponse()
                       ^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/urllib3/connection.py", line 466, in getresponse
            httplib_response = super().getresponse()
                               ^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/http/client.py", line 1390, in getresponse
            response.begin()
          File "/opt/conda/lib/python3.11/http/client.py", line 325, in begin
            version, status, reason = self._read_status()
                                      ^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/http/client.py", line 286, in _read_status
            line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/socket.py", line 706, in readinto
            return self._sock.recv_into(b)
                   ^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/ssl.py", line 1314, in recv_into
            return self.read(nbytes, buffer)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/ssl.py", line 1166, in read
            return self._sslobj.read(len, buffer)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        TimeoutError: The read operation timed out

        The above exception was the direct cause of the following exception:

        Traceback (most recent call last):
          File "/opt/conda/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
            resp = conn.urlopen(
                   ^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 847, in urlopen
            retries = retries.increment(
                      ^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/urllib3/util/retry.py", line 470, in increment
            raise reraise(type(error), error, _stacktrace)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/urllib3/util/util.py", line 39, in reraise
            raise value
          File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
            response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 539, in _make_request
            self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
          File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 370, in _raise_timeout
            raise ReadTimeoutError(
        urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='mriqc.nimh.nih.gov', port=443): Read timed out. (read timeout=15)

        During handling of the above exception, another exception occurred:

        Traceback (most recent call last):
          File "/opt/conda/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run
            runtime = self._run_interface(runtime)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/mriqc/interfaces/webapi.py", line 148, in _run_interface
            response, payload = upload_qc_metrics(
                                ^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/mriqc/interfaces/webapi.py", line 272, in upload_qc_metrics
            response = requests.post(
                       ^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/requests/api.py", line 115, in post
            return request("post", url, data=data, json=json, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/requests/api.py", line 59, in request
            return session.request(method=method, url=url, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
            resp = self.send(prep, **send_kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
            r = adapter.send(request, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/requests/adapters.py", line 532, in send
            raise ReadTimeout(e, request=request)
        requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='mriqc.nimh.nih.gov', port=443): Read timed out. (read timeout=15)

@mckenziephagen
Copy link
Collaborator Author

`crash-20240814-110617-mphagen-UploadMetrics.a1-9b22ba04-6602-4b06-abe0-9ba3f380cc26.txt` 
Working directory: /lscratch/mphagen/work/mriqc_wf/funcMRIQC/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..func..sub-0029_task-emomatchi
ng_acq-seq_bold.nii.gz/UploadMetrics
Node inputs:
auth_token = 8sSYVI0XjFqacEMZ8wF4
email = <undefined>
endpoint = https://mriqc.nimh.nih.gov:443/api/v1
in_iqms = ['/scratch/users/mphagen/mriqc-protocol/ds002785/derivatives/mriqc-24.0.0/sub-0029/func/sub-0029_task-emomatching_acq-seq_bold.json']
strict = False
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 64, in run_node
    result['result'] = node.run(updatehash=updatehash)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 1380, in _run_interface
    result = self._collate_results(
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 1249, in _collate_results
    for i, nresult, err in nodes:
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/utils.py", line 94, in nodelist_runner
    result = node.run(updatehash=updatehash)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
                       ^^^^^^^^^^^^^^^^^^^
    return self._run_command(execute)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node _UploadMetrics0.

Traceback:
        Traceback (most recent call last):
          File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 537, in _make_request
            response = conn.getresponse()
                       ^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/urllib3/connection.py", line 466, in getresponse
            httplib_response = super().getresponse()
                               ^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/http/client.py", line 1390, in getresponse
            response.begin()
          File "/opt/conda/lib/python3.11/http/client.py", line 325, in begin
            version, status, reason = self._read_status()
                                      ^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/http/client.py", line 286, in _read_status
            line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/socket.py", line 706, in readinto
            return self._sock.recv_into(b)
                   ^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/ssl.py", line 1314, in recv_into
            return self.read(nbytes, buffer)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/ssl.py", line 1166, in read
            return self._sslobj.read(len, buffer)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        TimeoutError: The read operation timed out

        The above exception was the direct cause of the following exception:

        Traceback (most recent call last):
          File "/opt/conda/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
            resp = conn.urlopen(
                   ^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 847, in urlopen
            retries = retries.increment(
                      ^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/urllib3/util/retry.py", line 470, in increment
            raise reraise(type(error), error, _stacktrace)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/urllib3/util/util.py", line 39, in reraise
            raise value
          File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
            response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 539, in _make_request
            self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
          File "/opt/conda/lib/python3.11/site-packages/urllib3/connectionpool.py", line 370, in _raise_timeout
            raise ReadTimeoutError(
        urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='mriqc.nimh.nih.gov', port=443): Read timed out. (read timeout=15)

        During handling of the above exception, another exception occurred:

        Traceback (most recent call last):
          File "/opt/conda/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run
            runtime = self._run_interface(runtime)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/mriqc/interfaces/webapi.py", line 141, in _run_interface
            response = upload_qc_metrics(
                       ^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/mriqc/interfaces/webapi.py", line 244, in upload_qc_metrics
            response = requests.post(
                       ^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/requests/api.py", line 115, in post
            return request("post", url, data=data, json=json, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/requests/api.py", line 59, in request
            return session.request(method=method, url=url, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
            resp = self.send(prep, **send_kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
            r = adapter.send(request, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/requests/adapters.py", line 532, in send
            raise ReadTimeout(e, request=request)
        requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='mriqc.nimh.nih.gov', port=443): Read timed out. (read timeout=15)

As for the other crashfile, I found this:

`crash-20240814-114516-mphagen-EPI2MNI .a0-f02fd06b-9c09-4b39-b35c-5cd905232ae8.txt` 
Node: mriqc_wf.dwiMRIQC.SpatialNormalization.EPI2MNI
Working directory: /lscratch/mphagen/work/mriqc_wf/dwiMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..dwi..sub-
0029_dwi.nii.gz/EPI2MNI
Node inputs:
compress_report = auto
explicit_masking = False
flavor = precise
float = False
initial_moving_transform = <undefined>
lesion_mask = <undefined>
moving = boldref
moving_image = /lscratch/mphagen/work/mriqc_wf/dwiMRIQC/SpatialNormalization/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..dwi..sub-0029
_dwi.nii.gz/SharpenEPI/sub-0029_dwi_valid_average_corrected.nii.gz
moving_mask = /lscratch/mphagen/work/mriqc_wf/dwiMRIQC/synthstrip_wf/_in_file_..scratch..users..mphagen..mriqc-protocol..ds002785..sub-0029..dwi..sub-0029_dwi.nii
.gz/synthstrip/clipped_corrected_desc-brain_mask.nii.gz
num_threads = 8
orientation = RAS
out_report = report.svg
  File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 64, in run_node
reference = boldref
reference_image = /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-fMRIPrep_boldref.nii.gz
reference_mask = /templateflow/tpl-MNI152NLin2009cAsym/tpl-MNI152NLin2009cAsym_res-02_desc-brain_mask.nii.gz
settings = <undefined>
template = MNI152NLin2009cAsym
template_resolution = <undefined>
template_spec = <undefined>

Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/site-packages/mriqc/engine/plugin.py", line 64, in run_node
    result['result'] = node.run(updatehash=updatehash)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node EPI2MNI.

Traceback:
        Traceback (most recent call last):
          File "/opt/conda/lib/python3.11/site-packages/nipype/interfaces/base/core.py", line 398, in run
            runtime = self._post_run_hook(runtime)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/niworkflows/interfaces/reportlets/registration.py", line 82, in _post_run_hook
            return super()._post_run_hook(runtime)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/nipype/interfaces/mixins/reporting.py", line 50, in _post_run_hook
            self._generate_report()
          File "/opt/conda/lib/python3.11/site-packages/niworkflows/interfaces/reportlets/base.py", line 102, in _generate_report
            plot_registration(
          File "/opt/conda/lib/python3.11/site-packages/niworkflows/viz/utils.py", line 388, in plot_registration
            svg = extract_svg(display, compress=compress)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/niworkflows/viz/utils.py", line 142, in extract_svg
            image_svg = svg_compress(image_svg, compress)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/site-packages/niworkflows/viz/utils.py", line 66, in svg_compress
            pout = subprocess.run(
                   ^^^^^^^^^^^^^^^
          File "/opt/conda/lib/python3.11/subprocess.py", line 571, in run
            raise CalledProcessError(retcode, process.args,
        subprocess.CalledProcessError: Command 'svgo -i - -o - -q -p 3 --pretty' returned non-zero exit status 135.

However, this might not be representative - this is the only crashfile for EPI2MNI in the directory, and this error was more widespread than this one participant. The other crashfiles are mostly for ComputeQI2, so I'll see if those are related to this problem.

@oesteban
Copy link
Member

oesteban commented Sep 5, 2024

The first error in the UploadMetrics node was addressed in #1336, which was released with 24.0.2. Basically, yes, when the server is down you get a timeout and it wasn't handled gracefully (now it should). Please reopen this issue if 24.0.2 crashes again this way.

@mckenziephagen for the second error in EPI2MNI let's open a new issue.

@oesteban oesteban closed this as completed Sep 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@oesteban @jbwexler @mckenziephagen