Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Galaxy-stable] Very slow file upload #164

Open
pcm32 opened this issue Mar 7, 2018 · 9 comments
Open

[Galaxy-stable] Very slow file upload #164

pcm32 opened this issue Mar 7, 2018 · 9 comments
Assignees
@pkrog

Comments

@pcm32
Copy link
Member

pcm32 commented Mar 7, 2018

During workshop we see very slow file uploads. I think this started after the addition of ISATools dependencies, that might be conflicting with uWSGI.
@pkrog
Copy link
Member

pkrog commented Mar 15, 2018

What is uWSGI?
Could you be more specific about "slow file uploads"?

@pcm32
Copy link
Member Author

pcm32 commented Mar 15, 2018

uWSGI is the default HTML cacher and web server invoking Galaxy on the recommended admin setup for Galaxy. Slow update can be several minutes, noticeably slow. It happens with at file types and sizes.

@pcm32
Copy link
Member Author

pcm32 commented Mar 15, 2018 •
edited
Loading

Log of upload by @pkrog using Safari, where no major delay is noticeable.

galaxy.tools DEBUG 2018-03-15 12:40:51,736 Validated and populated state for tool request (25.423 ms)
galaxy.tools.actions.upload DEBUG 2018-03-15 12:40:51,736 Persisted uploads (0.029 ms)
galaxy.tools.actions.upload DEBUG 2018-03-15 12:40:51,791 Checked and cleaned uploads (54.394 ms)
galaxy.tools.actions.upload_common INFO 2018-03-15 12:40:51,817 tool upload1 created job id 132
galaxy.tools.actions.upload DEBUG 2018-03-15 12:40:51,872 Created upload job (81.060 ms)
galaxy.tools.execute DEBUG 2018-03-15 12:40:51,872 Tool [upload1] created job [132] (135.773 ms)
galaxy.tools.execute DEBUG 2018-03-15 12:40:51,886 Executed 1 job(s) for tool upload1 request: (149.869 ms)
151.77.171.148 - - [15/Mar/2018:12:40:51 +0000] "POST /api/tools HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
[pid: 190|app: 0|req: 2206/3167] 10.233.68.0 () {60 vars in 1266 bytes} [Thu Mar 15 12:40:51 2018] POST /_upload => generated 1143 bytes in 281 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 3)
151.77.171.148 - - [15/Mar/2018:12:40:51 +0000] "GET /api/histories/eafb646da3b7aac5/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-15T12%3A40%3A28.612Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
[pid: 187|app: 0|req: 962/3168] 10.233.68.0 () {52 vars in 1292 bytes} [Thu Mar 15 12:40:51 2018] GET /api/histories/eafb646da3b7aac5/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-15T12%3A40%3A28.612Z&qv=False&qv=False => generated 491 bytes in 51 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 3)
galaxy.jobs DEBUG 2018-03-15 12:40:52,585 (132) Working directory for job is: /export/galaxy-central/database/job_working_directory/000/132
galaxy.jobs.handler DEBUG 2018-03-15 12:40:52,592 (132) Dispatching to local runner
galaxy.jobs DEBUG 2018-03-15 12:40:52,615 (132) Persisting job destination (destination id: local)
galaxy.jobs.runners DEBUG 2018-03-15 12:40:52,624 Job [132] queued (31.330 ms)
galaxy.jobs.handler INFO 2018-03-15 12:40:52,629 (132) Job dispatched
galaxy.jobs.command_factory INFO 2018-03-15 12:40:52,803 Built script [/export/galaxy-central/database/job_working_directory/000/132/tool_script.sh] for tool command [python /export/tools/data_source/upload.py /export/galaxy-central /export/galaxy-central/database/job_working_directory/000/132/registry.xml /export/galaxy-central/database/files/tmpbAUDAn 1536:/export/galaxy-central/database/job_working_directory/000/132/dataset_1536_files:/export/galaxy-central/database/files/001/dataset_1536.dat]
galaxy.jobs.runners DEBUG 2018-03-15 12:40:52,958 (132) command is: rm -rf working; mkdir -p working; cd working; /export/galaxy-central/database/job_working_directory/000/132/tool_script.sh; return_code=$?; cd '/export/galaxy-central/database/job_working_directory/000/132'; 
[ "$GALAXY_VIRTUAL_ENV" = "None" ] && GALAXY_VIRTUAL_ENV="$_GALAXY_VIRTUAL_ENV"; _galaxy_setup_environment True
python "/export/galaxy-central/database/job_working_directory/000/132/set_metadata_ztEpJC.py" "/export/galaxy-central/database/job_working_directory/000/132/registry.xml" "/export/galaxy-central/database/job_working_directory/000/132/working/galaxy.json" "/export/galaxy-central/database/job_working_directory/000/132/metadata_in_HistoryDatasetAssociation_1536_h27Ehf,/export/galaxy-central/database/job_working_directory/000/132/metadata_kwds_HistoryDatasetAssociation_1536_lsTTta,/export/galaxy-central/database/job_working_directory/000/132/metadata_out_HistoryDatasetAssociation_1536_XAPD2c,/export/galaxy-central/database/job_working_directory/000/132/metadata_results_HistoryDatasetAssociation_1536_nU1Snu,/export/galaxy-central/database/files/001/dataset_1536.dat,/export/galaxy-central/database/job_working_directory/000/132/metadata_override_HistoryDatasetAssociation_1536_FBVhwp" 5242880; sh -c "exit $return_code"
galaxy.jobs.runners.local DEBUG 2018-03-15 12:40:52,998 (132) executing job script: /export/galaxy-central/database/job_working_directory/000/132/galaxy_132.sh
galaxy.jobs DEBUG 2018-03-15 12:40:53,021 (132) Persisting job destination (destination id: local)
151.77.171.148 - - [15/Mar/2018:12:40:56 +0000] "GET /api/histories/eafb646da3b7aac5/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-15T12%3A40%3A52.000Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
[pid: 187|app: 0|req: 963/3169] 10.233.68.0 () {52 vars in 1292 bytes} [Thu Mar 15 12:40:56 2018] GET /api/histories/eafb646da3b7aac5/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-15T12%3A40%3A52.000Z&qv=False&qv=False => generated 2 bytes in 49 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 0)
151.77.171.148 - - [15/Mar/2018:12:41:00 +0000] "GET /api/histories/eafb646da3b7aac5/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-15T12%3A40%3A56.000Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
[pid: 190|app: 0|req: 2207/3170] 10.233.68.0 () {52 vars in 1292 bytes} [Thu Mar 15 12:41:00 2018] GET /api/histories/eafb646da3b7aac5/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-15T12%3A40%3A56.000Z&qv=False&qv=False => generated 2 bytes in 52 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 1)
[pid: 187|app: 0|req: 964/3171] 10.233.68.0 () {52 vars in 1292 bytes} [Thu Mar 15 12:41:04 2018] GET /api/histories/eafb646da3b7aac5/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-15T12%3A41%3A00.000Z&qv=False&qv=False => generated 2 bytes in 61 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 1)
151.77.171.148 - - [15/Mar/2018:12:41:08 +0000] "GET /api/histories/eafb646da3b7aac5/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-15T12%3A41%3A04.000Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
[pid: 190|app: 0|req: 2208/3172] 10.233.68.0 () {52 vars in 1292 bytes} [Thu Mar 15 12:41:08 2018] GET /api/histories/eafb646da3b7aac5/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-15T12%3A41%3A04.000Z&qv=False&qv=False => generated 2 bytes in 63 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 3)

@pkrog pkrog self-assigned this Mar 15, 2018
@pcm32
Copy link
Member Author

pcm32 commented Mar 20, 2018

Upload with Chrome, of a 22 kb file, taking nearly a minute:

galaxy.jobs DEBUG 2018-03-20 13:22:50,420 (137) Working directory for job is: /export/galaxy-central/database/job_working_directory/000/137
galaxy.jobs.handler DEBUG 2018-03-20 13:22:50,467 (137) Dispatching to local runner
galaxy.jobs DEBUG 2018-03-20 13:22:50,491 (137) Persisting job destination (destination id: local)
galaxy.jobs.runners DEBUG 2018-03-20 13:22:50,501 Job [137] queued (33.811 ms)
galaxy.jobs.handler INFO 2018-03-20 13:22:50,506 (137) Job dispatched
galaxy.jobs.command_factory INFO 2018-03-20 13:22:50,806 Built script [/export/galaxy-central/database/job_working_directory/000/137/tool_script.sh] for tool command [python /export/tools/data_source/upload.py /export/galaxy-central /export/galaxy-central/database/job_working_directory/000/137/registry.xml /export/galaxy-central/database/files/tmpjmD5HW 1877:/export/galaxy-central/database/job_working_directory/000/137/dataset_1877_files:/export/galaxy-central/database/files/001/dataset_1877.dat]
galaxy.jobs.runners DEBUG 2018-03-20 13:22:50,975 (137) command is: rm -rf working; mkdir -p working; cd working; /export/galaxy-central/database/job_working_directory/000/137/tool_script.sh; return_code=$?; cd '/export/galaxy-central/database/job_working_directory/000/137'; 
[ "$GALAXY_VIRTUAL_ENV" = "None" ] && GALAXY_VIRTUAL_ENV="$_GALAXY_VIRTUAL_ENV"; _galaxy_setup_environment True
python "/export/galaxy-central/database/job_working_directory/000/137/set_metadata_xtwjbT.py" "/export/galaxy-central/database/job_working_directory/000/137/registry.xml" "/export/galaxy-central/database/job_working_directory/000/137/working/galaxy.json" "/export/galaxy-central/database/job_working_directory/000/137/metadata_in_HistoryDatasetAssociation_1877_OgOsRX,/export/galaxy-central/database/job_working_directory/000/137/metadata_kwds_HistoryDatasetAssociation_1877_MN43R3,/export/galaxy-central/database/job_working_directory/000/137/metadata_out_HistoryDatasetAssociation_1877_uRMCBr,/export/galaxy-central/database/job_working_directory/000/137/metadata_results_HistoryDatasetAssociation_1877_N1ZpmV,/export/galaxy-central/database/files/001/dataset_1877.dat,/export/galaxy-central/database/job_working_directory/000/137/metadata_override_HistoryDatasetAssociation_1877_sKrdFw" 5242880; sh -c "exit $return_code"
galaxy.jobs.runners.local DEBUG 2018-03-20 13:22:51,003 (137) executing job script: /export/galaxy-central/database/job_working_directory/000/137/galaxy_137.sh
galaxy.jobs DEBUG 2018-03-20 13:22:51,015 (137) Persisting job destination (destination id: local)
193.62.194.250 - - [20/Mar/2018:13:22:54 +0000] "GET /api/histories/1cd8e2f6b131e891/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A22%3A49.000Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"
[pid: 142|app: 0|req: 9/18] 10.233.68.0 () {52 vars in 1325 bytes} [Tue Mar 20 13:22:54 2018] GET /api/histories/1cd8e2f6b131e891/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A22%3A49.000Z&qv=False&qv=False => generated 489 bytes in 153 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 0)
193.62.194.250 - - [20/Mar/2018:13:22:58 +0000] "GET /api/histories/1cd8e2f6b131e891/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A22%3A54.000Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"
[pid: 142|app: 0|req: 10/19] 10.233.68.0 () {52 vars in 1325 bytes} [Tue Mar 20 13:22:58 2018] GET /api/histories/1cd8e2f6b131e891/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A22%3A54.000Z&qv=False&qv=False => generated 2 bytes in 59 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 2)
193.62.194.250 - - [20/Mar/2018:13:23:02 +0000] "GET /api/histories/1cd8e2f6b131e891/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A22%3A58.000Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"
[pid: 142|app: 0|req: 11/20] 10.233.68.0 () {52 vars in 1325 bytes} [Tue Mar 20 13:23:02 2018] GET /api/histories/1cd8e2f6b131e891/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A22%3A58.000Z&qv=False&qv=False => generated 2 bytes in 59 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 3)
193.62.194.250 - - [20/Mar/2018:13:23:06 +0000] "GET /api/histories/1cd8e2f6b131e891/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A23%3A02.000Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"
[pid: 142|app: 0|req: 12/21] 10.233.68.0 () {52 vars in 1325 bytes} [Tue Mar 20 13:23:06 2018] GET /api/histories/1cd8e2f6b131e891/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A23%3A02.000Z&qv=False&qv=False => generated 2 bytes in 62 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 1)
193.62.194.250 - - [20/Mar/2018:13:23:10 +0000] "GET /api/histories/1cd8e2f6b131e891/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A23%3A06.000Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"
[pid: 145|app: 0|req: 10/22] 10.233.68.0 () {52 vars in 1325 bytes} [Tue Mar 20 13:23:10 2018] GET /api/histories/1cd8e2f6b131e891/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A23%3A06.000Z&qv=False&qv=False => generated 2 bytes in 53 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 2)
193.62.194.250 - - [20/Mar/2018:13:23:14 +0000] "GET /api/histories/1cd8e2f6b131e891/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A23%3A10.000Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"
[pid: 145|app: 0|req: 11/23] 10.233.68.0 () {52 vars in 1325 bytes} [Tue Mar 20 13:23:14 2018] GET /api/histories/1cd8e2f6b131e891/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A23%3A10.000Z&qv=False&qv=False => generated 2 bytes in 59 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 1)
193.62.194.250 - - [20/Mar/2018:13:23:18 +0000] "GET /api/histories/1cd8e2f6b131e891/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A23%3A14.000Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"
[pid: 145|app: 0|req: 12/24] 10.233.68.0 () {52 vars in 1325 bytes} [Tue Mar 20 13:23:18 2018] GET /api/histories/1cd8e2f6b131e891/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A23%3A14.000Z&qv=False&qv=False => generated 2 bytes in 53 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 3)
193.62.194.250 - - [20/Mar/2018:13:23:22 +0000] "GET /api/histories/1cd8e2f6b131e891/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A23%3A18.000Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"
[pid: 145|app: 0|req: 13/25] 10.233.68.0 () {52 vars in 1325 bytes} [Tue Mar 20 13:23:22 2018] GET /api/histories/1cd8e2f6b131e891/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A23%3A18.000Z&qv=False&qv=False => generated 2 bytes in 49 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 0)
193.62.194.250 - - [20/Mar/2018:13:23:26 +0000] "GET /api/histories/1cd8e2f6b131e891/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A23%3A22.000Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"
[pid: 142|app: 0|req: 13/26] 10.233.68.0 () {52 vars in 1325 bytes} [Tue Mar 20 13:23:26 2018] GET /api/histories/1cd8e2f6b131e891/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A23%3A22.000Z&qv=False&qv=False => generated 2 bytes in 57 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 0)
193.62.194.250 - - [20/Mar/2018:13:23:30 +0000] "GET /api/histories/1cd8e2f6b131e891/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A23%3A26.000Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"
[pid: 142|app: 0|req: 14/27] 10.233.68.0 () {52 vars in 1325 bytes} [Tue Mar 20 13:23:30 2018] GET /api/histories/1cd8e2f6b131e891/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A23%3A26.000Z&qv=False&qv=False => generated 2 bytes in 52 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 2)
193.62.194.250 - - [20/Mar/2018:13:23:34 +0000] "GET /api/histories/1cd8e2f6b131e891/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A23%3A30.000Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"
[pid: 142|app: 0|req: 15/28] 10.233.68.0 () {52 vars in 1325 bytes} [Tue Mar 20 13:23:34 2018] GET /api/histories/1cd8e2f6b131e891/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A23%3A30.000Z&qv=False&qv=False => generated 2 bytes in 47 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 3)
193.62.194.250 - - [20/Mar/2018:13:23:38 +0000] "GET /api/histories/1cd8e2f6b131e891/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A23%3A34.000Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"
[pid: 142|app: 0|req: 16/29] 10.233.68.0 () {52 vars in 1325 bytes} [Tue Mar 20 13:23:38 2018] GET /api/histories/1cd8e2f6b131e891/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A23%3A34.000Z&qv=False&qv=False => generated 2 bytes in 72 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 1)
193.62.194.250 - - [20/Mar/2018:13:23:42 +0000] "GET /api/histories/1cd8e2f6b131e891/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A23%3A38.000Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.162 Safari/537.36"
[pid: 145|app: 0|req: 14/30] 10.233.68.0 () {52 vars in 1325 bytes} [Tue Mar 20 13:23:42 2018] GET /api/histories/1cd8e2f6b131e891/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A23%3A38.000Z&qv=False&qv=False => generated 2 bytes in 55 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 2)
galaxy.jobs.runners.local DEBUG 2018-03-20 13:23:45,171 execution finished: /export/galaxy-central/database/job_working_directory/000/137/galaxy_137.sh
galaxy.model.metadata DEBUG 2018-03-20 13:23:45,288 loading metadata from file for: HistoryDatasetAssociation 1877
galaxy.jobs INFO 2018-03-20 13:23:45,476 Collecting metrics for Job 137
galaxy.jobs DEBUG 2018-03-20 13:23:45,520 job 137 ended (finish() executed in (348.884 ms))

@pcm32
Copy link
Member Author

pcm32 commented Mar 20, 2018

Upload of the same file, with Safari, also taking around a minute:

galaxy.jobs DEBUG 2018-03-20 13:30:55,319 (138) Working directory for job is: /export/galaxy-central/database/job_working_directory/000/138
galaxy.jobs.handler DEBUG 2018-03-20 13:30:55,333 (138) Dispatching to local runner
galaxy.jobs DEBUG 2018-03-20 13:30:55,365 (138) Persisting job destination (destination id: local)
galaxy.jobs.runners DEBUG 2018-03-20 13:30:55,374 Job [138] queued (40.797 ms)
galaxy.jobs.handler INFO 2018-03-20 13:30:55,380 (138) Job dispatched
galaxy.jobs.command_factory INFO 2018-03-20 13:30:55,565 Built script [/export/galaxy-central/database/job_working_directory/000/138/tool_script.sh] for tool command [python /export/tools/data_source/upload.py /export/galaxy-central /export/galaxy-central/database/job_working_directory/000/138/registry.xml /export/galaxy-central/database/files/tmpNWGKBX 1878:/export/galaxy-central/database/job_working_directory/000/138/dataset_1878_files:/export/galaxy-central/database/files/001/dataset_1878.dat]
galaxy.jobs.runners DEBUG 2018-03-20 13:30:55,728 (138) command is: rm -rf working; mkdir -p working; cd working; /export/galaxy-central/database/job_working_directory/000/138/tool_script.sh; return_code=$?; cd '/export/galaxy-central/database/job_working_directory/000/138'; 
[ "$GALAXY_VIRTUAL_ENV" = "None" ] && GALAXY_VIRTUAL_ENV="$_GALAXY_VIRTUAL_ENV"; _galaxy_setup_environment True
python "/export/galaxy-central/database/job_working_directory/000/138/set_metadata_2ln2zp.py" "/export/galaxy-central/database/job_working_directory/000/138/registry.xml" "/export/galaxy-central/database/job_working_directory/000/138/working/galaxy.json" "/export/galaxy-central/database/job_working_directory/000/138/metadata_in_HistoryDatasetAssociation_1878_m7ksVR,/export/galaxy-central/database/job_working_directory/000/138/metadata_kwds_HistoryDatasetAssociation_1878_tvK48V,/export/galaxy-central/database/job_working_directory/000/138/metadata_out_HistoryDatasetAssociation_1878_pyFa5f,/export/galaxy-central/database/job_working_directory/000/138/metadata_results_HistoryDatasetAssociation_1878_EsLLnT,/export/galaxy-central/database/files/001/dataset_1878.dat,/export/galaxy-central/database/job_working_directory/000/138/metadata_override_HistoryDatasetAssociation_1878_pr7zql" 5242880; sh -c "exit $return_code"
galaxy.jobs.runners.local DEBUG 2018-03-20 13:30:55,756 (138) executing job script: /export/galaxy-central/database/job_working_directory/000/138/galaxy_138.sh
galaxy.jobs DEBUG 2018-03-20 13:30:55,768 (138) Persisting job destination (destination id: local)
193.62.194.250 - - [20/Mar/2018:13:30:59 +0000] "GET /api/histories/9752b387803d3e1e/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A30%3A55.000Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
[pid: 142|app: 0|req: 25/50] 10.233.68.0 () {52 vars in 1288 bytes} [Tue Mar 20 13:30:59 2018] GET /api/histories/9752b387803d3e1e/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A30%3A55.000Z&qv=False&qv=False => generated 2 bytes in 55 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 0)
193.62.194.250 - - [20/Mar/2018:13:31:03 +0000] "GET /api/histories/9752b387803d3e1e/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A30%3A59.000Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
[pid: 142|app: 0|req: 26/51] 10.233.68.0 () {52 vars in 1288 bytes} [Tue Mar 20 13:31:03 2018] GET /api/histories/9752b387803d3e1e/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A30%3A59.000Z&qv=False&qv=False => generated 2 bytes in 56 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 2)
193.62.194.250 - - [20/Mar/2018:13:31:07 +0000] "GET /api/histories/9752b387803d3e1e/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A31%3A03.000Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
[pid: 142|app: 0|req: 27/52] 10.233.68.0 () {52 vars in 1288 bytes} [Tue Mar 20 13:31:07 2018] GET /api/histories/9752b387803d3e1e/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A31%3A03.000Z&qv=False&qv=False => generated 2 bytes in 47 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 3)
193.62.194.250 - - [20/Mar/2018:13:31:11 +0000] "GET /api/histories/9752b387803d3e1e/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A31%3A07.000Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
[pid: 145|app: 0|req: 26/53] 10.233.68.0 () {52 vars in 1288 bytes} [Tue Mar 20 13:31:11 2018] GET /api/histories/9752b387803d3e1e/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A31%3A07.000Z&qv=False&qv=False => generated 2 bytes in 58 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 2)
193.62.194.250 - - [20/Mar/2018:13:31:15 +0000] "GET /api/histories/9752b387803d3e1e/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A31%3A11.000Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
[pid: 142|app: 0|req: 28/54] 10.233.68.0 () {52 vars in 1288 bytes} [Tue Mar 20 13:31:15 2018] GET /api/histories/9752b387803d3e1e/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A31%3A11.000Z&qv=False&qv=False => generated 2 bytes in 52 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 1)
193.62.194.250 - - [20/Mar/2018:13:31:19 +0000] "GET /api/histories/9752b387803d3e1e/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A31%3A15.000Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
[pid: 145|app: 0|req: 27/55] 10.233.68.0 () {52 vars in 1288 bytes} [Tue Mar 20 13:31:19 2018] GET /api/histories/9752b387803d3e1e/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A31%3A15.000Z&qv=False&qv=False => generated 2 bytes in 57 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 1)
193.62.194.250 - - [20/Mar/2018:13:31:23 +0000] "GET /api/histories/9752b387803d3e1e/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A31%3A19.000Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
[pid: 145|app: 0|req: 28/56] 10.233.68.0 () {52 vars in 1288 bytes} [Tue Mar 20 13:31:23 2018] GET /api/histories/9752b387803d3e1e/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A31%3A19.000Z&qv=False&qv=False => generated 2 bytes in 51 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 3)
193.62.194.250 - - [20/Mar/2018:13:31:27 +0000] "GET /api/histories/9752b387803d3e1e/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A31%3A23.000Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
[pid: 145|app: 0|req: 29/57] 10.233.68.0 () {52 vars in 1288 bytes} [Tue Mar 20 13:31:27 2018] GET /api/histories/9752b387803d3e1e/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A31%3A23.000Z&qv=False&qv=False => generated 2 bytes in 55 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 0)
193.62.194.250 - - [20/Mar/2018:13:31:31 +0000] "GET /api/histories/9752b387803d3e1e/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A31%3A27.000Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
[pid: 142|app: 0|req: 29/58] 10.233.68.0 () {52 vars in 1288 bytes} [Tue Mar 20 13:31:31 2018] GET /api/histories/9752b387803d3e1e/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A31%3A27.000Z&qv=False&qv=False => generated 2 bytes in 58 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 0)
193.62.194.250 - - [20/Mar/2018:13:31:35 +0000] "GET /api/histories/9752b387803d3e1e/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A31%3A31.000Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
[pid: 142|app: 0|req: 30/59] 10.233.68.0 () {52 vars in 1288 bytes} [Tue Mar 20 13:31:35 2018] GET /api/histories/9752b387803d3e1e/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A31%3A31.000Z&qv=False&qv=False => generated 2 bytes in 62 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 2)
193.62.194.250 - - [20/Mar/2018:13:31:39 +0000] "GET /api/histories/9752b387803d3e1e/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A31%3A35.000Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
[pid: 142|app: 0|req: 31/60] 10.233.68.0 () {52 vars in 1288 bytes} [Tue Mar 20 13:31:39 2018] GET /api/histories/9752b387803d3e1e/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A31%3A35.000Z&qv=False&qv=False => generated 2 bytes in 51 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 3)
193.62.194.250 - - [20/Mar/2018:13:31:43 +0000] "GET /api/histories/9752b387803d3e1e/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A31%3A39.000Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
[pid: 145|app: 0|req: 30/61] 10.233.68.0 () {52 vars in 1288 bytes} [Tue Mar 20 13:31:43 2018] GET /api/histories/9752b387803d3e1e/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A31%3A39.000Z&qv=False&qv=False => generated 2 bytes in 131 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 2)
193.62.194.250 - - [20/Mar/2018:13:31:48 +0000] "GET /api/histories/9752b387803d3e1e/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A31%3A43.000Z&qv=False&qv=False HTTP/1.1" 200 - "https://publicdev.phenomenal-h2020.eu/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6"
[pid: 142|app: 0|req: 32/62] 10.233.68.0 () {52 vars in 1288 bytes} [Tue Mar 20 13:31:48 2018] GET /api/histories/9752b387803d3e1e/contents?order=hid&v=dev&q=update_time-ge&q=deleted&q=purged&qv=2018-03-20T13%3A31%3A43.000Z&qv=False&qv=False => generated 2 bytes in 58 msecs (HTTP/1.1 200) 3 headers in 124 bytes (1 switches on core 1)
galaxy.jobs.runners.local DEBUG 2018-03-20 13:31:49,525 execution finished: /export/galaxy-central/database/job_working_directory/000/138/galaxy_138.sh
galaxy.model.metadata DEBUG 2018-03-20 13:31:49,657 loading metadata from file for: HistoryDatasetAssociation 1878
galaxy.jobs INFO 2018-03-20 13:31:49,817 Collecting metrics for Job 138
galaxy.jobs DEBUG 2018-03-20 13:31:49,848 job 138 ended (finish() executed in (323.458 ms))

@pkrog
Copy link
Member

pkrog commented Apr 9, 2018

Hi @ilveroluca , @pcm32 ,
I've been trying to build our Galaxy container for debugging this issue. I've been using the branch develop of container-galaxy-k8s-runtime for this, running the script simplified_galaxy_stable_container_creation.sh with the following command:

 CONTAINER_USER=pierrickroger PUSH=no sh simplified_galaxy_stable_container_creation.sh

The PUSH option comes from the fact I use the branch refact/container_creation_script_cleaning I've created for cleaning a bit the bash script. PUSH=no avoid pushing anything.

However I keep having the following error message while building galaxy-init container:

ASK [galaxyprojectdotorg.galaxyextras : Install proxy dependencies.] **********                                                                                                                                                                                                   fatal: [localhost]: FAILED! => {"changed": true, "cmd": "cd /export/galaxy-central/lib/galaxy/web/proxy/js && npm install", "delta": "0:01:11.522230", "end": "2018-04-05 16:34:44.649715", "msg": "non-zero return code", "rc": 1, "start": "2018-04-05 16:33:33.127485", "stderr": "
npm http GET https://registry.npmjs.org/commander
npm http GET https://registry.npmjs.org/sqlite3/3.1.8
npm http GET https://registry.npmjs.org/http-proxy/1.6.0
npm http GET https://registry.npmjs.org/eventemitter3/0.1.6
npm http GET https://registry.npmjs.org/http-proxy/1.6.0
npm http GET https://registry.npmjs.org/commander
npm http GET https://registry.npmjs.org/sqlite3/3.1.8
npm http GET https://registry.npmjs.org/eventemitter3/0.1.6
npm http GET https://registry.npmjs.org/sqlite3/3.1.8
npm http GET https://registry.npmjs.org/eventemitter3/0.1.6
npm http GET https://registry.npmjs.org/commander
npm http GET https://registry.npmjs.org/http-proxy/1.6.0
npm ERR! Error: CERT_UNTRUSTED
npm ERR!     at SecurePair.<anonymous> (tls.js:1370:32)
npm ERR!     at SecurePair.EventEmitter.emit (events.js:92:17)
npm ERR!     at SecurePair.maybeInitFinished (tls.js:982:10)
npm ERR!     at CleartextStream.read [as _read] (tls.js:469:13)
npm ERR!     at CleartextStream.Readable.read (_stream_readable.js:320:10)
npm ERR!     at EncryptedStream.write [as _write] (tls.js:366:25)
npm ERR!     at doWrite (_stream_writable.js:223:10)
npm ERR!     at writeOrBuffer (_stream_writable.js:213:5)
npm ERR!     at EncryptedStream.Writable.write (_stream_writable.js:180:11)
npm ERR!     at write (_stream_readable.js:583:24)
npm ERR!     at flow (_stream_readable.js:592:7)
npm ERR!     at Socket.pipeOnReadable (_stream_readable.js:624:5)
npm ERR! If you need help, you may report this log at:
npm ERR!     <http://github.com/isaacs/npm/issues>
npm ERR! or email it to:
npm ERR!     <[email protected]>

npm ERR! System Linux 4.9.87-linuxkit-aufs
npm ERR! command \"/usr/bin/nodejs\" \"/usr/bin/npm\" \"install\"
npm ERR! cwd /galaxy-export/galaxy-central/lib/galaxy/web/proxy/js
npm ERR! node -v v0.10.25
npm ERR! npm -v 1.3.10
npm ERR! 
npm ERR! Additional logging details can be found in:
npm ERR!     /galaxy-export/galaxy-central/lib/galaxy/web/proxy/js/npm-debug.log
npm ERR! not ok code 0",
	"stderr_lines": ["npm http GET https://registry.npmjs.org/commander", "npm http GET https://registry.npmjs.org/sqlite3/3.1.8", "npm http GET https://registry.npmjs.org/http-proxy/1.6.0", "npm http GET https://registry.npmjs.org/eventemitter3/0.1.6", "npm http GET https://registry.npmjs.org/http-proxy/1.6.0", "npm http GET https://registry.npmjs.org/commander", "npm http GET https://registry.npmjs.org/sqlite3/3.1.8", "npm http GET https://registry.npmjs.org/eventemitter3/0.1.6", "npm http GET https://registry.npmjs.org/sqlite3/3.1.8", "npm http GET https://registry.npmjs.org/eventemitter3/0.1.6", "npm http GET https://registry.npmjs.org/commander", "npm http GET https://registry.npmjs.org/http-proxy/1.6.0", "npm ERR! Error: CERT_UNTRUSTED", "npm ERR!     at SecurePair.<anonymous> (tls.js:1370:32)", "npm ERR!     at SecurePair.EventEmitter.emit (events.js:92:17)", "npm ERR!     at SecurePair.maybeInitFinished (tls.js:982:10)", "npm ERR!     at CleartextStream.read [as _read] (tls.js:469:13)", "npm ERR!     at CleartextStream.Readable.read (_stream_readable.js:320:10)", "npm ERR!     at EncryptedStream.write [as _write] (tls.js:366:25)", "npm ERR!     at doWrite (_stream_writable.js:223:10)", "npm ERR!     at writeOrBuffer (_stream_writable.js:213:5)", "npm ERR!     at EncryptedStream.Writable.write (_stream_writable.js:180:11)", "npm ERR!     at write (_stream_readable.js:583:24)", "npm ERR!     at flow (_stream_readable.js:592:7)", "npm ERR!     at Socket.pipeOnReadable (_stream_readable.js:624:5)", "npm ERR! If you need help, you may report this log at:", "npm ERR!     <http://github.com/isaacs/npm/issues>", "npm ERR! or email it to:", "npm ERR!     <[email protected]>", "", "npm ERR! System Linux 4.9.87-linuxkit-aufs", "npm ERR! command \"/usr/bin/nodejs\" \"/usr/bin/npm\" \"install\"", "npm ERR! cwd /galaxy-export/galaxy-central/lib/galaxy/web/proxy/js", "npm ERR! node -v v0.10.25", "npm ERR! npm -v 1.3.10", "npm ERR! ", "npm ERR! Additional logging details can be found in:", "npm ERR!     /galaxy-export/galaxy-central/lib/galaxy/web/proxy/js/npm-debug.log", "npm ERR! not ok code 0"], "stdout": "", "stdout_lines": []}                                                                                                                                                                                                                   to retry, use: --limit @/ansible/provision.retry

The galaxy branch is fixed inside the script simplified_galaxy_stable_container_creation.sh, thus it has been already tested and used by us. So I don't get it... Any idea of what would be happening?

@sneumann
Copy link
Member

sneumann commented Jun 6, 2018 •
edited
Loading

On a just deployed 17.09 based on the galaxy-stable-1.5.5 I observed that also the import
of a smallish mzML file via http URL download feels very slow.
Fetching https://msbi.ipb-halle.de/~cruttkie/data/PE_PG_AutoMS_2-B_3_03_18583.mzML.zip
takes around 120 seconds until green, the same file on usegalaxy.eu takes around 30 + 15 seconds (grey + yellow until green). Yours, Steffen

@sneumann
Copy link
Member

I also observed on a just deployed 17.09 based on the galaxy-stable-1.5.5 that the FTP upload was really slow. I had ~400MB/s for the aspera connection into /tmp on the master, and then ~4MB/s from the master via SFTP into the K8S cluster. Could that be the overlay network or something ? Yours, Steffen

@sneumann
Copy link
Member

Around 80 sec for PE_PG_AutoMS_2-B_3_03_18583.mzML.zip on a just deployed 18.01.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@pkrog pkrog

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@sneumann @pcm32 @pkrog