Out of Memory error

[bpmckinnon]

Hi, I just started using the software and I'm getting an issue. The issue repeats every few hours, and I'm running it on many smallish file (15000 uploads for 50GB). The system has only 2GB of ram, but if I restart the application it continues uploading.

So far I have:

dmesg
[158309.634505] Out of memory: Kill process 15899 (mdadm) score 870 or sacrifice child
[158309.634510] Killed process 15899 (mdadm) total-vm:1818512kB, anon-rss:1801668kB, file-rss:0kB

I've restarted the process with strace and I'll let you know if I find anything.
I have 2 strace stacks. Let me know if you want the full files (I'm not 100% how much data is in there, so I don't know that I just post the entire file).
The first is:
write(16, "00000036", 8) = 8
write(16, "16493\tupload_part\t9142\t354\t40024"..., 36) = 36
write(16, "{"mtime":1279327706,"part_final_"..., 354) = 354
write(16, "RIFF\26\270b\2AVI LISTR\3\0\0hdrlavih8\0\0\0"..., 40024094) = 40024094
select(24, [3 4 5 7 9 11 13 15 17 19], NULL, NULL, NULL) = 1 (in [9])
read(9, "00000026", 8) = 8
read(9, "16498\tresponse\t9141\t238\t0\n", 26) = 26
read(9, "{"console_out":"Created an uploa"..., 238) = 238
write(1, "PID 16498 Created an upload_id K"..., 124) = 124
mmap(NULL, 268439552, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
brk(0x15c94000) = 0x5c94000
mmap(NULL, 268570624, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap(NULL, 268439552, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
The second is:
read(32, "", 8192) = 0
brk(0x40f0000) = 0x40f0000
brk(0x41f0000) = 0x41f0000
write(10, "00000036", 8) = 8
write(10, "16201\tupload_part\t8582\t296\t62247"..., 36) = 36
write(10, "{"relfilename":"Pre-2012 incl. e"..., 296) = 296
write(10, "RIFF>\323\265\3AVI LISTR\3\0\0hdrlavih8\0\0\0"..., 62247750) = 62247750
select(24, [3 4 5 7 9 11 13 15 17 19], NULL, NULL, NULL) = 1 (in [19])
read(19, "00000026", 8) = 8
read(19, "16211\tresponse\t8576\t118\t0\n", 26) = 26
read(19, "{"console_out":"Uploaded part fo"..., 118) = 118
write(1, "PID 16211 Uploaded part for Pre-"..., 96) = 96
munmap(0x7fb517273000, 268439552) = 0
mmap(NULL, 268439552, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
brk(0x1414e000) = 0x41f0000
mmap(NULL, 268570624, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap(NULL, 268439552, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
brk(0x1414e000) = 0x41f0000
mmap(NULL, 268570624, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 ENOMEM (Cannot allocate memory)
mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7fb51f274000
Let me know if I there is a way to provide more precise error data.

Originally posted by @bpmckinnon in #105 (comment)

[bpmckinnon]

Just for a test I've dropped the concurrency down to 1, to help confirm if this is an issue with an old system thats just short on ram.

[vsespb]

need to know

• perl version (perl -V)

• full command line and config file (exclude secret keys and bucket names)

• how much free RAM before running upload

• process size in the beginning and how it increases till crash (and how free ram decreases).

basically, it's all about proof that something wrong is happening here.
I don't see yet how it differs from "there was no enough free RAM".

[bpmckinnon]

So this does appear to be a problem with the amount of ram. When I run only a single upload process it appears to be stable. Sorry for the bother.