IPv4 to INT LUA error on large files

[jeromekleinen]

Hello,

First of all thank you for implementing my feature request of exposing url_type to LUA!

I am trying to use it today but I am experiencing issues processing large files.

To reproduce:

yes "127.0.0.1" | head -n 500000 > 500k
cat 500k | faup -m ipv4toint.lua -q

On my system I am getting "stack overflow" and "error in error handling" after roughly 400k lines. I imagine this is related to how variables are processed in LUA. Perhaps the local keyword in variable triggers LUA to create a new variable for each line?

I would appreciate your help! I will keep searching in the meantime as well and will update my issue if I find a solution.

Jerome

[stricaud]

I confirm I could catch lots of evil stuff. Not ready yet then, in progress. I will close this ticket with a few more commits :) Thanks!

± valgrind faup -q -m ipv4toint.lua 89 > out
==8508== Memcheck, a memory error detector
==8508== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==8508== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==8508== Command: faup -q -m ipv4toint.lua 89
==8508== 
==8508== Invalid write of size 8
==8508==    at 0x4021D0: main (faup.c:283)
==8508==  Address 0x9894de0 is 0 bytes inside a block of size 1 alloc'd
==8508==    at 0x4C2DAE8: malloc (vg_replace_malloc.c:298)
==8508==    by 0x4C2FD4F: realloc (vg_replace_malloc.c:785)
==8508==    by 0x4021AC: main (faup.c:282)
==8508== 
==8508== Syscall param openat(filename) contains uninitialised byte(s)
==8508==    at 0x602AF4E: open (in /usr/lib64/libc-2.27.so)
==8508==    by 0x5FBC281: _IO_file_open (in /usr/lib64/libc-2.27.so)
==8508==    by 0x5FBC42C: _IO_file_fopen@@GLIBC_2.2.5 (in /usr/lib64/libc-2.27.so)
==8508==    by 0x5FB0658: __fopen_internal (in /usr/lib64/libc-2.27.so)
==8508==    by 0x4E52B92: faup_modules_load_from_arg (modules.c:133)
==8508==    by 0x4E52FE3: faup_modules_new (modules.c:49)
==8508==    by 0x4E56C42: faup_init (faup.c:41)
==8508==    by 0x4024C4: main (faup.c:369)
==8508== 
==8508== Use of uninitialised value of size 8
==8508==    at 0x4C30C06: strlen (vg_replace_strmem.c:458)
==8508==    by 0x4E52BEC: faup_modules_load_from_arg (modules.c:147)
==8508==    by 0x4E52FE3: faup_modules_new (modules.c:49)
==8508==    by 0x4E56C42: faup_init (faup.c:41)
==8508==    by 0x4024C4: main (faup.c:369)
==8508== 
==8508== Use of uninitialised value of size 8
==8508==    at 0x4C30C14: strlen (vg_replace_strmem.c:458)
==8508==    by 0x4E52BEC: faup_modules_load_from_arg (modules.c:147)
==8508==    by 0x4E52FE3: faup_modules_new (modules.c:49)
==8508==    by 0x4E56C42: faup_init (faup.c:41)
==8508==    by 0x4024C4: main (faup.c:369)
==8508== 
==8508== Conditional jump or move depends on uninitialised value(s)
==8508==    at 0x5F8FFC0: vfprintf (in /usr/lib64/libc-2.27.so)
==8508==    by 0x5FB8A48: vasprintf (in /usr/lib64/libc-2.27.so)
==8508==    by 0x5F988B3: asprintf (in /usr/lib64/libc-2.27.so)
==8508==    by 0x4E52C17: faup_modules_load_from_arg (modules.c:148)
==8508==    by 0x4E52FE3: faup_modules_new (modules.c:49)
==8508==    by 0x4E56C42: faup_init (faup.c:41)
==8508==    by 0x4024C4: main (faup.c:369)
==8508== 
==8508== Use of uninitialised value of size 8
==8508==    at 0x4C30C06: strlen (vg_replace_strmem.c:458)
==8508==    by 0x5F91916: vfprintf (in /usr/lib64/libc-2.27.so)
==8508==    by 0x5FB8A48: vasprintf (in /usr/lib64/libc-2.27.so)
==8508==    by 0x5F988B3: asprintf (in /usr/lib64/libc-2.27.so)
==8508==    by 0x4E52C17: faup_modules_load_from_arg (modules.c:148)
==8508==    by 0x4E52FE3: faup_modules_new (modules.c:49)
==8508==    by 0x4E56C42: faup_init (faup.c:41)
==8508==    by 0x4024C4: main (faup.c:369)
==8508== 
==8508== Use of uninitialised value of size 8
==8508==    at 0x4C30C14: strlen (vg_replace_strmem.c:458)
==8508==    by 0x5F91916: vfprintf (in /usr/lib64/libc-2.27.so)
==8508==    by 0x5FB8A48: vasprintf (in /usr/lib64/libc-2.27.so)
==8508==    by 0x5F988B3: asprintf (in /usr/lib64/libc-2.27.so)
==8508==    by 0x4E52C17: faup_modules_load_from_arg (modules.c:148)
==8508==    by 0x4E52FE3: faup_modules_new (modules.c:49)
==8508==    by 0x4E56C42: faup_init (faup.c:41)
==8508==    by 0x4024C4: main (faup.c:369)
==8508== 
==8508== Use of uninitialised value of size 8
==8508==    at 0x5FBDCD0: _IO_default_xsputn (in /usr/lib64/libc-2.27.so)
==8508==    by 0x5F9004D: vfprintf (in /usr/lib64/libc-2.27.so)
==8508==    by 0x5FB8A48: vasprintf (in /usr/lib64/libc-2.27.so)
==8508==    by 0x5F988B3: asprintf (in /usr/lib64/libc-2.27.so)
==8508==    by 0x4E52C17: faup_modules_load_from_arg (modules.c:148)
==8508==    by 0x4E52FE3: faup_modules_new (modules.c:49)
==8508==    by 0x4E56C42: faup_init (faup.c:41)
==8508==    by 0x4024C4: main (faup.c:369)
==8508== 
==8508== Use of uninitialised value of size 8
==8508==    at 0x5FBDCDF: _IO_default_xsputn (in /usr/lib64/libc-2.27.so)
==8508==    by 0x5F9004D: vfprintf (in /usr/lib64/libc-2.27.so)
==8508==    by 0x5FB8A48: vasprintf (in /usr/lib64/libc-2.27.so)
==8508==    by 0x5F988B3: asprintf (in /usr/lib64/libc-2.27.so)
==8508==    by 0x4E52C17: faup_modules_load_from_arg (modules.c:148)
==8508==    by 0x4E52FE3: faup_modules_new (modules.c:49)
==8508==    by 0x4E56C42: faup_init (faup.c:41)
==8508==    by 0x4024C4: main (faup.c:369)
==8508== 

[stricaud]

Here's more. I especially like the "error in error handling". Big fan of this!

*** Error(faup_modules_url_output)[in script:/usr/local/share/faup/modules_available/ipv4toint.lua]: error in error handling
*** Error(faup_modules_url_output)[in script:/usr/local/share/faup/modules_available/ipv4toint.lua]: error in error handling
==8508== 
==8508== Process terminating with default action of signal 11 (SIGSEGV): dumping core
==8508==  Access not within mapped region at address 0x2DF8E000
==8508==    at 0x5082A7F: lua_pushstring (in /usr/lib64/liblua-5.3.so)
==8508==    by 0x4E531E5: _faup_add_keyval_dict (modules.c:324)
==8508==    by 0x4E532D1: _faup_add_feature (modules.c:345)
==8508==    by 0x4E533A2: faup_modules_url_output (modules.c:376)
==8508==    by 0x4E584F0: faup_output (output.c:384)
==8508==    by 0x402A55: run_from_stream (faup.c:161)
==8508==    by 0x4024FD: main (faup.c:403)
==8508==  If you believe this happened as a result of a stack
==8508==  overflow in your program's main thread (unlikely but
==8508==  possible), you can try to increase the size of the
==8508==  main thread stack using the --main-stacksize= flag.
==8508==  The main thread stack size used in this run was 8388608.
==8508== 
==8508== HEAP SUMMARY:
==8508==     in use at exit: 472,660,278 bytes in 1,034,057 blocks
==8508==   total heap usage: 5,714,901 allocs, 4,680,844 frees, 2,332,788,021 bytes allocated
==8508== 
==8508== LEAK SUMMARY:
==8508==    definitely lost: 99 bytes in 3 blocks
==8508==    indirectly lost: 0 bytes in 0 blocks
==8508==      possibly lost: 471,948,007 bytes in 1,004,454 blocks
==8508==    still reachable: 712,172 bytes in 29,600 blocks
==8508==         suppressed: 0 bytes in 0 blocks
==8508== Rerun with --leak-check=full to see details of leaked memory
==8508==