realloc failed during char_array_push

[kylebarron]

I get the error

> ./interpolate polyline street.db < road_network.polylines
realloc failed during char_array_push

I'm using the road_network.polylines file linked to from the Polylines README. This error occurred 15 hours after starting the command. I'm using the Production branch of the interpolation repository.

[missinglink]

Sorry for the very late reply, I have never seen this error myself, I suspect it's due to either insufficient RAM or insufficient disk space.

[kylebarron]

That may have been the case. I haven't worked on this for a couple months so I'll close this.

[nilsnolde]

This happens to me too.. After 5 hours approx the exact same command exits with the same error. Data came from your S3, @missinglink. But I guess it's not a data corruption problem.

140 GB free on / and 20 GB RAM free. I assume that should be enough space in either domain.

If anyone has any idea, I'd appreciate a hint on this one.

[missinglink]

reopening issue, I will run a planet build and see if I can reproduce the error.

[missinglink]

I got the same error when running a full planet import yesterday.

The issue is due to libpostal hanging, which results in the nodejs process memory filling up and eventually crashing.

Steps to reproduce:

$ node
> var postal = require('node-postal');
undefined

> postal.expand.expand_address('улица 40 лет Победы');

... hangs indefinately

I'll file an upstream bug and will also look at implementing some sort of timeout if possible on our end to abort long-running calls to libpostal.

[missinglink]

Issue opened openvenues/libpostal#378

[missinglink]

There doesn't seem to be an easy way to abort long-running calls to the C code, the libpostal API uses synchronous function calls, if it were async then it might be easier to do.

The bug itself is quite nasty in nodejs because it blocks the event loop, there is a setInterval timer set up prior to this point in the code and it doesn't even fire, meaning that the program is essentially stalled waiting for libpostal to return.

It's probably possible to write an abort logic into the node binding but might require reworking the function interface, it would require getting into the C binding code which I don't have any experience with, nor the time for learning it right now.

Let's wait a week or so to see what Al says, it might be a simple enough thing for him to patch it and fix it in libpostal core.

[orangejulius]

Would it be worth trying the libpostal-service PR in #146? It has been tested and works on smaller builds, and even if there was an issue in the underlying libpostal service, the network calls between services would be an ideal way to isolate that problem.

[missinglink]

hmm.. interestingly the go-whosonfirst-libpostal service seems unaffected:

$ curl -s -X GET 'http://localhost:8080/expand?address=улица+40+лет+Победы' | jq .
[
  "улица 40 лет победы",
  "ulitsa 40 let pobedy"
]

maybe because the code is quite old now and built off a prior version.

[edit] yes, it's running this commit openvenues/libpostal@5f0e394

[jpan127]

Would like to report having the same issue, all up to date production branches, January 2017 planet polylines file.

[missinglink]

{comment deleted to avoid confusion}

[missinglink]

today I published a development docker image pelias/interpolation:patch, you can try using that version instead of pelias/interpolation:latest.

I will be doing some more testing over the next few days, I'm currently running a full planet street.db build using the patched libpostal and it appears to be going fine.

cc/ @kylebarron @nilsnolde @jpan127

[edit] the build completed successfully, please consider this as a fix until we can resolve the issue in libpostal or we merge that fix to master/latest.

[missinglink]

the patch I applied is not the same as the one posted above, here is the code I'm using:

# clone libpostal repo
cd /code
git clone https://github.com/openvenues/libpostal
cd /code/libpostal
git log -1

# generate libpostal patch file
# https://github.com/openvenues/libpostal/issues/378
cat > /tmp/libpostal.patch <<-EOF
diff --git a/src/transliterate.c b/src/transliterate.c
index 7f5fff7..83e502b 100644
--- a/src/transliterate.c
+++ b/src/transliterate.c
@@ -938,7 +938,9 @@ char *transliterate(char *trans_name, char *str, size_t len) {
                     if (context_no_match && !prev_state.empty_transition && prev_state.phrase_len > 0) {
                         log_debug("Previous phrase stays as is %.*s\n", (int)prev_state.phrase_len, str+prev_state.phrase_start);
                         char_array_cat_len(new_str, str + prev_state.phrase_start, prev_state.phrase_len);
-                        state = start_state;
+                        if (match_candidate_state.state != TRANS_STATE_PARTIAL_MATCH) {
+                            state = start_state;
+                        }
                     }
                     
                     if (state.state == TRANS_STATE_BEGIN && !prev_state.empty_transition) {
EOF

# patch libpostal
git apply /tmp/libpostal.patch
git log -1

[stephenlacy]

@missinglink You saved my endless failed imports, thanks!

[missinglink]

I have patched our docker images with the C patch above and released them under the latest tag on dockerhub.

Specifically, the patched versions are:

pelias/libpostal_baseimage:latest
pelias/libpostal_baseimage:production-2018-09-18-61f072a16e3b3d74454e9371b448b4da0e35bc58

pelias/interpolation:latest
pelias/interpolation:production-2018-09-18-c27102f29f0db4ee00c473c5b3c4af3262c1fd1e

for more info:

• https://hub.docker.com/r/pelias/interpolation/tags/
• https://hub.docker.com/r/pelias/libpostal_baseimage/

I'm going to mark this issue as resolved.

Please pull from dockerhub to update the tags above and re-test.

If you encounter the error again, please post a hash of the docker image you are using and the error message and I will re-open the issue.

[missinglink]

also worth noting the pelias/interpolation:patch image is now deprecated, please replace it with pelias/interpolation:latest in your configs to ensure you continue receiving updates.