java.lang.NegativeArraySizeException after loading freebase hdt

[lruppe]

Hi
NegativeArraySizeException gets thrown in org.rdfhdt.hdt.util.io.IOUtil.readBuffer while trying to load freebase hdt from http://www.rdfhdt.org/datasets/.

Any ideas?

All the best,
Leon

[v4ss4llo]

Are you using hdt-java to use HDT with Fuseki? I think a better option is to look for alternative ways to query your HDT file, without using hdt-java. But I don't know any other alternative... or I'd use it myself. If you find any, let me know.

[v4ss4llo]

@lruppe did you try to increase the Java MV heap? Freebase seems rather large, so you probably need like 4-6GB of heap.

[v4ss4llo]

oh BTW, if you want to query Freebase you'll also need the .hdt.index file. I hope you have that file, or that you have a machine powerful enough to generate it. If you don't have the file and also don't have a powerful server, you're definitely out of luck (the rdf2hdt tool is really inefficient and requires terabytes of RAM for a graph that large).

[lruppe]

sad sad, worked out nicely for me till now for subsets of the knowdledge graphs.

Yes, I increased it to 50gb, running it on a cluster - still fails. Allthough the stuff I try to do worked with the wikidata hdt file, which has a similar size and a subset of the dbpedia graph.

btw: if you want to use hdt java with large graphs, I needed to add another jar file "JLargeArrays" to the classpath. (https://github.com/IcmVis/JLargeArrays)

Probably going to try out sparql now.

[v4ss4llo]

ah good to know... of course the documentation doesn't mention that. Did you use JLargeArrays for Fuseki? Or for your own Java program? Or for what else? I was able to load Fuseki+HDT and it didn't require me to use that library (as far as I can tell).

[wouterbeek]

@lruppe I don't know anything about Fuseki, but for loading an HDT dataset you need 3% of the disk size of the HDT file in memory. With 50GB of RAM you are able to load a 1.5TB HDT, which means that you are loading at least 50B triples.

[lruppe]

The problem (JLargeArrays Class not found) do not seem to occure when you use Fuseki, but only when you directly load a big hdt file with the HDTManager.loadHDT function.

@wouterbeek Ah interessting, where do you have the numbers from? True, I also do not think it is about the memory but more about an implementation detail as NegativeArraySizeException indicates.

[wouterbeek]

@lruppe Here you can find the numbers: http://lod-a-lot.lod.labs.vu.nl/

[mielvds]

@lruppe does this issue still exist?