Is it parallel?

[marchezinixd]

I'm trying on a really large dataset and checking the resources usage. Apparently it is using only one core. is it possible to set it to use all cores and make it faster?

[chirayukong]

It should be. Which algorithm are you running? I'll take a look.

[marchezinixd]

The parameters i used were:
FGES
Sem-Bic
Sem-Bic
Penalty: 100

I have 4 cores and it is using 100% of one but nothing of the others.
The dataset have 105 features, 2.6 million rows
The memory is ok, it is using 14gb and i have a total of 32gb

[jdramsey]

Just one note: the part of FGES that parallelizes the best is the initial (usually most time-consuming) part. After that, there is a period where the parallelization isn't quite as good. You might for sanity's sake check to see if you're using more than one core when you first call the process. Joe

…

On Thu, Sep 13, 2018 at 7:12 PM Guilherme Fernandes Marchezini < ***@***.***> wrote: The parameters i used were: FGES Sem-Bic Sem-Bic Penalty: 100 I have 4 cores and it is using 100% of one but nothing of the others. The dataset have 105 features, 2.6 million rows The memory is ok, it is using 14gb and i have a total of 32gb — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#74 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AJZZRw65AzuibZqNEJZJ04B63HdAb9YCks5uauZsgaJpZM4WoTbn> .

-- Joseph D. Ramsey Special Faculty and Director of Research Computing Department of Philosophy 135 Baker Hall Carnegie Mellon University Pittsburgh, PA 15213 [email protected] Office: (412) 268-8063 http://www.andrew.cmu.edu/user/jdramsey

[marchezinixd]

Well i checked it.
I reduced the penalty to 25 and ran it again. The attached image shows how it behaves. Basically there was a few seconds peak that used all cores.
The second graph shows that it uses just one core at a time, but in sequentially uses all cores.

[chirayukong]

Try it on the causal-cmd cli. Attachment is its distribution. Run it with java -Xmx14G -jar causal-cmd-0.4.0-SNAPSHOT-jar-with-dependencies.jar --algorithm fges --data-type <discrete|continuous> --delimiter <comma|tab> --dataset <your_dataset> --score sem-bic --test sem-bic --penaltyDiscount 100 --json-graph. More about causal-cmd.
causal-cmd-0.4.0-SNAPSHOT-distribution.zip

[marchezinixd]

Hello @chirayukong sorry for the long time to awnser, i was having trouble with the dataset and how to handle the full size. I did the test, in the cmd it ran in ~5 minutes and had the behaviour of the attached image. When running with python it took ~1 hour and had the same behavior of the previous images. Apparently the new jar have a better performance and parallelize more than the python one. Is it possible to update the pycausal?

CMD test

[chirayukong]

The jar file is updated. Please try it. @marchezinixd

[marchezinixd]

The beginning was a little different, but still following the same old pattern, while the jar ran in 4 minutes, the python is running for 20 minutes and it seems it will not end soon.
Apparently it is a python problem, maybe the way it handles parallelism?

[chirayukong]

Maybe it's a problem on the javabridge library, which I don't know how to fix it. You can run it on causal-cmd and load the json result back to python.

[chirayukong]

This is the latest one.
causal-cmd-0.4.0-SNAPSHOT-distribution.zip

[marchezinixd]

Well i'll do it for now. I'll leave the issue open in case you guys have any ideas how to solve the python problem.
Thankyou