Analysis remains stuck

[GianlucaDeStefano]

Hi, I am trying to analyze a set of ~500 python scripts using codeql.
To do that, I am running the following commands:

Create the db:

./codeql database create /tmp/1685480387.4439943/database --source-root /tmp/1685480387.4439943/Samples --language python

Run the analysis:

./codeql database analyze /tmp/1685480387.4439943/database --format sarifv2.1.0 --output /tmp/1685480387.4439943/codeql_results.json --threads 20 python-security-extended.qls

As you can see, I am running the default queries in the python-security-extended.qls suite.
However, the analysis seems to remain stuck immediately as I get no output.
If I add the following flag --verbosity=progress+++.

I only get the following outputs:

Writing logs to /tmp/1685480387.4439943/database/log/database-analyze-20230530.214415.290.log.
Running queries.
Writing logs to /tmp/1685480387.4439943/database/log/execute-queries-20230530.214415.741.log.

This is run on a server that may not have the fastest disk but I don't see how this should cause the analysis to not even start.
Does someone know how to further debug this problem or fix it?

The version I am using is the 2.12.3

[alexet]

The logs in execute-queries-20230530.214415.741.log should contain more details that we can use to diagnose what is happening. Are you able to share the end of one of those logs?

[GianlucaDeStefano]

Yes, It contains:

[2023-05-31 10:47:33] This is codeql execute queries -J-Xmx15344M --off-heap-ram=15344 --verbosity=progress+ --logdir=/tmp/1685529240.656937/CWE-732/database/log --evaluator-log-level=5 --threads=20 --warnings=show --qlconfig-file=/home/codeql/qlconfig.yml --no-rerun --output=/tmp/1685529240.656937/CWE-732/database/results -- /tmp/1685529240.656937/CWE-732/database/db-python python-security-extended.qls
[2023-05-31 10:47:33] Calling plumbing command: codeql resolve queries --qlconfig-file=/home/codeql/qlconfig.yml --format=json -- python-security-extended.qls

Also If I manually run

./codeql resolve queries --qlconfig-file=/home/codeql/qlconfig.yml --format=json -- python-security-extended.qls

and

./codeql pack download python-queries

I still get stuck.

I suspect that there may be an issue in connecting to the server to download the queries. I am adding this step in the docker to take care of it beforehand.

[GianlucaDeStefano]

My previous guesses were wrong, unfortunately. However, I may have another lead:

I have noticed that, sometimes, during the creation of the database, codeql includes many dependencies installed on the system but not used by the scripts under analysis.
Why does this happen? Is this a normal behaviour from codeql?

Can this be the reason why sometimes I remain stuck? I noticed that, sometimes, the database creation process is much longer and includes many of these dependencies.
I suspect these "unwanted" addictions cause the analysis to become insufferably slow.

Can someone explain what may cause codeql to include the dependencies in the analysis?
How can I avoid it? Should I run the codeql command in a virgin environment with no installed dependencies?

[GianlucaDeStefano]

I think these dependencies are the issue because they are not included in cases where I do not get stuck (even if I run the exact same commands).
Since the only thing between running and stuck runs is the analyzed data, the cause of their inclusion has to be in the Python scripts under analysis. However, I don't understand how they may cause all dependencies in my environment to be included in the codeql db if they have no direct references to these dependencies. Looking at some of them, they appear to be normal scripts.

[GianlucaDeStefano]

Yep, by running the codeql process in a separate container (with no dependencies installed) the analysis is much faster. I still do not get how this is possible.

Is there a quick way to understand why a certain dependency gets included in the database?
How can i avoid this problem?