Using common_rules in malware with yara-python doesn't work

[supremeliter]

First I'll say this might be an issue with yara-python but I have a proposed solution here so it might be worth the time commenting on it for anyone else who is having this same issue.

The issue

1. When using yara.compile on certain malware rules I get an error:

>>> rule = yara.compile('malware/MALW_TinyShell_Backdoor_gen.yar')
Traceback (most recent call last):
  File "<input>", line 1, in <module>
yara.SyntaxError: malware/MALW_TinyShell_Backdoor_gen.yar(126): undefined identifier "is__elf"

2. If I try using the file generated by the index_gen.sh script I get another error:

>>> rule = yara.compile('malware_index.yar')
Traceback (most recent call last):
  File "<input>", line 1, in <module>
yara.SyntaxError: ./malware/APT_Grasshopper.yar(374): can't open include file: ./malware/TOOLKIT_Wineggdrop.yar

A work around

To be honest, I have no idea what's going on in the second error but I have a solution for the first. To fix it I simply put include "000_common_rules.yar" at the top of all .yar files giving me this issue. Simple but even I admit it isn't the best solution.

I haven't seen any other posts on this and I accept if the answer is that I'm not using these rules or yara-python properly.

Last, the program I'm working on is doing something like this, trying to provide a report on which of your rules these binaries are hitting on. I'm not a malware analyst but I want to tell you our customer swears up and down by your rules!

[supremeliter]

$ yara-3.8.1/yarac malware_index.yar rules_compiled/malware
./malware/APT_DPRK_ROKRAT.yar(43): warning: $b2 is slowing down scanning
./malware/MALW_AZORULT.yar(5): error: unknown module "cuckoo"
./malware/MALW_AZORULT.yar(23): error: invalid field name "sync"
./malware/MALW_AZORULT.yar(107): error: can't open include file: ./malware/MALW_BackdoorSSH.yar
./malware/MALW_AZORULT.yar(108): error: can't open include file: ./malware/MALW_Backoff.yar
./malware/MALW_AZORULT.yar(109): error: can't open include file: ./malware/MALW_Bangat.yar
./malware/MALW_AZORULT.yar(110): error: can't open include file: ./malware/MALW_Batel.yar

...  < This goes on and on for every line in the file > ...

./malware/MALW_AZORULT.yar(373): error: can't open include file: ./malware/TOOLKIT_THOR_HackTools.yar
./malware/MALW_AZORULT.yar(374): error: can't open include file: ./malware/TOOLKIT_Wineggdrop.yar

Removing the file MALW_AZORULT.yar from the malware_index.yar fixes the problem. I can load the compiled rule with yara-python but I still can't use yara.compile.

[BrunoJesus84]

I removed the file MALW_AZORULT.yar, but many other files shows the same error "error: can't open include file:". I tried to remove 1 by 1, but they are many. I saw this same error with a simple rule that I try to create. I don`t understand why the file can't be included...

[Xumeiquer]

I've just regenerate the index files, it could fix the issue.

[supremeliter]

I wanted to quickly write that at the time of posting I did try using the generate indexes shell script if that's what your referring to. I quit the poorly managed job that used this package so I don't care about this anymore but I don't think it's solved. I'll let someone else close it.

[jdsnape]

I'm seeing exactly the same issue. I find if I remove the MALW_AZORULT.yar file from the malware_index.yar flie then it generally works on my machine, but if I copy it onto a server I get errors about lots of different files (like @BrunoJesus84 ). I'm honestly a bit stumped, as it works fine from yara on the command line.

Update: I’ve also noticed this with other rule sets, so I think it’s a Yara issue...

[jholgui]

@jdsnape, Can you tell us the yara version? Next days, i'm going to test this issue ..

[jdsnape]

@jdsnape, Can you tell us the yara version? Next days, i'm going to test this issue ..

Sorry for slow reply, I was away for a bit. I've just retested, and I'm seeing the issue with yara-python-3.10.0 and Python 3.7.4

Edit - I've now tried removing the MALW_AZORULT.yar file from malware_index.yar and I can use yara.compile OK with these versions.

[jdsnape]

I opened an issue on the yara-python project about this (VirusTotal/yara-python#112) and someone has recommended using absolute paths rather than relative paths. I've not tested it but thought I'd put it here in case anyone else is having the issue

[Bondey]

The main problem with Azorult rule is the "cuckoo" module usage.
The rule works fine on a system where Yara has been compiled with support for this module, but, 99% times, people don't compile yara with support for this module, as far as most of us dont have a cuckoo installation on the host...

An option could be to simply delete the usage of this module on that rule...