YAML Configuration: Test Fails When Whitelisted Requests Trigger WAF Rule ID Alerts

[xu-xiang]

Could you please explain how to configure YAML so that if none of the alert rules are satisfied, we can collect certain whitelisted requests. These whitelisted requests should not match any alerts. After configuring the YAML test file, if any WAF rule ID alerts are triggered, it should be considered a test failure, indicating that the rules need to be optimized.

rule_id: 932140
tests:
  - test_id: 1
    desc: for %
    stages:
      - input:
          dest_addr: 127.0.0.1
          headers:
            Accept: "text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5"
            Host: localhost
            User-Agent: "OWASP CRS test agent"
          method: GET
          port: 80
          uri: "/get?foo=for%20%25variable%20in%20%28set%29%20do%20command"
          version: HTTP/1.0
        output:
          log:
            expect_ids: [932140]

no rule_id expect_ids，just test case id

[theseion]

I don't understand your question. Could you maybe try to explain specifically, which actions you want to perform and which information you need? What do you mean by "collect certain whitelisted requests"? How would you want to collect those? Where would they be "whitelisted"?

[xu-xiang]

We will organize historical false positive alerts into whitelisted requests (legitimate requests) based on our normal business traffic, sampling records, and other methods. We will have a large set of whitelist sample data, such as .http format files or Burp logs, etc. We can batch convert these into the FTW test file format, but FTW doesn't seem to support writing a rule file for false positive testing (i.e., recording a false positive whenever any rule triggers an alert).

[theseion]

I see, interesting. If I'm not mistaken, you could work around that using no_match_regex (adapt the regex to match your log format):

rule_id: 932140
tests:
  - test_id: 1
    desc: for %
    stages:
      - input:
          dest_addr: 127.0.0.1
          headers:
            Accept: "text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5"
            Host: localhost
            User-Agent: "OWASP CRS test agent"
          method: GET
          port: 80
          uri: "/get?foo=for%20%25variable%20in%20%28set%29%20do%20command"
          version: HTTP/1.0
        output:
          log:
            no_match_regex: 'id: "\d+"'

The test will then fail when a request triggers any rule.

[xu-xiang]

I didn't think of directly using id: "\d+" to match any rule alert, I'll give it a try, thanks!

[xu-xiang]

There is still one issue: it is possible to test whether there was a false positive, but in practice, it is not possible to know which rule caused the false positive (in this scenario, the ID written in the test case is actually the unique ID of the test case, and it is not possible to predict in advance the rule ID to be tested. We may collect generic false positive test cases).

[theseion]

True. Well, it looks like you might need a dedicated mode for getting what you want. It shouldn't be hard to do. Would this cover your needs?

• new flag --report-triggered-rules
• no need to specify anything for output
• summary at the end of the run of triggered rules for each test

[theseion]

@xu-xiang I've created a PR with what I think should solve your problem (#432). The PR adds reporting of triggered rules by test (and test stage) to the JSON output (--output json). You'll have to manually filter by failed tests if you need that, but the JSON should now contain everything you need. Could you give it a try and let me know what you think?

[xu-xiang]

You're very welcome! I will test to see if it meets the false positive test scenario and will provide you with the results.

[xu-xiang]

I conducted some tests and found that it can satisfy the scenario of managing false positive test sets. For example, I maintain a false positive test case "1000000-1" and, when executing the test, I use the --output json parameter (without needing to use --report-triggered-rules). By constructing the following test case:

---
meta:
    author: "x"
    description: "test case"
rule_id: 1000000
tests:
    - test_id: 1
      stages:
          - input:
                dest_addr: 127.0.0.1
                method: POST
                port: 80
                headers:
                    Accept: application/json, text/plain, */*
                    Accept-Encoding: gzip, deflate
                    Accept-Language: zh-CN,zh;q=0.9,en-US;q=0.8,en;q=0.7
                    Content-Length: "1798"
                    Content-Type: application/json;charset=utf-8
                    Host: otheve.beacon.qq.com
                    Origin: https://console.cloud.tencent.com
                    Referer: https://console.cloud.tencent.com/
                    Sec-Ch-Ua: '"Not.A/Brand";v="8", "Chromium";v="114", "Google Chrome";v="114"'
                    Sec-Ch-Ua-Mobile: ?0
                    Sec-Ch-Ua-Platform: '"Windows"'
                    Sec-Fetch-Dest: empty
                    Sec-Fetch-Mode: cors
                    Sec-Fetch-Site: cross-site
                    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36
                uri: /analytics/v2_upload?appkey=0WEB0OEX9Y4SQ244
                version: HTTP/1.1
                data: |-
                    {"appVersion":"","sdkId":"js","sdkVersion":"4.5.14-web","mainAppKey":"0WEB0OEX9Y4SQ244","platformId":3,"common":{"atversion":"3.3.3","system":"Windows(10)-Blink","browser":"Chrome(114.0.0.0)","uid":"100003290640","fromsource":"gwzcw.7455492.7455492.7455492","from":"","from_column":"","islogined":"true","pagetag":"","trafficparams":"***$;timestamp%3D1688383695816;from_type%3Dserver;track%3D209e824a-63cb-4713-b428-eba9e8ddf6d8;$***","owneruin":"100003290640","qclouduid":"d25addea5c3c4ce19e1bcc9f0a430ff0@devS","A2":"BxWSJw8tHc0T3KPB1c9Z2idy0hQ85Ddf","A8":"","A12":"zh-CN","A17":"1920*1080*1","A23":"","A50":"","A76":"0WEB0OEX9Y4SQ244_1688383701023","A101":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36","A102":"https://console.cloud.tencent.com/vpc/conn?rid%3D9","A104":"https://cloud.tencent.com/act/pro/seckill_season?fromSource%3Dgwzcw.7455492.7455492.7455492%26utm_medium%3Dcpc%26utm_id%3Dgwzcw.7455492.7455492.7455492","A119":"","A153":""},"events":[{"eventCode":"click","eventTime":"1688384036580","mapValue":{"viewportWidth":"1920","viewportHeight":"921","url":"https://console.cloud.tencent.com/vpc/conn?rid%3D9","pathname":"/vpc/conn","query":"?rid%3D9","hostname":"console.cloud.tencent.com","qr_rid":"9","pagefrom":"console.cloud.tencent.com$$/vpc/nat$$?rid%3D9","urlfrom":"https://console.cloud.tencent.com/vpc/nat?rid%3D9","title":"腾讯云-控制台","text":"","classnames":"tc-15-input-text m","attributes":"{\"type\":\"text\",\"data-name\":\"peeringConnectionName\",\"maxlength\":\"60\"}","top":"220","left":"802","domPositon":"1","queryPath":".form-list > li > div > div > input","queryDom":".form-unit > [type%3D\"text\"]","eid":"13586117151","A99":"Y","A100":"93","A72":"4.5.14-web","A88":"1688383701022"}}]}
            output:
                log:
                    no_match_regex: 'id "\d+"'

This can yield valuable test results.

{
    "run": 1,
    "success": [

    ],
    "failed": [
        "1000000-1"
    ],
    "skipped": [

    ],
    "ignored": [

    ],
    "forced-pass": [

    ],
    "forced-fail": [

    ],
    "runtime": {
        "1000000-1": 28734292
    },
    "TotalTime": 28734292,
    "triggered-rules": {
        "1000000-1": [
            [
                942130,
                942370,
                949110
            ]
        ]
    }
}

[theseion]

Awesome, thank you. We'll soon merge the PR and release a new version.

[xu-xiang]

During the testing process, we encountered another issue. For example, the following rule

SecRule &REQUEST_HEADERS:User-Agent "@eq 0" \
    "id:920320,\
    phase:1,\
    pass,\
    t:none,\
    msg:'Missing User Agent Header',\
    tag:'application-multi',\
    tag:'language-multi',\
    tag:'platform-multi',\
    tag:'attack-protocol',\
    tag:'paranoia-level/2',\
    tag:'OWASP_CRS',\
    tag:'capec/1000/210/272',\
    tag:'PCI/6.5.10',\
    ver:'OWASP_CRS/4.10.0-dev',\
    severity:'NOTICE',\
    setvar:'tx.inbound_anomaly_score_pl2=+%{tx.notice_anomaly_score}'"

generates alert logs (we are currently analyzing blocked scenarios), which affects the determination of success or failure. However, the current regular expression cannot be used, such as match_regex: 'id "(?!920320)\d+"', to exclude alerts from specific rule IDs, as the regex is not supported.

4:28PM FTL ftw/waflog: bad regexp error parsing regexp: invalid or unsupported Perl syntax: `(?!`

Apart from disabling this rule, is there a way to exclude certain rule IDs without affecting the result determination?

[xu-xiang]

2025/01/02 12:19:00 [info] 117#117: *16117 ModSecurity: Warning. Matched "Operator `Rx' with parameter `%u[4e00-9fa5]{3,}' against variable `TX:matched' (Value: `{"anonym":0,"key":"","sn":"","id":"168838233072272454","from":4,"token":"bridge"}' ) [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-942-APPLICATION-ATTACK-SQLI.conf"] [line "973"] [id "942370"] [rev ""] [msg "Detects classic SQL injection probings 2/3"] [data "Matched Data: \x22:\x22\x22,\x22 found within TX:matched: {\x22anonym\x22:0,\x22key\x22:\x22\x22,\x22sn\x22:\x22\x22,\x22id\x22:\x22168838233072272454\x22,\x22from\x22:4,\x22token\x22:\x22bridge\x22}"] [severity "2"] [ver "OWASP_CRS/3.3.2"] [maturity "0"] [accuracy "0"] [tag "modsecurity"] [tag "modsecurity"] [tag "modsecurity"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-sqli"] [tag "OWASP_CRS"] [tag "capec/1000/152/248/66"] [tag "PCI/6.5.2"] [tag "paranoia-level/2"] [tag "CVE-2018-2380"] [tag "APP/SAP CRM"] [hostname "172.18.0.3"] [uri "/cps5/site/aust"] [unique_id "173582034044.678876"] [ref "o1,38v79,261t:urlDecodeUnio16,6v1542,81t:urlDecodeUnit:utf8toUnicode"], client: 172.18.0.1, server: localhost, request: "GET /cps5/site/aust?cb=jsonp_bridge_1688382331340_2474076564328216&op=0&s_info=%7B%22lang%22%3A%22zh-CN%22%2C%22cbit%22%3A24%2C%22rsl%22%3A%221920*1080%22%2C%22tz%22%3A%22UTC%2B8%3A0%22%2C%22xst%22%3A%22%22%2C%22referrer%22%3A%22https%253A%252F%252Fwww.baidu.com%252Flink%253Furl%253DPukzFxKXUAPWz2bGom-5-N5FroXNtyu0tc9pzXJz8ma%2526wd%253D%2526eqid%253Df68a4a19000c43c70000000464a2ab71%22%2C%22xstlink%22%3A%22http%253A%252F%252Fwww.authing.co%252F%22%7D&url=http%3A%2F%2Fwww.authing.co%2F&siteToken=fa8cc78cf376a0ce56fe3eeed5c06e5f&dev=0&ser=3&bst=1688382327210&AFDbiz=%7B%22ev%22%3A%22page_enter%22%2C%22customer%22%3A%2230208105%22%2C%22bid%22%3A%22168838233072272454%22%2C%22length%22%3A0%7D&AFDjt=31%24eyJrIj4iNyI0Iix5IkciQEZJSkZMR0lKSUxNUyJJIkFqIjwiNTs%2BPztBPD4%2FPkFCSCI%2BIjYzIlEiSlBTVFBWUTM0MzM3OCIzIit5IkYiQz9AIj4iOCJQIktHTklUIkoiaiI8IlQ%2Bd3VJS```

id error in json output：

"1000000-1275":[[920274,942370,949110,168838233072272454]]

"id":\s*"?(\d+)"?

[theseion]

We don't support look-around expressions. But if you're expecting certain IDs you can just use expect_ids: [920320].

For the bad ID I'll have to create a fix. Thanks.

[theseion]

I've opened #436 to address the ID detection.