Towards better inference: bits → nibbles

[originalsouth]

Changes

Bits → Nibbles

Issue link

N/A

Demo

T.B.D.

QA notes

Test all ported bits/new nibbles from front-end:

• disallowed_csp_hostnames (with/without config)
• check_cve_2021_41773
• domain_owner_verification
• expiring_certificate
• ask_url_params_to_ignore
• website_discovery
• missing_certificate
• cipher_classification
• default_findingtype_risk
• spf_discovery
• max_url_length_config (with config for testing only)
• oois_in_headers (with/without config)
• ask_port_specification
• ask_disallowed_domains
• check_hsts_header (with/without config)
• url_classification
• missing_spf

---

Code Checklist

• All the commits in this PR are properly PGP-signed and verified.
• This PR only contains functionality relevant to the issue.
• I have written unit tests for the changes or fixes I made.
• I have checked the documentation and made changes where necessary.
• I have performed a self-review of my code and refactored it to the best of my abilities.

• Tickets have been created for newly discovered issues.
• For any non-trivial functionality, I have added integration and/or end-to-end tests.
• I have informed others of any required .env changes files if required and changed the .env-dist accordingly.
• I have included comments in the code to elaborate on what is not self-evident from the code itself, including references to issues and discussions online, or implicit behavior of an interface.

---

Checklist for code reviewers:

Copy-paste the checklist from the docs/source/templates folder into your comment.

---

Checklist for QA:

Copy-paste the checklist from the docs/source/templates folder into your comment.

[stephanie0x00]

The following Nibbles were tested and possibly contain bugs:

1. • Onboarding
     • 2 findings remain pending. These are the KAT-NO-SPF findings.
2. • disallowed_csp_hostnames (with/without config)
     • CSP with config: applied the Question to create a config. Scanned a host with one of the default bad CSP hostnames, but it is not picked up.
     • without config: not sure what is meant by this. I removed the list with bad CSP hostnames from the Question object, this is applied. However when editing this in the config it is not allowed.
3. • check_cve_2021_41773
   • Added a HTTPHeader with Server: Apache/2.4.49 (Unix) and also a Software version for 2.4.49, but it doesn't seem to create any findings.
4. • expiring_certificate: doesn't create expired certificates. Tried against expired.badssl.com with nmap TCP + l2 clearance and all SSL boefjes enabled.
5. • missing_certificate: The No-Certificate finding remains pending. Is this also from this nibble? Tested against mispo.es (which does have a certificate) and httpforever.com (http only)
6. • cipher_classification: seems to work mostly, got a finding KAT-MEDIUM-BAD-CIPHER and also some TLS1.0 and TLS1.1 Support findings. There seems to be a bug though. There is apparently TLS 1.0 support for port 80?
7. • check_hsts_header (with/without config)
   • Tested without config: the absence of the HSTS header is not triggered on mispo.es.
   • with config: Not sure how to test this as there isn't a question or config by default?
8. • url_discovery bit was broken (but not on the list), but should be fixed with one of the fixes below.

The following nibbles have not been tested yet, but are picked up in between other PRs.
- [ ] ask_url_params_to_ignore
- [ ] default_findingtype_risk
- [ ] spf_discovery
- [ ] max_url_length_config (with config for testing only)
- [ ] oois_in_headers (with/without config)
- [ ] ask_port_specification
- [ ] ask_disallowed_domains
- [ ] check_hsts_header (with/without config)
- [ ] url_classification
- [ ] missing_spf

The following nibbles were tested and seem to work. Please double check the short reproduction steps if this was the intended way.

• domain_owner_verification: this nibble seems to work if you take the following actions:
  • 1. Manually add a Hostname: NS1.REGISTRANT-VERIFICATION.ISPAPI.NET.
  • 2. Gave the Hostname L1 clearance.
  • 3. Added a DNS NS record for mispo.es with the value "NS1.REGISTRANT-VERIFICATION.ISPAPI.NET" on the Name Server NS1.REGISTRANT-VERIFICATION.ISPAPI.NET (as added in step 1).
  • 4. Then the finding is triggered and resolves into a severity.
• website_discovery: Seems to work, Website objects are created and on the object details page I can see the web_discovery nibblet has run to create the object.

[noamblitz]

Interesting about the pending findings, ill pick it up!

The others ill test myself to confirm.

[noamblitz]

I spent some time debugging the pending finding types and here's my conclusion on what's happening. Let's consider the missing-spf finding since it's the most straightforward.

1. Hostname is created
2. missing-spf nibble runs and creates missing-spf finding
3. In the cache, the default finding type risk has also run
4. Both the finding and the hydrated finding type (with pending risk) are written to XTDB
5. Boefje kat-finding-types runs and hydrates the finding type with actual risk scores

So far so good!

6. Another hostname is created
7. missing-spf nibble runs and creates missing-spf finding
8. In the cache, the default finding type risk has also run
9. Both the finding and the hydrated finding type (with pending risk) are written to XTDB

Now, the finding type is set back to pending risk (because of the caching) and the boefje will not run anymore because it had run already. Now... this problem will fix itself every day but that's not sustainable. I will play around with this.

[noamblitz]

The following Nibbles were tested and possibly contain bugs:

1. • Onboarding

     • 2 findings remain pending. These are the KAT-NO-SPF findings. This should be fixed with the new pr

2. • disallowed_csp_hostnames (with/without config)

     • CSP with config: applied the Question to create a config. Scanned a host with one of the default bad CSP hostnames, but it is not picked up. Will check this out
     • without config: not sure what is meant by this. I removed the list with bad CSP hostnames from the Question object, this is applied. However when editing this in the config it is not allowed. Will check this out
3. • check_cve_2021_41773
   • Added a HTTPHeader with Server: Apache/2.4.49 (Unix) and also a Software version for 2.4.49, but it doesn't seem to create any findings. Will check this out
4. • expiring_certificate: doesn't create expired certificates. Tried against expired.badssl.com with nmap TCP + l2 clearance and all SSL boefjes enabled. This only checks if certs are expiring, it does not create the certs itself. That should be done by a boefje
5. • missing_certificate: The No-Certificate finding remains pending. Is this also from this nibble? Tested against mispo.es (which does have a certificate) and httpforever.com (http only) Should be fixed in new pr
6. • cipher_classification: seems to work mostly, got a finding KAT-MEDIUM-BAD-CIPHER and also some TLS1.0 and TLS1.1 Support findings. There seems to be a bug though. There is apparently TLS 1.0 support for port 80? Thats a bug in the boefje then! Good catch! But then the nibble should be working!
7. • check_hsts_header (with/without config)
   • Tested without config: the absence of the HSTS header is not triggered on mispo.es. This nibble does not check the existence but checks the content.
   • with config: Not sure how to test this as there isn't a question or config by default?

The following nibbles have not been tested yet, but are picked up in between other PRs.

• ask_url_params_to_ignore

• website_discovery

• default_findingtype_risk

• spf_discovery

• max_url_length_config (with config for testing only)

• oois_in_headers (with/without config)

• ask_port_specification

• ask_disallowed_domains

• check_hsts_header (with/without config)

• url_classification

• missing_spf

  The following nibbles were tested and seem to work. Please double check the short reproduction steps if this was the intended way.

  • domain_owner_verification: this nibble seems to work if you take the following actions:

    • 1. Manually add a Hostname: NS1.REGISTRANT-VERIFICATION.ISPAPI.NET.
    • 2. Gave the Hostname L1 clearance.
    • 3. Added a DNS NS record for mispo.es with the value "NS1.REGISTRANT-VERIFICATION.ISPAPI.NET" on the Name Server NS1.REGISTRANT-VERIFICATION.ISPAPI.NET (as added in step 1).
    • 4. Then the finding is triggered and resolves into a severity.

[underdarknl]

I spent some time debugging the pending finding types and here's my conclusion on what's happening. Let's consider the missing-spf finding since it's the most straightforward.

1. Hostname is created

2. missing-spf nibble runs and creates missing-spf finding

3. In the cache, the default finding type risk has also run

4. Both the finding and the **hydrated** finding type (with pending risk) are written to XTDB

5. Boefje kat-finding-types runs and hydrates the finding type with actual risk scores

So far so good!

6. Another hostname is created

7. missing-spf nibble runs and creates missing-spf finding

8. In the cache, the default finding type risk has also run

9. Both the finding and the **hydrated** finding type (with pending risk) are written to XTDB

Now, the finding type is set back to pending risk (because of the caching) and the boefje will not run anymore because it had run already. Now... this problem will fix itself every day but that's not sustainable. I will play around with this.

Is this really related to caching the bits/nibbles?
In step 9 of your steps, we write out the finding, and the findingtype to best to the ability of the discvering code. (eg, it does not do the hydration) and as such can only set the score of the findingtype to None.
Should we not write out an empty field for the finding, as we cannot make an assumption about its value, and as such should not overwrite any existing data for that field? (eg, the hydrated value previously created by the hydration boefje)
I believe we already had this issue earlier and decided on the same solution?

[noamblitz]

I spent some time debugging the pending finding types and here's my conclusion on what's happening. Let's consider the missing-spf finding since it's the most straightforward.

1. Hostname is created

2. missing-spf nibble runs and creates missing-spf finding

3. In the cache, the default finding type risk has also run

4. Both the finding and the **hydrated** finding type (with pending risk) are written to XTDB

5. Boefje kat-finding-types runs and hydrates the finding type with actual risk scores

So far so good!

6. Another hostname is created

7. missing-spf nibble runs and creates missing-spf finding

8. In the cache, the default finding type risk has also run

9. Both the finding and the **hydrated** finding type (with pending risk) are written to XTDB

Now, the finding type is set back to pending risk (because of the caching) and the boefje will not run anymore because it had run already. Now... this problem will fix itself every day but that's not sustainable. I will play around with this.

Is this really related to caching the bits/nibbles? In step 9 of your steps, we write out the finding, and the findingtype to best to the ability of the discvering code. (eg, it does not do the hydration) and as such can only set the score of the findingtype to None. Should we not write out an empty field for the finding, as we cannot make an assumption about its value, and as such should not overwrite any existing data for that field? (eg, the hydrated value previously created by the hydration boefje) I believe we already had this issue earlier and decided on the same solution?

No this is another problem, there is a nibble (bit before) that's called default-finding-risk which puts pending on a FindinType the first time it sees it. So with bits this worked well, when there was a change to a FindingType, it was not the first time KAT saw it, so there would be no change.

Since this nibble now runs in cache, when the FindingType is created, it is really the first time we see it (in cache :) ). So it gives Pending.

[noamblitz]

So FindingType is not set to None by either a boefje or nibble, this is already going well, problem is that status is set to pending.

[sonarqubecloud]

Quality Gate failed

Failed conditions
4 Security Hotspots
37.8% Coverage on New Code (required ≥ 80%)
10.6% Duplication on New Code (required ≤ 3%)
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE