[Security Solution] Adds normalization for filter meta field diff

[dplumlee]

Fixes: #206527
Partially addresses: #209518

Summary

Adds a normalization to the filters field in the rule diffing calculation that omits all filter fields other than the query field and the negate and disabled fields within the meta object. This makes our diffing logic much more robust and resilient as we only compare data in the rule fields that have an impact on the query itself and not the fields that relate to UI implementation (alias, key, etc).

To test

• Open a prebuilt rule with filters in the non-customized rule parameters (e.g. PowerShell Script with Discovery Capabilities)
• Edit the rule and save without editing
• The rule should remain unmodified even though more fields have been added to the rule's filters field

Unless the user adds or deletes a filter on the rule, the rule should only be marked as customized under 3 circumstances:

• The user negates the filter (adds NOT to the beginning of the filter)
• The user disables the filter
• The user changes the filter query

All other scenarios (such as adding a custom name for the filter) should not change the rule's customized status

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

• Unit or functional tests were updated or added to match the most common scenarios

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[maximpn]

@dplumlee Thanks for that improvement 🙏

There are a couple of comments to improve tests I believe should be addressed before merging the PR.

---

The PR was tested locally by going through the following scenarios

• Saving rule unchanged
• Negating and/or disabling filters
• Preview customized filters in Rule Upgrade Flyout
• Upgrading rules with customised filters

Conclusion: It works as expected.

Customization state is properly determined for a prebuilt rule with filters. It resets to not modified when filter's are restored to it's original state.

Rule upgrade works as expected. Upgraded rule has customized value unchanged.

There is a recorded video
https://github.com/user-attachments/assets/88b1a9c3-de6e-4eec-85e2-88bf5c421158

---

Future improvements (out of scope of this PR)

• Editing filters in Rule Upgrade Flyout leads to showing diff with fields normalization removes. It looks like we can apply the same normalization in the Rule Upgrade Diff View. But it's important to preserve values for rule upgrade API call to have values saved upon upgrade.
• Filter's editing doesn't look handling when specified fields are missing in ES. This is the existing behavior persisting in Kibana for long time. The same is applicable to editing filters in Rule Upgrade Flyout
  

[maximpn]

Ideally this test should be improved. Let's use it.each with inline filters to tested the following normalization scenarios

• all fields exists normalization
• negate field missing (removable fields missing)
• disabled field missing (removable fields missing)
• query field missing (removable fields missing)

[maximpn]

@dplumlee We can further improve tests isolation by having separate tests for filters[].query and filters[].meta. I gave it a try and got the tests below you can simply replace normalization tests with

describe('filters normalization', () => {
      it('normalizes filters[].query when all fields present', () => {
        const extractedKqlQuery = extractRuleKqlQuery(
          'some:query',
          'kuery',
          [mockFilter],
          undefined
        );

        expect(extractedKqlQuery).toMatchObject({
          filters: [
            {
              query: {
                term: {
                  field: 'value',
                },
              },
            },
          ],
        });
      });

      it('normalizes filters[].query when query object is missing', () => {
        const extractedKqlQuery = extractRuleKqlQuery(
          'some:query',
          'kuery',
          [{ ...mockFilter, query: undefined }],
          undefined
        );

        expect(extractedKqlQuery).not.toMatchObject({
          filters: [
            {
              query: expect.anything(),
            },
          ],
        });
      });

      it.each([
        {
          caseName: 'when all fields present',
          filter: mockFilter,
          expectedFilterMeta: {
            negate: false,
            disabled: false,
          },
        },
        {
          caseName: 'when disabled field is missing',
          filter: { ...mockFilter, meta: { ...mockFilter.meta, disabled: undefined } },
          expectedFilterMeta: {
            negate: false,
            disabled: false,
          },
        },
        {
          caseName: 'when negate field is missing',
          filter: { ...mockFilter, meta: { ...mockFilter.meta, negate: undefined } },
          expectedFilterMeta: {
            disabled: false,
          },
        },
        {
          caseName: 'when query object is missing',
          filter: { ...mockFilter, query: undefined },
          expectedFilterMeta: {
            negate: false,
            disabled: false,
          },
        },
      ])('normalizes filters[].meta $caseName', ({ filter, expectedFilterMeta }) => {
        const extractedKqlQuery = extractRuleKqlQuery('some:query', 'kuery', [filter], undefined);

        expect(extractedKqlQuery).toMatchObject({
          filters: [
            {
              meta: expectedFilterMeta,
            },
          ],
        });
      });

      it('normalizes filters[].meta when query object is missing', () => {
        const extractedKqlQuery = extractRuleKqlQuery(
          'some:query',
          'kuery',
          [{ ...mockFilter, query: undefined }],
          undefined
        );

        expect(extractedKqlQuery).toMatchObject({
          filters: [
            {
              meta: {
                negate: false,
                disabled: false,
              },
            },
          ],
        });
      });
    });

[maximpn]

In that test we are interested in filters field. Let's move type, query and language assertions to separate tests.

[banderror]

@elasticmachine merge upstream

[maximpn]

@dplumlee I left a comment to further improve unit tests you added in this PR. Since it's not a critical comment I approve the PR.

[maximpn]

@dplumlee We can further improve tests isolation by having separate tests for filters[].query and filters[].meta. I gave it a try and got the tests below you can simply replace normalization tests with

describe('filters normalization', () => {
      it('normalizes filters[].query when all fields present', () => {
        const extractedKqlQuery = extractRuleKqlQuery(
          'some:query',
          'kuery',
          [mockFilter],
          undefined
        );

        expect(extractedKqlQuery).toMatchObject({
          filters: [
            {
              query: {
                term: {
                  field: 'value',
                },
              },
            },
          ],
        });
      });

      it('normalizes filters[].query when query object is missing', () => {
        const extractedKqlQuery = extractRuleKqlQuery(
          'some:query',
          'kuery',
          [{ ...mockFilter, query: undefined }],
          undefined
        );

        expect(extractedKqlQuery).not.toMatchObject({
          filters: [
            {
              query: expect.anything(),
            },
          ],
        });
      });

      it.each([
        {
          caseName: 'when all fields present',
          filter: mockFilter,
          expectedFilterMeta: {
            negate: false,
            disabled: false,
          },
        },
        {
          caseName: 'when disabled field is missing',
          filter: { ...mockFilter, meta: { ...mockFilter.meta, disabled: undefined } },
          expectedFilterMeta: {
            negate: false,
            disabled: false,
          },
        },
        {
          caseName: 'when negate field is missing',
          filter: { ...mockFilter, meta: { ...mockFilter.meta, negate: undefined } },
          expectedFilterMeta: {
            disabled: false,
          },
        },
        {
          caseName: 'when query object is missing',
          filter: { ...mockFilter, query: undefined },
          expectedFilterMeta: {
            negate: false,
            disabled: false,
          },
        },
      ])('normalizes filters[].meta $caseName', ({ filter, expectedFilterMeta }) => {
        const extractedKqlQuery = extractRuleKqlQuery('some:query', 'kuery', [filter], undefined);

        expect(extractedKqlQuery).toMatchObject({
          filters: [
            {
              meta: expectedFilterMeta,
            },
          ],
        });
      });

      it('normalizes filters[].meta when query object is missing', () => {
        const extractedKqlQuery = extractRuleKqlQuery(
          'some:query',
          'kuery',
          [{ ...mockFilter, query: undefined }],
          undefined
        );

        expect(extractedKqlQuery).toMatchObject({
          filters: [
            {
              meta: {
                negate: false,
                disabled: false,
              },
            },
          ],
        });
      });
    });

[kibanamachine]

Starting backport for target branches: 8.18, 8.x, 9.0

https://github.com/elastic/kibana/actions/runs/13510692338

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: c5b0935

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	8.9MB	8.9MB	+8.0B

History

• 💛 Build #278680 was flaky aa7cf42
• 💚 Build #278528 succeeded 56e79c8
• 💚 Build #277447 succeeded 0a299fb
• 💔 Build #277428 failed 164a8b1
• 💔 Build #276335 failed 834e1d9
• 💚 Build #274507 succeeded 577171d

cc @dplumlee

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.18
✅	8.x
✅	9.0

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation