feat(tests): new e2e tests [1/N] #1661

therealbobo · 2024-01-30T19:36:18Z

What type of PR is this?

Uncomment one (or more) /kind <> lines:

/kind bug

/kind cleanup

/kind design

/kind documentation

/kind failing-test

/kind feature

Any specific area of the project related to this PR?

Uncomment one (or more) /area <> lines:

/area API-version

/area build

/area CI

/area driver-kmod

/area driver-bpf

/area driver-modern-bpf

/area libscap-engine-bpf

/area libscap-engine-gvisor

/area libscap-engine-kmod

/area libscap-engine-modern-bpf

/area libscap-engine-nodriver

/area libscap-engine-noop

/area libscap-engine-source-plugin

/area libscap-engine-savefile

/area libscap

/area libpman

/area libsinsp

/area tests

/area proposals

Does this PR require a change in the driver versions?

/version driver-API-version-major

/version driver-API-version-minor

/version driver-API-version-patch

/version driver-SCHEMA-version-major

/version driver-SCHEMA-version-minor

/version driver-SCHEMA-version-patch

What this PR does / why we need it:

This is the initial batch of new e2e tests.

Which issue(s) this PR fixes:

Part of #1650

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

leogr

Hey @therealbobo

Can you rebase this on the master?

This #1664 should fix the failing tests. 🙏

leogr

🥳

poiana · 2024-02-01T08:59:32Z

LGTM label has been added.

Git tree hash: 472ea3c0f282d766bada55f6a7991b1df036dc27

Andreagit97 · 2024-02-01T09:19:42Z

/hold i would like to review this before merging it

CMakeLists.txt

test/libsinsp/CMakeLists.txt

Andreagit97 · 2024-02-06T10:46:36Z

test/libsinsp/libsinsp_test_var.h.in

+// Absolute path to the bpf probe .o file
+#define LIBSINSP_TEST_BPF_PROBE_PATH "${CMAKE_BINARY_DIR}/driver/bpf/probe.o"
+
+#define LIBSINSP_TEST_CAPTURES_PATH "${CMAKE_BINARY_DIR}/test/libsinsp/captures"


This seems relative to scap-files, but we already have a framework to support them https://github.com/falcosecurity/libs/tree/master/userspace/libsinsp/test/scap_files, maybe it could be enough WDYT?

LIBSINSP_TEST_CAPTURES_PATH has a different motivation: each test dumps the captured events in a scap file so that we can inspect them in a second moment. This is not meant to test scap file correctness. 😅

Andreagit97 · 2024-02-06T10:49:21Z

test/libsinsp/main.cpp

+	std::string captures_dir = LIBSINSP_TEST_CAPTURES_PATH;
+
+	if(!std::filesystem::exists(captures_dir))
+	{
+		if (!std::filesystem::create_directory(captures_dir)) {
+			std::cerr << "Failed to create captures directory." << std::endl;;
+			return EXIT_FAILURE;
+		}
+	}


Let's see what we decide to do with scap-files

Andreagit97 · 2024-02-06T10:52:09Z

test/libsinsp/main.cpp

+	/* Get current cwd as a base directory for the driver path */
+	char driver_path[FILENAME_MAX];
+	if(!getcwd(driver_path, FILENAME_MAX))
+	{
+		std::cerr << "Unable to get current dir" << std::endl;
+		return EXIT_FAILURE;
+	}


it seems we can remove this

Andreagit97 · 2024-02-06T10:55:28Z

test/libsinsp/CMakeLists.txt

Can we add a README.md in this folder with instructions to run these tests?

More precisely I would try to follow the same structure we have in the other tests (e.g. the drivers one):

/libsinsp_e2e ├─ `helpers` ├─ `test_suites` ├─ `event_capture` ├─ `CMakeLists.txt` ├─ `README.md` ├─ ...

Yes, totally on board on this! On the other hand I'd like to do this kind on changes later on (I already have some PRs in the working). Are you ok with that?

Andreagit97 · 2024-02-06T11:45:03Z

.github/workflows/e2e_ci.yml

+      - name: Run e2e tests with modern bpf 🏎️
+        run: |
+          cd build/test/libsinsp/
+          sudo ./tests -m
+
+      - name: Run e2e tests with bpf 🏎️
+        run: |
+          cd build/test/libsinsp/
+          sudo ./tests -b
+
+      - name: Run e2e tests with kmod 🏎️
+        run: |
+          cd build/test/libsinsp/
+          sudo ./tests -k


On QEMU we should be able to inject eBPF drivers so I don't think we need to touch this job

Another suggestion I'd add here is that we may want to run the tests on different drivers by using a matrix without soft fail, so that we don't skip tests on a driver if we find errors in another.

Andreagit97 · 2024-02-06T12:01:50Z

test/libsinsp/sys_call_test.cpp

+
+using namespace std;
+
+TEST_F(sys_call_test, stat)


Uhm, all the tests in this file seem to do the exact thing as drivers_test.
They run a syscall or a combination of syscalls and then assert that the event is here and something is happening in userspace... why don't we extend the class event_test to support some inspections in sinsp instead of introducing all these executables/classes + callbacks, WDYT?

Although some tests are similar in this first batch, the whole contribution process of the test suite we foresee can potentially cover use cases that diverge with the goals of the driver-level tests. It's hard to predict it at the moment, and refactor everything right now may be outside of the engineering bandwidth we currently have available. Plus, having this suite being its own standalone thing will allow us to converge it with the drivers one once we have a more high-level view than now. If this was a library code contribution, I'd agree with you of this being a strong blocker. But since this is a test-based contribution, I think it can provide tons of value as-is and we can postpone cleaning it up deeply as a secondary priority. Moreover @therealbobo is also curating each test while contributing them.

Andreagit97 · 2024-02-06T12:02:58Z

I left some initial comments but my main point is this one #1661 (comment)

LucaGuerra · 2024-02-06T16:43:26Z

When attempting to build the test suite I get

In file included from /usr/include/features.h:392,
                 from /usr/include/arpa/inet.h:21,
                 from /home/ubuntu/dev/falcosecurity/libs/test/libsinsp/test_helper.cpp:1:
/usr/include/features-time64.h:20:10: fatal error: bits/wordsize.h: No such file or directory
   20 | #include <bits/wordsize.h>
      |          ^~~~~~~~~~~~~~~~~
compilation terminated.

Am I missing some dependency?

therealbobo · 2024-02-06T17:04:42Z

This is probably related to the build of the is needed gcc with multilib support: there are some tests that run also for x86.

.github/workflows/e2e_ci.yml

LucaGuerra · 2024-02-07T10:34:42Z

Yep, I was missing gcc-multilib and g++-multilib. I would suggest documenting how to run these tests in the libs README (we can mark them as experimental)

Signed-off-by: Roberto Scolaro <[email protected]>

mrgian

Probably just a matter of setting the right directory

Hi @LucaGuerra
You are right, it's just a path issue caused by the new name of the test's directory.
I've tried running these tests with the correct paths and they are working 🎉

test/libsinsp_e2e/sys_call_test.cpp

LucaGuerra · 2024-02-20T13:53:47Z

I had tested this before and it seems like cmake had issues generating the binary, I'll clean and retry. Thank you for your efforts in making the e2e tests more compatible!

Signed-off-by: Roberto Scolaro <[email protected]>

LucaGuerra

After the latest fixes I was able to run the tests as specified in the readme. Thanks!

poiana · 2024-02-21T11:55:32Z

LGTM label has been added.

Git tree hash: 7d47e4c299049d641e824eed88027c0d654f1548

poiana · 2024-02-21T13:22:14Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: leogr, LucaGuerra, therealbobo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [LucaGuerra,leogr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

LucaGuerra · 2024-02-21T14:08:40Z

/unhold

I believe review comments have been addressed

poiana added release-note-none kind/feature New feature or request dco-signoff: yes area/CI area/tests labels Jan 30, 2024

poiana requested review from hbrueckner and leogr January 30, 2024 19:36

poiana added the size/XXL label Jan 30, 2024

leogr reviewed Jan 31, 2024

View reviewed changes

therealbobo force-pushed the new-e2e-tests branch from bd8ad48 to 2131dda Compare January 31, 2024 17:27

leogr previously approved these changes Feb 1, 2024

View reviewed changes

poiana assigned leogr Feb 1, 2024

poiana added the lgtm label Feb 1, 2024

poiana added the approved label Feb 1, 2024

poiana added the do-not-merge/hold label Feb 1, 2024

Andreagit97 reviewed Feb 6, 2024

View reviewed changes

therealbobo dismissed leogr’s stale review via 77e5a6b February 7, 2024 09:51

poiana removed the lgtm label Feb 7, 2024

poiana requested a review from leogr February 7, 2024 09:51

poiana removed approved labels Feb 7, 2024

LucaGuerra reviewed Feb 7, 2024

View reviewed changes

.github/workflows/e2e_ci.yml Outdated Show resolved Hide resolved

therealbobo force-pushed the new-e2e-tests branch 2 times, most recently from 7aa8ee4 to a242e36 Compare February 7, 2024 14:54

therealbobo added 15 commits February 16, 2024 09:56

fix(tests): fix minor unshare_ test issues

21a961a

Signed-off-by: Roberto Scolaro <[email protected]>

chore(tests): document libsinsp e2e tests

ec6ecf9

Signed-off-by: Roberto Scolaro <[email protected]>

chore(cmake): warning users on usage of libsinsp e2e tests

5b131c7

Signed-off-by: Roberto Scolaro <[email protected]>

fix(ci): remove typo

b736653

Signed-off-by: Roberto Scolaro <[email protected]>

fix(ci): correctly handle job matrix

bf9051c

Signed-off-by: Roberto Scolaro <[email protected]>

fix(ci): use correct e2e libsinsp cmake flag

38e989a

Signed-off-by: Roberto Scolaro <[email protected]>

fix(ci): change libsinsp_e2e dir

4be981a

Signed-off-by: Roberto Scolaro <[email protected]>

fix(tests): use new libsinsp_e2e comm

53318c2

Signed-off-by: Roberto Scolaro <[email protected]>

fix(tests): build test_helper_32 only on x86_64

5bf2e18

Signed-off-by: Roberto Scolaro <[email protected]>

fix(ci): remove s390x e2e tests

8aa9c0f

Signed-off-by: Roberto Scolaro <[email protected]>

fix(ci): exclude quotactl_ok test on arm64

a2e3246

Signed-off-by: Roberto Scolaro <[email protected]>

fix(tests): fix load of misaligned addresses

543a886

Signed-off-by: Roberto Scolaro <[email protected]>

fix(test): use builtin parameter type converter

15ab1cf

Signed-off-by: Roberto Scolaro <[email protected]>

fix(tests): invoke helper scripts with absolute paths

0009a01

Signed-off-by: Roberto Scolaro <[email protected]>

fix(tests): improved error handling with subprocesses

b23361c

Signed-off-by: Roberto Scolaro <[email protected]>

mrgian reviewed Feb 19, 2024

View reviewed changes

test/libsinsp_e2e/sys_call_test.cpp Outdated Show resolved Hide resolved

test/libsinsp_e2e/sys_call_test.cpp Outdated Show resolved Hide resolved

therealbobo force-pushed the new-e2e-tests branch from fa4f3ae to b23361c Compare February 20, 2024 09:58

fix(test/libsinsp_e2e): correctly manage cmake deps

82a1f3c

Signed-off-by: Roberto Scolaro <[email protected]>

LucaGuerra approved these changes Feb 21, 2024

View reviewed changes

poiana assigned LucaGuerra Feb 21, 2024

poiana added the lgtm label Feb 21, 2024

poiana added the approved label Feb 21, 2024

leogr approved these changes Feb 21, 2024

View reviewed changes

poiana removed the do-not-merge/hold label Feb 21, 2024

poiana merged commit d8bfc3c into falcosecurity:master Feb 21, 2024
42 checks passed

Andreagit97 added this to the 0.15.0 milestone Feb 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(tests): new e2e tests [1/N] #1661

feat(tests): new e2e tests [1/N] #1661

therealbobo commented Jan 30, 2024 •

edited by leogr

Loading

leogr left a comment

leogr left a comment

poiana commented Feb 1, 2024

Andreagit97 commented Feb 1, 2024

Andreagit97 Feb 6, 2024

therealbobo Feb 6, 2024

Andreagit97 Feb 6, 2024

Andreagit97 Feb 6, 2024

Andreagit97 Feb 6, 2024

Andreagit97 Feb 6, 2024

therealbobo Feb 6, 2024

Andreagit97 Feb 6, 2024

jasondellaluce Feb 6, 2024

Andreagit97 Feb 6, 2024

jasondellaluce Feb 6, 2024

Andreagit97 commented Feb 6, 2024

LucaGuerra commented Feb 6, 2024

therealbobo commented Feb 6, 2024

LucaGuerra commented Feb 7, 2024

mrgian left a comment •

edited

Loading

LucaGuerra commented Feb 20, 2024

LucaGuerra left a comment •

edited

Loading

poiana commented Feb 21, 2024

poiana commented Feb 21, 2024

LucaGuerra commented Feb 21, 2024

feat(tests): new e2e tests [1/N] #1661

feat(tests): new e2e tests [1/N] #1661

Conversation

therealbobo commented Jan 30, 2024 • edited by leogr Loading

leogr left a comment

Choose a reason for hiding this comment

leogr left a comment

Choose a reason for hiding this comment

poiana commented Feb 1, 2024

Andreagit97 commented Feb 1, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Andreagit97 commented Feb 6, 2024

LucaGuerra commented Feb 6, 2024

therealbobo commented Feb 6, 2024

LucaGuerra commented Feb 7, 2024

mrgian left a comment • edited Loading

Choose a reason for hiding this comment

LucaGuerra commented Feb 20, 2024

LucaGuerra left a comment • edited Loading

Choose a reason for hiding this comment

poiana commented Feb 21, 2024

poiana commented Feb 21, 2024

LucaGuerra commented Feb 21, 2024

therealbobo commented Jan 30, 2024 •

edited by leogr

Loading

mrgian left a comment •

edited

Loading

LucaGuerra left a comment •

edited

Loading