Skip to content

Commit 5315234

Browse files
andreistan26andreistan26
committed
assignments/parallel-firewall: Add parallel firewall assignment
Add parallel firewall assignment for the compute chapter. The goal is to introduce students to parallel programming with a single producer multiple consumer problem that uses a thread-safe ring buffer to transfer data between threads. Signed-off-by: Andrei Stan <[email protected]> Signed-off-by: Anton-Fabian Patras <[email protected]>
1 parent 5e4424b commit 5315234

23 files changed

+1551
-1
lines changed

‎config.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -303,7 +303,7 @@ docusaurus:
303303
subsections:
304304
- Mini Libc/: chapters/software-stack/libc/projects/mini-libc/
305305
- Memory Allocator/: content/assignments/memory-allocator/
306-
- Parallel Graph/: content/assignments/parallel-graph/
306+
- Parallel Firewall/: content/assignments/parallel-firewall/
307307
- Mini Shell/: content/assignments/minishell/
308308
- Asynchronous Web Server/: content/assignments/async-web-server/
309309
- Exams:

‎content/assignments/parallel-firewall/README.md

+214
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,214 @@
1+
# Parallel Firewall
2+
3+
## Objectives
4+
5+
- Learn how to design and implement parallel programs
6+
- Get experienced at utilizing the POSIX threading API
7+
- Learn how to convert a serial program into a parallel one
8+
9+
## Statement
10+
11+
A firewall is a program that checks network packets against a series of filters which provide a decision regarding dropping or allowing the packets to continue to their intended destination.
12+
13+
**In a real setup**, the network card will receive real packets (e.g. packets having [`Ethernet`](https://en.wikipedia.org/wiki/Ethernet_frame), [`IP`](https://en.wikipedia.org/wiki/IPv4) headers plus payload) from the network and will send them to the firewall for processing.
14+
The firewall will decide if the packets are to be dropped or not and, if not, passes them further.
15+
16+
**In this assignment**, instead of real network packets, we'll deal with made up packets consisting of a made up source (a number), a made up destination (also a number), a timestamp (also a number) and some payload.
17+
And instead of the network card providing the packets, we'll have a **producer thread** creating these packets.
18+
19+
The created packets will be inserted into a [`circular buffer`](https://en.wikipedia.org/wiki/Circular_buffer), out of which **consumer threads** (which implement the firewall logic) will take packets and process them in order to decide whether they advance to the destination.
20+
21+
The result of this processing is a log file in which the firewall will record the decision taken (PASS or DROP) for each packet, along with other information such as timestamp.
22+
23+
The purpose of this assignment is to:
24+
25+
- implement the circular buffer, along with synchronization mechanisms for it to work in a multithreaded program
26+
27+
- implement the consumer threads, which consume packets and process them
28+
29+
- provide the log file containing the result of the packet processing
30+
31+
## Support Code
32+
33+
The support code consists of the directories:
34+
35+
- `src/` contains the skeleton for the parallelized firewall and the already implemented serial code in `src/serial.c`.
36+
You will have to implement the missing parts marked as `TODO`
37+
38+
- `utils/` contains utility files used for debugging and logging.
39+
40+
- `tests/` contains tests used to validate and grade the assignment.
41+
42+
## Implementation
43+
44+
### Firewall Threads
45+
46+
In order to parallelize the firewall we have to distribute the packets to multiple threads.
47+
The packets will be added to a shared data structure (visible to all threads) by a `producer` thread and processed by multiple `consumer` threads.
48+
Each `consumer` thread picks a packet from the shared data structure, checks it against the filter function and writes the packet hash together with the drop/accept decision to a log file.
49+
`consumer` threads stop waiting for new packets from the `producer` thread and exit when the `producer` thread closes the connection to the shared data structure.
50+
51+
The `consumer` threads **must not do any form of busy waiting**.
52+
When there are new packets that need to be handled, the `consumer` threads must be **notified**.
53+
**Waiting in a `while()` loop or sleeping is not considered a valid synchronization mechanism and points will be deducted.**
54+
55+
Implement the `consumer` related functions marked with `TODO` in the `src/consumer.c` file.
56+
**The number of consumer threads will be passed as the 3rd command-line argument**
57+
58+
### Ring Buffers
59+
60+
A ring buffer (or a circular buffer) is a data structure that stores its elements in a circular fixed size array.
61+
One of the advantages of using such a data structure as opposed to an array is that it acts as a FIFO, without the overhead of moving the elements to the left as they are consumed.
62+
Thus, the shared ring buffer offers the following fields:
63+
64+
- `write_pos` index in the buffer used by the `producer` thread for appending new packets.
65+
- `read_pos` index in the buffer used by the `consumer` threads to pick packets.
66+
- `cap` the size of the internal buffer.
67+
- `data` pointer to the internal buffer.
68+
69+
Apart from these fields you have to add synchronization primitives in order to allow multiple threads to access the ring buffer in a deterministic manner.
70+
You can use mutexes, semaphores, conditional variables and other synchronization mechanisms offered by the `pthread` library.
71+
72+
You will have to implement the following interface for the ring buffer:
73+
74+
- `ring_buffer_init()`: initialize the ring buffer (allocate memory and synchronization primitives).
75+
- `ring_buffer_enqueue()`: add elements to the ring buffer.
76+
- `ring_buffer_dequeue()`: remove elements from the ring buffer.
77+
- `ring_buffer_destroy()`: free up the memory used by the ring_buffer.
78+
- `ring_buffer_stop()`: finish up using the ring buffer for the calling thread.
79+
80+
### Log File
81+
82+
The output of the firewall will be a log file with the rows containing the firewall's decision, the hash of the packet and its timestamp.
83+
The actual format can be found in the serial implementation (at `src/serial.c`).
84+
85+
When processing the packets in parallel the threads will finish up the work in a non deterministic order.
86+
The packet processing functions are already implemented in `src/packet.c`
87+
88+
We would like the logs to be sorted by the packet timestamp, the order that they came in from the producer.
89+
Thus, the `consumers` should insert the packet information to the log file such as the result is ordered by timestamp.
90+
The printing format can be found in `./src/serial.c`
91+
92+
The logs must be written to the file in ascending order during packet processing.
93+
**Sorting the log file after the consumer threads have finished processing is not considered a valid synchronization mechanism and points will be deducted.**
94+
95+
## Operations
96+
97+
### Building
98+
99+
To build both the serial and the parallel versions, run `make` in the `src/` directory:
100+
101+
```console
102+
student@so:~/.../content/assignments/parallel-firewall$ cd src/
103+
104+
student@so:~/.../assignments/parallel-firewall/src$ make
105+
```
106+
107+
That will create the `serial` and `firewall` binaries.
108+
109+
## Testing and Grading
110+
111+
Testing is automated.
112+
Tests are located in the `tests/` directory.
113+
114+
To test and grade your assignment solution, enter the `tests/` directory and run `grade.sh`.
115+
116+
```console
117+
student@so:~/.../content/assignments/parallel-firewall$ cd tests/
118+
```
119+
120+
```console
121+
student@so:~/.../content/assignments/parallel-firewall/tests$ ./grade.sh
122+
```
123+
124+
Note that this requires linters being available.
125+
The easiest way to test the project is to use a Docker-based setup with everything installed and configured (see the [README.checker.md](README.checker.md) file for instructions).
126+
127+
To create the tests, run:
128+
129+
```console
130+
student@so:~/.../content/assignments/parallel-firewall/tests$ make check
131+
```
132+
133+
To remove the tests, run:
134+
135+
```console
136+
student@so:~/.../content/assignments/parallel-firewall/tests$ make distclean
137+
```
138+
139+
When using `grade.sh` you will get a maximum of 90/100 points for general correctness and a maximum of 10/100 points for coding style.
140+
141+
### Restrictions
142+
143+
- Threads must yield the cpu when waiting for empty/full buffers i.e. not doing `busy waiting`.
144+
- The logs must be written as they are processed and not after the processing is done, in ascending order by the timestamp.
145+
- The number of running threads must be at least `num_consumers + 1`, where `num_consumers` is the 3rd command-line argument of the `firewall` binary.
146+
147+
### Grades
148+
149+
- 10 points are awarded for a single consumer solution that also implements the ring buffer
150+
- 50 points are awarded for a multi consumer solution
151+
- 30 points are awarded for a multi consumer solution that writes the logs in the sorted manner (bearing in mind the above restrictions)
152+
153+
### Running the Checker
154+
155+
Each test is worth a number of points.
156+
The maximum grade is `90`.
157+
158+
A successful run will show the output:
159+
160+
```console
161+
student@so:~/.../assignments/parallel-firewall/tests$ make check
162+
[...]
163+
Test [ 10 packets, sort False, 1 thread ] ...................... passed ... 3
164+
Test [ 1,000 packets, sort False, 1 thread ] ...................... passed ... 3
165+
Test [20,000 packets, sort False, 1 thread ] ...................... passed ... 4
166+
Test [ 10 packets, sort True , 2 threads] ...................... passed ... 5
167+
Test [ 10 packets, sort True , 4 threads] ...................... passed ... 5
168+
Test [ 100 packets, sort True , 2 threads] ...................... passed ... 5
169+
Test [ 100 packets, sort True , 4 threads] ...................... passed ... 5
170+
Test [ 1,000 packets, sort True , 2 threads] ...................... passed ... 5
171+
Test [ 1,000 packets, sort True , 4 threads] ...................... passed ... 5
172+
Test [10,000 packets, sort True , 2 threads] ...................... passed ... 5
173+
Test [10,000 packets, sort True , 4 threads] ...................... passed ... 5
174+
Test [20,000 packets, sort True , 2 threads] ...................... passed ... 5
175+
Test [20,000 packets, sort True , 4 threads] ...................... passed ... 5
176+
Test [ 1,000 packets, sort False, 4 threads] ...................... passed ... 5
177+
Test [ 1,000 packets, sort False, 8 threads] ...................... passed ... 5
178+
Test [10,000 packets, sort False, 4 threads] ...................... passed ... 5
179+
Test [10,000 packets, sort False, 8 threads] ...................... passed ... 5
180+
Test [20,000 packets, sort False, 4 threads] ...................... passed ... 5
181+
Test [20,000 packets, sort False, 8 threads] ...................... passed ... 5
182+
183+
Checker: 90/100
184+
```
185+
186+
### Running the Linters
187+
188+
To run the linters, use the `make lint` command in the `tests/` directory:
189+
190+
```console
191+
student@so:~/.../assignments/parallel-firewall/tests$ make lint
192+
[...]
193+
cd .. && checkpatch.pl -f checker/*.sh tests/*.sh
194+
[...]
195+
cd .. && cpplint --recursive src/ tests/ checker/
196+
[...]
197+
cd .. && shellcheck checker/*.sh tests/*.sh
198+
```
199+
200+
Note that the linters have to be installed on your system: [`checkpatch.pl`](https://.com/torvalds/linux/blob/master/scripts/checkpatch.pl), [`cpplint`](https://github.com/cpplint/cpplint), [`shellcheck`](https://www.shellcheck.net/).
201+
They also need to have certain configuration options.
202+
It's easiest to run them in a Docker-based setup with everything configured.
203+
204+
### Fine-Grained Testing
205+
206+
Input tests cases are located in `tests/in/` and are generated by the checker.
207+
The expected results are generated by the checker while running the serial implementation.
208+
If you want to run a single test, use the below commands while in the `src/` directory:
209+
210+
```console
211+
student@so:~/.../assignments/parallel-firewall/src$ ./firewall ../tests/in/test_<num_packets>.in <output_file> <number_of_consumers>
212+
```
213+
214+
Results provided by the serial and parallel implementation must be the same for the test to successfully pass.

‎content/assignments/parallel-firewall/src/.gitignore

Whitespace-only changes.

‎content/assignments/parallel-firewall/src/Makefile

+32
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
BUILD_DIR := build
2+
UTILS_PATH ?= ../utils
3+
CPPFLAGS := -I$(UTILS_PATH)
4+
CFLAGS := -Wall -Wextra
5+
6+
CFLAGS += -ggdb -O0
7+
LDLIBS := -lpthread
8+
9+
SRCS:= ring_buffer.c producer.c consumer.c packet.c $(UTILS_PATH)/log/log.c
10+
HDRS := $(patsubst %.c,%.h,$(SRCS))
11+
OBJS := $(patsubst %.c,%.o,$(SRCS))
12+
13+
.PHONY: all pack clean always
14+
15+
all: firewall serial
16+
17+
firewall: $(OBJS) firewall.o
18+
$(CC) $(CPPFLAGS) $(CFLAGS) -o $@ $^ $(LDLIBS)
19+
20+
serial: $(OBJS) serial.o
21+
$(CC) $(CPPFLAGS) $(CFLAGS) -o $@ $^ $(LDLIBS)
22+
23+
$(UTILS_PATH)/log/log.o: $(UTILS_PATH)/log/log.c $(UTILS_PATH)/log/log.h
24+
$(CC) $(CPPFLAGS) $(CFLAGS) -c -o $@ $<
25+
26+
pack: clean
27+
-rm -f ../src.zip
28+
zip -r ../src.zip *
29+
30+
clean:
31+
-rm -f $(OBJS) serial.o firewall.o
32+
-rm -f firewall serial

‎content/assignments/parallel-firewall/src/consumer.c

+35
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
// SPDX-License-Identifier: BSD-3-Clause
2+
3+
#include <pthread.h>
4+
#include <fcntl.h>
5+
#include <unistd.h>
6+
7+
#include "consumer.h"
8+
#include "ring_buffer.h"
9+
#include "packet.h"
10+
#include "utils.h"
11+
12+
void consumer_thread(so_consumer_ctx_t *ctx)
13+
{
14+
/* TODO: implement consumer thread */
15+
(void) ctx;
16+
}
17+
18+
int create_consumers(pthread_t *tids,
19+
int num_consumers,
20+
struct so_ring_buffer_t *rb,
21+
const char *out_filename)
22+
{
23+
(void) tids;
24+
(void) num_consumers;
25+
(void) rb;
26+
(void) out_filename;
27+
28+
for (int i = 0; i < num_consumers; i++) {
29+
/*
30+
* TODO: Launch consumer threads
31+
**/
32+
}
33+
34+
return num_consumers;
35+
}

‎content/assignments/parallel-firewall/src/consumer.h

+20
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
/* SPDX-License-Identifier: BSD-3-Clause */
2+
3+
#ifndef __SO_CONSUMER_H__
4+
#define __SO_CONSUMER_H__
5+
6+
#include "ring_buffer.h"
7+
#include "packet.h"
8+
9+
typedef struct so_consumer_ctx_t {
10+
struct so_ring_buffer_t *producer_rb;
11+
12+
/* TODO: add synchronization primitives for timestamp ordering */
13+
} so_consumer_ctx_t;
14+
15+
int create_consumers(pthread_t *tids,
16+
int num_consumers,
17+
so_ring_buffer_t *rb,
18+
const char *out_filename);
19+
20+
#endif /* __SO_CONSUMER_H__ */

‎content/assignments/parallel-firewall/src/firewall.c

+78
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
// SPDX-License-Identifier: BSD-3-Clause
2+
3+
#include <stdio.h>
4+
#include <stdlib.h>
5+
#include <sys/types.h>
6+
#include <sys/wait.h>
7+
#include <pthread.h>
8+
9+
#include "ring_buffer.h"
10+
#include "consumer.h"
11+
#include "producer.h"
12+
#include "log/log.h"
13+
#include "packet.h"
14+
#include "utils.h"
15+
16+
#define SO_RING_SZ (PKT_SZ * 1000)
17+
18+
pthread_mutex_t MUTEX_LOG;
19+
20+
void log_lock(bool lock, void *udata)
21+
{
22+
pthread_mutex_t *LOCK = (pthread_mutex_t *) udata;
23+
24+
if (lock)
25+
pthread_mutex_lock(LOCK);
26+
else
27+
pthread_mutex_unlock(LOCK);
28+
}
29+
30+
void __attribute__((constructor)) init()
31+
{
32+
pthread_mutex_init(&MUTEX_LOG, NULL);
33+
log_set_lock(log_lock, &MUTEX_LOG);
34+
}
35+
36+
void __attribute__((destructor)) dest()
37+
{
38+
pthread_mutex_destroy(&MUTEX_LOG);
39+
}
40+
41+
int main(int argc, char **argv)
42+
{
43+
so_ring_buffer_t ring_buffer;
44+
int num_consumers, threads, rc;
45+
pthread_t *thread_ids = NULL;
46+
47+
if (argc < 4) {
48+
fprintf(stderr, "Usage %s <input-file> <output-file> <num-consumers:1-32>\n", argv[0]);
49+
exit(EXIT_FAILURE);
50+
}
51+
52+
rc = ring_buffer_init(&ring_buffer, SO_RING_SZ);
53+
DIE(rc < 0, "ring_buffer_init");
54+
55+
num_consumers = strtol(argv[3], NULL, 10);
56+
57+
if (num_consumers <= 0 || num_consumers > 32) {
58+
fprintf(stderr, "num-consumers [%d] must be in the interval [1-32]\n", num_consumers);
59+
exit(EXIT_FAILURE);
60+
}
61+
62+
thread_ids = calloc(num_consumers, sizeof(pthread_t));
63+
DIE(thread_ids == NULL, "calloc pthread_t");
64+
65+
/* create consumer threads */
66+
threads = create_consumers(thread_ids, num_consumers, &ring_buffer, argv[2]);
67+
68+
/* start publishing data */
69+
publish_data(&ring_buffer, argv[1]);
70+
71+
/* TODO: wait for child threads to finish execution*/
72+
(void) threads;
73+
74+
free(thread_ids);
75+
76+
return 0;
77+
}
78+

0 commit comments

Comments
 (0)