zstd package not present in agent:production docker image

[hrimlinger]

Agent cannot start measurement because zstd package does not exits for agent:production package tag. This prevent any measurement to start and makes the whole process to fail when trying to insert csv file into clickhouse.

Agent logs:

INFO:iris:AGENT :: ddd8541d-b4f5-42ce-b163-e3e9bfcd0a47 :: Registering agent for 30 seconds
INFO:iris:AGENT :: ddd8541d-b4f5-42ce-b163-e3e9bfcd0a47 :: Setting agent state to AgentState.Working
INFO:iris:AGENT :: d7e95cdf-1af7-4688-9fcd-84e33b8b74ac :: ddd8541d-b4f5-42ce-b163-e3e9bfcd0a47 :: Launch measurement procedure
INFO:iris:AGENT :: d7e95cdf-1af7-4688-9fcd-84e33b8b74ac :: ddd8541d-b4f5-42ce-b163-e3e9bfcd0a47 :: Create local measurement directory
INFO:iris:AGENT :: d7e95cdf-1af7-4688-9fcd-84e33b8b74ac :: ddd8541d-b4f5-42ce-b163-e3e9bfcd0a47 :: Download CSV probe file locally
INFO:iris:AGENT :: ddd8541d-b4f5-42ce-b163-e3e9bfcd0a47 :: Registering agent for 30 seconds
INFO:iris:AGENT :: d7e95cdf-1af7-4688-9fcd-84e33b8b74ac :: ddd8541d-b4f5-42ce-b163-e3e9bfcd0a47 :: Probe file: next_round_1:0:0.csv.zst
INFO:iris:AGENT :: d7e95cdf-1af7-4688-9fcd-84e33b8b74ac :: ddd8541d-b4f5-42ce-b163-e3e9bfcd0a47 :: Round#1.0
INFO:iris:AGENT :: d7e95cdf-1af7-4688-9fcd-84e33b8b74ac :: ddd8541d-b4f5-42ce-b163-e3e9bfcd0a47 :: Requested probing rate: 100000
INFO:iris:AGENT :: d7e95cdf-1af7-4688-9fcd-84e33b8b74ac :: ddd8541d-b4f5-42ce-b163-e3e9bfcd0a47 :: Running zstd -cd iris_data/agent/targets/next_round_1:0:0.csv.zst | caracal --meta-round 1 --probing-rate 100000 --filter-from-prefix-file-excl statics/excluded_prefixes | zstd -c > iris_data/agent/results/d7e95cdf-1af7-4688-9fcd-84e33b8b74ac/results_1:0:0.csv.zst
/bin/sh: 1: zstd: not found
/bin/sh: 1: zstd: not found
caracal v0.15.1 (Release build)
[2023-11-24 16:36:03.346] [info] Reading from stdin, press CTRL+D to stop...
[2023-11-24 16:36:03.346] [info] caracal_id=21842 n_packets=1 probing_rate=100000 sniffer_wait_time=1 integrity_check=1 interface=eno1 rate_limiting_method=auto prefix_excl_file="statics/excluded_prefixes" round=1
[2023-11-24 16:36:03.346] [info] Loading excluded prefixes...
[2023-11-24 16:36:03.359] [info] sniffer_filter=(ip and icmp and (icmp[icmptype] = icmp-echoreply or icmp[icmptype] = icmp-timxceed or icmp[icmptype] = icmp-unreach)) or (ip6 and icmp6 and (icmp6[icmp6type] = icmp6-echoreply or icmp6[icmp6type] = icmp6-timeexceeded or icmp6[icmp6type] = icmp6-destinationunreach))
[2023-11-24 16:36:03.468] [info] Resolving the gateway MAC address...
[2023-11-24 16:36:03.503] [info] dst_mac=f4:cf:e2:d4:4b:00
[2023-11-24 16:36:03.503] [info] src_ip_v4=132.227.123.8 src_ip_v6=2001:660:3302:287b:b27b:25ff:feb7:abd8
INFO:iris:AGENT :: d7e95cdf-1af7-4688-9fcd-84e33b8b74ac :: ddd8541d-b4f5-42ce-b163-e3e9bfcd0a47 :: Upload probing statistics to Redis
INFO:iris:AGENT :: ddd8541d-b4f5-42ce-b163-e3e9bfcd0a47 :: Setting measurement statistics
INFO:iris:AGENT :: d7e95cdf-1af7-4688-9fcd-84e33b8b74ac :: ddd8541d-b4f5-42ce-b163-e3e9bfcd0a47 :: Upload results file into S3
INFO:iris:AGENT :: d7e95cdf-1af7-4688-9fcd-84e33b8b74ac :: ddd8541d-b4f5-42ce-b163-e3e9bfcd0a47 :: Empty local results directory
INFO:iris:AGENT :: d7e95cdf-1af7-4688-9fcd-84e33b8b74ac :: ddd8541d-b4f5-42ce-b163-e3e9bfcd0a47 :: Empty local targets directory
INFO:iris:AGENT :: ddd8541d-b4f5-42ce-b163-e3e9bfcd0a47 :: Setting agent state to AgentState.Idle
INFO:iris:AGENT :: ddd8541d-b4f5-42ce-b163-e3e9bfcd0a47 :: Registering agent for 30 seconds

Worker logs:

[2023-11-24 16:37:29,037] [PID 33] [Thread-4] [dramatiq.middleware.retries.Retries] [INFO] Retrying message '006aae48-d0e0-4ebe-a8f8-90ec536e735b' in 60577 milliseconds.
	
RuntimeError: coroutine raised StopIteration
	
    await clickhouse.insert_csv(measurement_uuid, agent_uuid, results_filepath)
	
  File "/app/iris/worker/inner_pipeline/yarrp.py", line 45, in yarrp_inner_pipeline
	
    n_probes_to_send = await inner_pipeline_for_tool[tool](**inner_pipeline_kwargs)
	
  File "/app/iris/worker/outer_pipeline.py", line 92, in outer_pipeline
	
    result = await outer_pipeline(
	
  File "/app/iris/worker/watch.py", line 135, in watch_measurement_agent_with_deps
	
    await watch_measurement_agent_with_deps(
	
  File "/app/iris/worker/watch.py", line 52, in watch_measurement_agent_
	
    return future.result()
	
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
	
    return loop.run_until_complete(main)
	
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
	
    asyncio.run(
	
  File "/app/iris/worker/watch.py", line 33, in watch_measurement_agent
	
    return self.fn(*args, **kwargs)
	
  File "/app/.venv/lib/python3.10/site-packages/dramatiq/actor.py", line 182, in __call__
	
    res = actor(*message.args, **message.kwargs)
	
  File "/app/.venv/lib/python3.10/site-packages/dramatiq/worker.py", line 485, in process_message
	
Traceback (most recent call last):
	
The above exception was the direct cause of the following exception:
	
StopIteration
	
    next(f)
	
  File "/app/iris/commons/filesplit.py", line 88, in split_compressed_file
	
    split_compressed_file(
	
  File "/app/iris/commons/clickhouse.py", line 98, in insert_csv
	
Traceback (most recent call last):

[hrimlinger]

help, I cannot create a branch

[maxmouchet]

Fixed by #167.