This repository contains code recipes to demonstrate how to access CAIDA's 100G passive pcap data using various methods.
Clone the repository into an environment that has a minimum of 2TB of disk space
git clone https://github.com/CAIDA/100g-passive-tools.git
In order to access CAIDA's swift server, a swift_config.ini file will have to be configured in the repository directory, using the AWS S3 credentials provided in the confirmation email (aws_access_key_id and aws_secret_access_key).
Note: One can use the following config file template to get started: swift_config-example.ini
Each one-directional anonymized pcap file, captured monthly, can reach up to approximately 1TB in size, so users will need more than 2TB of space to download the entire one-hour capture.
Refer to the following Wiki Page to troubleshoot commonly found errors.
Note: Refer to the requirements.txt file to install the dependencies needed for the recipes.
Note: This method requires a swift_config.ini file to be configured, and assumes you've installed boto3 locally.
Refer to the following recipes to download or list out files for a given capture, 100g-anon_download-objects.py and 100g-anon_list-objects.py
Note: This method requires your ~/.aws/credentials file to be configured, and assumes you've set up AWS CLI locally.
For installing instructions, reference AWS Documentation.
[default]
aws_access_key_id =
aws_secret_access_key =
List out files in the 100g-anon-pcap bucket/container
aws s3api list-objects --bucket 100g-anon-pcap --endpoint-url https://hermes.caida.org --output text
Note: --output {text|json|table}
Download a file in the 100g-anon-pcap bucket/container
aws s3api get-object --bucket 100g-anon-pcap --key monitor=100g-01/year=2024/mon=05/date=20240523-210000.UTC/20240523-210000.dira.stats --endpoint-url https://hermes.caida.org --output text 20240523-210000.dira.stats