Home

Welcome to the wa-tools wiki!

Heritrix initiation

scripts/pre-harvest/prepare-job-dir.sh
pscp -h crawlers seeds.txt, crawler-beans.cxml /opt/heritrix/jobs/job-dir/
check for /opt/heritrix/shared-config
pssh -h crawlers -i su heritrix -c '/opt/heritrix/start3.sh'
pssh -h crawlers -i 'hostname=$(hostname); curl -sslv3 -v -d "action=add&addpath=/opt/heritrix/jobs/job-dir" -k -u user:pass --anyauth --location https://$hostname:7778/engine'