Skip to content

LetsMT

Jump to bottom
tiedemann edited this page Aug 24, 2018 · 6 revisions

Setting up the resource repository from the LetsMT! project

Basic setup

  • create a virtual machine on cPouta with Ubuntu 14.04
  • connect with a floating IP and find the DNS name with dig -x xxx.xxx.xx.xxx
  • download the repository software
  • set HOSTNAME with the fully qualified DNS name, for example, vm1637.kaj.pouta.csc.fi
  • install with make
sudo apt-get install git
git clone https://github.com/Helsinki-NLP/LetsMT-repository.git
cd LetsMT-repository.git
export HOSTNAME=vm1637.kaj.pouta.csc.fi
sudo make install

Trouble shooting

  • querying TokyoCabinet database from command line, for example search all files with status=imported
sudo service ttservctl_meta stop
tctmgr search /var/lib/letsmt/metadata.tct status eq imported
sudo service ttservctl_meta start
  • Debugging

edit log4perl config: /usr/local/etc/repository/log4perl.conf and activate DEBUG level

log4perl.logger=DEBUG, AppDebug

restart apache and monitor logfiles in /var/log/letsmt/www-data_debug.log Edit code LetsMT code with more calls to get_logger(__PACKAGE__)->info or get_logger(__PACKAGE__)->debug. Don't forget to install the new code and restart with make code-update

SLURM instead of grid engine (obsolete)

  • SLURM is now default in the repository software! The information below is just for keeping the info about its setup.
sudo apt-get install slurm-llnl
sudo apt-get install munge
sudo /usr/sbin/create-munge-key
sudo service munge start
sudo echo 'CgroupAutomount=yes' > /etc/slurm-llnl/cgroup.conf

ControlMachine=letsmt
AuthType=auth/munge
CryptoType=crypto/munge
MpiDefault=none
ProctrackType=proctrack/cgroup
ReturnToService=1
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmctldPort=6817
SlurmdPidFile=/var/run/slurmd.pid
SlurmdPort=6818
SlurmdSpoolDir=/var/spool/slurmd
SlurmUser=root
StateSaveLocation=/var/spool/slurm.state
SwitchType=switch/none
TaskPlugin=task/affinity
TaskPluginParam=Sched
InactiveLimit=0
KillWait=30
MinJobAge=300
SlurmctldTimeout=120
SlurmdTimeout=300
Waittime=0
FastSchedule=1
SchedulerType=sched/backfill
SelectType=select/cons_res
SelectTypeParameters=CR_Core
AccountingStorageType=accounting_storage/none
AccountingStoreJobComment=YES
ClusterName=cluster
JobCompType=jobcomp/none
JobAcctGatherFrequency=30
JobAcctGatherType=jobacct_gather/none
SlurmctldDebug=3
SlurmdDebug=3
NodeName=letsmt CPUs=1 State=UNKNOWN
PartitionName=debug Nodes=letsmt Default=YES MaxTime=1440 State=UP
  • start slurm daemons:
sudo service slurm-llnl start

sudo slurmctld -Dvvv
sudo slurmd -Dvvv

Links and documentation

Releases

Clone this wiki locally