This repository contains the linux files to upload minION data to minoTour.
There are two ways of monitoring minION data using minoTour. The first is by directly processing the fast5 read files containing basecalled minION data. The second is by directly monitoring the activity of the minKNOW software itself directly. These two options can be combined to monitor both reads and minKNOW activity. This helps give remote real time feedback on minION activity. The speed of data analysis is affected by many things, but by enabling direct monitoring of minKNOW, some of these effects can be mitigated.
Brief running instructions are provided below. Further details can be found by watching the tutorial videos available on minoTour.
MinUP assumes the following ports and connections are open:
port 3306 outgoing to minotour.nottingham.ac.uk
port 80 to minotour.nottingham.ac.uk
port 8080 to minotour.nottingham.ac.uk
minUP checks for new versions. We strongly recommend updating to the latest version whenever it is released.
Changes are noted in the 'changes.txt' document.
minUP can align reads to a reference in real time. However - you are cautioned not to use this for references larger than 10 Mb at this time. We are working on methods to track real time aligning on larger genomes in the future. At this time, algining to large genomes will significantly slow down the speed of data upload to minoTour.
minUP and minoTour assume that you are uploading data as they are generated by minKNOW and any of the available basecallers (metrichor, minKNOW, nanonet or albacore). If this is not the case, then the time dependent data sets that are reported in minoTour will only be correct at the end of a run.
Files are uploaded using the program minUP.py . An optional GUI wrapper is included - minUPgui.py.
To run the gui, type:
python minUPgui.py
from the command line in the minUP folder.
This will launch the minUP graphical user interface (GUI). The fields in the GUI will be retained from launch to launch, so once you have configured it, you only need to change individual commands each time you run it. The gui provides a simpler method of running the command line script and provides access to the most common options.
To run the command line script, type:
python minUP.py
Help is available by typing:
python minUP.py -h
If all this is too long to read, a simple command for uploading data to minoTour in Nottingham would be:
python minUP.py -w /path/to/basecalled/fast5/ -u yourusername -dbu yourusername -pw yourpassword -dbh minotour.nottingham.ac.uk
If you wish to align to a reference:
python minUP.py -w /path/to/basecalled/fast5/ -u yourusername -dbu yourusername -pw yourpassword -dbh minotour.nottingham.ac.uk -f /path/to/your/reference.fasta -bwa
More detailed notes on all the options are available below.
If you are using the GUI, these options are set within the GUI interface itself.
We note that this is a beta function - it is still in development. It may break with minKNOW updates and changes. It may break with lots of users. It may break at any time.
To enable real time monitoring of minKNOW you need to add two extra flags to your minUP command. These are:
-ip
-pin
-ip is the IP address of the machine running minKNOW. Typically this is 127.0.0.1. Advanced users may wish to run this software on another machine - but note you need extremely permissive firewall settings to do so. -pin is a security pin number. This is required for any user of minoTour to set automated control features.
So - to run real time monitoring enter (but please don't use the pin 1234!):
python minUP.py -w /path/to/basecalled/fast5/ -u yourusername -dbu yourusername -pw yourpassword -dbh minotour.nottingham.ac.uk -ip 127.0.0.1 -pin 1234
It is possible to just monitor minKNOW using our software without processing read files. This is done using:
python mincontrol.py
No gui interface is available for this option at this time.
A run command for mincontrol.py to connect to minotour.nottingham.ac.uk is (again - please don't use 1234!):
python mincontrol.py -ws minotour.nottingham.ac.uk -dbh minotour.nottingham.ac.uk -dbu yourusername -pw yourpassword -pin 1234 -ip 127.0.0.1
The command line options are described below:
-h, --help show this help message and exit
-w WATCHDIR, --watch-dir WATCHDIR
The path to the folder containing the downloads
directory with fast5 reads to analyse - e.g.
C:\data\minion\downloads (for windows).
-downloads DOWNLOADS, --downloads-dir DOWNLOADS
The path to the folder containing the downloads
directory with fast5 reads to analyse - e.g.
C:\data\minion\downloads (for windows).
-uploaded UPLOADED, --uploaded-dir UPLOADED
The path to the folder containing the uploaded
directory with fast5 reads to analyse - e.g.
C:\data\minion\downloads (for windows).
-dbh DBHOST, --mysql-host DBHOST
The location of the MySQL database. default is
'localhost'.
-dbu DBUSERNAME, --mysql-username DBUSERNAME
The MySQL username with create & write privileges on
MinoTour.
-dbp DBPORT, --mysql-port DBPORT
The MySQL port number, else the default port '3306' is
used.
-pw DBPASS, --mysql-password DBPASS
The password for the MySQL username with permission to
upload to MinoTour.
-f REF_FASTA, --align-ref-fasta REF_FASTA
The reference fasta file to align reads against. Using
this option enables read alignment provided LastAl and
LastDB are in the path. Leaving this entry blank will
upload the data without any alignment. To use multiple
reference fasta files input them as one text string
seperated by commas (no white spaces)
-b, --align-batch-fasta
Align reads in batch processing mode. Assumes the
watch-dir (-w) is pointed at a directory with one or
more "downloads" folders below it somewhere. Each
"downloads" folder can have a subfolder named
"reference" containing the fasta file(s) to align the
fast5 reads in the corresponding "downloads" folder to
-procs PROCS, --proc_num PROCS
The number of processors to run this on.
-u MINOTOURUSERNAME, --minotour-username MINOTOURUSERNAME
The MinoTour username with permissions to upload data.
-s VIEW_USERS, --minotour-sharing-usernames VIEW_USERS
A comma seperated list (with no whitespaces) of other
MinoTour users who will also be able to view the data.
-o FLOWCELL_OWNER, --flowcell-owner FLOWCELL_OWNER
The name of the minion owner. 'minionowner' is the
default
-r RUN_NUM, --run-number RUN_NUM
The run number of the flowcell. The default value is
0.
-c, --commment-true Add a comment to the comments field for this run.
Follow the prompt once minup starts .
-last, --last-align-true
align reads with LAST
-bwa, --bwa-align-true
align reads with BWA
-bwa-opts BWA_OPTIONS, --bwa-align-options BWA_OPTIONS
BWA options: Enter a comma-seperated list of BWA
options without spaces or '-' characters e.g. k12,T0
-last-opts LAST_OPTIONS, --last-align-options LAST_OPTIONS
LAST options: Enter a comma-seperated list of LAST
options without spaces or '-' characters e.g.
s2,T0,Q0,a1
-pin PIN, --security-pin PIN
pin number for remote control
-ip IP_ADDRESS, --ip-address IP_ADDRESS
Used for remote control with option '-pin'. Provide IP
address of the computer running minKNOW. The default
is the IP address of this computer
-t, --insert-tel-true
DEPRECATED
-pre, --insert-pre-true
Process raw reads prior to basecalling by metrichor.
-prealign, --align-pre-true
Align raw reads prior to basecalling by metrichor.
-d, --drop-db-true Drop existing database if it already exists.
-v VERBOSE, --verbose-true VERBOSE
Print detailed messages while processing files.
-name CUSTOM_NAME, --name-custom CUSTOM_NAME
Provide a modifier to the database name. This allows
you to upload the same dataset to minoTour more than
once. The additional string should be as short as
possible.
-cs ADDED_COMMENT [ADDED_COMMENT ...], --commment-string ADDED_COMMENT [ADDED_COMMENT ...]
Add given string to the comments field for this run
-largerRef, --larger-reference
DEPRECATED
-customup, --custom-upload
Stop minUP when cached has remained zero for more than
30 seconds.
-qScale, --scale-query
DEPRECATED
-qryStartEnd, --prealign-query-start-end-only
DEPRECATED
-useHdfTimes, --use-hdf5_file-timestamps
Sort read files using hdt file timestamps rather than
file modified times.
-indexToRefDir, --store-indexes-in-ref-folder
Store the bwa/glast index files in the reference
genome folder
-debug, --disable-MyHandler-try-except
Disable MyHandler try excep
-standalone, --standalone-mode
DEPRECATED
-abs ALIGNERBLOCKSIZE, --aligner_block_size ALIGNERBLOCKSIZE
Number of reads to align on each call to the aligner
-ver, --version show program's version number and exit
-h, --help show this help message and exit
-ws WSHOST, --websocket-host WSHOST
The location of the minotour server being connected
to.
-dbh DBHOST, --mysql-host DBHOST
The location of the MySQL database. default is
'localhost'.
-dbu DBUSERNAME, --mysql-username DBUSERNAME
The MySQL username with create & write privileges on
MinoTour.
-dbp DBPORT, --mysql-port DBPORT
The MySQL port number, else the default port '3306' is
used.
-pw DBPASS, --mysql-password DBPASS
The password for the MySQL username with permission to
upload to MinoTour.
-pin PIN, --security-pin PIN
This is a security feature to prevent unauthorised
remote control of a minION device. You need to provide
a four digit pin number which must be entered on the
website to remotely control the minION.
-ip IP, --ip-address IP
The IP address of the minKNOW machine.
-v, --verbose Display debugging information.
To install on Linux, the following packages are required. We assume python2.7 and pip are installed.
System packages required include:
python python-setuptools python-dev build-essential libmysqlclient-dev python-wxtools python-wxgtk3.0
Python Modules required are:
pip install watchdog MySQL-Python configargparse psutil BioPython numpy progressbar ws4py thrift dictdiffer twisted autobahn gooey h5py xmltodict dateutil