-
Notifications
You must be signed in to change notification settings - Fork 1
3. Reference
Queue API is an integral part of Styk.tv virtual host. Access to the API is through the SSH interface. It is a special shell designed to execute commands and get output in XML format. It is a simple way for a non-human to interact with a host.
Standard way of starting and stopping the node is to use init.d script named stock-footage-node which supports typical commands like start,stop,restart and status. Another way is to run node.py directly with appropriate parameter:
start - starts the node if not running, do nothing if already running
stop - stop the node if running
check - return 0 if node is running or 3 if not
run - run the node without detaching from console and redirecting output. Used as debug toolNode is normally run as root. It immediately drops permissions and switches to user 'node'.
queue.py can be used to modify workflows in the queue. Available commands:
remove <workflow> - removes workflow with given ID
retry <workflow> - sets workflow status back to 'pending'. Returns an error if workflow has not completed (successfully or with error) yet
clear <status> - removes all workflows with given status. Returns number of removed workflows.
clear all - completely clears the queue. Returns 0.Call queue.py list. Output will contain all workflows (past, current and pending) along with their statuses. You can also filter output by status and to omit all workflows before specified one.
Syntax: queue.py list [<numeric status>|All] [<task guid>]
If task guid is provided, queue.py will find workflow containing this task and remove all previous workflows from output
Example:
<?xml version="1.0" ?>
<queue>
<workflow dateCompleted="Thu Jul 14 05:50:42 2011" dateStart="Thu Jul 14 05:50:32 2011" guid="a" progress="100" status="2">
<task action="MOVE" dateCompleted="Thu Jul 14 05:50:42 2011" dateStart="Thu Jul 14 05:50:32 2011" destAssetItem="asset1" destStore="63eabc6ff34c42459dca17e622123f85" guid="b" progress="100" srcAssetItem="asset1" srcStore="722ffa90e2e54751ac8f7a072eea3b0b" status="2"/>
</workflow>
</queue>There is also way to get information about node status (running or not), disk usage, cpu usage and memory. It is done by calling heartbeat.py. Without parameter it will include all information except for cpu usage. To include cpu usage call it with parameter --cpu. Example:
<?xml version="1.0" ?>
<status>
<node failed="0" finished="0" pending="0" processing="0" running="True"/>
<disk freespace="4270694400" guid="dc5785794cc140ea9c8ba12b4c17a881" online="True"/>
<memory avail_physical="332828672" avail_virtual="1911402496" total_physical="4160036864" total_virtual="2154983424"/>
<cpu usage="18.1"/>
</status>Workflow or task status:
pending=0
working=1
finished=2
error=3Node attributes running="True" and processing="0" means that node is running, but idle. If number of processing workflow is higher, then node is busy.
ENCODE - encodes an asset into different format using one of defined encoders. Both source and destination stores have to be local. Parameters:
srcStore, srcAssetItem, srcAssetItemType - source file. Source asset can also be set of images instead of video file. In this case srcAssetItem should be in form of 'multi:<format>:<fps>' just like for animated watermarks. See Watermarks section for details.
encoder - guid of one of encoders defined in Encoders.xml
destStore,destAssetItem - destination store and asset itemENCRYPT - encrypts an asset item using AES-128 CBC. Both source and destination stores have to be local. Parameters:
srcStore, srcAssetItem - source asset item
key - 128 bit key in base64 form
destStore, destAssetItem - destination store and asset itemDECRYPT - decrypts an asset item previously encryped using AES-128 CBC. Both source and destination stores have to be local. Parameters:
srcStore, srcAssetItem - source asset item
key - 128 bit key in base64 form
destStore, destAssetItem - destination store and asset itemMOVE - moves an asset from one store to another. Destination store can be local or remote. Parameters:
srcStore, srcAssetItem - source asset
destStore,destAssetItem - destination store and assetCOPY - copies an asset from one store to another. Destination store can be local or remote. Parameters:
srcStore, srcAssetItem - source asset
destStore,destAssetItem - destination store and assetTHUMBS - generates set of thumbnails from a video file
srcStore, srcAssetItem, srcAssetItemType - source file
encoder - guid of one of thumbnail encoders defined in Encoders.xml
destStore,destAssetItem - destination store and asset itemIMGRESIZE - resizes an image and optionally adds a border and/or watermark
srcStore, srcAssetItem, srcAssetItemType - source file
encoder - guid of one of image encoders defined in Encoders.xml
destStore,destAssetItem - destination store and asset itemIMGROTATE - rotates an image
srcStore, srcAssetItem, srcAssetItemType - source file
encoder - guid of one of image encoders defined in Encoders.xml. The encoder is used only for save image quality and destination image type
destStore,destAssetItem - destination store and asset item
direction - one of following: CW90,CW180,CW270,CCW90,CCW180,CCW270,VFLIP,HFLIPCOMPRESS - adds an asset to zip file. If this action is called several times with different source assets but the same destination asset, files will be added to the zipfile without overwriting previous ones. Destination asset item always have extension .zip
srcStore, srcAssetItem - source asset
destStore,destAssetItem - destination store and assetMEDIAINFO - retrieves information about media asset and stores it in the same directory, with the same name and extension xml. Output format is the same as used by mediainfo --Language=raw --Output=XML with one addition: element has attribute "sha1sum" containing hash of the file
srcStore, srcAssetItem, srcAssetItemType - source fileDOWNLOAD - download a file from the internet and stores it as asset item
destStore,destAssetItem - destination assetitem
destAssetItemType - extension of destination file
url - url to retrieveDELETE - deletes an local asset
srcStore, srcAssetItem - asset to deleteRENDER - renders set of set of frames from other image assets and description in JSON format
destStore,destAssetItem - destination assetitem
destAssetItemType - destination asset type. Should be 'multi:png:0" or "multi:jpg:0"
srcStore, srcAssetItem, srcAssetItemType - source file. The only handled asset type is 'jsonrender'<queue guid="78D6012F21264C50802247930B8DBE9D" destination="192.168.1.211" queue_version="0.1" dateCreated="" dateStart="" dateCompleted=""/>
<workflow guid="D67D32FBE72846DA9400A6A883F1F4B9" dateCreated="" dateStart="" dateCompleted="">
<task guid="809D540132DE4A759E9C5E140B3451F3"
action="ENCODE"
srcStore="B57378404D3A4105A8FF3222A47224EE"
srcAssetItem="3613951C17154BC2808D91BA7BCD03C5"
srcAssetItemType="mov"
encoder="8A5D25EB9D234C9992062C08823BD9F5"
destStore="2CEC0552EC60481C9FEF44831A8F6A19"
taskSequence="1"
dateCompleted=""/>
<task guid="809D540132DE4A759E9C5E140B3451F3"
action="MOVE"
srcStore="2CEC0552EC60481C9FEF44831A8F6A19"
destStore="70ED4230977A41A6BCDA66CF93D10B6D"
taskSequence="2"
dateCompleted=""/>
</workflow>
</queue> Render module allows creation of multi-frame assets from set of images (each one is an asset) and description file in JSON format. Example:
{
"main": {
"width": 640,
"height" : 480,
"background" : [ 0,0,0, 0 ]
},
"objects" : [
{ "store" : "store1", "assetItem" : "asseti1", "assetItemType" : "png", "x" : 0, "y" : 0 , "z" : 1 , "start" : 0, "finish" : 48, "alpha" :0.1 },
{ "store" : "store1", "assetItem" : "asseti2", "assetItemType" : "jpg", "x" : 100, "y" : 0 , "z" : 2 , "start" : 0, "finish" : 48, "alpha" :0.5 },
{ "store" : "store1", "assetItem" : "asseti1", "assetItemType" : "png", "x" : 50, "y" : 50 , "z" : 0 , "start" : 20, "finish" : 58, "alpha" :1 }
]
}"main" section specifies attributes of each created frame -width, height and background color (as R,G,B,A values in 0 - 255 range). Following is list of objects that should be rendered in frames. Attributes explanation:
- store, assetItem, assetItemType - source image to render on a frame. Should be png or jpg
- x,y - where the image should be pasted
- z - controls order of drawing. Objects with lowest z will be drawn first
- start - first frame this object should appear in
- finish - last frame this object should appear in
- alpha - modify transparency of the object: 0 - completely transparent , 1- no changesNumber of frames rendered depends on highest value of 'finish' attribute
New task can be added to the queue by calling addworkflow.py . No command line parameters are required. Workflow definition is provided by writing workflow XML (in the same format as in queue) to standard input and closing the stream.
Note that you don't have to first upload source asset and then create workflow operating on this asset. If queue encounters a workflow with source asset missing used by first task missing, it will simply skip this workflow without setting error status. Queue always attempts to run earlier created workflows first. If for example there are three workflows: A,B,C and there is source asset for A missing, the queue will skip it and start working on B. However if source asset for A appears in meantime, the queue after completing B will proceed to A instead of C.
For the source task to be processed it has to be present in the specified folder. When files are moved to the source folder they have a temporary name of (ex: guid.mp4_UP), when file upload is completed the name is changed to guid.mp4
Example of creating encoding job:
Prerequisites:
- Encoders.xml - must be valid
- at least two local stores have to be mounted and described in LocalStores.xml
- encoding queue has to be runningCall addworkflow.py with one parameter:
<workflow guid="D67D32FBE72846DA9400A6A883F1F4B9" dateCreated="" dateStart="" dateCompleted="">
<task guid="809D540132DE4A759E9C5E140B3451F3"
action="ENCODE"
srcStore="B57378404D3A4105A8FF3222A47224EE"
srcAssetItem="3613951C17154BC2808D91BA7BCD03C5"
srcAssetItemType="mov"
encoder="8A5D25EB9D234C9992062C08823BD9F5"
destStore="2CEC0552EC60481C9FEF44831A8F6A19" />
</workflow>Make sure that whole text above is treated as single parameter.
Change values of srcStore, encoder, destStore and srcAssetItem to match the system. Addworkflow.py will return guid of the workflow or error message. Verify that task appeared in /opt/node/queue/Queue.xml.
Transfer of assets between stores connected to different nodes is implemented using sftp protocol. It is always initiated from source - files are sent, not downloaded. Every node has list of all stores in the system, including hosting machine address so there is no need to look up destination store location in central server.
In order for one node to connect to another it has to be authorized using RSA key. Authorization is handled by keys.py:
- get - returns public key of the host
- authorize <key> - adds given key to list of authorized keys
- unauthorize <key> - removes given key from authorized listThe console is simple interactive text application that allows to monitor status of the queue, change network settings, passwords, start and stop the node. It is starts automatically on boot (before login) and later can be started using /opt/node/console.py. It has to be run as root.
Node also has ability to make an asset available in other store for defined time. This can be used for creating temporary download links: keep all assets in non-published store and only link them to published one for limited time. Removal of expired links is done by a node, so limits will not be enforced if it is not running. Links are managed by links.py. Commands:
create <src store> <dest store> <src asset uuid> <dest asset uuid> <expiry date and time> - creates a link from source asset in source store to dest asset in dest store. When this command is invoked, source asset must already exist in source store. Expiry date and time is given in "dd/mm/yyyy hh:mm:ss" format. It is interpreted as UTC.
modify <dest store> <dest asset uuid> <expiry date and time> - modify expire time of existing link. It cannot be used after link expired and has been removed
list - lists existing links
remove <dest store> <dest asset uuid> - remove a link from list of links and deletes it from disk