First Release

.vscode/settings.json

@@ -0,0 +1,17 @@
+{
+    "python.linting.pylintEnabled": false,
+    "python.linting.pep8Enabled": true,
+    "python.linting.enabled": true,
+    "python.testing.unittestEnabled": true,
+    "python.testing.pyTestEnabled": false,
+    "python.testing.nosetestsEnabled": false,
+    "python.pythonPath": "${env:GAFFER_ROOT}/bin/python.exe",
+    "python.testing.cwd": "./python",
+    "python.testing.unittestArgs": [
+        "-v",
+        "-s",
+        "./python/GafferDeadlineTest",
+        "-p",
+        "*.py"
+    ]
+}

GafferDeadline.code-workspace

@@ -0,0 +1,8 @@
+{
+	"folders": [
+		{
+			"path": "."
+		}
+	],
+	"settings": {}
+}

LICENSE.md

@@ -1,5 +1,5 @@
 
-Copyright (c) 2019, Eric Mehl
+Copyright (c) 2019, Hypothetical Inc.
 All rights reserved.
 
 Redistribution and use in source and binary forms, with or without
@@ -12,7 +12,7 @@ modification, are permitted provided that the following conditions are met:
   this list of conditions and the following disclaimer in the documentation
   and/or other materials provided with the distribution.
 
-* Neither the name of [project] nor the names of its
+* Neither the name of Hypothetical Inc. nor the names of its
   contributors may be used to endorse or promote products derived from
   this software without specific prior written permission.
 

README.md

@@ -1 +1,47 @@
-GafferDeadline
+# GafferDeadline #
+Deadline Dispatcher for Gaffer. There are three components - the Gaffer dispatcher, Deadline plugin and Python dependency script called by Deadline to check for task dependencies getting released.
+
+Gaffer can generate arbitrarily complex node trees with dependencies between Task Nodes that are more complicated than Deadline supports through it's standard job and task dependencies. To support Gaffer's DAG style of dependencies, a dependency script is included. Deadline runs this script periodically for each job to determine which, if any, tasks for that job are ready to be released. Using the included dispatcher and plugin this should be transparent to the user once everything is setup as described below.
+
+GafferDeadline will auto-detect the most efficient method of setting up dependencies by default (see not in Usage about overriding this behavior). Most Gaffer scripts will likely be submitted with frame-to-frame dependencies, but it will fall back to the scripted task dependency system or job-to-job on a per-node basis if it doesn't fit within Deadline's dependency scheme.
+
+It is tested on Linux and the beta Windows Gaffer build. OS X compatibility is unknown.
+
+## Installing ##
+ 1. Extract the archive / clone the repostory to a directory accessible to Gaffer.
+ 2. Move the "Gaffer" subdirectory to your Deadline repository "custom/plugins" directory. This is the Deadline plugin that will run Gaffer jobs on your render farm.
+ 3. Add the directory where you extracted / cloned the repository to the GAFFER_EXTENSION_PATHS environment variable before running Gaffer.
+ 4. Move the gaffer_batch_dependency.py file to a location where all of your Deadline Slaves and Pulse machines can access the file. Deadline will run that script according to your repository settings to check for tasks that can be released from pending status based on their dependencies being completed.
+ If you have multiple operating systems in your Deadline installation, you will likely need to set up path mapping for machines to locate the script.
+ 5. Set the DEADLINE_DEPENDENCY_SCRIPT_PATH environment variable to the full path (including filename) where you saved the gaffer_batch_dependency.py file before running Gaffer. GafferDeadline dispatcher uses this variable as the location for the dependency script when submitting jobs to Deadline.
+ 6. Ensure that the DEADLINE_PATH environment variable is set to the directory where the "deadlinecommand" executable lives. This is typically set system-wide when you install the Deadline Client. GafferDeadline uses this environment variable to locate "deadlinecommand" for interacting with your Deadline repository.
+
+## Using ##
+With everything set up correctly, Task Nodes in Gaffer will have a Deadline section on their Dispatcher tab. This section is where you setup the Deadline configuration for that task. You can set most common settings like groups, pools, priority, description, etc.
+
+When you are ready to submit the node(s) press the node's "Execute" button and select Deadline from the dropdown box of available dispatchers.
+
+You need a Deadline Client installed and connected to your repository on the machines you will be running GafferDeadline from. GafferDeadline uses the Deadline installation on the host machine, similar to other integrated submitters Deadline includes such as for Nuke, Houdini, etc.
+
+The Deadline settings in Gaffer include an override for the dependency method for that node. This override controls downstream Task Nodes that depend on the node on which it was set. Most of the time it should be left on Auto to let the dispatcher determine the most efficient method. If you know a node needs to be handled in a particular way, you can force its dendency method with the override plug. Usually the "Full Job" setting will be the safest but least flexible because downstream tasks will wait for all frames of that job to complete before being released.
+
+## Running Unit Tests ##
+You don't need to run the unit tests for normal use of GafferDeadline, but if you want to make customizations it is recommended that you add unit tests as appropriate and run the existing tests to ensure compatibility.
+
+To run the unit tests, you need to have an installation of Gaffer and have your Python environment setup to point to that installation. The easiest way to do that is to use the included gaffer_env (Linux) and gaffer_env.bat (Windows) files to setup the environment first. Then you can use regular Python unit test runners to run tests.
+
+More specifically:
+1. PATH environment variable needs to include the gaffer/bin, gaffer/lib (on Windows) and gaffer/python directories.
+2. PYTHONPATH environment variable needs to include the gaffer/python directory.
+3. On Linux the LD_LIBRARY_PATH needs to be set to the gaffer/lib directory.
+
+There is also a Visual Studio Code environment included that may be helpful.
+
+## Contributing ##
+Feedback and pull requests are welcome! If you have ideas about how to improve the dispatcher, find bugs or would like to submit improvements, please create an issue on GitHub for discussion or a pull request.
+
+## Copyright and License ##
+
+© 2019 Hypothetical Inc. All rights reserved.
+
+Distributed under the [BSD license](LICENSE).

custom/Gaffer/gaffer.options

@@ -0,0 +1,60 @@
+[Script]
+Type=string
+Label=Script
+Category=Gaffer Options
+CategoryOrder=0
+Index=0
+Description=The Gaffer script to execute.
+Required=true
+DisableIfBlank=true
+
+[Version]
+Type=label
+Label=Version
+Category=Gaffer Options
+Index=1
+Description=The version of Gaffer to execute.
+Required=false
+DisableIfBlank=true
+
+[IgnoreScriptLoadErrors]
+Type=boolean
+Label=Ignore Script Load Errors
+Category=Gaffer Options
+CategoryOrder=0
+Index=2
+Description=Causes error which occur while load the script to be ignored. Not recommended.
+Required=false
+Default=false
+DisableIfBlank=false
+
+[Nodes]
+Type=string
+Label=Nodes
+Category=Gaffer Options
+CategoryOrder=0
+Index=2
+Description=The names of the nodes to execute. If not specified then all executable nodes will be found automatically.
+Required=false
+DisableIfBlank=false
+
+[Frames]
+Type=string
+Label=Frames
+Category=Gaffer Options
+CategoryOrder=0
+Index=3
+Description=The frames to execute. The default value executes the current frame as stored in the script.
+Required=false
+DisableIfBlank=false
+Default=false
+
+[Context]
+Type=string
+Label=Context
+CategoryOrder=0
+Index=4
+Category=Gaffer Options
+Description=The context used during the execution. Note that the frames parameter will be used to vary the context frame entry.
+Required=false
+DisableIfBlank=false

custom/Gaffer/gaffer.param

@@ -0,0 +1,43 @@
+[About]
+Type=label
+Label=About
+Category=About Plugin
+CategoryOrder=-1
+Index=0
+Default=Gaffer for Deadline
+Description=Not configurable
+
+[ConcurrentTasks]
+Type=label
+Label=ConcurrentTasks
+Category=About Plugin
+CategoryOrder=-1
+Index=0
+Default=True
+Description=Not configurable.
+
+[Executable0_45_0_0]
+Type=multilinemultifilename
+Category=Gaffer 0.45.0.0 Executables
+CategoryOrder=0
+CategoryIndex=0
+Label=Gaffer 0.45.0.0 Render Executable
+Default=%HOME%/gaffer_0.45.0.0;~/gaffer_0.45.0.0
+Description=The path to the Gaffer 0.45.0.0 executable (gaffer.bat on Windows) file used for executing. Enter alternative paths on separate lines.
+[Executable0_53_0_0]
+Type=multilinemultifilename
+Category=Gaffer 0.53.0.0 Executables
+CategoryOrder=0
+CategoryIndex=0
+Label=Gaffer 0.53.0.0 Render Executable
+Default=%HOME%/gaffer_0.53.0.0;~/gaffer_0.53.0.0
+Description=The path to the Gaffer 0.53.0.0 executable (gaffer.bat on Windows) file used for executing. Enter alternative paths on separate lines.
+
+[EnablePathMapping]
+Type=boolean
+Category=Path Mapping (For Mixed Farms)
+CategoryOrder=70
+CategoryIndex=0
+Label=Enable Path Mapping
+Default=true
+Description=If enabled, environment variables will be set for the process running Gaffer. This feature can be turned off if there are no Path Mapping entries defined in the Repository Options.

custom/Gaffer/gaffer.py

@@ -0,0 +1,170 @@
+##########################################################################
+#
+#  Copyright (c) 2019, Hypothetical Inc. All rights reserved.
+#
+#  Redistribution and use in source and binary forms, with or without
+#  modification, are permitted provided that the following conditions are
+#  met:
+#
+#      * Redistributions of source code must retain the above
+#        copyright notice, this list of conditions and the following
+#        disclaimer.
+#
+#      * Redistributions in binary form must reproduce the above
+#        copyright notice, this list of conditions and the following
+#        disclaimer in the documentation and/or other materials provided with
+#        the distribution.
+#
+#      * Neither the name of Hypothetical Inc. nor the names of
+#        any other contributors to this software may be used to endorse or
+#        promote products derived from this software without specific prior
+#        written permission.
+#
+#  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
+#  IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
+#  THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+#  PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
+#  CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
+#  EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
+#  PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
+#  PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
+#  LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
+#  NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+#  SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+#
+##########################################################################
+
+import os
+import re
+
+from System.IO import *
+from System.Text.RegularExpressions import *
+
+from Deadline.Scripting import *
+from Deadline.Plugins import *
+
+from FranticX.Processes import *
+
+######################################################################
+# This is the function that Deadline calls to get an instance of the
+# main DeadlinePlugin class.
+######################################################################
+
+
+def GetDeadlinePlugin():
+    return GafferPlugin()
+
+
+def CleanupDeadlinePlugin(deadlinePlugin):
+    deadlinePlugin.Cleanup()
+
+######################################################################
+# This is the main DeadlinePlugin class for the Gaffer plugin.
+######################################################################
+
+
+class GafferPlugin(DeadlinePlugin):
+    def __init__(self):
+        self.InitializeProcessCallback += self.InitializeProcess
+        # self.RenderTasksCallback += self.RenderTasks
+        self.RenderExecutableCallback += self.GetRenderExecutable
+        self.RenderArgumentCallback += self.GetRenderArguments
+        # Some tasks like Ply2Vrmesh and Houdini sims handle multiple frames rather than a separate Deadline task per frame
+        self.currentFrame = 0.0
+        self.totalFrames = 0.0
+
+    def Cleanup(self):
+        for stdoutHandler in self.StdoutHandlers:
+            del stdoutHandler.HandleCallback
+
+        del self.InitializeProcessCallback
+        del self.RenderTasksCallback
+
+    def InitializeProcess(self):
+        self.PluginType = PluginType.Simple
+        self.StdoutHandling = True
+
+        # Generic Gaffer progress
+        self.AddStdoutHandlerCallback(".*Progress: (\d+)%.*").HandleCallback += self.HandleProgress
+
+        # Vray's ply2vrmesh prints out lines for each frame and also each voxel within the frame
+        self.AddStdoutHandlerCallback(".*Subdividing frame ([0-9]+) of ([0-9]+).*").HandleCallback += self.HandlePly2VrmeshFrameProgress
+        self.AddStdoutHandlerCallback(".*Processing voxel ([0-9]+) of ([0-9]+).*").HandleCallback += self.HandlePly2VrmeshVoxelProgress
+
+    def GetRenderExecutable(self):
+        self.Version = self.GetPluginInfoEntry("Version")
+        gafferExeList = self.GetConfigEntry("Executable" + str(self.Version).replace(".", "_"))
+        gafferExe = FileUtils.SearchFileList(gafferExeList)
+        if(gafferExe == ""):
+            self.FailRender("Gaffer %s render executable could not be found in the semicolon separated list \"%s\". The path to the render executable can be configured from the Plugin Configuration in the Deadline Monitor." % (
+                self.Version, gafferExeList))
+
+        return gafferExe
+
+    def GetRenderArguments(self):
+        script = RepositoryUtils.CheckPathMapping(self.GetPluginInfoEntryWithDefault("Script", "").strip())
+        script = self.replaceSlashesByOS(script)
+        local_script = os.path.join(self.GetJobsDataDirectory(), script)
+        if os.path.isfile(local_script):
+            script = local_script
+
+        ignoreErrors = self.GetPluginInfoEntryWithDefault("IgnoreScriptLoadErrors", "False")
+        nodes = self.GetPluginInfoEntryWithDefault("Nodes", "")
+        frames = self.GetPluginInfoEntryWithDefault("Frames", "")
+        frames = re.sub(r"<(?i)STARTFRAME>", str(self.GetStartFrame()), frames)
+        frames = re.sub(r"<(?i)ENDFRAME>", str(self.GetEndFrame()), frames)
+        frames = self.ReplacePaddedFrame(frames, "<(?i)STARTFRAME%([0-9]+)>", self.GetStartFrame())
+        frames = self.ReplacePaddedFrame(frames, "<(?i)ENDFRAME%([0-9]+)>", self.GetEndFrame())
+        context = self.GetPluginInfoEntryWithDefault("Context", "")
+
+        arguments = "execute -script \"{}\"".format(script)
+        arguments += " -ignoreScriptLoadErrors" if ignoreErrors.lower() == "true" else ""
+        arguments += " -nodes {}".format(nodes) if nodes != "" else ""
+        arguments += " -frames {}".format(frames) if frames != "" else ""
+        arguments += " -context {}".format(context) if context != "" else ""
+
+        return arguments
+
+    def ReplacePaddedFrame(self, arguments, pattern, frame):
+        frameRegex = Regex(pattern)
+        while True:
+            frameMatch = frameRegex.Match(arguments)
+            if frameMatch.Success:
+                paddingSize = int(frameMatch.Groups[1].Value)
+                if paddingSize > 0:
+                    padding = StringUtils.ToZeroPaddedString(frame, paddingSize, False)
+                else:
+                    padding = str(frame)
+                arguments = arguments.replace(frameMatch.Groups[0].Value, padding)
+            else:
+                break
+
+        return arguments
+
+    def HandleProgress(self):
+        progress = float(self.GetRegexMatch(1))
+        self.SetProgress(progress)
+
+    def HandlePly2VrmeshFrameProgress(self):
+        self.currentFrame = float(self.GetRegexMatch(1)) - 1.0
+        self.totalFrames = float(self.GetRegexMatch(2))
+
+        self.SetProgress(self.currentFrame / self.totalFrames * 100)
+        self.SetStatusMessage("Ply2Vrmesh: frame {}/{}".format(self.currentFrame, self.totalFrames))
+
+    def HandlePly2VrmeshVoxelProgress(self):
+        currentVoxel = float(self.GetRegexMatch(1)) - 1.0
+        totalVoxels = float(self.GetRegexMatch(2))
+
+        voxelProgress = currentVoxel / totalVoxels
+
+        self.SetProgress(((self.currentFrame / self.totalFrames) + (voxelProgress * 1.0 / self.totalFrames)) * 100)
+        self.SetStatusMessage("Ply2Vrmesh: Processing Voxel {}/{} @ frame {}/{}".format(currentVoxel, totalVoxels, self.currentFrame, self.totalFrames))
+
+    def replaceSlashesByOS(self, value):
+        if SystemUtils.IsRunningOnWindows():
+            value = value.replace('/', '\\')
+        else:
+            value = value.replace("\\", "/")
+
+        return value