terraform

VFX and Animation Rendering with HPC Cache and Avere vFXT on Azure

To meet tight deadlines and reduce total cost of ownership (TCO), VFX and Animation studios use Azure for on-demand access to compute capacity for their render workloads. Avere technology enables the cloud burst rendering scenario and makes it easy to extend an on-premises rendering pipeline to Azure with minimal workflow changes.

Customers burst render to Azure for the main reasons of controlling and tracking costs, security, ease of use, and collaboration. The Azure rendering white paper goes into detail on each of these topics.

The common cloud burst rendering architecture is shown in the following diagram:

There are 4 major cloud infrastructure components that make up a cloud rendering solution:

1. Network connection - the burst rendering solution is connected via an Azure Virtual Private Network (VPN) Gateway or an Azure ExpressRoute.
2. Custom Image - customers create a linux or Windows based custom image that contains the necessary rendering and render management software to mount the storage cache and also connect to on-premises services such as a render manager, AD server, and metrics server.
3. Storage Cache - a managed HPC Cache or Avere vFXT is used to cache data, and hide latency to on-premises data.
4. Render Farm - Virtual Machine Scalesets are used to scale the custom image across thousands of virtual machines. Additionally Azure Cycle Cloud is used to help manage the virtual machine scale-sets.

To get started the recommended approach is to follow the First Render Pilot. The first render pilot provides a phased approach to build out the burst rendering architecture on Azure. The resources below will help you through each of the phases.

Learning

There are multiple video resources to help with learning about burst rendering on Azure:

Description	Video Link
Siggraph 2020 Videos - Six videos from Siggraph 2020 review customer solutions and why render on Azure.
Studio gains 70 percent compute capacity with Azure Virtual Machine Scale Sets, HBv2 virtual machines - By using Azure ExpressRoute, Virtual Machine Scale Sets, and Avere vFXT for Azure Jellyfish Pictures was able to increase their compute capacity by 70%.
Securing a custom image - This example shows an Azure administrator how to take an on-prem image, upload it to Azure, and then provide secure access to a contributor.
Avere vFXT in a Proxy Environment - This example shows how to configure an Avere vFXT in a secured locked down internet environment where access to outside resources is via a proxy.
Visual Effects and Animation Rendering in Azure	Whitepaper
Head-Turning Animation: ‘Bobbleheads: The Movie’ First Feature Completed Remotely on Microsoft Azure Using NVIDIA GPUs - With post-production struck by the COVID-19 pandemic, Threshold Entertainment and Incessant Rain Studios use the cloud to bring 3D animators and VFX artists together from around the world.	Blog Link
GitOps for Azure Rendering	Blog

The remainder of this page provides Terraform infrastructure examples to build out the rendering architecture:

1. Full End-To-End Examples - Full end to end examples in Linux and Windows.
2. Rendering Best Practices for Azure Compute, Network, and Storage - learn about the best practices for rendering to reduce TCO on Compute, Network, and Storage.
3. Storage Cache Infrastructure - Use HPC Cache
4. Rendering Accessories Infrastructure - this provides examples of a dns server, NFS ephemeral filer, secure image, and jumpbox.
5. Terraform Modules - these are the common infrastructure building blocks that make up the rendering architecture.
6. Avere vFXT Terraform Provider - this is the resource page that provides the full reference to using the Avere vFXT Terraform provider.

Full End-To-End Examples

The following examples provide end-to-end examples that implement the burst rendering architecture in Linux and Windows environments.

1. Create a Linux Render Farm On Azure - shows how to create a Linux Render Farm On Azure.
2. Create a Windows Render Farm On Azure - this walks through a deployment of a Houdini render environment on Azure.
3. Create a Linux based OpenCue managed render farm on Azure - deploy an end to end render solution on Azure using OpenCue as your render manager.
4. Create a multi-region render farm with a Hammerspace global file system on Azure - this walks through a deployment of a Houdini render environment on Azure.

Rendering Best Practices for Azure Compute, Network, and Storage

The highest priority for VFX and Animation Studios is the lowest total cost of ownership (TCO). The following best practices supplement existing Azure documentation with guidance on how to achieve the lowest TCO.

1. Best Practices for a New Subscription - it may be useful for a studio to create a subscription for each office, or each new show to separate out billing. If this is the case, we recommend creating a one-time process described in this document.
2. Best Practices for using Azure Virtual Machine Scale Sets (VMSS) or Azure Cycle Cloud for Rendering
3. Networking Best Practices for Rendering
4. Storage Cache Best Practices for Rendering
5. Security Best Practices for Rendering
6. Support Best Practices for Rendering - describes what to do when you encounter an issue on Azure.

Storage Cache Infrastructure

Both HPC Cache and Avere vFXT for Azure provide file caching for high-performance computing (HPC). We recommend to always choose HPC Cache for greater user manageability and only choose Avere vFXT for Azure for custom scenarios where HPC is unable to fit. If you need to use Avere vFXT for Azure because of a missing feature in HPC Cache, please submit an issue so we can track and add to HPC Cache.

HPC Cache

1. no-filer example
2. HPC Cache mounting Azure Blob Storage cloud core filer example
3. HPC Cache mounting 1 IaaS NAS filer example
4. HPC Cache mounting 3 IaaS NAS filers example
5. HPC Cache mounting an Azure NetApp Files volume
6. HPC Cache extends Azure NetApp Files across regions
7. HPC Cache and VDBench example
8. HPC Cache and VMSS example
9. HPC Cache and CacheWarmer
10. HPC Cache mounting Hammerspace filer example - WIP

Avere vFXT for Azure

1. no-filer example
2. Avere vFXT mounting Azure Blob Storage cloud core filer example
3. Avere vFXT mounting 1 IaaS NAS filer example
4. Avere vFXT mounting 3 IaaS NAS filers example
5. Avere vFXT mounting an Azure NetApp Files volume
6. Avere vFXT extends Azure NetApp Files across regions
7. Avere vFXT and VDBench example
8. Avere vFXT and VMSS example
9. Avere vFXT and CacheWarmer
10. Avere vFXT mounting Hammerspace filer example

Specialized Avere vFXT for Rendering and Artists](examples/vfxt)

1. Avere vFXT optimized for Houdini
2. Avere vFXT and Cloud Workstations
3. Avere vFXT only - this example is useful for when the cloud environment is already configured.
4. Avere vFXT in a Proxy Environment - this example shows how to deploy the Avere in a locked down internet environment, with a proxy.
5. Deploy Avere vFXT directly from the controller - this example shows how to deploy the Avere directly from the controller.
6. Specify a custom VServer IP Range with the Avere vFXT - this example shows how to specify a custom VServer IP Range with the Avere vFXT.
7. Avere vFXT using User Assigned Managed Identity - this example shows how to use a user assigned managed identity with the Avere vFXT.
8. Backup Restore - Backup any FXT or vFXT cluster and build terraform to restore to HPC Cache or Avere vFXT for Azure.
9. Zone Redundant NFS / SMB Storage - Availability Zone Redundant Avere vFXT mounting Blob Storage example

Rendering Accessories Infrastructure

The following terraform examples build out accessory rendering infrastructure such as DNS Servers, high speed NFS ephemeral filers, and a jumpbox:

VPN

1. Azure VPN Gateway to VyOS IPSec Tunnel - this example is useful for experimenting with building an IPSec tunnel with Azure VPN Gateway.
2. WireGuard Multi-tunnel implementation using ECMP on Azure - this example shows how to deploy a WireGuard VPN with multiple tunnels to achieve high throuhput.

Workstations

1. Windows 10 + Nvidia Grid + Teradici PCoIP - this example deploys Windows 10 + Nvidia Grid + Teradici PCoIP.
2. CentOS7 + Gnome + Nvidia Grid + Teradici PCoIP - this example deploys CentOS7 with Gnome + Nvidia Grid + Teradici PCoIP.

NFS Filers

1. NFS Ephemeral Filer - builds a very high performance NFS filers for use as a scratch filer.
2. NFS Managed Disk Filer - builds an NFS Filer with highly available managed disk storage.
3. NFS Filer using NFS-Ganesha - builds high performance NFS filers.
4. NFS Filer using Hammerspace - builds a Hammerspace filer capable of spanning across multiple regions.

DNS, Security, and Jumpbox

1. Secure VNET - shows how to create a simple secure VNET that has locked down internet for burst render.
2. DNS Server to Override Filer Address - This deploys an Azure virtual machine that installs and configures Unbound and and configures it to override the address of an on-premises filer so that the render nodes mount the Avere to hide the latency. All other dns requests are forwarded to pre-configured on-premises dns servers.
3. SecuredImage - shows how to create, upload, and deploy a custom image with an introduction to RBAC, Azure Governance, and Network.
4. Jumpbox - this deploys a VM pre-installed with pre-installed with az cli, terraform, golang, and the built avere provider Security.

Terraform Modules

These modules provide core components for use with HPC Cache or Avere vFXT for Azure:

Network

1. Render Network - the render network module creates a sample render network complete with five subnets: cloud cache, filer, jumpbox, and two render node subnets
2. Secure Render Network - the secure render network module where the internet is locked down to all subnets but the proxy subnet. This module creates a sample render network complete with six subnets: cloud cache, filer, jumpbox, two render node subnets, and a proxy subnet.
3. Proxy - this installs a proxy VM running the Squid Proxy.
4. DNS Server - this installs a DNS Server to be used with HPC Cache or Avere vFXT.

Compute

1. Mountable VMSS - this deploys a Linux based VMSS and runs a script off an NFS share.
2. Azure CycleCloud - this deploys an Azure CycleCloud instance.
3. VMSS Mount Script - This module installs the mount bootstrap script for VMSS.
4. VD Bench Config Script - this module configures an NFS share with the VDBench install tools.
5. VMSS Config Script - this module configures an NFS share with a round robin mount script.
6. Opencue Config Script - this module sets up the OpenCue RQD clients module.
7. Gnome + Nvidia Grid + Teradici PCoIP - this module deploys Gnome + Nvidia Grid + Teradici PCoIP.

Controller and Jumpbox

1. Controller3 - Deploys a python 3 controller that is used to create and manage an Avere vFXT for Azure.
2. Controller - (Deprecated) Deploys a python 2 controller that is used to create and manage an Avere vFXT for Azure.
3. Jumpbox - the jumpbox has the necessary environment for building the terraform-provider-avere. It is also useful for when experimenting in virtual networks where there is no controller.

NFS Filers

1. NFS Ephemeral Filer - the NFS ephemeral filer provides a high IOPs, high throughput filer that can be used for scratch data.
2. NFS Managed Disk Filer - the NFS managed disk filer provides NFS access to highly available storage. There is an offline mode to destroy the VM and cool the storage for maximum cost savings when not in use.
3. NFS-Ganesha Filer - the NFS-Ganesha provides a user space NFS Server and provides NFS access to highly available storage. There is an offline mode to destroy the VM and cool the storage for maximum cost savings when not in use.
4. Hammerspace Anvil - the Hammerspace Anvil module deploys a Standalone or Highly Available Hammerspace Anvil cluster.
5. Hammerspace DSX - the Hammerspace DSX module deploys a configurable number of Hammerspace DSX nodes.

Cache Warmer

The Cache Warmer is deployed through 4 Terraform modules. Examples that use these 4 modules are the following:

• HPC Cache and CacheWarmer
• Avere vFXT and CacheWarmer

A good explanation of each of the modules is described in the HPC Cache or vFXT documentation. For Locked down internet environments, customers will manually have to run the shell script of the cachewarmer module.

Avere vFXT Terraform Provider

The following provider creates, destroys, and manages an Avere vFXT for Azure:

• terraform-provider-avere

To add the provider to a pipeline on either Ubuntu or CentOS read the following examples:

• Building a Pipeline for the vFXT Provider