Assimilate Announces Open Beta of New Live FX for Live Compositing of Virtual Productions
Live Compositing Enables Real-Time, On-Set Pre-Visualization of Final Content
October 19, 2021 – Assimilate – San Jose, CA – Assimilate® announced today an open beta for its new and innovative Live FX® software that enables real-time, live compositing for green-screen and LED wall-based virtual productions on set. Live FX is also ideal for creating quick comps when scouting locations. Live FX features the latest technology in keying, camera tracking, DMX light control, Notch Block integration, and a live link to Unreal Engine that simplifies on-set virtual production for not only previsualization, but even final pixel for in-camera VFX. By recording any incoming dynamic metadata along with the imagery, Live FX can automatically prep all content for VFX/post, saving time and money. Filmmakers and artists – from freelancers to big studios – can now work in a familiar film/video workflow rather than in a programming style environment.
“The future of filmmaking is in virtual productions for the obvious reasons of time and cost savings in reduced logistics, travel, equipment, shipping, location set-ups, and more,” said Jeff Edson, CEO at Assimilate. “Assimilate is supporting this process by pioneering live compositing on set, which provides a significant leap in productivity and creativity for users. Live FX enables the most realistic pre-visualization up front in the workflow before the content moves down the pipeline and into post production. Additionally, we wanted to develop a tool that meets both needs of running on laptops for doing quick comps for location scouting or on-set quick checks, as well as on powerful systems to drive entire productions.”
Camera Tracking
Live FX supports a broad range of tracking systems for a variety of on-set situations, such as indoor, outdoor, big stage or small set, tethered or untethered. To track camera movement, Live FX allows use of Intel RealSense tracking cameras, Mo-Sys StarTracker, HTC Vive trackers, and even gyro and ARKit technology inside smartphones mounted to the cinema camera. Even without a dedicated tracking device, Live FX is able to create camera tracks using its own live tracking algorithm. All tracking data can be refined and synced to the incoming live camera signal, so all data stays in sync throughout the entire pipeline. Through its built-in virtual camera calibration, Live FX is able to calculate lens distortion and create final composites live on-set.
Easy Green Screen Replacement
The green screen can be easily replaced on set with a virtual background of choice – whether it’s a simple 2D texture, 360 equirectangular footage, a 3D Notch Block, or a live feed from another camera or Unreal Engine. This is made possible by a variety of keying algorithms within Live FX. Users can create and combine multiple keyers using the layer stack and add multiple garbage masks to their composite. RGB and Alpha channels can be output separately via SDI or NDI to be captured by other tools on-set, such as Unreal Engine.
LED Rear Projection Creates Photorealistic Environment
Live FX supports color-managed playback of any footage, at any resolution, and at any frame rate.
It allows projection of conventional 2D files, 360 equirectangular footage, Notch Blocks, or live feeds, such as 3D environments directly from Unreal Engine, onto LED walls. Once the content is loaded, it can be color graded, framed and tweaked further.
DMX Lighting Control
For a realistic and natural look, accurate lighting is a key aspect of virtual productions. The DMX lighting control engine in Live FX allows users to mark a region of interest on the footage that is being projected onto the LED wall. Live FX analyzes and averages the color inside the marked region for every frame and sends the color down to the stage lighting, making it easy and straight forward to adjust the lighting of any scene. Think big – not only bright blue sky, but even dynamic scenes like a house on fire with flickering ARRI Sky panels or a car-chase scene through tunnels are now easy to create and set up.
Live Links
Live FX features Live Links, which allows any piece of dynamic metadata entering the system to be used inside a live composition. The metadata comes in from a dedicated tracker system, via the SDI signal of the camera, or through the Open Sound Control (OSC) protocol from another system. Examples include using the dynamic lens metadata that comes in from the camera through SDI to directly animate the blur of a texture. Or, link the camera’s own gyro metadata to the virtual camera inside Live FX to link camera movement completely without additional tracking devices.
Timecode Sync Signal Routing
One of the most important aspects in live compositing is managing latency. Live FX allows for compositing multiple streams together: Live streams, dynamic renders like those from Unreal Engine, and any pre-recorded footage. If an input is timecode-managed, Live FX can auto sync multiple feeds by delaying the fastest feeds to ensure 100 percent frame sync in the composite. The camera tracking can be manually delayed to more precisely align the tracking data with the video feed. The additional latency from SDI output is kept to a minimum in Live FX and the output can be genlocked. Users can also exchange image data using NDI, or even direct GPU texture sharing with zero latency.
Recording, Metadata, and Post Prep
Anything that goes into Live FX can be recorded and instantly played back for review. The media can be recorded in H264, Avid DNx or even Apple ProRes, including alpha channel. In the case of multiple live feeds, users can decide to record the final baked result or the individual raw feeds separately. And not only the media is recorded; all metadata that is captured and used in the live composite – including camera tracking data – is saved to file as well. This all allows for a unique and highly efficient workflow: Start with the live composite and record the raw feeds and metadata separately. As soon as the recording stops, Live FX automatically creates an offline composite that is ready for instant playback and review. The offline composite is a duplicate of the live composite, using the just recorded files instead of the live signals, and includes all other elements and animation channels for further manipulation.
Once the high resolution media is offloaded from the camera, Live FX automatically assembles the online composite and replaces the on-set recorded clips with the high quality camera raw media.
These online composites can be loaded into Assimilate’s Scratch® to create metadata-rich dailies or VFX plates in the OpenEXR format, including all the recorded frame-based metadata, which saves time in post-production.
“Live FX is not an island. It’s part of Assimilate’s extensive Scratch ecosystem of advanced digital-cinema tools that are designed not only to simplify workflows and increase productivity for DITs, post-production artists, directors, and all filmmakers, but also to achieve the highest quality productions,” said Mazze Aderhold, chief strategic workflow engineer at Assimilate. “For Live FX, Assimilate is now expanding its connections to third-party software with an Unreal Engine plug-in for metadata communication, support for Open Sound Control (OSC) protocol, and direct GPU texture sharing to enable ultimate flexibility on-set. We’ve also added support for a wide range of different camera tracking devices and workflows that equally serve high-end productions and indie filmmakers with cost-effective solutions for virtual production – all running on off-the-shelf (OTF) hardware. This makes Live FX the easy go-to solution for all virtual productions.”
Participate in Open Beta
To participate in the open beta, simply visit the Assimilate webpage and click the “Try” button for Live FX. Fill in your email address and receive a return email with the download-link and license key.
For beta users to get started immediately, watch the Live FX short hands-on video tutorial series.
About Assimilate
Assimilate develops and offers advanced, real-time tools for on-set and post-production workflows that deliver groundbreaking features, speed, performance, flexibility, and stability for industry professionals. The recently launched Live FX is a state-of-the-art live compositor, shipping with camera tracking, DMX control and the latest green-screen keying technology for virtual production. Live Assist is the multi-cam VTR tool for video assist, offering support for any number of cameras, any resolution, with easy green-screen comp and local-clip server functionality. Live Looks is the optimum tool for live grading and direct-look creation with an instant connection to Assimilate’s Scratchâ for live streaming and keying capabilities. Used by DITs and post artists worldwide, Scratch is known for its advanced on-set and post-production tools that offer stability, speed and metadata support for any camera format, including 2D/3D/VR. Play Pro is the go-to solution for a professional player for VFX-reviews, ProRes RAW QC and genuine ProRes transcoding on Windows.