Exploring 3D Worldbuilding with AI & Hybrid Filmmaking Techniques
From Prompt to Presence:
The Evolution of Our Hybrid Filmmaking Pipeline
Bringing Performance Into AI Assisted Workflows
AI filmmaking is moving quickly, but one of the biggest questions we kept coming back to was simple: how much control can the filmmaker actually have? During our work leading up to the HollyShorts Film Festival workshop, we tested three different approaches to AI filmmaking. Each pipeline built on the previous one, moving from generating everything with AI to giving filmmakers more control over performance, movement, camera, and character.
Pipeline 1.0: Start with Previsualization
Our first approach was the most straightforward: create the scene directly inside AI filmmaking tools. We experimented with tools like Luma, Kling, Seedance and other generative AI platforms, using prompts to generate characters, movement, environments, and cinematic shots.
It was fast and opened up a lot of creative possibilities, but there was also a limitation. The filmmaker had less control over exactly how the character moved and performed. That led us to the next question: what happens if we bring real human performance into the pipeline?
Pipeline 2.0: Adding Motion Capture
For Pipeline 2.0, we introduced Xsens motion capture. Instead of asking AI to generate the movement, we started with a real performer. Their movement was captured through Xsens and brought into Unreal Engine, where we could work with the digital character, blocking, timing, and virtual camera aka Vcam.
From there, we brought the rendered performance into our AI workflow and used AI to transform the visual style. The pipeline became Xsens to Unreal Engine to AI. This gave us a lot more control over the performance while still allowing AI to handle the look and style transfer.
Pipeline 3.0: Adding Face Capture
For Pipeline 3.0, we wanted to push that control even further. The timing was ideal because we had recently led a roundtable discussion at Oxen.ai exploring this very idea: how filmmakers can build more directable AI workflows. The conversation gave us a chance to unpack our own experiments alongside insights from other creators, technologists, and filmmakers exploring new approaches to directing with AI.
This time, we added facial capture through Live Link, allowing us to capture not just the performer's body movement, but also their facial performance. We also leveraged Luma Agents and then Higgsfield Genjutsu for motion transfer, giving us another way to carry the performance into the AI-generated result.
The pipeline became:
Xsens + Live Link → Unreal Engine → Higgsfield Genjutsu
Now we had control over more of the performance before AI entered the process.
Why the Pipeline Matters
The biggest takeaway from testing these three workflows was that AI doesn't have to mean giving up human authorship and creative control.
Pipeline 1.0 showed us what was possible with AI alone.
Pipeline 2.0 gave us control over movement and performance.
Pipeline 3.0 added another layer of human performance through facial capture.
Each version moved us closer to a workflow where the filmmaker can collaborate on worldbuilding and decide how the character moves, how they perform, how the camera sees them, and how AI transforms the final image maintaining collaboration that’s Human led, AI-enabled.
That's where we see the opportunity. Not just using AI to generate a beautiful shot, but building a filmmaking pipeline where human performance and creative direction remain at the center.
Connect and Collaborate
Interested in immersive storytelling for yourself, organization or school? Reach out to us!
Our Skool just hit over 100 members! Want to be part of the movement? Click below to join our Skool community to meet members of the team, other creators, and gain access to resources, workshops, labs and events!
