Back to The New Full-Stack Designer, with Aaron Epstein & Marvin Schwaibold.
How Does Figma Help Codex Build Motion?
Marvin Schwaibold uses Figma as a scratch pad so a broad sketch can carry motion he cannot put into words, then he gives the idea to Codex. He thinks Codex interprets the image the way he interpreted it. Codex understood what he was doing and how he was thinking about the motion.
On a side project over a weekend, he wanted a two by three overview of grass studies and videos he took. Opening any video and dragging down closes it, and the interaction works the same in the simulator. He described it to AI while working in Codex, and Codex built it, but the close was not interruptible. Every time he clicked an asset and pulled it down, he had to wait until the asset was all the way back in position and sat down before he could tap another image. The pattern has to stay very responsive very quickly, or the interface gets annoying very fast. In Figma he drew rudimentary simple boxes. Click the box and red comes up, ready to swipe away. While the swipe is still moving, a click on blue brings blue up, and green works the same way.
The scratch pad makes decompiling an idea easier for him, and Codex decompiles it easily when Codex sees the picture. Articulating the motion by text or voice failed. Drawing it out worked. Codex and current AI software are extremely capable of creating anything, as long as he knows the work, takes a harder concept such as a shader, breaks it into small components, understands each component, and puts the pieces back together.
From the transcript
A lot of the way I'm talking to AI or I'm still using Figma at the moment is I can get really far with a very broad sketch. Here's an example of a side project I've been working on over the weekend, which I can go into detail a bit. But one of the things I wanted to build out for this project was a two by three overview of these grass studies and these videos I took.
And the interaction here is you can open up any one of these videos and you can drag down to close them. And it works the same in the simulator.
I explained this, I described this to AI and I was working in Codex and Codex built this out for me. But the interaction wasn't interruptible. So every time I clicked on an asset and pulled it down, I had to wait until the asset was all the way back in its position and sat down before I was able to tap on another image. And that's just not how you want to use this pattern.