I really want to talk to a video model.
There are so many small details about life in these models and I feel like we barely have access to them. We write a prompt, generate a video, look at the result.
Think about a mother holding her child. You can describe it as gentle, protective, nervous, tired. Each of these words changes how you imagine the whole scene. Where her hands go, how close she holds the child, how often she looks down. A good video generation can carry those differences without you specifying every movement.
That is the part I find crazy. How much does a model need to learn about people to fill in all of that?
I think this is why video models are so interesting. So much of life happens without anyone describing it. When you write about someone picking up a sleeping child, you probably leave out how they shift their weight first, support the head, then slowly pull their other hand away. In a video, all of that is there. Learning to generate that movement means learning something about how these things fit together.
Maybe we need some sort of decoder between the video model’s internal representations and language. Something we could train to connect what the model has learned about movement and interactions to things we can ask about.
I don’t know what the right architecture is. I feel like video models have way more knowledge about everyday life than we give them credit for. Maybe even more than LLMs in some ways. So much of what we know about people comes from watching them, and it’s hard to put all of that into words. The same thing happens with a smile. Ask for a small girl smiling slowly, then starting to laugh. The model gets the order right: eyes first, then the mouth, then the shoulders join, then the head tips back. It gets the duration right. It knows a real laugh has a buildup and that the buildup is where the emotion is.
No text corpus on earth contains the timing of a laugh.
Now prompt something the model has never seen: a clown, in full costume, teaching an advanced physics lecture to a university class.
For that to work, the model cannot have memorized scenes. It has to have factored the world into parts that recombine: who, where, doing what, in what manner. That is compositional generalization, the thing people spent years arguing neural networks could not do.
We built a mind and gave it a projector.
We stitch the parts together and train the whole thing on one objective: produce frames that look like the dataset.
This is Frankensteining. We take a creature that has absorbed more about human behavior than any model before it and we give it exactly one way to express anything: render a clip. The only door in is a short text prompt with some references. The only door out is pixels.
