Make-A-Storyboard:

A General Framework for Storyboard with Disentangled and Merged Control

Center for Future Media, University of Electronic Science and Technology of China (UESTC)
"A storyboard is a visual representation of how a story will play out, scene by scene."

Abstract

Story Visualization aims to generate images aligned with story prompts, reflecting the coherence of storybooks through visual consistency among characters and scenes.Whereas current approaches exclusively concentrate on characters and neglect the visual consistency among contextually correlated scenes, resulting in independent character images without inter-image. To tackle this issue, we propose a new presentation form for Story Visualization called Storyboard, inspired by film-making. Specifically, a Storyboard unfolds a story into visual representations scene by scene. Within each scene in Storyboard, characters engage in activities at the same location, necessitating both visually consistent scenes and characters.For Storyboard , we design a general framework coined as Make-A-Storyboard that applies disentangled control over the consistency of contextual correlated characters and scenes and then merge them to form harmonized images.Extensive experiments demonstrate 1) Effectiveness.the effectiveness of the method in story alignment, character consistency, and scene correlation; 2) Generalization. Our method could be seamlessly integrated into mainstream Image Customization methods, empowering them with the capability of story visualization.

Generation Storyboard Results

Scene 1: The little girl arrived in an unfamiliar town, feeling very curious as she explored everything around her.
Scene 2: The little girl followed the town streets and arrived at a bridge, where she discovered a very beautiful house ahead.
Scene 3: The little girl crossed the bridge, but she sensed an eerie atmosphere in the surroundings, and she felt very frightened.
Scene 4: To escape from there, she discovered tram tracks on the water, so she waited here for the tram to depart.

Generation Video Storyboard with AnimateDiff

Click to Play the Storyboard
#1 Long ago, in a little house, lived a small beaver named Loopy, she like to play the guitar. #2 Loopy joyfully unwrapped a delicious hamburger.
#3 Loopy rushed to the beach and accidentally found a boat and rowed happily on the sea. #4 Loopy felt very comfortable floating on the crystal-like sea


Multi-character Generation Storyboard

#1 A long time ago, SpongeBob lives in a house under the sea. #2 Mr. Krabs also lives there.
#3 Spongebob especially likes to eat burgers. #4 But Mr. Krabs thinks the burger tastes terrible.
#5 SpongeBob decides to go to the lab to research improved burgers. #6 Mr. Krabs also comes to the laboratory.
#7 Mr.Krabs is looking for SpongeBob everywhere. #8 Finally, he finds SpongeBob in lab.
#9 SpongeBob is holding a test tube in his hand. #10 SpongeBob's experiment failed and explored.
#11 Mr.Krabs is also injured by the explosion. #12 Eventually, they fled back to the house together.

Video for Pipeline

Comparison

BibTeX

@article{su2023make,
title={Make-A-Storyboard: A General Framework for Storyboard with Disentangled and Merged Control},
author={Su, Sitong and Guo, Litao and Gao, Lianli and Shen, Heng Tao and Song, Jingkuan},
journal={arXiv preprint arXiv:2312.07549},
year={2023}
}