Machine vision can create Harry Potter–style photos for muggles (2024)

In J.K. Rowling’s Harry Potter novels, magical photographs are similar to ordinary ones except that the characters in them are animated; they wave, smile, or sometimes disappear altogether to attend to other business.

Magical photos are the creation of Rowling’s extraordinary imagination. But something like them may soon be available to ordinary muggles thanks to the work of Chung-Yi Weng at the University of Washington in Seattle and a couple of pals. These folks have created a piece of software called Photo Wake-Up that can animate the central character in a photograph while leaving the rest of the image untouched.

This task is easier said than done because of an important unsolved problem in computer science. This is the problem of body pose estimation. Given a two-dimensional image of a human, the question that machine vision struggles to answer is: what three-dimensional pose is the person taking?

Machine vision can create Harry Potter–style photos for muggles (1)

That’s difficult because bodies can be partially occluded, often by other body parts, as when someone stands with arms folded. That makes it hard for a machine to determine the three-dimensional structure from a 2D image.

A wide range of computer science teams have attempted to tackle this problem. In this work, Weng and co use a program called SMPL, developed by a team at Microsoft and the Max Planck Institute for Intelligent Systems in Germany.

This begins with a 2D cutout of a human body and superimposes a 3D skeleton onto the shape. The skeleton can then be animated to create the sense of movement. That solves the problem of pose estimation, albeit for a limited set of circumstances.

The code needs to see a head-to-toe cutout of a body seem from the front. It can handle some types of occlusion, such as an arm in front of the body, but cannot handle more complex occlusions, such as somebody sitting with legs crossed. Even still, mapping the cutout from a photograph onto a 3D skeleton does not produce realistic animations.

That’s where Weng and co come in. Their main achievement is to develop a way to warp the 2D cutout in a way that creates a realistic 3D model of the body. “Our key technical contribution, then, is a method for constructing an animatable 3D model that matches the silhouette in a single photo,” they say.

In the past, computer scientists have tried to solve this problem by deforming a three-dimensional body-shaped mesh to reflect the 2D cutout. That does not always work well, so Weng and co try a different approach.

Their idea is to map the body-shaped mesh into 2D space and then align it with the 2D cutout using a warping algorithm. This identifies specific parts of the body—head, right arm, right leg, left arm, left leg, and torso—and warps each one in a way that matches the cutout.

Having performed the alignment in 2D, they transform it back into 3D. “This 2D warping approach works well for handling complex silhouettes,” they say.

The team pay special attention to the head, which human viewers tend to focus on. “Accuracy in head pose is important for good animation,” they say. So their algorithm also identifies features such as gaze direction and head angle, and then use this to get a precise angle for the body-mesh head pose.

These automated techniques are good, but they are not perfect. So the team have also developed a user interface that allows anybody to change the skeleton’s orientation relative to the body. That allows users to correct any errors and to fine-tune the animation.

The end result is an impressive kind of animated photo. The algorithm isolates a human body in the photograph, cuts it out of the image, and fills in gap with a patch-filling algorithm. It then animates this body in three dimensions to make it walk out of the photo, to run, jump, or wave, rather like the magical photos imagined by Rowling. The algorithm even works in augmented-reality settings.

“Our method works with large variety of whole-body, fairly frontal photos, ranging from sports photos, to art, and posters,” they say. The team have produce a video showing their method and results here. It’s worth watching!

That’s interesting work with potential to entertain and inform. As Weng and co put it: “We believe the method not only enables new ways for people to enjoy and interact with photos, but also suggests a pathway to reconstructing a virtual avatar from a single image while providing insight into the state of the art of human modelling from a single photo.”

Even Rowling would surely be impressed.

Ref: arxiv.org/abs/1812.02246 : Photo Wake-Up: 3D Character Animation from a Single Photo

Machine vision can create Harry Potter–style photos for muggles (2024)

FAQs

Machine vision can create Harry Potter–style photos for muggles? ›

A clever algorithm animates characters in still images, allowing them to walk out of photographs.

How are Muggle photographs different from magical ones? ›

Magical photographs are similar to Muggle photographs, except that the people portrayed in them are animate, moving, waving, and apparently dropping out of the picture occasionally to attend to other business.

What makes magical photographs move Harry Potter? ›

The developing solution was a potion, that, when used in the developing of photographs, allowed the pictures to move.

Why can't a Muggle use magic? ›

In J. K. Rowling's Harry Potter series, a Muggle (/ˈmʌɡəl/) is a person who lacks any sort of magical ability and was not born in a magical family. Muggles can also be described as people who do not have any magical blood inside them.

What happens if a Muggle sees magic? ›

Wizarding law

Thus, in the modern age, most Muggles believed magic to be nothing but a childish fantasy. Wizards and witches hid their world with Muggle-Repelling Charms, and if a Muggle witnessed a magical event or saw a magical creature such as a dragon, their memories were erased.

What makes Harry Potter pictures move? ›

Key Takeaways. Magical portraits in the wizarding world capture the essence of people and bring them to life through enchantment, allowing them to move, talk, and interact with others.

How do Harry Potter portraits work? ›

Whether a photo or a painting, wizards and witches are able to magically enchant the images to bring them to life. Even Colin Creevey, with his Muggle camera, is able to magically enhance his many, many photos – as long as he develops them in 'the right potion.

What do you call the moving pictures in Harry Potter? ›

When a magical portrait is taken, the witch or wizard artist will naturally use enchantments to ensure that the painting will be able to move in the usual way. The portrait will be able to use some of the subject's favourite phrases and imitate their general demeanour.

What is the difference between the magic world and the Muggle world? ›

The term "Muggle world" refers to a society inhabited by non-magical people ("Muggles"), while the term "wizarding world" refers to a society of wizards that live parallel to Muggles.

What makes wizards different from Muggles? ›

Wizards had the power to cure 'mundane' illnesses and injuries, and contact with non-magical creatures that Muggles could not. However, they could struggle to repair any damage caused by magical means such as the Memory Charm and Unforgivable Curses.

Can wizards tell if someone is a Muggle? ›

Most of the times, they could. Whenever there is a Muggle in the scenes, they are almost always described as Muggles. There are plenty of things to differentiate between Muggles and Wizards.

How do Muggle borns know they have magic? ›

Muggle-borns inherit magic from a distant ancestor; they are descended from Squibs who have married Muggles and whose families had lost the knowledge of their wizarding legacy. The magic resurfaces unexpectedly many generations later.

Top Articles
Latest Posts
Recommended Articles
Article information

Author: Greg O'Connell

Last Updated:

Views: 6414

Rating: 4.1 / 5 (42 voted)

Reviews: 89% of readers found this page helpful

Author information

Name: Greg O'Connell

Birthday: 1992-01-10

Address: Suite 517 2436 Jefferey Pass, Shanitaside, UT 27519

Phone: +2614651609714

Job: Education Developer

Hobby: Cooking, Gambling, Pottery, Shooting, Baseball, Singing, Snowboarding

Introduction: My name is Greg O'Connell, I am a delightful, colorful, talented, kind, lively, modern, tender person who loves writing and wants to share my knowledge and understanding with you.