Valve’s Portal video game has spawned thousands of videos that appear to bring the game’s holographic windows to real life. This video is different.
Creative software designer Roland Smeenk built a system that leverages Microsoft’s Kinect V2 sensor to recreate Portal’s portals in real time. He uses one Kinect to record a 3D view of the world in real-time and another track the head of the user to see where they are looking.
In this way, Smeenk can reconstruct the camera view as if the user is looking through a virtual window by using a projector connected to his laptop. Taking it a step further, he uses both Kinects for 3D world reconstruction and head tracking to recreate a two-way portal effect:
The projector shows one portal and the laptop shows the other. To ensure proper head tracking, the setup opens a portal as soon as a user is detected by the Kinect.
Smeenk notes there is a size limitation to consider: smaller portals hide more of the other side of the scene and allow for larger head movements because the field of view is limited. There is also a distance limitation (about 7 meters when using USB3 extensions cables) because latency is being minimized by only transferring the recorded bodyframes through a network connection (as opposed to transferring the whole recorded 3D scene).
This is quite the technical achievement given that Kinect wasn’t built for this type of thing. Yet Microsoft’s HoloLens was.
Smeenk envisions how technologies like Kinect and HoloLens could transform human communications:
Looking through a window is a familiar experience for almost everyone. Imagine being in a Skype conversation and having the ability to move your head to see who your caller is looking at or talking to (when it’s not you). A holographic window has the power to give people the feeling of being in the same space and allows for interesting new interactions.
In other words, one day you could put on Microsoft’s HoloLens and see holographic windows into not just virtual worlds, but other parts of the real world in real time.