Watch the Apple RoomPlan API empty a piece of furniture, ready for augmented reality purchases

Augmented reality shopping is about to be taken to the next level, thanks to a new feature introduced in iOS 16. The new Apple RoomPlan API uses the LiDAR scanner in recent iPhone Pro models to create a 3D room plan – and an awesome new demo shows off exactly what it’s capable of.

Shopify has created a proof of concept that scans a room and then lets you virtually empty it, ready to see what new furniture would look like in the space. Alternatively, you can remove specific items, like a sofa, and then insert new ones into the space…

If that sounds familiar, that’s because Ikea has done the same thing before in more raw form, but we noted at the time that RoomPlan was likely to let furniture companies create fancier apps. .

Shopify Russ Maschmeyer says there has been a lot of interest in the ability. It refers to the feature as a reset button for a part and explains what it can do.

Augmented reality makes it easier to try out new sofas, but the sofa already in your space may get in the way. What if every room had a reset button? Space Eraser models spaces in high fidelity, capturing the objects defining the room, their size, their position, their orientation.

Live pixel data combines with LiDAR depth data to enable a realistic digital twin of your room that can be overlaid onto your real space using augmented reality and then digitally edited.

With an empty room, shoppers can now browse room sets that take advantage of the orientation of existing furniture to automatically organize themselves.

With a digital twin of your bedroom and furniture, you can simply scan your original sofa to explore options that better match the rest of the room.

Here you can see the reset process:

And here, removing individual elements from the room:

Maschmeyer explains how they created it and some of the challenges they faced.

RoomPlan generates an untextured USDZ model composed of unit cubes; one for each door, window, wall and room-defining object, but surprise! The models do not contain floor or ceiling elements. We align the model with the real world using ARWorldMap.

Since RoomPlan exports untextured models, we explored ways to automatically texture them. The best way would be to locate and map the texture on capture, but we are satisfied with an aligned model and real-time camera feed.

We reversed the graphics pipeline: instead of sampling from a texture and writing to the screen, we sample from the camera and write to a texture. RealityKit made this difficult, so we used SceneKit instead.

This approach creates problems when furniture obscures a surface. In the video above our sofa obscured the wall. RoomPlan provides bounding box references for the objects defining the room. With this we can determine which parts of the texture should be redacted.

We can now use inpainting techniques to fill in the redacted texture. We were pleasantly surprised by the results produced by out-of-the-box tools like Photoshop and theinpaint.com.

AR and #RoomPlan are exciting tools for commerce. The ability to select and remove real objects from a scene brings us closer to a fully editable reality.

I’m a big fan of augmented reality shopping, which gives a good idea of ​​what something in your home would be like without anything as barbaric as visiting a store in person. What do you think? Please share your thoughts on our social networks.


Check out this video below for more Apple news:

Leave a Comment