Apple and Pixar created a new file format for augmented reality on your phone | Popular Science

Apple and Pixar created a new file format for augmented reality on your phone | Popular Science:

Apple and Pixar created a new file format for augmented reality on your phone

The new USDZ file format will make it easier than ever to look at AR objects in iOS.
Sharing photos and videos on a computer or a smartphone is, for the most part, pretty easy. File formats like JPG and PNG for still photos, and MOV for videos, display just as intended on pretty much any device. Now that augmented, and even virtual reality are gaining popularity, device and software makers are looking to standardize the way we share three-dimensional models for use in various apps. We know how to share a photo or video of a globe, but how do we share an interactive model of it?

At its WWDC conference earlier this month, Apple announced that it has teamed up with Pixar to develop a new file format called USDZ, which the upcoming iOS 12 operating system will natively support. It was a small piece of the presentation, but this kind of standardization will be increasingly important as AR gains traction with users.

What is USDZ?

USD stands for Universal Scene Description, and it’s a file format that’s fundamental to Pixar’s 3D animation process. You can read the technical classifications for USD here, but it’s essentially a way for content creators to code 3D objects and package-save them with enough information about things like geometry and shading to stay consistent across editing applications.

USDZ takes the concept one step further and packages all the USD elements an application needs to display that 3D object, and combines them into a single, uncompressed zip file. Each USDZ file is an archive full of individual elements that applications can reference in real time to make an object appear on a screen and in augmented reality situations.
Apple ARKit 2 integration
As part of the iOS 12 release later this year, Apple is introducing the updated version of its augmented reality platform, ARKit. It has some interesting features, including the ability for multiple users to simultaneously view the same augmented reality scene from different angles at the same time for things like catapult battles and collaborative learning. But, it will also include native support for USDZ files.

If you’re part of the Apple Developer’s program and you’re trying out iOS 12, you can go to this link and find a collection of USDZ objects to interact with using Apple’s AR Quick Look. Once you download the object, you can virtually place it in the scene in front of you, like you can with typical AR apps like Ikea’s Place app, which lets you drop virtual furniture into your real-world surroundings using your phone’s camera.

Who can use USDZ?

During the WWDC keynote, Apple announced that it’s partnering with Adobe to integrate the USDZ format into the Creative Cloud apps like the ubiquitous photo editing app Photoshop and its 3D compositing software, Dimension. Adobe’s overall AR creation strategy is called Project Aero.
The fundamental idea is to allow content creators to make 3D objects and allow users in apps and on the web to interact with them in a standardized way across AR-capable devices. In the video above, USDZ is the container that holds the elements that make up the 3D sculpture and allows the iOS device to view it in real time.

What’s the competition look like?

According to Stefano Corazza, head of augmented reality at Adobe, the company is currently working with USDZ as well as a competing format called glTF, which already has support from behemoths like Google, Microsoft, and Facebook.

Corazza says that format is primarily designed for the web and is still working on adding some of the features that are inherent to USDZ right from the outset.

Some other people in the 3D world have concerns about USDZ as well. Apple’s sheer number of iOS devices can make a file format relevant for content creators, even if it’s not the strongest candidate.

Look for it later this year

Whether Apple and Pixar can make the open USDZ standard the go-to format for 3D objects will shake out over time. But, USDZ support will be fully integrated into iOS 12 when it launches later this year. Then, the floodgates for 3D dancing objects will officially open.

Google is working on bringing AR to Chrome with downloadable 3D objects

Google is working on bringing AR to Chrome with downloadable 3D objects:

Google today unveiled its experimental effort to integrate augmented reality features into the mobile and desktop web using its Chrome browser. That way, web designers, media organizations, and other creative professionals could create virtual 3D objects, embed them into websites for viewing on desktop, and make them downloadable on mobile so users could place those objects into their real world surroundings.

“In the next few months, there will be hundreds of millions of Android and iOS devices that are able to provide augmented reality experiences — meaning you’ll be able to look at the world through your phone, and place digital objects wherever you look,” writes Reza Ali and Josh Carpenter, who work on user experience on Google’s Daydream WebXR team. “To help bring this to as many users as possible, we’ve been exploring how to bring augmented reality to the web platform, so someday anyone with a browser can access this new technology.”

The blog post details how the working prototype version of AR on Chrome would work. It starts with a product tentatively called Article that Ali and Carpenter describe as a 3D model viewer for browsers. If Article is loaded on a desktop browser with a 3D model, it will display it as an interactive image you can drag to rotate. When placed in a webpage, the model could be animated similar to a GIF as a user scrolls to indicate it’s an interactive 3D model.

Image: Google

On mobile, the experience is much more sophisticated thanks to built-in cameras and sensors on modern smartphones. “The unique power of AR is to blend digital content with the real world,” the duo write. “So we can, for example, surf the web, find a model, place it in our room to see just how large it truly is, and physically walk around it.” You can see how this would work in action:

Image: Google

It’s easy to imagine how neat this could be for all manner of activities, from entertainment to education. Just being able to load a Wikipedia page for the Moon landing, as Google suggests here, and drop a 3D model of an astronaut in a classroom would be a fascinating new way to teach children with interactive experiments.

“Article is just one in a series of prototypes, and there’s so much left to explore — from using light estimation to more seamlessly blend 3D objects with the real world, to adding diegetic UI annotations to specific positions on the model,” concludes Ali and Carpenter. “Mobile AR on the web is incredibly fun right now because there’s a lot to be discovered.”

Of course, Google has a reason to want to develop mobile, browser-based AR. The company is currently competing with Apple and the iPhone maker’s ARKit framework to be the toolkit developers use to make apps and other AR-focused services. Google, which owns Android and the Play Store and has its own ARCore framework, nonetheless has a presence on iOS via the mobile web, where Chrome is a popular browser alternative to the iPhone’s built-in Safari. By making AR work on mobile browsers, Google is able to maintain its cross-platform advantage. It’s also in Google’s interest to prevent AR from becoming an entirely app-based technology on mobile, as that would lock out Android users from AR experiences developed solely for iOS.

(Via The Verge – All Posts)

Airbnb wants to offer virtual reality previews of rental spots

Airbnb wants to offer virtual reality previews of rental spots:

Airbnb is developing virtual and augmented reality features to help guests find and navigate rental listings, the company announced on its blog today. Three-dimensional scans and 360-degree photos would allow users to get a better sense of a listing, and augmented reality overlays could help guests better understand the homes on a smaller scale once they’re in it. The company has been looking into VR to build trust between guests and hosts since last year, and this announcement confirming experiments and prototypes could mean a feature is coming soon.

It makes sense to take a look around a listing in VR before booking, but Airbnb also lists a few instances where augmented reality might be helpful during your trip. If you’re staying overseas and everything’s in a foreign language, it could be difficult to unlock the door, or figure out the thermostat or the hot water in the shower. It also lists a fun example of AR usefulness like “pulling up a mobile device to get directions to the coffee mugs … first thing in the morning.” This is a problem that could easily be solved by just opening up a few cupboards and figuring it out, but I get what they’re saying.


Virtual tours have been around in the real estate industry for a while, and Airbnb hosts have been asking for a feature to integrate them into their own listings. While the VR technologies could be implemented by providing Airbnb hosts with 360-degree cameras, it might be harder to integrate the AR concept for individual listings. Since ARKit was released on iOS 11, we’ve seen some concepts of what an AR overlay for Airbnb listings might look like. It seems like a useful and worthwhile idea, but it might be a more difficult challenge for Airbnb to take on.

But until this is a reality, I’ll still be satisfied with charming hand-written binders with tips on the local sights and illustrations on how to figure out the thermostat.

(Via The Verge – All Posts)