Forem: Max Cobb

What's New in Apple AR/VR 2023

Max Cobb — Wed, 07 Jun 2023 02:32:27 +0000

AR/VR/XR Highlights from Apple's WWDC Developer Conference 2023

It's been six years since ARKit's debut at WWDC 2017, and we've seen big milestones since then; but none come close to what was announced at WWDC 2023 this week.

From a full AR/VR simulator, to Apple Vision Pro, to full skeletal hand tracking, you'll find the AR/VR highlights from the main keynote listed below; as well as other things hiding around in Apple's beta API References. It won't go into too much technical details on how to implement some of these new features and new frameworks, you'll have to follow me on twitter.

Although many of the APIs are not yet public (expected later in June 2023), we'll cover an outline of the main new features coming our way soon.

ARKit

Starting with the core of AR at Apple, there only seems to be one feature available for now, and only on the newly introduced OS.

Skeletal Hand Tracking

As well as tracking surfaces, images and bodies, ARKit will now have skeletal hand tracking, starting with visionOS. This seems to be enabled due to a new device that has multiple cameras to better percieve depth.

RealityKit

Not known to be the fastest horse in terms of rolling out features, especially when compared to its older sister SceneKit. But this year there were some much needed advancements, which were of course necessary for a certain piece of hardware.

RealityKit x SwiftUI

With the introduction of a whole new SwiftUI view, RealityView, RealityKit can now fully take advantage of SwiftUI layouts, and vice versa.

You can also add SwiftUI views into RealityKit in the form of attachments.

This now allows us to add content with sharp, readable text, especially those with complex layouts and animations.

MaterialX

RealityKit is adopting the open standard MaterialX for surface and geometry shaders.

MaterialX is supported across a broad variety of tools. Meaning people already familiar with those tools can now write already shaders in RealityKit, as well as those already using RealityKit can now have more developer tools and countless shader examples at their disposal.

visionOS

Apple's exciting new operating system, visionOS brings many of the AR/VR concepts that RealityKit developers may be familiar with together in a stunning new headset dubbed "Vision Pro". While many of the

Fundamentals

There are a few key objects that you can create within Vision Pro's OS, visionOS:

Windows, planes in space.
Volumes, which can hold 3d objects, such as a game board.
Full Space, which are fully immersive environments, and can not necessarily be manipulated by the user.

Apps

Existing iOS/iPadOS apps are supported as a scalable 2d window shown in 3d space on visionOS. This window keeps their original look and feel.

If you simply add the build target of visionOS to your application, it immediately gets those nice visionOS materials, a fully resizable window, and displays the eye focus of the user, as you find in all visionOS applications.

With visionOS SwiftUI or UIKit can be used display content, RealityKit for 3d content, and ARKit to understand the user's space. The apps designed specifically for visionOS can create a collection of windows, volumes, and spaces.

Simulator

My favourite part of the developer tooling at WWDC 2023. This simulator lets you move and look around the scene, and interact with the app by simulating the system gestures. There are several different builtin scenes, with both day and night coditions to help better understand how users will interact with your apps.

Mac Virtual Display

Simply by looking at your Mac through a Vision Pro, your Mac's screen will pop out and instead a high fidelity 4k monitor will appear in its place. This enables a whole new end-to-end development experience, where you can code, test, and debug all in the same place.

Bluetooth keyboards and mice can be used for data input across visionOS, so of course you won't have to type on a virtual keyboard.

Reality Composer Pro

Using Reality Composer Pro, you can add and prepare 3d content for visionOS apps.

What's next?

You may have noticed some sections are a little bit bare, I'll be expanding upon the content over the next week or so as more videos and software are released.

To stay tuned, drop a follow on dev.to as well as twitter. I'm always posting about the latest advancements in AR/VR, and specifically those in the Swift ecosystem.

Getting Started with RealityKit: Brilliant Colours

Max Cobb — Mon, 15 May 2023 01:50:24 +0000

Have you ever noticed that the materials in RealityKit appear dull and lackluster? Even when using UnlitMaterials, the colors tend to look muted and unimpressive. In this article, we'll explore why this is the case and provide a solution to achieve more vibrant colors in your RealityKit scenes.

The Current Issue

By default, the materials in RealityKit are designed to mimic real-world lighting conditions. This means that even unlit materials will have some degree of shading and shadowing, resulting in colors that appear less saturated than they would in a perfectly lit environment.

While this approach may be appropriate for many use cases, such as architectural visualizations or product renderings, it can be frustrating when you're trying to create a more stylized or vibrant scene, or when just want some objects to stand out amongst the other objects that should blend into the scene.

Take a look at this example, showing a simple sphere in a 3d scene. It looks okay, but in on way does that white colour stand out as a true white:

The actual pixel value of the cube shows it's only 214 out of a maximum of 255 for true white.

The same can be said when applying a texture.
The colours are much more dull than they're intended to be, which is frustrating when wanting to make objects in your AR scene that really pop.

The Solution

Fortunately, there's a simple solution to this problem that doesn't involve custom shaders, using the new(ish) material class PhysicallyBasedMaterial.

This material type is actually made for simulating the appearance of real-world objects, but there's a trick to making it appear just like you would expect an UnlitMaterial to act, which is to set its baseColor and sheen to black, which then means the object will not respond to lighting, then apply your colour or texture to the emissiveColor, and bump up the emissiveIntensity to a value greater than its default of 1.

Here's an example of what that might look like:

var mat = PhysicallyBasedMaterial()
mat.baseColor = .init(tint: .black)
mat.sheen = .init(tint: .black)
mat.emissiveColor = .init(color: .white)
mat.emissiveIntensity = 2

The new material is on the right.

Here's what the pixel values are in the screenshot:

Similarly, instead of setting the color in the emissiveColor, you can set a texture as such:

mat.emissiveColor = .init(texture: MaterialParameters.Texture(try! .load(named: "rgb_test")))

The above image shows an UnlitMaterial on the left, a SimpleMaterial in the middle, and a BoldMaterial on the right, using this image:

By making this small change, you'll be able to achieve more vibrant and true-to-life colors in your RealityKit scenes, without sacrificing the benefits of an unlit material.

The type of BoldMaterial is included in RealityToolkit, which is a lightweight set of tools to make RealityKit more powerful, with remote image and USDZ file loading, and now BoldMaterial too.

Here's a few examples with other colours, with the unlit material on the left, simple material in the middle, then the adaptation of the physically based material on the right:

But What About Lighting?

This technique works best for materials that have no response to lighting, giving them an appearance that really stands out. However, if you do what your materials to respond to lighting, a few small tweaks could help. These tweaks include keeping the sheen as .white, and increasing the baseColor to instead a dark gray.

Play around with PhysicallyBasedMaterial to find the exact appearance you're looking for in your RealityKit app.

Conclusion

While RealityKit is a powerful tool for creating immersive AR and VR experiences, but things like this can sometimes make it more challenging than necesary to achieve the desired look and feel for your scenes. By using alternative type of material types, you can create more vibrant and engaging scenes that capture your user's attention.
As mentioned above, check out the RealityToolkit Swift Package to make adding bold materials to your project easy.

I hope that this simple guide has helped you achieve the vibrant colors you're looking for in your RealityKit projects!

Create a Voice Changing Video Call App with SwiftUI

Max Cobb — Thu, 31 Mar 2022 16:45:46 +0000

Creating a video call app with Agora is super straightforward, but what about adding cool features to enhance the overall user experience or adding monetisation options? That's where Agora's Extensions Marketplace comes in.

The Extensions Marketplace launched last month, and already several extensions are available, with more to come.

In this post, we'll show you how to easily add a voice changer to your existing video call app and choose between the available voice options. Thanks to the Voicemod Voice Changer, in just three easy steps you will be able to modify your voice with this extension:

Enable the extension
Pass extension credentials
Choose the voice

Now, let's get started.

Prerequisites

An Agora Developer Account - Sign up here
Xcode 12.3 or later
A physical iOS device with iOS 13.0 or later
A basic understanding of iOS development

Setup

Let’s start with a new, single-view iOS project. Create the project in Xcode, choosing SwiftUI for the interface, and then add the Agora UIKit package.

Add the package by selecting File > Swift Packages > Add Package Dependency, and then paste the link into this Swift package:

https://github.com/AgoraIO-Community/iOS-UIKit.git

When asked for the version, insert 4.0.0-preview, and select "Up to Next Major". This should install at least version 4.0.0-preview.8, which is released at the time of writing.

When in doubt, you can always select "Exact" and insert 4.0.0-preview.8.

Once that package installed, the camera and microphone usage descriptions need to be added. To see how to do that, check out Apple's documentation here:

https://developer.apple.com/documentation/avfoundation/cameras_and_media_capture/requesting_authorization_for_media_capture_on_ios#2962313

Video Call UI

I won't go into much detail about the UI here, because we're mainly focusing on the Voicemod extension, but on a high level:

The ContentView contains two views: a VideoCallView (defined in the project, and a button to join the channel.
ContentView holds the active AgoraViewer, which is brought in from Agora UIKit.
When the button is pressed, the Agora UIKit method join(channel:,with:,as:) is called, which joins the Agora video channel.

This is what the ContentView looks like:

And the VideoCallView, without any of the voicemod additions looks like this:

struct VideoCallView: View {
    @Binding var joinedChannel: Bool
    var body: some View {
        ZStack {
            ContentView.agview
        }
    }
}

We now have a working video call app with SwiftUI and Agora UIKit. The next step is to integrate the Voicemod extension!

Integrating Voicemod

Voicemod Credentials

If you have an account with Agora and are currently signed in, follow this link to activate the Voicemod extension for your account.

First, you will need to activate Voicemod for your account by clicking "Activate" here:

Next, enable Voicemod for the project, the same project as for the Agora App ID used in the app:

Once that's done, you can grab your Voicemod API Key and API Secret by clicking “view” below the credentials tab:

Add to Xcode

We need to add the Voicemod extension, which can also be installed via the Swift Package Manager with the following URL:

https://github.com/AgoraIO-Community/Extension-Voicemod-iOS.git

The latest release at the time of writing is 0.1.2.

This package doesn't need importing into your Swift code. It just needs to be bundled alongside the Agora SDK in your app target.

Enable the Extension

When using Agora extensions, you need to enable them before joining a channel.

To do this with Agora UIKit, you can call AgoraVideoViewer.enableExtension().

This method needs to be passed the vendor ("Voicemod"), the vendor extension ("VoicemodExtension") in this case:

ContentView.agview.viewer.enableExtension(
  withVendor: "Voicemod", extension: "VoicemodExtension", enabled: true
)

The above code snippet was added to the beginning of the joinChannel method defined earlier.

Choose the Voice

After joining the channel, we'll add a small menu to the top of our view. This menu will contain all the available voices from Voicemod. On selecting the voice, we'll call a method to make sure we're using that voice. On selecting an option, we update a state variable, selection.

The menu is created like this:

The names array is defined like this:

let names = [
    "baby",
    "cave",
    "lost-soul",
    "robot",
    "titan"
]

This is what the menu looks like:

Almost there now! The only thing left to do is define the setVoicemodParam method.

This method needs to do two things:

Initialise Voicemod, passing the API Key and API Secret values.
Set the voice property so that it knows which voice effect you want to have.

First, let's define a function registerVoicemod for setting the key and secret:

func registerVoicemod() {
  // Set API Credentials
  let dataDict = [
    "apiKey": AppKeys.voicemodApiKey,
    "apiSecret": AppKeys.voicemodApiSecret
  ]
  guard let setPropertyResp = ContentView.agview.viewer.setExtensionProperty(
    "Voicemod", extension: "VoicemodExtension",
    key: "vcmd_user_data", codable: dataDict
  ), setPropertyResp == 0 else {
    print("Could not set extension property")
    return
  }

  self.voicemodRegistered = true
}

Here we are using the method setExtensionProperty with the vcmd_user_data key. The API Key and API Secret properties are stored in a Swift dictionary of type [String: String]. This method from Agora UIKit allows us to pass a Swift dictionary, which is then encoded for us and passed to Agora's engine.

If something goes wrong, either setPropertyResp will be null or the response will be less than zero. But if all the steps are followed, no issues should arise!

Now the method setVoicemodParam can be defined. This method must call registerVoicemod (first time only), and then set the voice value (robot, baby etc.):

func setVoicemodParam(to voiceName: String) {
  if !self.voicemodRegistered {
    self.registerVoicemod()
  }

  // SET VOICE
  ContentView.agview.viewer.setExtensionProperty(
    "Voicemod", extension: "VoicemodExtension",
    key: "vcmd_voice", strValue: voiceName
  )
}

Conclusion

You now have a video call app complete with the new extension from Voicemod!

There are other extensions I'd encourage you to try out, and they can all be found here:

https://console.agora.io/marketplace

The same settings can be applied to an application made directly with the Agora SDK, but you will need to make sure you're using the 4.x version of the SDK. Details can be found here:

https://docs.agora.io/en/video-call-4.x-preview/landing-page?platform=iOS

The methods, such as setExtensionProperty, used in this post are adaptations of the SDK's built-in methods relating to extension. They can be found in this file from Agora Video UIKit:

https://github.com/AgoraIO-Community/iOS-UIKit/blob/4.0.0-preview.8/Sources/Agora-UIKit/AgoraVideoViewer%2BAgoraExtensions.swift

Testing

You can try out this app by following the GitHub link:

https://github.com/AgoraIO-Community/Extension-Voicemod-SwiftUI

Other Resources

For more information about building applications using Agora SDKs, take a look at the Agora Video Call Quickstart Guide and Agora API Reference.

I also invite you to join the Agora Developer Slack community to meet with our developer team as well as other like-minded developers and technical enthusiasts.

Universal Links and SwiftUI Video Calls

Max Cobb — Tue, 15 Mar 2022 12:31:16 +0000

When making a video call application, you may want to share links to let your users' friends jump into the same video call or live stream that they're watching. The second-best way to achieve this is by sharing a room code for people to type in to their version of the app. But a much better option is to share a link that opens the app right to the place you want. That is where we can use universal links.

What are Universal Links?

Often, if there is an equivalent web page for a view that exists in an app. For example, if you're viewing an Instagram profile on your phone's web browser, there is also a way to show the same content in the Instagram app. Universal links are the gateway for going from the web browser content to the app's equivalent content.

In some cases, universal links can’t have a web equivalent; for example, a complex game that will not run in a web browser. A universal link in that scenario could lead to a template page, which can then be passed through to your application to render the full content or join the game.

Check out Apple's documentation for more information on universal links:

https://developer.apple.com/ios/universal-links/

Universal Links also work on Android devices, but the setup on your website as well as the device is a little different.

Prerequisites

An Agora developer account - Sign up here
Xcode 12.3 or later
A physical iOS device with iOS 13.0 or later
A public server with an SSL certificate (https)

Setting Up Universal Links

Setting up universal links has been made easier in recent years, but it still has two main steps:

Setting Up the Website
Setting Up the App

Let's start with the website side.

Setting Up the Website

This section is probably the easiest part of universal links. All you need to do is add a file named apple-app-site-association, which contains the correct values for your application:

{
  "applinks": {
    "apps": [],
    "details": [{
        "appIDs": ["DEVELOPMENT_TEAM.PRODUCT_BUNDLE_IDENTIFIER"],
        "components": [
          {
            "/": "*"
          }
        ]
    }]
  }
}

The above example will put a universal link button on every page of your website. See below for how to specify specific URLs in your domain.

You just need to replace DEVELOPMENT_TEAM and PRODUCT_BUNDLE_IDENTIFIER with the associated values from your app. To find them, you can run a couple of commands in a Terminal from your project's root folder, such as this:

grep -r -m 1 "DEVELOPMENT_TEAM" .; grep -r -m 1 "PRODUCT_BUNDLE_IDENTIFIER" .

Your terminal will look similar to this:

 => grep -r -m 1 "DEVELOPMENT_TEAM" .; grep -r -m 1 "PRODUCT_BUNDLE_IDENTIFIER" .
./AppName.xcodeproj/project.pbxproj:                DEVELOPMENT_TEAM = 278494H572;
./AppName.xcodeproj/project.pbxproj:                PRODUCT_BUNDLE_IDENTIFIER = io.agora.AppName;

In this case, the values we want are 278494H572 and io.agora.AppName, respectively.

To check that your website is set up correctly, head to branch.io's validator. One common issue is that the content-type header cannot be found. If you face this issue, you will need to create or add these lines to .htaccess at the same location as the apple-app-site-association file:

<Files "apple-app-site-association">
ForceType 'application/json'
</Files>

With the current setup, all pages on the website will link through to the specified app. If you just want a specific part of your website to link to an app, you can modify the components array object or add another one. For example, if the current components array is replaced with:

"components": [
  { "/": "/wwdc/news" },
  { "/": "/videos/wwdc/2015/*" }
]

Then all the WWDC news articles would direct to the app, as well as all videos from 2015.

More examples can be found in Apple's documentation here.

For our app, we are going to add the following component:

{
  "/": "/join*",
  "?": { "channel": "?*" }
}

Using example.com as our domain address, this is what an example of the full URL might look like:

https://www.example.com/join?channel=dkvn3lw2

For more information about how to set up universal links with an apple-app-site-association file, Apple has a great resource from WWDC 2019 here.

The full file should look like this:

{
  "applinks": {
    "details": [
      {
        "appIDs": ["DEVELOPMENT_TEAM.PRODUCT_BUNDLE_IDENTIFIER"],
        "components": [
          {
            "/": "/join/*",
            "?": {
              "channel": "?*"
            }
          }
        ]
      }
    ]
  }
}

Now that that's set up, let's get to the app portion of our app.

Setting Up the App

Installing the Package

Set up a new SwiftUI app in Xcode, and add the following Swift package to your project with this URL:

https://github.com/AgoraIO-Community/iOS-UIKit.git

The current stable release of Agora UIKit is 1.7.1

If you're not sure how to add a Swift package, Apple's documentation has it covered with clear steps:

https://developer.apple.com/documentation/swift_packages/adding_package_dependencies_to_your_app

Setting the Views

We'll get this part out of the way quickly, as we want to get back to the universal links part of this guide. The first thing we need to do is add a button to create and join a channel:

Button {
  // click action here
} label: {
  Text("Create Channel")
    .padding().background(.green).cornerRadius(30)
}

This button's action should create a channel name and display an AgoraViewer. AgoraViewer is a subclass of View that is provided by Agora UIKit. This can be achieved with the following block of code:

import SwiftUI
import AgoraRtcKit
import AgoraUIKit_iOS

struct ContentView: View {
  @State var channelName: String?
  @State var isShowingVideo: Bool = false

  var videoCallView: some View {
    VStack {
      HStack { /* Other buttons will be placed here */ }
      ContentView.agview
    }
  }
  static var agview: AgoraViewer = {
    AgoraViewer(
      connectionData: AgoraConnectionData(
        appId: <#Agora App Id#>, rtcToken: <#Agora Token#>
      ), style: .floating
    )
  }()

  var body: some View {
    NavigationView {
      ZStack {
        NavigationLink(
          destination: self.videoCallView
            .navigationBarHidden(true)
            .onAppear(perform: { /* appear action */ })
            .onDisappear(perform: { /* disappear action */ }),
          isActive: $isShowingVideo
        ) { EmptyView() }
        HStack {
          Button {
            self.channelName = "test"
            self.isShowingVideo = true
          } label: {
            Text("Create Channel")
            .padding().background(.green).cornerRadius(30)
          }
        }
      }
    }
  }
}

[//]: <> "https://gist.github.com/maxxfrazer/d5cfacc3ba45e174dfe8168787f0075b?file=Agora-Univeral-Links-App-Shell.swift"

Above, the channel is set to a static string "test", and the property isShowingVideo is set to true. The "test" string will be replaced with a random string generator in the final product on GitHub.

isShowingVideo is a boolean which is used to decide if the video call view (videoCallView) should be presented or not by the NavigationView. The next things to add are quite straightforward, we need to join and leave the channel using the onAppear and onDisappear methods, and add some buttons to the top of the camera view.

.onAppear(perform: {
  guard let channelName = self.channelName else {
    self.isShowingVideo = false
    return
  }
  ContentView.agview.join(
    channel: channelName, with: nil, as: .broadcaster
  )
})
.onDisappear(perform: {
  ContentView.agview.viewer.leaveChannel()
})

If channelName is nil when the video view is appearing then we will close it again, this is because a channel name is required, so something must have gone wrong getting to this stage. After that we call the builtin function from Agora UIKit to join the channel as a broadcaster.

The second parameter in the join method is for a token, which I am not using in my test application, but is required for production apps that use Agora.

To find out more about working with tokens using Agora's SDKs, here are a few resources:

Now the final part of the UI is setting up the other buttons in that videoCallView above.

var videoCallView: some View {
  VStack {
    HStack {
      Button {
        self.isShowingVideo = false
      } label: {
        Text("Exit")
          .padding().background(.gray).cornerRadius(3)
      }
      Spacer()
      Button {
        guard let channelName = channelName,
        let urlShare = URL(
          string: "https://example.com/join?channel=\(channelName)"
        ) else { return }
        let activityVC = UIActivityViewController(
          activityItems: [urlShare],
          applicationActivities: nil
        )
        let scenes = UIApplication.shared.connectedScenes
        guard let windowScene = scenes.first as? UIWindowScene,
        let window = windowScene.windows.first else { return }
        window.rootViewController?.present(activityVC, animated: true, completion: nil)
      } label: {
        Text("Share")
          .padding().background(.gray).cornerRadius(3)
      }
    }
    ContentView.agview
  }
}

[//]: <> "https://gist.github.com/maxxfrazer/d5cfacc3ba45e174dfe8168787f0075b?file=Agora-Univeral-Links-App-Buttons.swift"

Two buttons have been added in the above code snippet, and since they're positioned in a HStack, they appear side by side, with a Spacer() in-between them.

The first button lets the user exit the current view, leaving the call and going back to the main screen. The second button creates a URL (using example.com as the domain) with the channel as a parameter. This generated URL will open up the default share screen, to send it natively via iMessage, AirDrop, email, etc.

The share screen looks like this:

Interpreting the Universal Link

To let your app interpret a Universal link, your app must know which domains it is allowed to talk to. This is a very straightforward project, and can now be done directly with Xcode.

Head to the Signing & Capabilities section of your app within Xcode, and add the Associated Domains capability like this:

Once that's done, set the domains within the new capability, here:

The above entries say applinks:example.com and applinks:www.example.com. Replace “example.com” with your domain that is hosting the apple-app-site-association.

The final step in this application is understanding when the app has been opened from a universal link, and getting the full URL used to open the app.

SwiftUI offers a very easy to implement method for capturing universal links that have opened the app, which is the onOpenURL method. This method can be appended to any View, and its only parameter is the URL object. We can attach this to our NavigationView:

NavigationView {
  // Navigation View content here
}.onOpenURL { url in
  print(url.absoluteString)
}

When opening our app via the universal link, the above code block will execute and print our full URL string.

All we really need from this is that channel parameter, the final part of the URL.

To extract parts of the URL into a dictionary, I am calling a property called queryDictionary, which has been grabbed from the following Stack Overflow answer:

https://stackoverflow.com/a/46603619/2156765

Using queryDictionary, we can get our channel name, and once again trigger the video feed to open as so:

if let channel = url.queryDictionary?["channel"] {
  self.channelName = channel
  self.isShowingVideo = true
}

The app is complete! Now when going to your domain address with the endpoint join?channel=test, it will launch your application and go straight to that channel name.

Testing

This entire project is available on GitHub here:

https://github.com/AgoraIO-Community/Universal-Links-SwiftUI

When running the application, be sure to head over to the Signing and Capabilities section to update the domain (it will be set to example.com by default). Also once changing the Bundle and Team IDs, run the following command in Terminal to get your new values:

grep -r -m 1 "DEVELOPMENT_TEAM" .; grep -r -m 1 "PRODUCT_BUNDLE_IDENTIFIER" .

Other Resources

For more information about building applications using Agora SDKs, take a look at the Agora Video Call Quickstart Guide and Agora API Reference.

I also invite you to join the Agora Developer Slack community to meet with our developer team as well as other like-minded developers and technical enthusiasts.

Agora Releases Native SDK v3.5.1

Max Cobb — Mon, 18 Oct 2021 09:29:05 +0000

Agora Native SDK 3.5.1 was released on 14 October 2021.

New Features ✨

Super Resolution (beta) 🪟

Effectively boost the resolution of a remote user's video seen by the local user.

You can call enableRemoteSuperResolution to enable super resolution, specifying the remote user's uid. Make sure the AgoraSuperResolutionExtension framework is also included in your project.

Background Blurring 😶‍🌫️

Virtual backgrounds was added to the SDK since 3.5.0, and now to further enrich that, there is an option to apply a blur to the background. Set the type of the custom background image to AgoraVirtualBackgroundBlur, apply a blur_degree, and the background will be blurred.

Device Flash 📸

isCameraTorchSupported and setCameraTorchOn have been added to the C++ APIs for both iOS and Android.

Media Stream Relay Pausing/Resuming ⏯

Methods have been added to quickly pause and resume cross-channel media stream relays; as well as additional delegate methods to check if the pause and resume commands were successful.

Pushing External Audio to a Specified Position 🎤

pushExternalAudioFrameRawData and pushExternalAudioFrameSampleBuffer have been updated to have an additional parameter. This parameter, sourcePos, allows you to push the external audio frames to one of three positions: after audio capture, before audio encoding, or before local playback.

Additionally, setExternalAudioSourceVolume has also been added; allowing you to set the volume of external audio frames at a specified position.

Music and Audio File Updates 🎶

Playback speed, audio track, and channel mode of a music file can be set in code.

This update deprecates getAudioMixingDuration method in favour of getAudioFileInfo. getAudioFileInfo still gets information such as duration, but returns it in an AgoraRtcAudioFileInfo object in the didRequestAudioFileInfo delegate method, found here.

Improvements

5G Quality Testing ⚡️

When monitoring the connection quality of a device, a new property has been added to the AgoraNetworkType enum, AgoraNetworkTypeMobile5G(6), or .mobile5G with Swift, and NETWORK_TYPE_MOBILE_5G on Android.

These are just a handful of the new features to the SDK in v3.5.0. To see the full release notes for the native platform you’re using, check out one of the following links: iOS, Android, macOS, and Windows. Head to Agora’s downloads page to get the latest SDK now.

How to Build a Live Video Streaming iOS App with Agora 4.x Preview

Max Cobb — Fri, 17 Sep 2021 15:04:29 +0000

Recording and streaming a live feed from your app can be difficult, especially if you want to reach a global audience with little to no latency. The Agora Voice and Video SDK is perfect for reliably sending low-latency messages to a global audience, where you can have one or more people streaming their feeds.

Introduction

In this tutorial, you learn how to create an application that enables users to be either a streamer or an audience member in a session using Agora’s newest SDK version, 4.0.0. The setup is very similar to creating a video call with Agora, with a slight difference of roles: audience and broadcaster.

Prerequisites

An Agora developer account (see How To Get Started with Agora)
Xcode 12.0 or later
iOS device with iOS 13.0 or later (as this project uses SF Symbols)
A basic understanding of iOS development

Note: If you need to target an earlier version of iOS, you can change how the buttons are created in the provided example project.

Setup

Let’s start with a new, single-view iOS project. Create the project in Xcode, and then add Agora’s 4.0.0 Preview Swift Package.

Add the package by opening selecting File > Swift Packages > Add Package Dependency, then paste in the link to this Swift Package:

https://github.com/agorabuilder/AgoraRtcEngine_iOS_Preview.git

At the time post is written, the latest release is 4.0.0.

If you want to jump ahead, you can find the full example project here on branch ng-sdk-update:

https://github.com/AgoraIO-Community/Agora-iOS-Swift-Example/tree/ng-sdk-update

Create the App

In this initial UIViewController, we can just have a button in the center to join the channel. This button will tell the application to open a view on top, enabling us to join as an audience member by default.

The view in this example is very basic, as you can see:

Next, create the ChannelViewController.swift file. This file will contain our UIViewController subclass called ChannelViewController, which displays the Agora pieces. In it, we will store values such as App ID, token, and channel name for connecting to the service, and add a button to change the user role between audience (default) and broadcaster. Also, initialise the Agora Engine with the correct client role. I have also preemptively added remoteUserIDs and userVideoLookup, which will keep track of the broadcasters/streamers.

When userVideoLookup is set or updated, reorganiseVideos is called to organise all of the streamed video feeds into a grid formation. Of course, a grid formation is not required, but if you want to implement the same thing, the reorganiseVideos method is provided in the example project. Here’s a link to it:

https://github.com/AgoraIO-Community/Agora-iOS-Swift-Example/blob/aebfd5c6fd6159c85906952acc9e1b2e9aaec117/Agora-iOS-Example/ChannelViewController%2BVideoControl.swift#L109

The hostButton has a target set as toggleBroadcast. This method is found right at the bottom of the gist. As you can see, it toggles the value of self.userRole between .broadcaster and .audience, and then sets the client role using setClientRole. When the local client starts streaming, additional buttons should appear (for audio and video settings). But those buttons are displayed only after the client role has been changed, which is signaled by the delegate callback.

In the gist above, the ChannelViewController is set as the AgoraRtcEngineDelegate, so we should add that protocol to our class, along with some callback methods.

The main callback methods we need for a basic streaming session are didJoinedOfUid, didOfflineOfUid, didClientRoleChanged, and firstRemoteVideoDecodedOfUid.

As mentioned in the gist, the didClientRoleChanged is the callback for when we have changed the local user role to broadcaster or audience member.

If a remote host is streaming to us, the nonhosting view should now look like this:

When the user is streaming, they should have additional options, including the ability to switch between the device’s front- and back-facing cameras and turning the camera or microphone on and off. In this case, a beautification toggle is added. To show and hide these options, the isHidden property of the button container is set to false or true, respectively.

In your application you may not want anyone to be able to start streaming. You could easily achieve this by requiring a password in the app (a static password or one authorised by a network request). Or you could offer separate apps for hosts and audience members.

Other Resources

For more information about building applications using Agora.io SDKs, take a look at the Agora Video Call Quickstart Guide and Agora API Reference.

I also invite you to join the Agora.io Developer Slack community.

Multi-User Augmented Reality Experiences with Agora (Part 2 of 2)

Max Cobb — Fri, 20 Aug 2021 13:46:08 +0000

CollaboratAR is an iOS example project created by Agora to show some ways you can use Agora SDKs to create an interactive experience in augmented reality. This post shows you how to connect the dots using Agora real-time messaging, making the experience shareable between multiple people from anywhere in the world.

To see an overview of how the example works, check out this post:

Multi-User Collaborative iOS AR Experiences with Agora (Part 1 of 2)

Max Cobb ・ Aug 20 '21

#swift #arkit #realitykit #realtime

And the full source code:

AgoraIO-Community / Collaborative-AR-RTM

A collaborative Augmented Reality iOS experience using Real-time Messaging with RealityKit

What is Agora Real-time Messaging (RTM)?

Agora real-time Messaging is an SDK that enables you to send data between devices that are connected to the network. These data could contain an encoded struct, plain text, or even larger files, ranging from PDF to 3D files.

For example, one set of messages we will send across the network include our device location in the scene and details about the room we have created for others to be able to join it.

How Real-time Messaging is Used in CollaboratAR

Currently, six different types of messages will be sent across the network in CollaboratAR. They are stored in an enum to make it easy to tell them apart when sending and receiving:

enum RawRTMMessageDataType: String {

  // MARK: Globe Scene

  /// A session is available to display on the globe
  case channelAvailable
  /// Request all available sessions
  case getSessionData

  // MARK: Collaborative Session

  /// A model has been created or transform modified
  case singleCollabModel
  /// Multiple models are available to display in the scene
  case multiCollabModels
  /// A model has been deleted
  case removeCollabModel
  /// Transform update for a remote user
  case peerUpdate
}

Globe Scene 🌎

Sending

In the initial scene a globe is visible. If a remote user is currently in a channel or experience, we see them on the globe as a red circle with an orange centre.

First, let's see how the presence of a channel is sent from the hosting device.

In our project, there are two methods called sendCollabChannel that have two different uses: one sends data to the channel, and the other sends data to a specific user. The channel-wide message is sent when a new experience is created. The message is sent to a specific user when a new user joins the globe lobby.

The data sent across starts in the form of a struct called ChannelData, which contains a few key data:

struct ChannelData: Codable {
  var channelName: String
  var channelID: String
  var position: SIMD3<Float>
  var systemImage: String?
}

The ChannelData struct conforms to Codable, which ensures we are able to encode all the data of our struct and provides methods to easily do so.

To send this ChannelData struct across the network, we need to create a message that can be sent across the Agora RTM network. The class for this message will be AgoraRtmRawMessage. The documentation for this class is available here:

https://docs.agora.io/en/Real-time-Messaging/API%20Reference/RTM_oc/Classes/AgoraRtmRawMessage.html?platform=iOS

Our data can be encoded with the Swift JSONEncoder like this:

// channelData: ChannelData
let jsonData: Data = try! JSONEncoder().encode(channelData)

Force unwrap here should be handled gracefully

The jsonData object is of type Data, which is ready to use in the creation of an AgoraRtmRawMessage instance. The second property when creating an AgoraRtmRawMessage is a description, which we will use to state the type of data in the message by the RawRTMMessageDataType's rawValue, which is a String.

We are sending a message of type RawRTMMessageDataType.channelAvailable, so our AgoraRtmRawMessage creation will look like this:

let channelDataRTM = AgoraRtmRawMessage(
  rawData: jsonData,
  description: RawRTMMessageDataType.channelAvailable.rawValue
)

And to send that to our lobby channel, which is used for the globe scene, we can just do this:

// rtmChannel: AgoraRtmChannel
rtmChannel.send(channelDataRTM)

Receiving

Now that our message has been sent across the network, we need to use the Agora RTM Delegate methods to catch the incoming messages and interpret them as channels available to join.

As this message is sent across the channel, we need to use the AgoraRtmChannelDelegate to catch that, as well as the AgoraRtmDelegate for when messages are sent directly to users.

The same message may be received either channel-wide or directly to a user, so instead of handling them individually, we can catch both delegate callbacks and send the messages through one set of logic in another method, handleIncomingMessage, like this:

extension CollaboratARDelegateEntity: AgoraRtmDelegate, AgoraRtmChannelDelegate {
  func channel(
    _ channel: AgoraRtmChannel,
    messageReceived message: AgoraRtmMessage,
    from member: AgoraRtmMember
  ) {
    handleIncomingMessage(message, channel: channel, from: member)
  }

  func rtmKit(
    _ kit: AgoraRtmKit, messageReceived message: AgoraRtmMessage,
    fromPeer peerId: String
  ) {
    handleIncomingMessage(message)
  }

  fileprivate func handleIncomingMessage(
    _ message: AgoraRtmMessage,
    channel: AgoraRtmChannel? = nil,
    from member: AgoraRtmMember? = nil
  ) {
    // Handle messages
  }
}

In the project on GitHub there is a big switch case that handles all the different messages, but we will initially discuss only the channelAvailable type we are interested in.

First, to determine the type of message, we can create an instance of RawRTMMessageDataType using the rawValue initialiser inside the handleIncomingMessage function:

let messageType = RawRTMMessageDataType(rawValue: message.text)

Then, if our messageType is .channelAvailable, we can decode the message back into a ChannelData type:

if messageType == .channelAvailable {
  let channelData = try! JSONDecoder().decode(
    ChannelData.self, from: rawMessage.rawData
  )
  // show channel data on globe...
}

In our example project, there is a method to take a ChannelData object and spawn a point on the globe, called spawnHitpoint. spawnHitpoint takes the data stored inside ChannelData to set the position of the target and the behaviour, when clicked.

The ChannelData object contains the unique channelID to join for both the real-time messaging channel and the audio channel. Read on to see how that is joined and what happens after joining the collaboration scene.

Globe Scene Conclusion

That's all the data that needs to be sent across Agora RTM network in the initial globe scene. All each device needs to know is what available channels there are and where to place them on the globe. No audio channels are joined when in the lobby. That comes in the next section.

Collaboration Scene

A lot more data are sent around in the collaboration scene, including the positions of all the other users, any models they add, move, and delete, as well as the data interpreted from the audio channel they are all members of.

The first step is joining the scene. This is triggered by a call to a method called setState, which contains an enum of the following type:

enum CollaborationState {
  case globe
  case collab(data: ChannelData)
}

We will just cover the collab case, leaving globe empty in the snippets:

switch collabState {
case .globe:
  // handle joining globe scene
  break
case .collab(let collabData):
  // the following two lines handle the creation of entities
  // that are not shared across the network
  self.tearDownGlobe()
  self.launchCollabScene()

  // Enable local microphone, then join Audio channel
  self.rtcKit.enableLocalAudio(true)
  self.rtcKit.joinChannel(
    byToken: AppKeys.appToken, channelId: collabData.channelID,
    info: nil, uid: self.rtcID ?? 0
  ) { chname, uid, _ in
     self.rtcID = uid
     self.updatePositions()
  }

  // Create + Join RTM Channel using collabData
  self.channels.collab = self.rtmKit.createChannel(
    withId: collabData.channelID, delegate: CollaboratARDelegateEntity.shared
  )
  self.channels.collab?.join(
    completion: { joinResponse in
      if joinResponse != .channelErrorOk {
        print("could not join collabs channel: \(joinResponse.rawValue)")
      }
      CollaboratARDelegateEntity.shared.joinedCollabChannel(
        self.channels.collab!
      )
    }
  )
}

In the above snippet, rtcKit is our AgoraRtcEngineKit instance for joining the audio channel, and rtmKit is our AgoraRtmKit for joining the real-time messaging channels.

The audio channel ID is taken from collabData.channelID, the same as our real-time messaging channel. These can be different, but they are the same strings in this project example. The token handling for audio and RTM are not covered in this post or the example project. For testing, you can use a development project that does not require a token, create a temporary token, or generate the token from a token server.

Once we have joined the RTM channel, the designated host of the channel will send information about all the objects that are currently in the scene to the new joiner.

To do so, we use the AgoraRtmChannelDelegate method, memberJoined. From here, we check that the new member's channelId is different from the lobby (the globe channel). And if we have been in the channel longer than anyone else, we gather all the data about entities in the scene and send that to the newest member.

The data shared will be an array of type [ModelData]. ModelData is defined here:

struct ModelData: Codable {
    /// Unique ID for this model
    var id: String
    /// Name of the USDZ file to use
    var usdz: String
    /// Scale of the entity
    var scale: SIMD3<Float>
    /// Orientation of the entity
    var rotation: simd_float4
    /// Position of the entity
    var translation: SIMD3<Float>
    /// Full Transform of the entity based on the scale, rotation and translation
    var transform: Transform {
      Transform(
        scale: scale, rotation: simd_quatf(vector: rotation),
        translation: translation
      )
    }
    /// Last user to move this entity
    var owner: String
    init(id: String, usdz: String, transform: Transform, owner: String? = nil) {
        self.id = id
        self.usdz = usdz
        self.scale = transform.scale
        self.rotation = transform.rotation.vector
        self.translation = transform.translation
        self.owner = owner ?? CollaborationExpEntity.shared.rtmID
    }
}

Sending All Models

All the entities we want to share conform to a custom protocol HasCollabModel. They are children of an entity we have a reference to, called collabBase.

guard let collabChildren = collabBase.children.compactMap{
  $0 as? HasCollabModel
} else { return }
let allModelData: [ModelData] = collabChildren.map { $0.modelData }

The struct ModelData conforms to Codable, which means that it and an array of ModelData can be encoded with JSON as easily as seen earlier with ChannelData, and then sent to our user the same way:

// member: AgoraRtmMember
let jsonData = try! JSONEncoder().encode(allModelData)
let rawMessge = AgoraRtmRawMessage(
  rawData: jsonData,
  description: RawRTMMessageDataType.multiCollabModels.rawValue
)
self.rtmKit.send(rawMessge, toPeer: member.userId)

Receiving All Models

On the other end, the device that just joined the channel needs to receive these models and place them in the scene.

We already have the RawRTMMessageDataType, as shown in the globe scene. We need to add a case to handleIncomingMessage to catch when our messageType is .multiCollabModels and then go on to add those models to the scene:

if messageType == .multiCollabModels {
  let modelDatas = try! JSONDecoder().decode([ModelData].self, from: rawMessage.rawData)
  CollaborationExpEntity.shared.collab?.update(with: modelDatas)
}

Let's take a look at what the update method does:

func update(with models: [ModelData]) {
  for modelData in models {
    if let child = self.findEntity(named: modelData.id) as? CollabModel {
      child.update(with: modelData)
    } else {
      self.createModelEntity(with: modelData)
    }
  }
}

First, it checks if an entity with the same ID already exists in the scene. If it does, then the CollabModel entity transform is updated with a very short animation. Otherwise, a new CollabModel is created based on the data contained inside ModelData:

func createModelEntity(with modelData: ModelData) {
  let collabMod = CollabModel(with: modelData)
  self.collabBase.addChild(collabMod)
}

Updating User Positions

All the remote augmented reality users will have their location relative to the ground square sent to the channel so that all the other users know where they are in the world. The called method is CollaborationExpEntity.updatePositions. It is called on an interval of 0.3 seconds after you have joined a channel and set the ground square. The 0.3 seconds value is arbitrary and can be altered if a faster update is desired.

The type of object sent over in this is of a custom struct, PeerData:

struct PeerData: Codable {
  var rtmID: String
  var rtcID: UInt?
  var usdz: String?
  var scale: SIMD3<Float>
  var rotation: simd_float4
  var translation: SIMD3<Float>
  var colorVector: [CGFloat]
  var color: Material.Color {
    Material.Color(
      red: self.colorVector[0], green: self.colorVector[1],
      blue: self.colorVector[2], alpha: self.colorVector[3]
    )
  }
  var transform: Transform {
    Transform(
      scale: scale, rotation: simd_quatf(vector: rotation),      translation: translation
    )
  }
}

The color and transform properties are created by using the other properties because Material.Color and Transform do not conform to Codable themselves. The above struct contains all the data needed to distinguish between the different remote users.

Sending User Position

In the updatePositions method, we first get the cameraTransform and then get that transform relative to the collabBase, which is the entity set when joining the collab session.

If we are running on an iOS device and not the simulator, we proceed to create a PeerData object, get the channel instance, and send our raw message with the description showing that it is of .peerUpdate type.

extension CollaborationExpEntity {
  @objc func updatePositions() {
    if let camTransform = self.arView?.cameraTransform,
    let localisedCam = self.collab?.collabBase.convert(
      transform: camTransform, from: nil
    ) {
      // Check that we are on iOS, and the cameraMode is .ar
      #if os(iOS) && !targetEnvironment(simulator)
      if self.arView?.cameraMode == .ar {
        // Create PeerData Codable object
        let peerData = PeerData(
          rtmID: self.rtmID, rtcID: self.rtcID, usdz: nil,
          scale: .one,
          rotation: localisedCam.rotation.vector,
          translation: localisedCam.translation,
          colorVector: self.assignedColor
        )
        // Encode PeerData
        let jsonData = try! JSONEncoder().encode(peerData)
        let rawMessage = AgoraRtmRawMessage(
          rawData: jsonData,
          description: RawRTMMessageDataType.peerUpdate.rawValue
        )
        // Send positional data to the entire channel
        self.channels.collab?.send(rawMessage)
      }
      #endif
    }
  }
}

Receiving User Position

Very similarly to receiving all the model updates in the previous example, we catch the case when our messageType is .peerUpdate, decode it, and send it to a collaboration update function.

if messageType == .peerUpdate {
  let peerData = try! JSONDecoder().decode(
    PeerData.self, from: rawMessage.rawData
  )
  CollaborationExpEntity.shared.collab?.update(with: peerData)
}

Inside the update function, we either create a new PeerEntity, or update the transform of the existing one:

func update(with peerData: PeerData) {
  if let child = self.findEntity(named: peerData.rtmID) as? PeerEntity {
    child.update(with: peerData)
  } else {
    self.createPeerEntity(with: peerData)
  }
}

Updating a Model

A similar update to .multiCollabModels is .singleCollabModel. This update is sent almost exactly the same way as .multiCollabModels, but it is a single model rather than an array.

We send this update instantly on creation of a model, and periodically if a model is selected and ready to be moved around the scene.

When joining the scene, a scheduled timer is initially fired to execute the function updatePositions(), which sends user location every 0.3 seconds (but could be more frequent if required). When a model is selected, updatePositions sees that selectedBox is not null, so it triggers a function (sendCollabData) that sends its position as an update to the channel:

func sendCollabData(for collabEnt: HasCollabModel) {
  let jsonData = try! JSONEncoder().encode(collabEnt.modelData)
  let rawMessge = AgoraRtmRawMessage(
    rawData: jsonData,
    description: RawRTMMessageDataType.singleCollabModel.rawValue
  )
  self.channels.collab?.send(rawMessge)
}

The message is then received by handleIncomingMessage on the other end and calls the same method as with .multiCollabModels:

if messageType == .singleCollabModel {
  let modelData = try! JSONDecoder().decode(ModelData.self, from: rawMessage.rawData)
  CollaborationExpEntity.shared.collab?.update(with: [modelData])
}

Deleting an Entity

When deleting an entity, the enum .removeCollabModel is used, and the individual model data is sent with the following message:

// model: CollabModel, collabChannel: AgoraRtmChannel
let jsonData = try! JSONEncoder().encode(model.modelData)
let rawMessge = AgoraRtmRawMessage(
  rawData: jsonData,
  description: RawRTMMessageDataType.removeCollabModel.rawValue
)
collabChannel.send(rawMessge)

On receiving the delete message, the model in question is located in the scene and is removed from its parent:

if messageType == .removeCollabModel {
  let modelData = try! JSONDecoder().decode(ModelData.self, from: rawMessage.rawData)
  if let foundEntity = CollaborationExpEntity.shared.collab?.findEntity(
    named: modelData.id
  ) {
    // Found the entity we want to delete, remove it from the scene
    foundEntity.removeFromParent()
  }
}

Testing

All of the above methods can be found in the full example project:

AgoraIO-Community / Collaborative-AR-RTM

A collaborative Augmented Reality iOS experience using Real-time Messaging with RealityKit

Some of the methods contain additional code to aid the rest of the experience. The above explanations focus mainly on using the Agora Real-time Messaging SDK to enable remote collaboration.

If anything is unclear, please feel free to send me a message on Twitter.

Other Resources

For more information about building applications using Agora Video and Audio Streaming SDKs, take a look at the Agora Video Call Quickstart Guide and Agora API Reference.

I also invite you to join the Agora Developer Slack community.

Conclusion

I hope you've enjoyed this post and the project that comes along with it. The idea of this project is to showcase how Agora Real-time Messaging SDK can be used to create interactive experiences with people around the world.

Multi-User Collaborative iOS AR Experiences with Agora (Part 1 of 2)

Max Cobb — Fri, 20 Aug 2021 13:45:36 +0000

Agora is much more than a video streaming SDK. One other SDK available from Agora is the Real-time Messaging SDK.

The Agora Real-time messaging SDK can be used to send chunks of data across the network, including encoded structs, files, and plain text.

CollaboratAR is an example project where people can either join an active session or create a new one, located at a place of their choosing on a 3D globe floating in front of them. Once joined, users can add, modify, and remove objects from the scene, with updates sent to everyone else in the channel.

When joining a session, the users also join an audio channel using the Agora Audio SDK, so they can hear each other as well as see each other's location.

This is what the session might look like from the macOS view, which does not have augmented reality:

This post shows you how to use the app to connect with other people around the world. If you want to see how the app is made, go to the follow-up blog post here:

Multi-User Augmented Reality Experiences with Agora (Part 2 of 2)

Max Cobb ・ Aug 20 ・ 11 min read

#swift #augmentedreality #ios #agora

App Flow

On first launching the app, the user is instructed to find a horizontal surface that the globe will spawn from. After finding a good horizontal surface, the globe is spawned. Any active channels will appear on the globe in the form of red and orange circular targets.

The user can either select one of these targets to join a current room, or create a room by clicking elsewhere on the globe model.

After a user joins a channel, the following things will happen:

The user will join an audio channel and can hear all other users who have their microphones connected.
A collection of models appear at the bottom that the user can place in the scene.
The locations of all remote users appear in the form of a translucent sphere and a visible microphone if their microphones are on.
Any models that have already been placed into the scene will appear in the place set by other users.

The user can tap on their model of choice and then tap on the scene to place the model anywhere they like. Any placements of objects will be immediately shared with everyone else in the channel at the relevant location of the scene for them.

A user can select a model to delete, scale, rotate, or move it around the scene. These updates are sent across the network. On receiving the update, the local session will animate the object from the old transform to the new one.

Technologies Used

How It Works

To see how the sample app works, you can check out our in-depth blog post on how each part interacts with the Agora Real-time Messaging SDK as well as the Agora Audio SDK:

Multi-User Augmented Reality Experiences with Agora (Part 2 of 2)

Max Cobb ・ Aug 20 ・ 11 min read

#swift #augmentedreality #ios #agora

The full source code is available here:

AgoraIO-Community / Collaborative-AR-RTM

A collaborative Augmented Reality iOS experience using Real-time Messaging

What You Can Do with This Technology

Using the techniques outlined in this project, you could create experiences for people to share ideas in augmented reality or virtual reality, as well as in 2D. Anything that someone does on their own device anywhere in the world could easily be represented in another user's experience using the Agora Real-time messaging SDK.

The same techniques could be used to make a multiplayer game, live interactive support, or something more basic, like a live chat feature.

Other Resources

For more information about building applications using the Agora Video and Audio Streaming SDKs, take a look at the Agora Video Call Quickstart Guide and the Agora API Reference.

I also invite you to join the Agora Developer Slack community.

Getting Started with RealityKit: Models

Max Cobb — Mon, 01 Feb 2021 09:48:34 +0000

There are two main ways of adding models to your RealityKit scene, importing from a USD (usd, usda, usdc, or usdz) or a Reality file, or creating a mesh using the MeshResource generators provided with Xcode. There are a few shapes that can be generated via the MeshResource generators, other shape must be imported via a USDZ model or experience file from Reality Composer.

This post also includes examples of how to import USD files from remote URLs, and customise imported models.

Importing from USDZ
- Loading with Entity Hierarchy
- Loading an AnchorEntity
- Loading a BodyTrackedEntity
- Loading a Flattened Model
- Loading a Remote USDZ File
Generating Shapes
Customising Models
- With Hierarchy
- Without Hierarchy

Importing from USDZ

One of the more popular and exciting methods of importing a model to RealityKit would be importing it from a USD file. There are several ways to do this, and not knowing the subtle differences can lead you to miss parts of the imported models such as animations.

All the import methods either require a name and an optional bundle (omit the bundle to use the app's main bundle), or require a url and a unique resource name. To see how to use each see this link for passing a name and bundle, and this link for the url and resource name methods. The rest of this post will only discuss the name + bundle examples for simplicity.

Loading with Entity Hierarchy

To maintain information bundled in the USD file such as animations, the best import methods are Entity.load and Entity.loadAsync. The difference between these two is that load will block the thread it is called on until the file has been imported, while loadAsync will use a LoadRequest (where the Output is an Entity) to get the model imported while allowing you to continue on the current thread.
Another difference between the two, is that load can throw an error, while the error found in loadAsync can be caught in the LoadRequest slightly differently.

Examples:

Simple loading of a USDZ file named toy_robot_vintage.usdz synchronously:

guard let entity = try? Entity.load(
  named: "toy_robot_vintage"
) else {
  // failed to load entity
  return
}
// do something with entity

Loading a USDZ file named toy_robot_vintage.usdz asynchronously:

_ = Entity.loadAsync(named: "toy_robot_vintage").sink(
  receiveCompletion: { loadCompletion in
    // Added this switch just as an example
    switch loadCompletion {
      case .failure(let loadErr):
        print(loadErr)
      case .finished:
        print("entity loaded without errors")
    }
  }, receiveValue: { entity in
    // do something with entity
  }
)

Loading multiple USDZ files asynchronously, where the parameter in receiveValue is of type [Entity], and will only reach there once all models have been imported:

_ = Entity.loadAsync(named: "toy_robot_vintage")
  .append(Entity.loadAsync(named: "gramophone"))   
  .append(Entity.loadAsync(named: "fender_stratocaster"))
  .collect()
  .sink(receiveCompletion: { loadCompletion in
    // Add switch case for loadCompletion
  }, receiveValue: { entities in
    // Do something with returned entities array.
  }
)

Note: You may notice that there is a small freeze and when you add the entity to your scene, even when using loadAsync. The freeze is due to large meshes or images stored in the materials being loaded into memory, which must be executed on the main thread.

Loading an AnchorEntity

In a very similar way to load and loadAsync, loadAnchor can load your USD file in as an AnchorEntity. This lets you place it directly in your scene without needing to make a separate anchor for it. Any animations in your USDZ file will also be available, as with load and loadAsync.
Example:

guard let entity = try? Entity.loadAnchor(
  named: "toy_robot_vintage"
) else {
  // failed to load entity
  return
}
entity.anchoring = AnchoringComponent(
  .plane(
    .horizontal, classification: .floor, minimumBounds: [1, 1]
  )
)
arview.scene.addAnchor(entity)

In this example, the AnchoringComponent is set to place the contents of the USDZ file to a horizontal plane categorised as the floor with a size of at least 1m x 1m. Only loadAnchor is shown here, but loadAnchorAsync works the same way as loadAsync, it will just return an AnchorEntity instead of an Entity.

Load a BodyTrackedEntity

When you want to import a model that is rigged up to be animated by a real person, you instead want to use loadBodyTracked or loadBodyTrackedAsync to import a BodyTrackedEntity.

guard let entity = try? Entity.loadBodyTracked(
  named: "robot"
) else {
  // failed to load entity
  return
}
// do something with body tracked entity

For more information on tracking human bodies in AR, this session from WWDC 2019 is a great resource:

https://developer.apple.com/videos/play/wwdc2019/607

Loading a Flattened Model

When you just want to import a Model from a USDZ file, removing entity hierarchy as well as animations, you can import using loadModel or loadModelAsync. This will turn your 3d model of what would be various levels of nodes and children into a single Entity containing meshes and an array of all the materials.

In my opinion, this difference can be confusing to someone who is new to RealityKit and wanting to import their model that has an animation from a USDZ file; as seen by various questions on Stack Overflow asking what happened to their USDZ animations when importing a file using these methods. However, once aware of this difference, it is very useful to not have all the overhead of a large hierarchy of entities that you may not need.

Loading a Remote USDZ File

Currently there is no straightforward or builtin way to import USDZ files from remote URLs, but thankfully there are a few solutions, here's one from StackOverflow:

answer re: How to download USDZ by URLSession?

Dec 9 '19

This problem is solved let entity call on the main thread.

let url = URL(string: "download usdz url")  
let documentsUrl = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first!  
let destinationUrl = documentsUrl.appendingPathComponent(url!.lastPathComponent)  
let session = URLSession(configuration: URLSessionConfiguration.default, delegate: nil, delegateQueue: nil)  
var request = URLRequest(url: url!)  
request.httpMethod = "GET"  
let downloadTask = session.downloadTask(with:

…

Open Full Answer

I've also altered the solution into a synchronous function with an example usage at the bottom in this gist:

Generating Shapes

In its current state, the provided generators include Box (cuboid), Plane, Sphere or Text. Ironically, out of the three shapes in the RealityKit logo, only one of those (a sphere) can be produced without importing USDZ files.

All the mesh initialisers can be found on this page:

https://developer.apple.com/documentation/realitykit/meshresource

The easiest way to add one of these shapes to your RealityKit scene would be to use the ModelEntity initialiser:

let newModel = ModelEntity(mesh: .generateBox(size: 1))

Notice here, no material was specified in the ModelEntity initialiser, which means this broken shader sort of look is applied to the cube shape. This is the case for all the other MeshResource initialisers.

For in-depth information on how to create materials in RealityKit and a few tricks with them, see my previous post:

Getting Started with RealityKit: Materials | by Max Cobb | AR/VR Journey: Augmented & Virtual Reality Magazine

Max Cobb ・ Feb 25, 2021 ・
maxxfrazer.Medium

Customising Models

Once you bring a USDZ file in your scene there may be some customisation you want to do to it, such as changing a material, or actually moving around one part of the model.

If you wanted to change the material of a model imported from a USDZ file, there are a couple of ways you can do so.

With Hierarchy

If you import your model with anything other than loadModel or loadModelAsync you can replace the material of your choosing with ease.
Take this file, gramophone.usdz:

In the above image I've selected the record in the middle, and on the right we can see its name is GramophoneRecord1, and if we select the materials tab we can see that it only has one material named pxrUsdPreviewSurface1SG:

There is actually only one material in this whole USDZ file, however different parts of it are used on different meshes in the file.

Next all we need to do is find the Entity named GramophoneRecord1 and replace its material with our own green one.

if let gramRecord1 = gramophoneModel.findEntity(
  named: "GramophoneRecord1"
) as! HasModel {
  gramRecord1.materials = SimpleMaterial(
    color: .green, isMetallic: true
  )
}

Original on the left, altered material on the right.

Without Hierarchy

If you imported your model using loadModel or loadModelAsync, then as discussed earlier there is no hierarchy of entities, just the one ModelEntity consisting of a ModelComponent with a single mesh and a collection of materials. Take this file, fender_stratocaster.usdz:

The USDZ file itself consists of two meshes one for the stand and one for the guitar. Each of those meshes contain materials, the stand has one, and the guitar actually has 19, so once imported with loadModel the ModelEntity will have one mesh and a list of 20 materials.

Let’s replace the guitar body with a generated SimpleMaterial. It could take a bit of guess work to figure out which one in those 20 is that material, but by clicking on the part of the model that has the material you want to change Xcode highlights it in the side bar, see that it is the very first material in the guitar’s list of materials. It isn’t incredibly clear, but the first material when imported with loadModel is actually that of the guitar stand, leaving the guitar body material the second in the list, index 1.

After importing the model, simply running this I was able to change the guitar body to an electric blue:

modelAdjusted.model?.materials[1] = SimpleMaterial(
  color: .blue, isMetallic: true
)

Original on the left, altered material on the right

The version without hierarchy leads to a bit less code, but doesn't allow for embedded animations, and can lead to issues if our USDZ file ever changes such that the material order differs. Whereas the version with hierarchy lets us find a specific part of the model, which would be preferred with a more complex model.

I hope you found something useful in this post, give it a like if you did! 👏👏👏

Send me a message on twitter or leave a comment if there’s something I’ve missed out or if there’s something that isn’t working for you in RealityKit.

Getting Started with RealityKit: Materials

Max Cobb — Mon, 25 Jan 2021 13:34:20 +0000

Setting the right material to your mesh in RealityKit is a crucial part of setting the scene and making your models look just right, and there are a few tools at your disposal with RealityKit.

I’ve already talked about the newest and most complex type, VideoMaterial here:

RealityKit VideoMaterials: Add Video textures to your ARKit Scene | Medium

Max Cobb ・ Jul 25, 2020 ・
maxxfrazer.Medium

But there are a few other types of materials you can set by code, and ways to tailor them for your needs that you may not already know, learn about all these other types and what you can do with them in this post.

Basic Materials
- UnlitMaterial
- SimpleMaterial
- OcclusionMaterial
Materials with Textures
- Material Transparency
- UIImage to TextureResource

Basic Materials

UnlitMaterial

UnlitMaterial is the most simple of all of these, it only has properties baseColor and tintColor. If you’re coming from SceneKit, or many other rendering engines, baseColor is essentially the diffuse of the material. Here is how you may set the baseColor of an UnlitMaterial to UIColor.systemGreen:

var mat = UnlitMaterial()
mat.baseColor = .color(.systemGreen)
// equivalent to
let mat = UnlitMaterial(color: .systemGreen)

Sphere with green UnlitMaterial

Notice that without any interaction with the light this sphere looks like it could be just a flat circle.

The other parameter, tintColor is also there, but is not very useful when applying a simple color as the baseColor, but for example you can do this:

var mat = UnlitMaterial()
mat.baseColor = .color(.white)
mat.tintColor = .cyan

Sphere with a cyan UnlitMaterial

We can see that the color ends up being cyan, rather than white, or a combination of them both. If you try any colours that have no crossover, such as red and blue you will get a result color of black, rather than purple which may be expected. tintColor is best left for when the baseColor is a white image rather than a color.

SimpleMaterial

Moving on to SimpleMaterial, we can use the initialiser with no parameters and set the baseColor like so:

var mat = SimpleMaterial()
mat.baseColor = MaterialColorParameter.color(.systemGreen)

The subtle darker shading at the bottom right, and lighter at the top left shows it’s interacting with light

There are two additional properties introduced with SimpleMaterial; metallic and roughness. By using the default initialiser those properties are set to float(0.0) and float(1.0) respectively, meaning that the material is not metallic and it has a rough coat. These properties can also be set to a texture, which will be covered below.

Note: If you print out the properties here it actually says the opposite, that metallic is 1.0 and roughness is 0.0, but this is a bug, if you set the properties as such you will get a different visual result. I’ve opened a feedback to Apple

Here’s that same sphere with metallic set to 1.0, and roughness set to 0.0:

Shiny green sphere

OcclusionMaterial

OcclusionMaterial is very different, instead of writing its own color or texture properties to the depth buffer it writes alpha, meaning that everything behind it in your RealityKit scene will not be rendered, and instead the camera feed or otherwise RealityKit environment background (grey in my examples) will be shown instead.

Here is an example where I have positioned two spheres with a cube in front of each of them. One cube has a SimpleMaterial with baseColor of blue, and the other has the same sized cube with OcclusionMaterial.

Notice how there is a bite taken out of the sphere on the right in place of where the cube should be. This can be used to place an OcclusionMaterial in place of a real world object, which can then occlude any rendered objects that are supposed to be behind them. This is what the scene understanding feature does, that automatically detects surfaces and objects to then occlude virtual objects behind them.

Materials with Textures

In many instances you will not want a simple color as the material of your object in the scene. You may want to add a texture, for example on a cube to give the look of a die rolling around your scene, or a map of Earth to have a spinning globe in your scene.

For SimpleMaterial and UnlitMaterial we can set the baseColor to a texture in the same way.

let unlitMat = UnlitMaterial()
let litMat = SimpleMaterial()
if let texParam = try? TextureResource.load(named: "planet_earth") {
  unlitMat.baseColor = .texture(texParam)
  litMat.baseColor = .texture(texParam)
}

Note: The planet earth image I used can be found here.

With SimpleMaterial we can also apply textures to metallic and roughness. When we apply a float value of 1.0 to either of those, that is equivalent to giving a completely white image, and similarly 0.0 is a black image.

If we then want the oceans to be metallic, and the land to be non metallic, we can apply an image such as the following to the metallic property:

planet_earth_ocean.png

All the ocean is set to white (1.0), and the land is black (0.0). If we also want to set the ocean to be smooth and the land to be rough, we can apply the inverted image to the roughness property:

planet_earth_land.png

Those can be applied like this:

if let earthParam = try? TextureResource.load(
  named: "planet_earth"
), let oceanParam = try? TextureResource.load(
  named: "planet_earth_ocean"
), let landParam = try? TextureResource.load(
  named: "planet_earth_land"
) {
  mat.baseColor = .texture(earthParam)
  mat.metallic = .texture(oceanParam)
  mat.roughness = .texture(landParam)
}

The resulting globe shows reflective oceans and matt land:

Material Transparency

You may have trouble using a transparent image on a model in RealityKit, I believe this is to do with the blend modeapplied to the model. When setting a charming smiley face image with transparent edges to the baseColor property of an UnlitMaterial, then setting that to a basic plane we get these black corners where the opacity is 0.

However, if we simply set the tintColor to white with an alpha component less than 1, we get the opacity back:

mat.tintColor = .init(white: 1, alpha: 0.999)

When trying the same with instead a SimpleMaterial there is one more issue, and that is that the transparency only works when the metallic property is set to 1.0, or true when using the initialiser property isMetallic.

This issue has been around since the first version of RealityKit (iOS 13) in one form or another, and I’m expecting it will be solved once we’re able to manually set the blend mode of models.

Translucency with a plane colored material works as you might expect - just change the alpha component of the UIColor. However, due to a bug in RealityKit, if you ever update the mesh with either a coloured or textured material, you will need to replace the entire ModelComponent like this:

let newModel = ModelEntity(mesh: .generatePlane(width: 1, height: 1))
var mat = SimpleMaterial(color: UIColor.blue.withAlphaComponent(0.5), isMetallic: true)
newModel.model?.materials = [mat]
// next line breaks transluency
newModel.model?.mesh = .generatePlane(width: 1, height: 1)
// next line fixes it
newModel.model = ModelComponent(
    mesh: .generatePlane(width: 1, height: 1),
    materials: [mat]
)

UIImage to TextureResource

There is currently no way (that I’m aware of) to convert a UIImage to a TextureResource, which is a huge blow if you want to download images to your applications and set them to models in the scene. I’m hoping that this too will be fixed one way or another in future versions of RealityKit.

Thanks to Ryan Kopinsky for suggesting I write about some of these things when we were discussing them. I hope many RealityKit developers find this a useful resource when creating materials for Augmented Reality scenes!

As usual, give this article some claps if you can, and feel free to message me on Twitter or leave a response here if you have any questions.

Getting started with RealityKit

Max Cobb — Sun, 24 Jan 2021 09:53:50 +0000

Before WWDC19, myself and other Augmented Reality (AR) enthusiasts were anticipating many updates to SceneKit to come along with some shiny new ARKit features; however this was not the case.

Instead Apple decided to create a whole new framework to implement AR, and dubbed this new framework RealityKit. Here we have a whole new rendering engine, made from the ground up with AR not just in mind, but as the top priority. Although there may be some features missing; with this as v1.0 I have high hopes that this will become the standard for high-performance AR. Due to the name I also believe it could be intended for use with both Augmented and Virtual Reality, but I have yet to see any evidence for this.

I started looking into RealityKit right from the start, however there’s currently limited materials out there helping people to find their feet, both if someone who is new to AR or is coming from other AR Frameworks, such as SceneKit. This collection of articles aims to get you started with understanding the new format of creating AR content programmatically with RealityKit, with new articles coming out as I use the features.

2020

To see what was changed with RealityKit during WWDC 2020, refer to this story:

WWDC20: What’s new in ARKit and RealityKit? | by Max Cobb | Medium

Max Cobb ・ Jun 28, 2020 ・
maxxfrazer.Medium

The Collection

If you want to see code examples without support material; take a look at this repository below. In this repo I’m adding the new features together into a game similar to one that Apple showed off at WWDC19, for which they have yet to release the code for.

maxxfrazer / RealityKit-CardFlip

An example app using Apple's new RealityKit framework

RealityKit CardFlip

I've been using RealityKit a lot since it was released, and am writing Medium posts about different aspects of it as well as adding examples to this app.

For specific details of what's covered, check out my Medium.

Example code for Apple's new AR framework RealityKit, demonstrating the following things:

The Entity Component System
- Inheritance/Protocol Oriented
- AnchoringComponent (with Targets)
- ModelComponent
- CollisionComponent
HitTests
Animations
AR Coaching

TODO:

Add game feedback
- Score counter
Audio Playback

Following the RealityKit examples from this keynote, as the code used has not yet been released:

Building Apps with RealityKit

Wishlist for RealityKit:

Particles
Custom Mesh Generation

View on GitHub

If you’ve enjoyed this article and want to see more in this RealityKit series follow me for updates or check out my Twitter.