Now Playing and Remote Commands on tvOS

Welcome to Now Playing and Remote Commands on tvOS. I'm Justin Voss, an engineer on tvOS and in this talk we'll cover how you can provide great metadata and playback experiences for your viewers. First, let's talk about providing Now Playing information. Having accurate and complete metadata about your content will help viewers to understand what they're watching across a variety of interfaces across tvOS and iOS. The metadata you provide is displayed in several places. For video content, the built-in AVPlayerViewController displays metadata along the top of the video screen.

For both video and audio content the TV Remote app for iOS would both display metadata and playback controls. And for audio content, your metadata will be displayed in a badge in the corner of the screen and as a full-screen display when the user is idle.

Let's look at some screenshots of these to better understand them. In this screenshot, you can already tell that we're watching a WWDC talk from last year because the metadata onscreen makes that super clear. This is an example of using the built-in AVPlayerViewController for video content. Here you can see the same talk, but displayed in the TV Remote app for iOS. You can also see the playback controls which we'll cover later in the talk. For audio content, when the Now Playing information changes the listener will see a notification on their screen with the updated metadata. In this screenshot here, you can see what this would look like if we were listening to an audio only version of that WWDC talk.

Finally, if the user puts down the remote for a while tvOS will automatically display the Now Playing metadata in a full-screen view like you see here which really shows off the artwork.

So, hopefully I've convinced you that providing the complete set of Now Playing information is worth your time.

Let's talk about some of the different ways you can do that. You have a few choices so it depends on what kind of technology you're using in your app.

Those of you using TVML can add metadata using the MediaItem JavaScript object.

This object has several properties on it for Now Playing info, including a title, subtitle and description, you can provide a URL to an artwork image and you can even provide information about the content rating such as PG-13 or R and whether the content is explicit. In the code sample, here you can see that I'm creating a MediaItem object with a URL to a video and then configuring a title and description on it and providing a URL to an artwork image. If you're using AVKit to play your content you'll want to use the AV metadata item class to provide Now Playing info.

For each piece of metadata you want to provide, such as a title or artwork, etcetera you'll create an AV mutable metadata item and configure several properties on it. You provide an identifier which tells AVKit what kind of metadata this represents. This is how we know to tell the difference between the object that represents a title versus the description and so on. You provide the value of the metadata itself, which may be a string or raw data bytes. You should set the extendedLanguageTag to specify what language the metadata is intended for. Unless you have a good reason to do otherwise we strongly recommend that you use the value und which means undefined.

The way this property works is that it will only be visible to the user if the user's locale matches the metadata's locale. So, for example, if you had a movie that was produced in the USA you may be tempted to provide the title of that movie with the language tag of English since the title is in English. But if you did that then a viewer in Germany wouldn't see any title at all because the metadata language doesn't match their language. But if you had used the tag und then everyone will see the metadata regardless of their language settings.

Finally, in some cases you need to give us a hint about how to interpret the value you've provided. In the case of providing artwork image you'll give us the raw bytes of the image as the value and then use the dataType to tell us if the image is a JPEG or a PNG. Let's look at some code samples to get a feel for how this works. Here you can see the code where I'm creating an AV playerItem and giving it a title and description.

In the first large code block I'm creating an AVMutableMetadataItem to represent the title. The first thing that I do is I set the identifier to AVMetadataCommon IdentifierTitle so that AVKit knows that this metadata item is the title. Then I set the actual value as a string. The AV metadata item API requires these values to be NS object types so casting the string to an NSString.

Finally, I'm setting the extendedLanguageTag to und because the title of this talk should appear the same worldwide. In the second large code block, I'm doing most of the same things to assign the description. Other than providing a different value you can see that on this line I'm setting the identifier to AVMetadataCommon IdentifierDescription so that AVKit knows how to interpret this item.

Finally, on the last line I assign an array of these metadata items to the AV playerItem's external metadata property. If you wanted to provide the release date of your content as a metadata item you can do that if you format your release date as a string in a specific format. This code sample shows you how you can do that. Here you see that I'm creating the date object for an arbitrary date. Then I create a date formatter and provide a specific date format string here on this line. To create the metadata item, I use the identifier AVMetadataCommon IdentifierCreationDate and finally, I use the date formatter to convert the date object into a formatted string and cast that string as an NSString.

Now in AVPlayerViewController viewers will be able to see the release year of the video alongside the other Now Playing info.

Finally, here's an example of how to provide image metadata for your content.

Here you can see that the first thing I need to do is to get my image as a data object. I'm doing that by loading the image from a JPEG that's in my app bundle, but you could get your artwork from any source. Then to create the metadata item the first thing I need to specify is my identifier, which in this case should be AVMetadataCommon IdentifierArtwork.

Next, I provide the artwork data itself and cast the Swift data object to an NSData.

Finally, to indicate what kind of image this is I need to set the dataType property. Since I know this image is a JPEG I'm going to provide this constant value here to indicate that. Okay to give you a visual reference of where some of these items appear onscreen here's a screenshot of the AVPlayerViewController with some annotations on it. The color coding will show where each metadata identifier is displayed. So, this is where the artwork will be displayed, the title, the creation date, and the description. And here's a similar screenshot of the TV Remote app. You can see here the Remote app doesn't display exactly the same set of metadata as AVPlayerViewController. It displays the same artwork and title, but instead of displaying the description it instead displays the artist and the album name.

You can see the AV foundation identifiers to use here.

Okay that's all for AVKit. If you're just playing your video using some other technology like Video Toolbox or if you're playing audio only content then you'll need another way to provide Now Playing info. In those cases, you can use the MPNowPlayingInfoCenter.

This is a singleton object that has a dictionary property that you can write your metadata into. So, there's a key for title, for album name, for artwork and so on. In addition to the metadata that you are probably expecting you can also specify explicitly if the content is audio or video. You can also provide some information about the content duration and the user's current playback position. Your artwork is not provided as raw bytes, but as an MPMediaItemArtwork object which we'll talk about more in just a minute.

Finally, it's your responsibility to update this metadata dictionary as the playback state changes. You do not need to update it every second or even every minute, but you should update it when certain events occur. We recommend that you update it when the currently playing item changes, if any metadata about the currently playing item changes, such as the title or artist name, if the user seeks to a new position of the content or if the playback rate changes and finally, if playback begins or stops. Let's take a look at some code.

Here in the first code block, I'm creating the object that's going to represent my artwork.

The way this MPMediaItemArtwork class works is that you provide us with the native size of the image in a block that we'll call later with specific image sizes. The way this MPMediaItemArtwork class works is that you provide us with a native size of the image in a block that we'll call later with specific image sizes. This block should return a UIImage object that closely fits inside the size that's passed to the block. So, for example, your app may have downloaded several different sizes of the same artwork say in small, medium and large sizes. When we call this block, you should take the size we provide and return the image that most closely fits the requested size.

We would discourage you from trying to perform expensive image resizing operations here, just return the image that you already have.

In the second block of code, this is where I'm actually providing all of my metadata to MPNowPlayingInfoCenter. You can see that this is just a plain Swift dictionary with some keys that are provided by the framework and I'm simply providing my values as regular Swift types.

There are two properties that I want to call out in particular though. The ElapsedPlaybackTime and the PlaybackDuration. Like I mentioned earlier, you should provide these keys so that tvOS knows how long your content is and where the user is currently at within the content. As various playback events occur you should update the ElapsedPlaybackTime to match where the user currently is.

There's no AVPlayerViewController for this API on tvOS, but I can show you what this metadata looks like in the TV Remote app. You can see where each property is displayed onscreen. These should look pretty familiar compared to the AVKit version.

Take note that in this version though the scrubber bar is under your control. The ElapsedPlaybackTime and the duration need to be provided by your app to be displayed correctly here.

All right, let's change gears a bit now and talk about how your app can handle external playback commands. Of course, your app will have its own controls that the user can interact with when your app is displayed, but the user may also want to control your app from the TV Remote app on iOS or if they're listening to background audio from your app on tvOS pressing the play pause button on the Siri remote can pause your app even while it's in the background. To support these kinds of interactions you will need to make sure your app can handle these Remote Commands.

The way to do that is with an API called MPRemoteCommandCenter.

The way to do that is with an API called MPRemoteCommandCenter. This is another singleton object that has a property for each different kind of command that your app can choose to support. For each command that you want support you can register either a target-action pair or a callback block which will be invoked when the command is performed. If you provide a target-action or a block the command is assumed to be supported. If you want to provide a handler for the command, but need to temporarily indicate that it's not available you can mark it as disabled. The method or block that you provide must return an enum value to indicate if your app was able to successfully perform the command. The definition of success here is pretty broad. For example, if the user is playing the last song in a playlist and requests to skip to the next track your app may choose to simply end playback. That would still be considered successfully handling the command so you should return the success value in that case.

Some commands like play and pause are pretty simple. If the handler for the pause command is invoked it's pretty clear what your app needs to do. But other commands are more flexible and both accept parameters from your app and can provide more parameters back to your handler.

For example, the MPSkipIntervalCommand is used to allow the user to skip forward or backward by several seconds.

Your app can express a preference for how many seconds the user should be allowed to skip. The skip command object allows you to configure that preference. When the user actually performs the skip command your handler will be invoked with an object parameter. That object is of the type MPSkipIntervalCommandEvent and it has an interval property that tells your app how many seconds you should skip. And be careful this may not be the same as the skip preference you provided earlier.

Let's look at an example. Here's how an app might implement the skip backward command.

You can see here that on the first line I would prefer that the user skips backward by 10 second intervals so I'm going to assign the preferred intervals property on the SkipBackwardCommand object. Then I'm going to provide my handler block. Now technically every command handler receives the generic type MPRemoteCommandEvent as the parameter, but I need to cast it to an MPSkipIntervalCommandEvent so that I can get to its interval property.

Then I'm going to actually perform the command. In this sample, I'm playing my content with an AVPlayer. Since AVPlayer expresses its time using CMTime structs that's how I'm going to need to calculate my new playback position.

You can see here that I'm getting the player's current composition then creating a new CMTime struct with the skip interval.

Finally, I'm subtracting that interval from the current time.

I ask the player to seek to that time and then return successful from the command handler. When the player finishes seeking to the new time my completion handler will be invoked and that's where I'm going to update my Now Playing info. Remember, if we modify playback state like seeking to a new time we need to publish new Now Playing info so we can update the elapsed playback time.

Here I'm calling a function called updateNowPlaying which I'll show you on this next slide.

Here you can see how I'm going to update my Now Playing info. This is basically the same as the example I showed you at the beginning of this section, but want to call out one specific detail.

Here you can see that I'm using the AVPlayer's current time property in order to provide an accurate value for the elapsed playback time property.

In this case, since AVPlayer is going to return a CMTime, but MPNowPlayingInfoCenter requires a floating-point value, that's the number of elapsed seconds, I need to use the CMTimeGetSeconds function to convert the value.

And that's all there is to providing a great playback experience with Remote Commands and Now Playing information.

For more information, you can visit the URL shown here to find links to documentation and related sessions. Thanks for watching.