How to: Fixing Windows 10 UWP Closed Captions client side

Standard

If you’ve ever dealt with serving video content through your Windows apps, you may have also seen the vast world of closed caption (or subtitles for my British countrymen) formats and methods to display them with the video content. I may be slightly exaggerating there but between the WebVTT, TTML, CC608 standards, and the option for in-stream closed captions or side-car caption files things can get quite complicated.

Microsoft’s new Media Element in Windows 10 has taken some big strides to improve compatibility with these different standards, previously Windows developers relied on the excellent Player Framework as a baseline against which to develop closed caption supporting media experiences, but with UWP we have a viable alternative out of the box.

However, closed caption standards are a tricky thing, and frequently media providers do not follow the standard specifications exactly to the letter, which can be a technical nightmare when it comes to using out of the box scenarios. I recently faced one such dilemma with one of our customer’s TTML implementations, so I’d like to share with you a trick to use when your captions will not display in the correct place on screen.

If you’d like to familiarise yourself with the TTML (or Timed Text Markup Language) standards, you can find the specifications here, they’re a little dry but worth reading if you have to get deeply involved with parsing this markup language.

The absolute basics to understand is that the file essentially contains a set of strings and a key time in the media file at which to display them. You will also see some basic formatting allowing for background and foreground colours, on screen positioning and other text styling attributes (alignment, sizing, fonts etc). Out of the box Windows 10 UWP implementations of this standard are worth your time pursuing because they allow for customization by the user in Windows 10’s settings screen – a requirement by US law as of this year.

Getting started with a TTML source is straight forward, just add it to your Media source in code behind as shown below (here the media source is created from an adaptive media URL, i.e. HLS streaming.

var amsSource = (await AdaptiveMediaSource.CreateFromUriAsync(new Uri("", UriKind.Absolute))).MediaSource;

var theSource = MediaSource.CreateFromAdaptiveMediaSource(amsSource);
var ttmlSource = TimedTextSource.CreateFromUri(new Uri("", UriKind.Absolute));

theSource.ExternalTimedTextSources.Add(ttmlSource);
Player.SetPlaybackSource(theSource);

This is all well and easy if you have a TTML source that strictly follows the standard’s specifications. However frequently I find clients have interpreted the specification a little differently, and Microsoft’s fairly rigid implementation of the parser can cause you headaches when your captions aren’t displaying correctly.

Faulty captions

Faulty captions

One of our recent clients at UI Centric faced this problem with their sidecar TTML files. In the screenshot sample above, you can see that whilst the captions are displaying, they have been locked to the top left of the screen with no background. Looking at the TTML styling in the document you’ll see the intention for a black background:


<style tts:color="white" id="s0" tts:fontFamily="sansSerif" tts:fontSize="16" tts:fontStyle="normal" tts:backgroundColor="black"/>

With the “body” text expecting to inherit that defined style of “s0” by default and be center aligned, and the particular paragraph inside the parser looking to use another style “s1”, cascading/inheriting down to the body “s0” for undeclared attributes.


<body style="s0" tts:textAlign="center">
<p style="s1" id="p6" end="00:00:26.64" begin="00:00:24.80">
If you don't stop prying...
<br/>
</p>

Unfortunately this is not understood by Microsoft’s parser, and without direct access to the display code for these captions it’s difficult to fix on the client side. Luckily after some digging around I’ve found it’s possible to access and alter the modelled TTML objects once the TTML has been downloaded using the following event:
ttmlSource.Resolved += TtmlSource_Resolved;
The resolved event has an arguments field which can be used to make adjustments to the track cues, and the region in which they are displayed, in this instance I used the following code to make the adjustments I was looking for, I’ve commented each line to explain its effect:

private void TtmlSource_Resolved(TimedTextSource sender, TimedTextSourceResolveResultEventArgs args)
        {
     sender.Resolved -= TtmlSource_Resolved;
            // Create a translucent black colour
            Color col = Color.FromArgb(150, 0, 0, 0);
            TimedTextRegion reg = new TimedTextRegion();
            // Display alignment after forces the captions to the bottom of the video
            reg.DisplayAlignment = TimedTextDisplayAlignment.After;
            // Allow the captions to fill the screen
            // NOTE: this can play havok with full region background colours, test carefully!
            reg.Extent = new TimedTextSize() { Unit = TimedTextUnit.Percentage, Width = 100, Height = 100 };
            foreach (var track in args.Tracks)
            {
                foreach (var cue in track.Cues)
                {
                    // Assign the region object created outside of the nested loop
                    (cue as TimedTextCue).CueRegion = reg;
                    // Align in the center of the 100% width
                    (cue as TimedTextCue).CueStyle.LineAlignment = TimedTextLineAlignment.Center;
                    // Use the background only on the cue
                    (cue as TimedTextCue).CueStyle.Background = col;
                }
            }
        }

You can take this method and expand upon it to fix TTML closed captions anywhere you’re using them in Windows UWP apps, I hope it saves you the time I spent digging through MSDN documentation! To finish let’s take another look at that screenshot with my changes:
Captions3
Much better.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>