Forem: Kubo Ryosuke

URL and Timestamp of MPEG-DASH Segments

Kubo Ryosuke — Wed, 03 Nov 2021 09:49:38 +0000

Introduction

MPD which is manifest file of MPEG-DASH indicates segment URLs and any attributes like HLS playlist.
HLS playlist has simple list of URLs and attributes and is very easy to read. However, MPD has compex structure and is more difficult to read than HLS playlist.
In this post, I describe things about segments of MPD.

MPD Examples

You can see Reference Client and many DASH streams at DASH Industry Forum.

https://reference.dashif.org/

BaseURL

If MPD has BaseURL tag, segment URL is indicated as relative path from value of the tag.

<BaseURL>http://localhost/mystream/hd/</BaseURL>

BaseURL can have relative URL from MPD.

<BaseURL>./hd/</BaseURL>

If all segments are concatenated as one file, BaseURL can have full URL of the file.

<BaseURL>http://localhost/mystream/video.mp4</BaseURL>

In this case, player will download those segments by range requests.
Using SegmentBase or SegmentList/SegmentURL which will be described bellow, MPD can indicates byte ranges of those segments to player.

SegmentTemplate

In live profile, SegmentTemplate tag is usually used.
This tag can be used to both of live stream (dynamic type MPD) and VOD stream (static type MPD).
ABEMA which is our video streaming service uses SegmentTemplate tag for all streams.

media attribute and initialization attribute

SegmentTemplate@media attribute and SegmentTemplate@initialization attribute indicate media segments (MP4 containing moof and mdat) and initialization segment (MP4 containing ftyp and moov), respectively.
However, you should resolve templates which is enclosed by $ and embedded to those values.

<SegmentTemplate 
    media="$RepresentationID$/$Time$.mp4"
    initialization="$RepresentationID$/init.mp4">

There are 5 types of template. In most cases, $RepresentationID$ and $Time$ or $Number$ will be used.
$RepresentationID$ template indicates value of Representation@id.
In following sample, There are 2 Representation tags at same level with SegmentTemplate tag.

<AdaptationSet contentType="video" mimeType="video/mp4" segmentAlignment="true">
    <SegmentTemplate timescale="90000" media="$RepresentationID$/$Time$.mp4" initialization="$RepresentationID$/init.mp4">
    <Representation id="video-hd" bandwidth="2000000" frameRate="30000/1001" height="720" width="1280" scanType="progressive" />
    <Representation id="video-sd" bandwidth="1000000" frameRate="30000/1001" height="480" width="854" scanType="progressive" />
</AdaptationSet>

Inserting video-hd and video-sd to $RepresentationId$ of @initialization attribute, you can get paths: video-hd/init.mp4 and video-sd/init.mp4. These are initialization segment of HD and SD, respectively.
On the other way, what does $Time$ template indicates?

With SegmentTimeline

SegmentTimeline tag enumerates relative time and duration of segments as follows:

<?xml version="1.0" encoding="utf-8"?>
<MPD xmlns="urn:mpeg:dash:schema:mpd:2011" availabilityStartTime="1970-01-01T00:00:00Z" profiles="urn:mpeg:dash:profile:isoff-live:2011" type="dynamic" minBufferTime="PT5.000000S" publishTime="2021-10-28T13:07:58Z" minimumUpdatePeriod="PT5.000000S" timeShiftBufferDepth="PT60.000000S" suggestedPresentationDelay="PT15.000000S">
  <BaseURL>http://localhost/mystream/</BaseURL>
  <Period id="1" start="PT1609426800S">
    <AdaptationSet mimeType="video/mp4" segmentAlignment="true">
      <SegmentTemplate timescale="90000" presentationTimeOffset="10786776" media="$RepresentationID$/$Time$.mp4" initialization="$RepresentationID$/init.mp4">
        <SegmentTimeline>
          <S d="357357" t="11771760" />
          <S d="360360" r="3"/>
          <S d="357357" />
        </SegmentTimeline>
      </SegmentTemplate>
      <Representation id="video-hd" bandwidth="2000000" frameRate="30000/1001" height="720" width="1280" scanType="progressive" />
      <Representation id="video-sd" bandwidth="1000000" frameRate="30000/1001" height="480" width="854" scanType="progressive" />
    </AdaptationSet>
    <AdaptationSet mimeType="audio/mp4" segmentAlignment="true">
      <SegmentTemplate timescale="48000" presentationTimeOffset="5752947" media="$RepresentationID$/$Time$.mp4" initialization="$RepresentationID$/init.mp4">
        <SegmentTimeline>
          <S d="191488" t="6278272"/>
          <S d="192512" r="4"/>
        </SegmentTimeline>
      </SegmentTemplate>
      <Representation id="audio-high" bandwidth="190000">
        <AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="2" />
      </Representation>
      <Representation id="audio-low" bandwidth="64000">
        <AudioChannelConfiguration schemeIdUri="urn:mpeg:dash:23003:3:audio_channel_configuration:2011" value="1" />
      </Representation>
    </AdaptationSet>
  </Period>
</MPD>

SegmentTimeline tag has S tags which indicates each segments.
However, if S tag has non-zero @r attribute, @r is repeat count of the S tag. (When @r is negative value, it means open-ended.) For example, r="3" means +3 segments, in other words, there are continuous 4 segments which have same duration.
S@d attribute is duration of segment and S@t attribute is earliest timestamp of segment.
The second and the subsequent segments can omit @t attribute, and its value is sum of previous segment's @t and @d.
Insert those @t values to $Time$ template, you can get media segment URLs.
For example, earliest segments of above sample is http://localhost/mystream/video-hd/11771760.mp4 and http://localhost/mystream/video-sd/11771760.mp4.

S@t and S@d are always integer values. Dividing those by SegmentTemplate@timescale attribute, you can get time and duration in seconds, respectively.
So, in above sample, the duration of earliest video segment is 357357 / 90000 = 3.97 seconds, and the timestamp is 11771760 / 90000 = 130.797 seconds. These values are synchronized with values in MP4 boxes.
Then, subtracting SegmentTemplate@presentationTimeOffset from S@t, you can get elapsed time from period start.
For above sample, you can calculate elapsed time of latest video segment from period start by following expression.

(11771760 + 357357 + 360360 * 4 - 10786776) / 90000 = 30.93 seconds

Adding Period@start to MPD@availabilityStartTime, you can get absolute time of period start.
So, in above sample, period start is 2020/12/31T15:00:00Z and composition time of latest video segment is 2020/12/31T15:00:30.930Z.

Only SegmentTemplate

As mentioned above, you can use SegmentTimeline to write timestamp (composition time) and duration of each segments.
On the other hand, if all segments have same duration, SegmentTimeline is not mandatory.

<AdaptationSet contentType="video" mimeType="video/mp4" segmentAlignment="true">
    <SegmentTemplate duration="2" startNumber="1000" initialization="$RepresentationID$/init.mp4" media="$RepresentationID$/$Number$.mp4" />
    <Representation id="video-300k" bandwidth="300000" codecs="avc1.64001e" frameRate="30" height="360" width="640" />
</AdaptationSet>

In above sample, SegmentTemplate@duration is segment duration and the all segments are 2 seconds.
@media attribute has $Number$ template, it is index number of segment which is started by @startNumber.
So, segment URLs of above MPD are followings.

video-300k/1000.mp4 --- earliest 2 seconds
video-300k/1001.mp4 --- 2nd 2 seconds
video-300k/1002.mp4 --- 3rd 2 seconds
    :
video-300k/1099.mp4 --- 1000th 2 seconds
    :

This post examples use $Time$ template with SegmentTimeline or $Number$ without SegmentTimeline.
However, not only these usages, you can use $Number$ template with SegmentTimeline.

Relation with Representation

In above samples, SegmentTemplate tag and Representation tags are placed in same layer. However, SegmentTemplate tag can be placed in each Representation tags.

<Representation id="video-hd" bandwidth="2000000" frameRate="30000/1001" height="720" width="1280" scanType="progressive">
    <SegmentTemplate timescale="90000" presentationTimeOffset="10786776" media="hd/$Time$.mp4" initialization="hd/init.mp4">
        ...
    </SegmentTemplate>
</Representation>
<Representation id="video-sd" bandwidth="1000000" frameRate="30000/1001" height="480" width="854" scanType="progressive">
    <SegmentTemplate timescale="90000" presentationTimeOffset="10786776" media="sd/$Time$.mp4" initialization="sd/init.mp4">
        ...
    </SegmentTemplate>
</Representation>

SegmentList, Initialization and SegmentURL

SegmentURL tags have URL or byte-range of each segments.
For example, you can place Initialization tag and SegmentURL tags in SegmentList tag as follows:

<SegmentList>
    <Initialization sourceURL="init.mp4" />
    <SegmentURL media="0.mp4" />
    <SegmentURL media="1.mp4" />
    <SegmentURL media="2.mp4" />
</SegmentList>

This is similar to HLS playlist and easy to understand.

SegmentBase

If BaseURL has URL of single MP4 file and MPD has neither SegmentList nor SegmentTemplate, player have no way knowing byte-range of segments.
SegmentBase@indexRange attribute indicates byte-range of information about byte-range of each segments (ex. sidx Box).

<BaseURL>http://localhost/sample.mp4</BaseURL>
<SegmentBase indexRange="896-1730"/>

For this example, player will send range-request of 896-1730 at first. The response data contains timestamp and offset of segments and player will send range-request for each segments.

Conclusion

There are many types of MPD usage. So it is difficult to understand.
This post described major ways to represent segments in MPD.

References

ISO/IEC 23009-1 Dynamic adaptive streaming over HTTP (DASH) - Part 1: Media presentation description and segment formats
フレームレートとタイムコード、タイムスタンプ
AbemaTV の MPEG-DASH 対応
MPEG-DASH (Dynamic Adaptive Streaming over HTTP, ISO/IEC 23009-1) - BITMOVIN Blog

go-mp4: Golang Library and CLI Tool for MP4

Kubo Ryosuke — Thu, 30 Jul 2020 03:20:24 +0000

Japanese Page

Introduction

I'm going to introduce go-mp4 which is Golang library for reading and writing MP4.

go-mp4 was developed by extending MP4 module of ABEMA which is the video streaming service in Japan.

GitHub

https://github.com/abema/go-mp4

What's MP4

There are many standards each for video codecs and audio codecs, for example, AVC, AV1, AAC, MP3, and etc.
However, it is not possible to record/playback the stream under synchronizing video and audio by only codecs.
You should use container like MP4 or MPEG2-TS for the purpose.
Containers can have video, audio, timecode, and other informations.

ISO/IEC 14496-12 "ISO Base Media File" stipulates the data unit called "Box" (also called "Atom"), tree structure of boxes, and the box types for each role.
And ISO/IEC 14496-14 "MP4 file format" is extension of that.

Structure of MP4 (ISO Base Media File Format)

The figure below shows an example of hexdump of MP4.
As mentioned above, it is composed of a collection of data units called box.
The first 4 bytes represent box-size, and the next 4 bytes represent box-type.
(If the box-size can not fit in 4 bytes, 0x00000001 is written in the first 4 bytes, and the box-size is written in 8 bytes of next to the box-type. And if the box-size is 0x00000000, the box continues until EOF.)

Looking at moov box, the size is 0x000004b3 (= 1203 bytes).
You can see that this section contains mvhd box and trak box.
Furthermore, the trak box contains tkhd box and mdia box.
Therefore, this data has a tree structure as shown in the following figure.

The standard describes the role of each box types, box type order, fields of each box types, and more.

About abema/go-mp4

Motivation

When rewriting or adding some boxes, in some situation, it can be implemented easily depending with no library.
However, in order to read or write boxes of deep layer, you have to seek repeatedly to reach to the target boxes. Furthermore, when the box size changes, you must seek back to the box-size of parent and correct it.
I was looking for Golang library which supports many box types and has various uses, but I couldn't find.

And, I often need to investigate the data of box.
There are many tools available for that, but many of them seems to support only some of boxes or display only some fields of box.

So, we developed abema/go-mp4 to be more general-purpose and to meet these requirements.

Integration with Your Go App

You can visit to the boxes by depth-first order with the ReadBoxStructure function as follows:



_, err := mp4.ReadBoxStructure(file, func(h *mp4.ReadHandle) (interface{}, error) {
    fmt.Println("depth", len(h.Path))

    // Box Type (e.g. "mdhd", "tfdt", "mdat")
    fmt.Println(h.BoxInfo.Type.String())

    // Box Size
    fmt.Println(h.BoxInfo.Size)

    if h.BoxInfo.Type.IsSupported() {
        // Payload
        box, _, _ := h.ReadPayload()
        fmt.Println(mp4.Stringify(box))

        // Expands children
        return h.Expand()
    }
    return nil, nil
})

You can also extract only boxes which has specific path using the ExtractBox function as follows:



// extract tkhd boxes which has path of moov->trak->tkhd
path := mp4.BoxPath{mp4.BoxTypeMoov(), mp4.BoxTypeTrak(), mp4.BoxTypeTkhd()}
boxes, err := mp4.ExtractBoxWithPayload(file, nil, path)

CLI Tool

go-mp4 has CLI tool.
For example, you can view the box tree with the mp4tool dump command.



$ mp4tool dump sample.mp4
[ftyp] Size=32 MajorBrand="isom" MinorVersion=512 CompatibleBrands=[{CompatibleBrand="isom"}, {CompatibleBrand="iso2"}, {CompatibleBrand="avc1"}, {CompatibleBrand="mp41"}]
[free] (unsupported box type) Size=8 Data=[...] (use -a option to show all)
[mdat] Size=6402 Data=[...] (use -mdat option to expand)
[moov] Size=1836 
  [mvhd] Size=108 ... (use -a option to show all)
  [trak] Size=743 
    [tkhd] Size=92 ... (use -a option to show all)
    [edts] Size=36 
      [elst] Size=28 Version=0 Flags=0x000000 EntryCount=1 Entries=[{SegmentDurationV0=1000 MediaTimeV0=2048 MediaRateInteger=1}]
    [mdia] Size=607 
      [mdhd] Size=32 Version=0 Flags=0x000000 CreationTimeV0=0 ModificationTimeV0=0 TimescaleV0=10240 DurationV0=10240 Pad=false Language="eng" PreDefined=0
      [hdlr] Size=44 Version=0 Flags=0x000000 PreDefined=0 HandlerType="vide" Name="VideoHandle"
              :
              :

By default, long lines are abbreviated. You can output everything except mdat box with the -a option.
Using -mdat option, you can view the mdat box as hex.

Implementation with Run-Time Reflection

You can add box types easily to go-mp4.
For example, tfdt box has implemented as follows:



package mp4

func BoxTypeTfdt() BoxType { return StrToBoxType("tfdt") }

func init() {
  AddBoxDef(&Tfdt{}, 0, 1)
}

type Tfdt struct {
  FullBox                      `mp4:"extend"`
  BaseMediaDecodeTimeV0 uint32 `mp4:"size=32,ver=0"`
  BaseMediaDecodeTimeV1 uint64 `mp4:"size=64,ver=1"`
}

// GetType returns the BoxType
func (*Tfdt) GetType() BoxType {
  return BoxTypeTfdt()
}

It has just a struct and few implementations.
It is interesting that the version number of tfdt box decides whether the size of BaseMediaDecodeTime is 32 bits or 64 bits.
mp4:"size=32,ver=0" tag and mp4:"size=64,ver=1" tag specifies that.
In most cases you can add a new box type without special logic.

It is a disadvantage that the evaluation is done at runtime using reflection.
However, the processing cost will be relatively insignificant for many applications.
(Creating a code generator, it will be completely resolved, and it will be easier to support other languages.)

Conclusion

I have introduced our Golang library for MP4.

In our company, we are developing more softwares related to video streaming.
If possible, I want to make the such softwares to open source too.