Using HTML5 MediaSource API as Video Source

More than half a year ago I developed a HTML5 Video Player that is available on github. The webserver (in our case lighttpd) accepts (byte) range requests. So we can feed the source (mp4, webm or ogg) directly as a source to the player/video element. So the video gets loaded as needed. This worked great for the last months and works automatically.

I recently came across a problem with Internet Explorer 11 on Windows 8.1. The browser does not play the video. It completely freezes instantly or after a few seconds. There is no chance to view the video except if you disable byte range request and load the video completely. It does this even on the popular videoplayer websites like flowplayer and jwplayer. But it didnt happen with YouTube videos.
After doing some research I found out that the source of the video is not a single URL but a blob (blob:<identifier>). shows the availability of HTML5 Video in your Browser and it also shows the availability of a feature called Media Source Extension (MSE). The MSE is used as a source for the HTML Media Elements (like <video> and <audio>).

Note that MSE is not available in IE11 on Windows 7 (with identical IE version as on Windows 8.1) for unknown reason.

The procedure for using MSE is the following:

  1. Create a new MediaSource (MS) object
  2. Create a blob URL for the MS object
  3. Set src of <video> to the blob URL
  4. Wait for the ’sourceopen‘ event to be fired on MS
  5. Create a new SourceBuffer (SB)
  6. Append data to the SB periodically

This works great for webm and ogg files. The only problem here is how do determine when new data has to be loaded.

Mp4 video files cannot be used in general for the MSE purposes. The keyword here is MPEG DASH (Dynamic Adaptive Streaming over HTTP). This is a technique that allows two things.

  1. It allows to split a mp4 file into segments that can be feed into the MS object
  2. It allows to switch quality while playing without the need two stop playback.

1) Is what we are most interested in because we want to load mp4 video data periodically if needed.

2) is a nice side effect that you may have encountered as well. When switching quality of a video on YouTube, you might have noticed that it take a few seconds until the video has better quality. This is because the current fragment will be played from the SB and after that the fragment with better quality is loaded and played.

I used GPAC’s MP4Box to create a fragmented mp4 file, that can be used for streaming. Mp4Box also created a XML file that contains the fragments. Not only the byte ranges are included but also the time offset in the video files. This way you can load a new segment when the segment is almost finished playing (e.g. 80% of the time of the segment is reached).

I hope this gives you a short introduction. I will post more blog posts when I have some demos/code I can present.