Downloading live TS streams

Given what I see in the Network tab of the Firefox developer tools, I wanted to use the power of firefox extensions to actually catch up all those TS requests, that I know are the actual meat of live streams. Concatenating those small TS fragments together ends up with the full file on your hard drive for personal consumption.

So that's how the journey begins. In fact those writing a firefox extension have a number of pre-built tools at their disposal. Namely the background script, which really acts as a background thread, where you will fetch and download the TS fragments.

Here is how you build the firefox extension, from scratch.

• manifest.json // that's how firefox recognizes a firefox extension
  
• background.js // your background code
  
• icons / border-48.png // a random icon
  

Here is manifest.json :

{

  "manifest_version": 2,
  "name": "TS download",
  "version": "1.0",

  "description": "---- -----",

  "icons": {
    "48": "icons/border-48.png"
  },

  "background": {
    "scripts": [ "background.js" ],
    "persistent": true
  },

  "permissions": [
    "webRequest",
    "unlimitedStorage",
    "storage",
    "downloads",
    "tabs",
    "http://*/*",
    "https://*/*"
  ]
}

The manifest.json file specifies the actual background script, that runs outside of firefox UI. Also the permissions, one of which is used to make sure we are able to fetch and download fragments and save them as a file on the system (firefox's download folder).

Which leads us to background.js :

var filenameFragmentId = 1;

function onStartedDownload(id) {
  console.log(`Started downloading: ${id}`);
}

function onFailed(error) {
  console.log(`Download failed: ${error}`);
}

function logURL(requestDetails) {

  filename = "tsFragment" + String(filenameFragmentId).padStart(5, '0') + ".ts";

  console.log(`Loading: ${requestDetails.url}, filename: ${filename}`);



 var downloading = browser.downloads.download(
      {
        url: requestDetails.url,
        saveAs: false,
        conflictAction: "uniquify",
        filename
      }
  );

  downloading.then(onStartedDownload, onFailed);

  filenameFragmentId++;

}

browser.webRequest.onBeforeRequest.addListener(
  logURL,
  {
      types:["xmlhttprequest"],
      urls:["http://*/*.ts","https://*/*.ts","http://*/*.ts?*","https://*/*.ts?*"]
  }
);

What we do is get notified of firefox actual requests to the live stream server. That's done by passing a function callback (logURL) to the onBeforeRequest listener, also passing a filter limiting our interest to .TS content types.

And in our function callback implementation, what we do is build and download the TS fragment which is a binary response to the incoming request. In order to make the download function work, we need to build a filename, so fragments don't overwrite them, and also we need to make sure that no UI dialog shows up, because that would stop our function from doing anything relevant.

When you get this running, you'll get the following file sequence created in the firefox download folder :

• tsFragment00001.ts
• tsFragment00002.ts
• tsFragment00003.ts
• tsFragment00004.ts
• ...

If you use VLC to view those .ts fragments, they work perfectly well. Of course, they are what they are, very short fragments of the actual live stream. What we need to put these together is a copy command.

For this, open up a command window, make sure your current folder is the firefox download folder, and then type this :

copy /B tsFragment*.ts full.ts

Running this copy command creates full.ts, which is the standalone video file for the live stream between the moment the firefox extension started and the moment it got stopped. Properly handling the start time and the end time lets you capture a video of what you actually want.

There is something that must be paid attention to, it is the video bitrate, especially the fact that it stays stable during capture. So if you are capturing a Twitch live stream for instance, just make sure to set a fix video bitrate for your player, for instance 720P if you are on fiber internet, 480P if you are on DSL internet.

To load the firefox extension, just type, in firefox address bar :

about:debugging#/runtime/this-firefox

and then click on Load temporary add-on and point to the manifest.json file.