{"id":2918,"date":"2019-09-25T10:09:02","date_gmt":"2019-09-25T10:09:02","guid":{"rendered":"http:\/\/www.max-sperling.bplaced.net\/?p=2918"},"modified":"2024-02-16T10:13:38","modified_gmt":"2024-02-16T10:13:38","slug":"download-stream-multiple-parts","status":"publish","type":"post","link":"http:\/\/www.max-sperling.bplaced.net\/?p=2918","title":{"rendered":"Download HLS streams (PowerShell)"},"content":{"rendered":"<p><strong>Introduction<\/strong><\/p>\n<p>Some web player are streaming videos in parts and not as a single file. The common reason is to be able to switch the quality within the movie if necessary. It&#8217;s called &#8220;Adaptive Bitrate Streaming&#8221; (ABS). One of the most famous ABS protocols is &#8220;HTTP Live Streaming&#8221; (HLS) from Apple.<\/p>\n<hr>\n<p><strong>1. Traffic analysis<\/strong><\/p>\n<p>If DevTools &#8230;<\/p>\n<ul>\n<li>\n<details>\n<summary>isn&#8217;t blocked \u25bc<\/summary>\n<p>Goto the Network tab.<\/p>\n<pre>\r\n File            | Request URL\r\n-----------------|------------------------------------------------------\r\nmaster.m3u8      | https:\/\/website.com\/path\/to\/video\/master.m3u8\r\nindex.m3u8       | https:\/\/website.com\/path\/to\/video\/high\/index.m3u8\r\nmov_part_000.ts  | https:\/\/website.com\/path\/to\/video\/high\/mov_part_000.ts\r\nmov_part_001.ts  | https:\/\/website.com\/path\/to\/video\/high\/mov_part_001.ts\r\n...\r\n<\/pre>\n<\/details>\n<\/li>\n<li>\n<details>\n<summary>is blocked \u25bc<\/summary>\n<p>Use Wireshark instead: <a href=\"http:\/\/www.max-sperling.bplaced.net\/?p=12484\">Follow this link<\/a><br \/>\n<\/details>\n<\/li>\n<\/ul>\n<hr>\n<p><strong>1. HLS playlists<\/strong><\/p>\n<ol>\n<li>\n<details>\n<summary>Master playlist \u25bc<\/summary>\n<p>Before the player starts to download\/play the movie segments it gets the master playlist.<\/p>\n<p>master.m3u8<\/p>\n<pre>\r\n#EXTM3U\r\n#EXT-X-VERSION:3\r\n#EXT-X-STREAM-INF:BANDWIDTH=1400000\r\nhttps:\/\/website.com\/path\/to\/video\/low\/index.m3u8\r\n#EXT-X-STREAM-INF:BANDWIDTH=2800000\r\nhttps:\/\/website.com\/path\/to\/video\/mid\/index.m3u8\r\n#EXT-X-STREAM-INF:BANDWIDTH=5000000\r\nhttps:\/\/website.com\/path\/to\/video\/high\/index.m3u8\r\n<\/pre>\n<\/details>\n<\/li>\n<li>\n<details>\n<summary>Media playlist \u25bc<\/summary>\n<p>The player then plays a media playlist based on bandwidth \/ screen size \/ user selection.<\/p>\n<p>high\/index.m3u8<\/p>\n<pre>\r\n#EXTM3U\r\n#EXT-X-VERSION:3\r\n#EXT-X-PLAYLIST-TYPE:VOD\r\n#EXT-X-TARGETDURATION:10\r\n#EXTINF:10.0,\r\nhttps:\/\/website.com\/path\/to\/video\/high\/mov_part_000.ts\r\n#EXTINF:10.0,\r\nhttps:\/\/website.com\/path\/to\/video\/high\/mov_part_001.ts\r\n...\r\n#EXTINF:8.0,\r\nhttps:\/\/website.com\/path\/to\/video\/high\/mov_part_500.ts\r\n#EXT-X-ENDLIST\r\n<\/pre>\n<\/details>\n<\/li>\n<\/ol>\n<hr>\n<p><strong>3. Media download<\/strong><\/p>\n<p>If the media playlist &#8230;<\/p>\n<ul>\n<li>\n<details>\n<summary>contains file names \u25bc<\/summary>\n<pre>\r\nfunction Download-PlaylistFiles()\r\n{\r\n    $ProgressPreference = \"SilentlyContinue\" # Suppresses progress bar\r\n    Get-Content index.m3u8 | Where-Object {$_ -match \"^.*`.ts$\"} | ForEach-Object -Parallel {\r\n        $Site = \"https:\/\/website.com\/path\/\"\r\n        $File = $_\r\n\r\n        try {\r\n            Invoke-WebRequest \"$Site\/$File\" -OutFile $File\r\n        } catch {\r\n            Write-Output \"$Site\/$File\"\r\n            Write-Output $_.Exception.Message\r\n        }\r\n    } -ThrottleLimit 10\r\n\r\n    [console]::beep(500,1000)\r\n}\r\n\r\nDownload-PlaylistFiles\r\n<\/pre>\n<\/details>\n<\/li>\n<li>\n<details>\n<summary>contains file links \u25bc<\/summary>\n<pre>\r\nfunction Download-PlaylistFiles()\r\n{\r\n    $ProgressPreference = \"SilentlyContinue\" # Suppresses progress bar\r\n\r\n    Get-Content index.m3u8 | Where-Object {$_ -match \"^.*`.ts$\"} | ForEach-Object -Parallel {\r\n        $Link = $_\r\n        $File = $_.Substring($_.LastIndexOf(\"\/\") + 1)\r\n\r\n        try {\r\n            Invoke-WebRequest $Link -OutFile $File\r\n        } catch {\r\n            Write-Output $Link\r\n            Write-Output $_.Exception.Message\r\n        }\r\n    } -ThrottleLimit 10\r\n\r\n    [console]::beep(500,1000)\r\n}\r\n\r\nDownload-PlaylistFiles\r\n<\/pre>\n<\/details>\n<\/li>\n<li>\n<details>\n<summary>wasn&#8217;t provided at all \u25bc<\/summary>\n<pre>\r\nfunction Download-PlaylistFiles()\r\n{\r\n    $ProgressPreference = \"SilentlyContinue\" # Suppresses progress bar\r\n\r\n    0..500 | ForEach-Object -Parallel {\r\n        $Site = \"https:\/\/website.com\/path\/\"\r\n        $File = \"mov_part_$($_.ToString('000')).ts\"\r\n\r\n        try {\r\n            Invoke-WebRequest \"$Site\/$File\" -OutFile $File\r\n        } catch {\r\n            Write-Output \"$Site\/$File\"\r\n            Write-Output $_.Exception.Message\r\n        }\r\n    } -ThrottleLimit 10\r\n\r\n    Get-ChildItem -File | where length -le 1024 | Remove-Item\r\n}\r\n\r\nDownload-PlaylistFiles\r\n<\/pre>\n<p>If there are files skipped in the playlist sequence, they will still be downloaded and removed afterwards based on their small size (<= 1KB).\n<\/ul>\n<\/details>\n<\/li>\n<\/ul>\n<hr>\n<p><strong>3.5 Watch during download<\/strong><\/p>\n<p>If the media playlist &#8230;<\/p>\n<ul>\n<li>\n<details>\n<summary>contains file names \u25bc<\/summary>\n<p>Just use it.<br \/>\n<\/details>\n<\/li>\n<li>\n<details>\n<summary>contains file links \u25bc<\/summary>\n<pre>\r\nfunction Create-LocalPlaylist([string]$FileName)\r\n{\r\n    $File = $FileName + \".m3u8\"\r\n\r\n    Get-Content index.m3u8 | ForEach-Object {\r\n        if ($_ -match \"^.*`.ts$\") {\r\n            $_ = $_.Substring($_.LastIndexOf(\"\/\") + 1)\r\n        }\r\n        $Content += \"$_`n\"\r\n    }\r\n\r\n    $Content > $File\r\n}\r\n\r\nCreate-LocalPlaylist \"index_local\"\r\n<\/pre>\n<\/details>\n<\/li>\n<li>\n<details>\n<summary>wasn&#8217;t provided at all \u25bc<\/summary>\n<pre>\r\nfunction Create-LocalPlaylist([string]$FileName)\r\n{\r\n    $File = $FileName + \".m3u8\"\r\n\r\n    $Content = \"#EXTM3U`n#EXT-X-VERSION:3`n\" +\r\n               \"#EXT-X-PLAYLIST-TYPE:VOD`n\" +\r\n               \"#EXT-X-TARGETDURATION:10`n\"\r\n\r\n    Get-ChildItem -Name *.ts | Sort {$_ -replace \"\\D+\" -as [int]} | ForEach-Object {\r\n        $Content += \"$_`n\"\r\n    }\r\n\r\n    $Content > $File\r\n}\r\n\r\nCreate-LocalPlaylist \"index\"\r\n<\/pre>\n<p>To set &#8216;EXT-X-TARGETDURATION&#8217; properly wait til some segments are downloaded and then use the highest length rounded up.<br \/>\n<\/details>\n<\/li>\n<\/ul>\n<hr>\n<p><strong>4. Segment merging<\/strong><\/p>\n<p>If the media playlist &#8230;<\/p>\n<ul>\n<li>\n<details>\n<summary>contains file names \u25bc<\/summary>\n<pre>\r\nfunction Concat-PlaylistFiles([string]$FileName)\r\n{\r\n    $File = $FileName + \".mkv\"\r\n    $Concat = \"concat:\"\r\n\r\n    Get-Content index.m3u8 | Where-Object {$_ -match \"^.*`.ts$\"} | ForEach-Object {\r\n        $Concat += \"$_|\"\r\n    }\r\n\r\n    ffmpeg -i $Concat -c copy $File\r\n}\r\n\r\nConcat-PlaylistFiles \"video\"\r\n<\/pre>\n<p>Alternative: Operate on the TS file names in the folder directly.<br \/>\n<\/details>\n<\/li>\n<li>\n<details>\n<summary>contains file links \u25bc<\/summary>\n<pre>\r\nfunction Concat-PlaylistFiles([string]$FileName)\r\n{\r\n    $File = $FileName + \".mkv\"\r\n    $Concat = \"concat:\"\r\n\r\n    Get-Content index.m3u8 | Where-Object {$_ -match \"^.*`.ts$\"} | ForEach-Object {\r\n        $_ = $_.Substring($_.LastIndexOf(\"\/\") + 1)\r\n        $Concat += \"$_|\"\r\n    }\r\n\r\n    ffmpeg -i $Concat -c copy $File\r\n}\r\n\r\nConcat-PlaylistFiles \"video\"\r\n<\/pre>\n<p>Alternative: Operate on the TS file names in the folder directly.<br \/>\n<\/details>\n<\/li>\n<li>\n<details>\n<summary>wasn&#8217;t provided at all \u25bc<\/summary>\n<pre>\r\nfunction Concat-PlaylistFiles([string]$FileName)\r\n{\r\n    $File = $FileName + \".mkv\"\r\n    $Concat = \"concat:\"\r\n\r\n    Get-ChildItem -Name *.ts | Sort {$_ -replace \"\\D+\" -as [int]} | ForEach-Object {\r\n        $Concat += \"$_|\"\r\n    }\r\n\r\n    ffmpeg -i $Concat -c copy $File\r\n}\r\n\r\nConcat-PlaylistFiles \"video\"\r\n<\/pre>\n<\/details>\n<\/li>\n<\/ul>\n<details>\n<summary>FFmpeg details \u25bc<\/summary>\n<p>The concat protocol (ffmpeg -i &#8220;concat:mov_part_000.ts|mov_part_001.ts|&#8230;&#8221;) gets used and not the concat demuxer (ffmpeg -f concat -i files.txt) to prevent video stutters.<br \/>\n<\/details>\n<hr>\n<p><strong>Addendum<\/strong><\/p>\n<ul>\n<li>\nHLS supports next to on-demand streaming also live streaming.\n<\/li>\n<li>\nOther famous ABS protocols are:<br \/>\n&#8211; &#8220;Dynamic Adaptive Streaming over HTTP&#8221; (DASH) from MPEG<br \/>\n&#8211; &#8220;HTTP Dynamic streaming&#8221; (HDS) from Adobe\n<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Introduction Some web player are streaming videos in parts and not as a single file. The common reason is to<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false},"categories":[45,55],"tags":[],"_links":{"self":[{"href":"http:\/\/www.max-sperling.bplaced.net\/index.php?rest_route=\/wp\/v2\/posts\/2918"}],"collection":[{"href":"http:\/\/www.max-sperling.bplaced.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.max-sperling.bplaced.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.max-sperling.bplaced.net\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.max-sperling.bplaced.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2918"}],"version-history":[{"count":1,"href":"http:\/\/www.max-sperling.bplaced.net\/index.php?rest_route=\/wp\/v2\/posts\/2918\/revisions"}],"predecessor-version":[{"id":16708,"href":"http:\/\/www.max-sperling.bplaced.net\/index.php?rest_route=\/wp\/v2\/posts\/2918\/revisions\/16708"}],"wp:attachment":[{"href":"http:\/\/www.max-sperling.bplaced.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2918"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.max-sperling.bplaced.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2918"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.max-sperling.bplaced.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2918"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}