Indeed this seems to be a pathological video where almost all video formats fail on the first fragment and 299 may fail later, regardless of Python 2.7/3.5/3.9 and User-Agent settings.

yt-dlp 2024.08.06 still works, apparently. It has fancy networking that we can't easily replicate: maybe punt to curl for all requests?

@dirkf the reason why yt-dlp works is because yt-dlp has abandoned the WEB client entirely. The potoken experiment rate has reached 100% for WEB / rollout is complete. TVHTML5_SIMPLY_EMBEDDED_PLAYER is reportedly the next victim, and I don't think we are currently able to detect the potoken requirement for that client

I have that with every single video I try. Curiously enough format '18' work all the time. Other formats that work are '136', '137', '248' and/or '160', but it depends on video - not always the case. Still, format '18' is the most reliable to work.

Yeah, format 18 (the last remaining "legacy" format) is unaffected, as are the m3u8 formats

So

1. poToken (I agree, this is always being detected today) randomly breaks the download rather than uniformly giving 403 as with the revised n-sig "throttling"?
2. different clients get more or less functional links for formats with the same itag?

1. in my experience, all formats other than 18 and m3u8/mpd formats will be 403'd, but you won't get the 403 until at least 1MB has been downloaded
2. WEB is fully potoken'd and TVHTML5_SIMPLY_EMBEDDED_PLAYER is reportedly partially potoken'd now via Botguard, and ANDROID is fully potoken'd via Droidguard. Also, just because it has the same itag doesn't necessarily mean it's the same format. e.g. DRC audio formats; WEB VR formats and TVHTML5 VR formats have different resolutions despite having the same itags

Can confirm that a lot of the video-only formats are just being 403-ed in the middle with their downloads, resulting in me getting files that stop after about 10-20 minutes into the video, but still have full sized audio.

By now I have written something into my scripts to just pick format 18 as long as a flag is set, because i foresee this issue happening again in the future once it is eventually fixed... >.>

So has anyone tried fetching fragments in fragments of <1MB? We already had a work-around to download in fragments to avoid throttling IIRC.

Otherwise:

1. actually ignore the data with poToken and use the existing "punt to API" logic with a selected unafflicted client.
2. extend to user-specified other, or selection of, clients similar to the yt-dlp extractor logic.

Apparently the latest fix worked for not even a day, that doesn't bode well. Personally I keep getting "giving up after 0 fragment retries" in my python stuff.

From what I read in yesterdays thread, it seems like this will just not work out with fake JS interpretation if they try to combat this in the slightest. Like, that almost doesn't deserve the name attack vector, that's an attack landscape.

So has anyone tried fetching fragments in fragments of <1MB? We already had a work-around to download in fragments to avoid throttling IIRC.

I've tested it with http chunk size and with the pseudo-dash fragmentation and both resulted in the 403 errors.

This change is significant. I checked old, pre quantum Firefox and videos don't work any more, when 3 days ago they did.

Maybe the new player JS uses some G JS syntax extension (aka ECMA2021+) that hadn't been contemplated in those FF versions. Is there an error in the JS console?

It used to work as embedded or as mobile (when used mobile user agent). Now all of them display all saying error:

An error occurred. Please try again later. (Playback ID: j-bZsC_YehYVyZZ8)
Learn More (https://support.google.com/youtube/?p=player_error1&hl=en)

Loading any video at https://www.youtube.com/embed/1234567890a:

Content Security Policy: Couldn't process unknown directive 'require-trusted-types-for' <unknown>
mutating the [[Prototype]] of an object will cause your code to run very slowly; instead create the object with the correct initial [[Prototype]] value using Object.create www-embed-player.js:26:77
InvalidStateError www-embed-player.js:1128:42

Then pressing play:

Error: WebGL: Error during native OpenGL init. base.js:11283:169
Error: WebGL: WebGL creation failed. base.js:11283:169

ED.

If it's of any help, despite what was said before, there are some videos that work.

First - this one doesn't, and gives following console log

Content Security Policy: Couldn't process unknown directive 'require-trusted-types-for' <unknown>
mutating the [[Prototype]] of an object will cause your code to run very slowly; instead create the object with the correct initial [[Prototype]] value using Object.create www-embed-player.js:26:77
InvalidStateError www-embed-player.js:1128:42
Empty string passed to getElementById(). zVhcVoOEv7o line 2 > eval:795:28944

Error: WebGL: getParameter: parameter: invalid enum value <enum 0x9246> base.js:11283:254

This one does and, with this log:

Content Security Policy: Couldn't process unknown directive 'require-trusted-types-for' <unknown>
mutating the [[Prototype]] of an object will cause your code to run very slowly; instead create the object with the correct initial [[Prototype]] value using Object.create www-embed-player.js:26:77
InvalidStateError www-embed-player.js:1128:42
Empty string passed to getElementById(). KAR4fAX5T7Y line 2 > eval:4650:33869

So, after clicking 'play' it gives this error: Error: WebGL: getParameter: parameter: invalid enum value <enum 0x9246> base.js:11283:254.

No matter what chunk size I use I'm seeing hard 403 errors at 1Meg as others have reported - I'm able to download as many fragments as I want up to 1Meg and then get a 403.

Have experimented with generating cpns (nonces) and adding them to the format fragment URLs without any luck as well as using rn (request numbers) in the URL query instead of byte ranges. Have tried sleeping between fragments to mimick video playback also without joy.

It feels like they've added a check somewhere which fails at the 1Meg mark but I haven't found anything yet where that might be.

Checking via the browser I can see that youtube is happily downloading /videoplayback fragment URLs above 1Meg without any issue...

But in the browser the media links have the pot parameter with its poToken challenge result, no? Which is what we can't haz.

In line with step 1 above, I'm gradually pulling stuff from the yt-dlp extractor, enough to download HLS with client ios, but plainly not yet enough to get unblocked links from tv or web_creator, eg with format 135. Should I be expecting that?

But in the browser the media links have the pot parameter with its poToken challenge result, no? Which is what we can't haz.

In line with step 1 above, I'm gradually pulling stuff from the yt-dlp extractor, enough to download HLS with client ios, but plainly not yet enough to get unblocked links from tv or web_creator, eg with format 135. Should I be expecting that?

I'm not seeing a pot parameter in the query strings, I am seeing post data in the /videoplayback requests which is referred to in the source as playbackCookie

Edit: Looks like the playbackCookie / POST data is extracted from the bytes of the previous fragment response somehow

This is the procedure that I am using in my own code.

Load https://www.youtube.com/embed/<id#> and find the base.js link. Do the usual to extract the sig and n-sig. Extract the signatureTimestamp for the next step.

Load https://www.youtube.com/youtubei/v1/player with the signatureTimestamp and TVHTML5_SIMPLY_EMBEDDED_PLAYER as the client name.

If the JSON response contains "formats" and/or "adaptiveFormats" then we're good. This covers most videos, including age-gated ones. The 403 problem occurs when we have go to the next step. We can't use "www.youtube.com". We must use "m.youtube.com" with the user agent set to something like "Mozilla/5.0 (Android 14)" which is what I'm using.

Load https://m.youtube.com/watch?v=<id#> and extract the JSON structures that you would otherwise have gotten from the previous step.

And that's it. The extra step is only required for videos that disallow embedding.

Please don't bother to supply any "me too" reports unless the log shows some novelty that may help with rectification. Just "Like", or whatever, an existing similar report.

You can see how a poToken is being sent in POST data by the browser in the Invidious code that shows how to capture the value. But I understood from yt-dlp discussions that a pot query parameter was used in the media links associated with the pot-ified session.

@8chanAnon's algorithm is what is currently done for age-gate videos, up to the last step with m.youtube.com which is new and interesting. What happens if you skip straight to that step?

Step 2 will only work if TVHTML5_SIMPLY_EMBEDDED_PLAYER is not pot-ified, and that seems to be in question.

Indeed, Android 14/FF 122 at m.youtube.com didn't list the poToken experiment IDs although yt-dlp has reported unsuccess with Android clients.

I don't think it's worth trying to get around the poToken, it will eventually be required in all clients.

I keep digging into base.js when I get some time trying to understand how the token is created, it does seem to be extracted from the bytes of at least the first video fragment as far as I can tell, but not all fragments?...

There's a Uint8Array which appears to be the fragment response data?... manipulated several times and then 82/84/68 bytes of that array are stored as playbackCookie which is then sent in the POST data

At least it would be good to have a program that is not not-youtube-dl while a long term solution to the twattery is being investigated.

repro with yt-dlp latest:

yt-dlp -v --ignore-config --extractor-arg "youtube:player_client=android" 40wkJJXfwQ0

and let it download for longer than 30 seconds

Or you could use your program with the same video id (40wkJJXfwQ0). The key is to actually try to download one of the videoplayback URLs beyond 30 seconds of download time, which is why I've provided the video id for a 4+ hour video

But yt-dlp has identified several InnerTube clients that are not "fucked": maybe try one of those. Which brings me back to the question as to whether MWEB is still unaffected.

sory build from source from c5098961b04ce83f4615f2a846c84f803b072639

The mobile UA isn't working with playlist pages and tabs. You should be able to work (further) around this by getting the playlist JSON normally and then getting the videos with the mobile UA.

It's plausible that a slow rollout of PO token enforcement on MWEB has begun (as that client is a likely candidate to be next), but I haven't yet been able to repro the 403. And I wouldn't hold my breath that the "experiment IDs" yt-dlp was using to detect PO token enforcement on WEB would be consistent across clients

I outlined the solution to this problem two weeks ago. Guess I have to prove that my method works. I put together a little demo:

https://8chananon.github.io/dl/yt-extract.htm

It runs in your browser. For that reason, I have to use a proxy server but my proxy is blocked by Youtube due to having an Amazon IP address so the proxy has to route through another proxy which isn't banned by YT. Five proxies have been provided and the app will cycle to the next one if it detects blocking (or the proxy stops working).

Note that videos that are pegged to the IP address won't work because of the proxy (like these two: 8Pa9x9fZBtY and 0pKfxbCHLoU).

You can input a full URL or just the video ID. The app just dumps all of the itags (except itag 18) as links to the actual video/audio. Click a link to play it in a new tab. You can play the audio and the video in different tabs and synchronize them (LOL).

The code is, of course, in Javascript. Sorry about that but I don't do Python. Also, the download speed seems to be throttled but videos play fine.

If you want itag 18 or you just want to play the video to check it out:

https://8chananon.github.io/dl/yt-player.htm

Having a fully-featured JS environment is equivalent to having a headless browser and doesn't address the yt-dl use case to archive or play media from websites without a browser.

@dirkf

How dismissive. Maybe you can take a hint and start taking advantage of what a browser can do instead of continuing to hack away on the command line.

Your website just uses the TVHTML5_SIMPLY_EMBEDDED_PLAYER client, which has been part of yt-dlp for ~2 years.
The solution to this issue is already known: Switch from the WEB client to something else.

@seproDev

If the solution is already known then why does this thread still exist???

Because someone needs to take the time and port the improvements made to yt-dlp, over to youtube-dl.
yt-dlp already switched away from the web client ~1 month ago and even before that used iOS as a secondary client.
If you require the Python 2 compatibility, you will have to wait for these improvements to be ported to youtube-dl (or even contribute something yourself).

@8chanAnon's heart is in the right place to judge from Alleycat BBS, but time spent understanding the context of an issue is rarely wasted.

Part 1 of the program above is working. That also opens the way to part 2 (multiple clients) but of course if pottery should be applied to all the InnerTube clients further invention would be needed.

The mobile UA isn't working with playlist pages and tabs.

I learned that the hard way in the last one hour investigating it. Having Mozilla/5.0 (Android 14; Mobile; rv:115.0) Gecko/115.0 Firefox/115.0 as the UA (as explained above) leads to an exception when performing a regex search.

Traceback (most recent call last):
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\YoutubeDL.py", line 879, in wrapper
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\YoutubeDL.py", line 975, in __extract_info
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\common.py", line 571, in extract
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\youtube.py", line 3472, in _real_extract
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\youtube.py", line 311, in _extract_yt_initial_data
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\common.py", line 1101, in _search_regex
youtube_dl.utils.RegexNotFoundError: Unable to extract yt initial data; please [...]

However when completely removing the uppercase letters in the parenthesis (Mozilla/5.0 (ndroid 14; obile; rv:115.0) Gecko/115.0 Firefox/115.0), the playlist data (name, amount of videos, etc.) can be retrieved but each video download fails with the expected HTTP Error 403: Forbidden.

Not sure if this is the actual reason it's not supposed to work.

Anything new on this?

I am using the latest snapshot and it's still not working, I'm getting the same errors as in this issue.

In addition other formats such as 140 or 18 aren't working as well. I'm getting the Unable to decode n-parameter there.

And interestingly, yt-dlp 2024.04.09 is working.

Edited to add:

This is the commandline I'm using:

youtube-dl --restrict-filenames -o "%(uploader)s-%(title)s-%(id)s-%(upload_date)s.%(ext)s" --cookies www.youtube.com_cookies.txt https://www.youtube.com/watch?v=650lXWfoPN8

I created the cookies file using the chrome extension "Get cookies.txt" right before I executed the command after visiting the uri I was about to download.

@TLINDEN thank you, yt-dlp 2024.04.09 can download normally, but not for all videos.
some videos still fail ERROR: unable to download video data: HTTP Error 403: Forbidden

It's been a long time, I don't know if this problem can still be solved. :(

@xyzxye

yt-dlp 2024.04.09

Update to 2024.08.06 or later. If the issue persists open an issue on the yt-dlp issue tracker.

@3052 https://www.youtube.com/watch?v=VaSV4NtZCXU

I have successfully downloaded this using the latest version of yt-dlp

@xyzxye

yt-dlp 2024.04.09

Update to 2024.08.06 or later. If the issue persists open an issue on the yt-dlp issue tracker.

Sure, but re-extracting videos every 30 seconds will get your IP blacklisted right quick

Sure, but re-extracting videos every 30 seconds will get your IP blacklisted right quick

@bashonly what is "re-extracting"? I am talking about

1. call /youtubei/v1/player
2. download as many segments as you can in 30 seconds
3. repeat

the only difference from the current method is calling /youtubei/v1/player more often, which 30 seconds is hardly enough to encounter any kind of rate limit

what is "re-extracting"? I am talking about

I'm talking about "extracting" in the YoutubeDL sense. This is a youtube-dl tracker, no? So, effectively the same thing you are talking about.

Have a look at some of the pinned yt-dlp issues if you want examples of why making repeat player endpoint requests is not viable. Youtube is flagging/blocking IP addresses that aren't using PO tokens when required (among other reasons)

Please confine comments to relevant matters and questions of fact. What anyone thinks about a member's behaviour belongs elsewhere.

Not "just" asking but imputing motives, which is not relevant.

yt-dlp reports may well offer good evidence of YT site behaviour (and its YT extractor can test the known InnerTube clients). There is no adverse competition with yt-dlp since each project has its own goals and code is compatibly (un)licensed.

For avoidance of doubt, that is the end of this meta-discussion.

For avoidance of doubt, that is the end of this meta-discussion.

fair enough. lets go ahead and put some data on the table. here is a test:

https://github.com/3052/youtube/blob/v1.0.2/get_watch_test.go

it calls /youtubei/v1/get_watch (the replacement for /youtubei/v1/player) 99 times, with no po token, with one second delay between each call. I ran it just now, with all calls from the same IP. no flags, no blocks. thats what is wrong with comments like this:

Have a look at some of the pinned yt-dlp issues if you want examples of why making repeat player endpoint requests is not viable. Youtube is flagging/blocking IP addresses that aren't using PO tokens when required (among other reasons)

this isnt the YT-DLP tracker. if user want to be taken seriously, they should be including evidence HERE to back up their claims. the comment is incomplete at best, and misleading at worst. in my mind these type of comments are worse than spam, and should not be allowed in a technical discussion.

One area where there could be differences is the https implementation in use, urllib2/OpenSSL 1 (here) vs urllib3/OpenSSL (n?) or other (yt-dlp) vs whatever the Go runtime library uses (@3052)

Hello, I want to ask about any updates on this issue.
I tried to download a youtube video today and I'm getting the same error.
I'm using youtube-dl nightly, updated to the latest 2024.08.07 version.

cookies were exported using EditThisCookie for Chrome

youtube-dl https://youtu.be/vGA4nXgq6PI --cookies cookies-youtube-com.txt --ffmpeg-location "ffmpeg\bin"

Like yt-dlp, yt-dl already applies 10MB chunking.

note unlike YouTube-DL/YT-DLP, no client below performs GET /videoplayback anymore, but instead POST, so it might be good to move away from that. here is WEB_EMBEDDED_PLAYER:

POST /videoplayback?ump=1 HTTP/1.1

note the above request returns application/vnd.yt-ump, but if you use ump=0 instead or remove the key, you get audio/webm or similar. here is ANDROID/WEB:

POST /videoplayback HTTP/1.1

these clients ignore ump key and always return application/vnd.yt-ump, or maybe you can configure it using request header or request body. also I found an implementation here:

https://github.com/LuanRT/googlevideo/blob/main/examples/downloader/main.ts