Support new YouTube JSON scheme #441

B0pol · 2020-10-26T12:30:13Z

I carefully read the contribution guidelines and agree to them.
I have tested the API against NewPipe.
I agree to create a pull request for NewPipe as soon as possible to make it compatible with the changed API.

fixes TeamNewPipe/NewPipe#4572 fixes #439
@Stypox @TobiGr

opusforlife2 · 2020-10-26T17:20:08Z

Yay!

opusforlife2 · 2020-10-26T17:21:09Z

^ 4572, not 4752.

xibr · 2020-10-26T18:50:34Z

I have tested this changes and I got this error
failed to find pattern ""assets":.+?"js":\s*("[^"]+")

Edit: For some videos, when you try again, the error does not appear.

opusforlife2 · 2020-10-26T19:21:06Z

Maybe try both patterns twice before moving on just to be safe?

Stypox

This isn't a proper fix, it only replaces the "retry multiple times" code with "try a different way", both of which are workarounds. I'll try to see if there is another way to really fix this

B0pol · 2020-10-26T20:23:50Z

@xibr I did some tests. I could reproduce your issue twice time in 300 tries. Before this patch, it's about 15 in 100 tries (BUT each exception try is already 3 tries, then more like 45-60 failed attempts out of 100) (with the same video though).

@Stypox @opusforlife2 So what I did is try to get info some missing info with new file from embed. It seemed to work everytime (yesterday), but actually I did not test enough, thank you @xibr. It's way more reliable as I said before, but yeah it's not a proper fix, and 1 / 600 is still way too much.

Here is what it tries to find pattern when it fails / when it is normal
newpipe.zip

Stypox · 2020-10-26T20:30:31Z

@B0pol the player url is there inside the fail html: "jsUrl": "/s/player/4a1799bd/player_ias.vflset/en_GB/base.js". It's just a matter of finding it, but it is there :-D

Stypox · 2020-10-26T20:43:03Z

I think this can be solved completely by caching the player url or even the deobfuscation code. I was able to use the player url from your HTMLs (it's the same in both of them: /s/player/4a1799bd/player_ias.vflset/en_GB/base.js) to deobfuscate the stream urls of requests made by me just now and of the HAR in #439

Stypox · 2020-10-26T21:08:40Z

Here is an APK with a hardcoded player js URL. For me it works perfectly. Obviously the URL probably will change over time, but just as a check I'm uploading this APK. @opusforlife2 does this also work in India, where YouTube might be doing things differently than in Europe? @xibr could you also test?
Btw, with this APK it even takes less time to load videos :-D
app-debug.zip

opusforlife2 · 2020-10-26T21:18:36Z

Werks ^. And it is indeed faster. :D

FireMasterK · 2020-10-26T21:29:49Z

Yep, can confirm it works in India :)

opusforlife2 · 2020-10-26T21:30:52Z

@FireMasterK Oh wow. I just noticed you're the one hosting that new Invidious instance I use. Thanks for that. ;)

B0pol · 2020-10-26T23:59:21Z

I pushed a fixed for

failed to find pattern ""assets":.+?"js":\s*("[^"]+")
Without hardcoding URL as @Stypox did previously.

New debug APK: TeamNewPipe/NewPipe#4675 (comment) ^{newpipe-FIXED.zip}
BTW, you may encounter bug TeamNewPipe/NewPipe#4653, just close it when it appears

geckolinux · 2020-10-27T01:53:56Z

New debug APK: newpipe-FIXED.zip

I was getting Could not load decryption code for the Youtube service for every video on a new install (CarbonROM Android 10). It was working normally on my previous installation of another Android 10 ROM. This debug APK fixes it, thanks!

Tetriser · 2020-10-27T01:59:34Z

Having the exact same problem here, now using the temporary APK.

xibr · 2020-10-27T03:52:37Z

I pushed a fixed for

failed to find pattern ""assets":.+?"js":\s*("[^"]+")
Without hardcoding URL as @Stypox did previously.

New debug APK: TeamNewPipe/NewPipe#4675 (comment)
BTW, you may encounter bug TeamNewPipe/NewPipe#4653, just close it when it appears

Exception

User Action: requested stream
Request: https://www.youtube.com/watch?v=O0StKlRHVeE
Content Country: KR
Content Language: ko
App Language: en
Service: YouTube
Version: 0.20.1
OS: Linux samsung/star2ltexx/star2lte:10/QP1A.190711.020/G965FXXUAETG3:user/release-keys 10 - 29

Crash log

org.schabi.newpipe.extractor.exceptions.ParsingException: Could not get name
	at org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeStreamInfoItemExtractor.getName(YoutubeStreamInfoItemExtractor.java:98)
	at org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeStreamInfoItemExtractor.isAd(YoutubeStreamInfoItemExtractor.java:81)
	at org.schabi.newpipe.extractor.stream.StreamInfoItemsCollector.extract(StreamInfoItemsCollector.java:39)
	at org.schabi.newpipe.extractor.stream.StreamInfoItemsCollector.commit(StreamInfoItemsCollector.java:94)
	at org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeStreamExtractor.getRelatedStreams(YoutubeStreamExtractor.java:601)
	at org.schabi.newpipe.extractor.utils.ExtractorHelper.getRelatedVideosOrLogError(ExtractorHelper.java:32)
	at org.schabi.newpipe.extractor.stream.StreamInfo.extractOptionalData(StreamInfo.java:328)
	at org.schabi.newpipe.extractor.stream.StreamInfo.getInfo(StreamInfo.java:73)
	at org.schabi.newpipe.extractor.stream.StreamInfo.getInfo(StreamInfo.java:64)
	at org.schabi.newpipe.util.ExtractorHelper.lambda$getStreamInfo$3(ExtractorHelper.java:116)
	at org.schabi.newpipe.util.-$$Lambda$ExtractorHelper$5fJcha6Sq5APJBLdG6osaJby-mc.call(Unknown Source:4)
	at io.reactivex.internal.operators.single.SingleFromCallable.subscribeActual(SingleFromCallable.java:44)
	at io.reactivex.Single.subscribe(Single.java:3666)
	at io.reactivex.internal.operators.single.SingleDoOnSuccess.subscribeActual(SingleDoOnSuccess.java:35)
	at io.reactivex.Single.subscribe(Single.java:3666)
	at io.reactivex.internal.operators.maybe.MaybeFromSingle.subscribeActual(MaybeFromSingle.java:41)
	at io.reactivex.Maybe.subscribe(Maybe.java:4290)
	at io.reactivex.internal.operators.maybe.MaybeConcatArray$ConcatMaybeObserver.drain(MaybeConcatArray.java:153)
	at io.reactivex.internal.operators.maybe.MaybeConcatArray$ConcatMaybeObserver.request(MaybeConcatArray.java:78)
	at io.reactivex.internal.operators.flowable.FlowableElementAtMaybe$ElementAtSubscriber.onSubscribe(FlowableElementAtMaybe.java:66)
	at io.reactivex.internal.operators.maybe.MaybeConcatArray.subscribeActual(MaybeConcatArray.java:42)
	at io.reactivex.Flowable.subscribe(Flowable.java:14935)
	at io.reactivex.internal.operators.flowable.FlowableElementAtMaybe.subscribeActual(FlowableElementAtMaybe.java:36)
	at io.reactivex.Maybe.subscribe(Maybe.java:4290)
	at io.reactivex.internal.operators.maybe.MaybeToSingle.subscribeActual(MaybeToSingle.java:46)
	at io.reactivex.Single.subscribe(Single.java:3666)
	at io.reactivex.internal.operators.single.SingleSubscribeOn$SubscribeOnObserver.run(SingleSubscribeOn.java:89)
	at io.reactivex.Scheduler$DisposeTask.run(Scheduler.java:578)
	at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66)
	at io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
	at java.lang.Thread.run(Thread.java:919)

Edit: just one time.

xibr · 2020-10-27T04:15:28Z

@xibr could you also test?
Btw, with this APK it even takes less time to load videos :-D
app-debug.zip

Works fine so far.

71walceli · 2020-10-27T04:30:35Z

I just installed the debug build. and even restored settings from stable one. Hereby I confirm it's working fine. Thanks a lot.

xibr · 2020-10-27T05:32:13Z

I have tested two apk files provided by @ Stypox and @ B0pol and it works fine no errors yet shown. There is one file error provided by @ B0pol but it never appears again after removing cache.
the crash log in my comment above.

FireMasterK · 2020-10-27T06:09:07Z

I got this error randomly on B0pol's build, I'm not sure if it's related to this tho.

Exception

User Action: requested comments
Request: https://m.youtube.com/watch?v=_83KqwEEGw4
Content Country: IN
Content Language: en-IN
App Language: en_IN
Service: YouTube
Version: 0.20.1
OS: Linux Android 10 - 29

Crash log

org.schabi.newpipe.extractor.exceptions.ParsingException: Could not parse json data for comments
	at org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeCommentsExtractor.getPage(YoutubeCommentsExtractor.java:97)
	at org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeCommentsExtractor.getInitialPage(YoutubeCommentsExtractor.java:52)
	at org.schabi.newpipe.extractor.utils.ExtractorHelper.getItemsPageOrLogError(ExtractorHelper.java:19)
	at org.schabi.newpipe.extractor.comments.CommentsInfo.getInfo(CommentsInfo.java:40)
	at org.schabi.newpipe.extractor.comments.CommentsInfo.getInfo(CommentsInfo.java:25)
	at org.schabi.newpipe.util.ExtractorHelper.lambda$getCommentsInfo$7(ExtractorHelper.java:155)
	at org.schabi.newpipe.util.-$$Lambda$ExtractorHelper$Xyj84g1UeNZqX9LVYNvWZsdT-kI.call(Unknown Source:4)
	at io.reactivex.internal.operators.single.SingleFromCallable.subscribeActual(SingleFromCallable.java:44)
	at io.reactivex.Single.subscribe(Single.java:3666)
	at io.reactivex.internal.operators.single.SingleDoOnSuccess.subscribeActual(SingleDoOnSuccess.java:35)
	at io.reactivex.Single.subscribe(Single.java:3666)
	at io.reactivex.internal.operators.maybe.MaybeFromSingle.subscribeActual(MaybeFromSingle.java:41)
	at io.reactivex.Maybe.subscribe(Maybe.java:4290)
	at io.reactivex.internal.operators.maybe.MaybeConcatArray$ConcatMaybeObserver.drain(MaybeConcatArray.java:153)
	at io.reactivex.internal.operators.maybe.MaybeConcatArray$ConcatMaybeObserver.request(MaybeConcatArray.java:78)
	at io.reactivex.internal.operators.flowable.FlowableElementAtMaybe$ElementAtSubscriber.onSubscribe(FlowableElementAtMaybe.java:66)
	at io.reactivex.internal.operators.maybe.MaybeConcatArray.subscribeActual(MaybeConcatArray.java:42)
	at io.reactivex.Flowable.subscribe(Flowable.java:14935)
	at io.reactivex.internal.operators.flowable.FlowableElementAtMaybe.subscribeActual(FlowableElementAtMaybe.java:36)
	at io.reactivex.Maybe.subscribe(Maybe.java:4290)
	at io.reactivex.internal.operators.maybe.MaybeToSingle.subscribeActual(MaybeToSingle.java:46)
	at io.reactivex.Single.subscribe(Single.java:3666)
	at io.reactivex.internal.operators.single.SingleSubscribeOn$SubscribeOnObserver.run(SingleSubscribeOn.java:89)
	at io.reactivex.Scheduler$DisposeTask.run(Scheduler.java:578)
	at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66)
	at io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:301)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
	at java.lang.Thread.run(Thread.java:919)
Caused by: com.grack.nanojson.JsonParserException: Unexpected character: < on line 1, char 1
	at com.grack.nanojson.JsonTokener.createParseException(Unknown Source:44)
	at com.grack.nanojson.JsonTokener.advanceToToken(Unknown Source:118)
	at com.grack.nanojson.JsonParser.advanceToken(Unknown Source:12)
	at com.grack.nanojson.JsonParser.parse(Unknown Source:1)
	at com.grack.nanojson.JsonParser$JsonParserContext.from(Unknown Source:19)
	at org.schabi.newpipe.extractor.services.youtube.extractors.YoutubeCommentsExtractor.getPage(YoutubeCommentsExtractor.java:95)
	... 30 more

B0pol · 2020-10-27T06:30:00Z

@xibr Could not reproduce the issue (I tried 100 times). If you can reproduce, please provide a HAR file

@FireMasterK it's not related, we already know this issue

xibr · 2020-10-27T07:00:56Z

@B0pol Well, so far I can't reproduce, if I can reproduce I will attach a HAR file.

This fix works fine.

FarisZR · 2020-10-27T07:05:33Z

I pushed a fixed for

failed to find pattern ""assets":.+?"js":\s*("[^"]+")
Without hardcoding URL as @Stypox did previously.

New debug APK: newpipe-FIXED.zip
BTW, you may encounter bug TeamNewPipe/NewPipe#4653, just close it when it appears

Can confirm this fixes all the issues with the current release.

S10+ LOS 17.1

vrhikaru · 2020-10-27T08:55:02Z

I pushed a fixed for

failed to find pattern ""assets":.+?"js":\s*("[^"]+")
Without hardcoding URL as @Stypox did previously.

New debug APK: newpipe-FIXED.zip
BTW, you may encounter bug TeamNewPipe/NewPipe#4653, just close it when it appears

It's work for me, I tested on Nexus 6P with Android 7.1.1, from Taiwan.

fixes TeamNewPipe/NewPipe#4572 fixes TeamNewPipe#439

Stypox

Not perfect, but perfectly valid for a hotfix. Thank you!

B0pol changed the title ~~fix teamnewpipe/newpipe#4752 #439~~ fix new json youtube scheme unsupported Oct 26, 2020

B0pol requested review from Stypox and TobiGr October 26, 2020 12:30

B0pol force-pushed the fix-three-attempts branch from 28fc20c to 309a42f Compare October 26, 2020 13:26

B0pol force-pushed the fix-three-attempts branch from 309a42f to a7d64eb Compare October 26, 2020 20:04

Stypox reviewed Oct 26, 2020

View reviewed changes

opusforlife2 mentioned this pull request Oct 26, 2020

unable to play any video: "YouTube did not provide player config even after three attempts" | "Could not parse website" error on every video TeamNewPipe/NewPipe#4572

Closed

B0pol changed the base branch from dev to master October 27, 2020 12:23

B0pol added 2 commits October 27, 2020 13:40

fix youtube decryption and three attemps bug

db0ef83

fixes TeamNewPipe/NewPipe#4572 fixes TeamNewPipe#439

find playerUrl in another place when assetsPattern regex fails

6dc5ab4

B0pol force-pushed the fix-three-attempts branch from 7d50f40 to 6dc5ab4 Compare October 27, 2020 12:49

B0pol changed the title ~~fix new json youtube scheme unsupported~~ Support new YouTube JSON scheme Oct 27, 2020

B0pol mentioned this pull request Oct 27, 2020

Release 0.20.2 (956) TeamNewPipe/NewPipe#4675

Merged

5 tasks

Stypox approved these changes Oct 27, 2020

View reviewed changes

cion49235 approved these changes Oct 28, 2020

View reviewed changes

TobiGr merged commit 354426f into TeamNewPipe:master Oct 28, 2020

B0pol deleted the fix-three-attempts branch October 29, 2020 18:23

szotsaki mentioned this pull request Nov 15, 2020

F-Droid version is not updated to v0.20.2 TeamNewPipe/NewPipe#4896

Closed

Pavlecom mentioned this pull request Dec 1, 2020

Enqueueing videos from browser or youtube app resets the existing queue TeamNewPipe/NewPipe#4946

Closed

4 tasks

Uh oh!

Support new YouTube JSON scheme #441

Support new YouTube JSON scheme #441

Uh oh!

Conversation

B0pol commented Oct 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

opusforlife2 commented Oct 26, 2020

Uh oh!

opusforlife2 commented Oct 26, 2020

Uh oh!

xibr commented Oct 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

opusforlife2 commented Oct 26, 2020

Uh oh!

Stypox left a comment

Choose a reason for hiding this comment

Uh oh!

B0pol commented Oct 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Stypox commented Oct 26, 2020

Uh oh!

Stypox commented Oct 26, 2020

Uh oh!

Stypox commented Oct 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

opusforlife2 commented Oct 26, 2020

Uh oh!

FireMasterK commented Oct 26, 2020

Uh oh!

opusforlife2 commented Oct 26, 2020

Uh oh!

B0pol commented Oct 26, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

geckolinux commented Oct 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Tetriser commented Oct 27, 2020

Uh oh!

xibr commented Oct 27, 2020 • edited by B0pol Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Exception

Uh oh!

xibr commented Oct 27, 2020

Uh oh!

71walceli commented Oct 27, 2020

Uh oh!

xibr commented Oct 27, 2020

Uh oh!

FireMasterK commented Oct 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Exception

Uh oh!

B0pol commented Oct 27, 2020

Uh oh!

xibr commented Oct 27, 2020

Uh oh!

FarisZR commented Oct 27, 2020

Uh oh!

vrhikaru commented Oct 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Stypox left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

B0pol commented Oct 26, 2020 •

edited

Loading

xibr commented Oct 26, 2020 •

edited

Loading

B0pol commented Oct 26, 2020 •

edited

Loading

Stypox commented Oct 26, 2020 •

edited

Loading

B0pol commented Oct 26, 2020 •

edited

Loading

geckolinux commented Oct 27, 2020 •

edited

Loading

xibr commented Oct 27, 2020 •

edited by B0pol

Loading

FireMasterK commented Oct 27, 2020 •

edited

Loading

vrhikaru commented Oct 27, 2020 •

edited

Loading