Fix ReCaptcha Activity for another type of recaptcha page #3414

Stypox · 2020-04-10T17:41:18Z

What is it?

Bug fix (user facing)
Feature (user facing)
Code base improvement (dev facing)
Meta improvement to the project (dev facing)

Description of the changes in your PR

Try to get cookies from pages even when they start loading, because non-html pages like ones with pbj=1 never stop loading.
Try to extract the cookie directly from the redirection url, by looking at the field google_abuse=.
Add GOOGLE_ABUSE_EXEMPTION= to the youtube recaptcha cookies.
Store cookies into shared preferences so that the user does not have to solve a recaptcha every time he opens the app

Fixes the following issue(s)

Fixes #3134 fixes #3272 fixes #3418 fixes #3310

Testing apk

@mehedihasanziku @baraa272 @LargePrime @tusch001 could you test this, if recaptchas still pop up for you? Also, @Ristovski, even though I didn't change anything about webviews, could you test, too?
NewPipe_recaptcha-debug_1.zip

Agreement

I carefully read the contribution guidelines and agree to them.

Stypox · 2020-04-11T09:59:25Z

Newest apk: NewPipe_recaptcha-debug_1.zip

Ristovski · 2020-04-11T14:08:14Z

@Stypox Just tested and it does indeed appear to fix the issue with non-native WebViews (#3310)

TobiGr · 2020-04-11T20:10:46Z

app/src/main/java/org/schabi/newpipe/ReCaptchaActivity.java

+                    e.printStackTrace();
+                    Log.d(TAG, "handleCookiesFromUrl: invalid google abuse starting at "
+                            + abuseStart + " and ending at " + abuseEnd + " for url " + url);


Is that information useful to find further cookies? We might want to show a snackbar to allow reporting the error and more importantly, the url

This piece of code has never been useful in my testing, since it just obtains a cookie that would anyway appear as a cookie in one of the subsequent requests. Though to prevent issues under different environment where the webview behaves strangely (who knows?), it doesn't hurt to obtain the cookie twice in a different way and call it a day.

I left a Log there just so that when I send people an apk to debug, I don't have to make a custom one that throws a random Exception with the log message inside (as I did with the other cookie testing apk), but I can just tell them to install a logcat viewer and I have the information needed for the whole execution of the activity.

Anyway, if the catch block is ever reached it means the url isn't useful to obtain a cookie, so there is no need to report anything to a normal user imho.

However logcat apps need root access since Jelly Bean.

Oh, do they? I didn't know ;-)
But what could I implement then, that would log everything and display it to the user after he presses "Ok" on a snackbar? Should I do some kind of bug report, but with logs instead of stack trace?

app/src/main/java/org/schabi/newpipe/ReCaptchaActivity.java

Try to get cookies from pages even when they start loading, because non-html pages like ones with `pbj=1` never stop loading. Try to extract the cookie directly from the redirection url, by looking at the field "google_abuse=". Add `GOOGLE_ABUSE_EXEMPTION=` to the youtube recaptcha cookies.

So that the user does not have to solve a recaptcha every time he opens the app

Stypox · 2020-04-15T16:41:51Z

@jniniinoinoino from #3418 (comment)
Unfortunately there is nothing that can be done to prevent recaptchas and tracking through the cookie being saved, because without that cookie nothing would work. Maybe an option to delete the current cookie could be created, though there are already far too many options, so I don't feel like adding another one.

the reCAPTCHA box disappeared

There has to be yet another different behaviour with recaptchas that only comes up sometimes. The recaptcha-solved page is never loaded and due to this no cookie is ever obtained. I'm investingating

mehedihasanziku · 2020-04-15T21:28:08Z

Yes I am using this latest debug apk, but i tried it so many times but captcha page not loading at all completely blank white screen with top tik (√) bar. I don't think it's a network issue, because i am running everything else very smoothly even youtube also in browser.

wb9688 · 2020-04-19T15:33:17Z

I think this should be merged now, and just leave the other reCAPTCHA page to another PR.

Stypox · 2020-04-19T16:47:44Z

I tried to understand in many ways what the problem is with the other recaptcha behaviour, but I still haven't understood anything because there seems to be no difference.

I tried to reproduce by connecting my pc to the Tor network and opening the app in an emulator, and the recaptcha could always be solved even after changing network multiple times.
I tried to reproduce by enabling the socks5 proxy using NetGuard, but that didn't trigger any recaptcha even though the IP had changed.
I was given (by someone facing this problem) the HAR file containing all of the connections made by Firefox during the resolution of a recaptcha under the same conditions where it didn't work in the app, but I couldn't find any meaningful information: everything seems to be the same as with a resolution that works in NewPipe
I tried to solve other manually-loaded recaptchas in the app (e.g. this: https://www.google.com/recaptcha/api2/demo), but it always worked

So yes, for now this can be merged, I'll leave further investigation for a future PR

Stypox added ASAP Issue needs to be fixed as soon as possible bug Issue is related to a bug youtube Service, https://www.youtube.com/ labels Apr 10, 2020

Stypox added this to the 0.19.3 milestone Apr 10, 2020

Stypox force-pushed the recaptcha branch from e721702 to bd0d7ce Compare April 11, 2020 09:52

TobiGr reviewed Apr 11, 2020

View reviewed changes

wb9688 reviewed Apr 15, 2020

View reviewed changes

app/src/main/java/org/schabi/newpipe/ReCaptchaActivity.java Outdated Show resolved Hide resolved

Stypox added 3 commits April 15, 2020 13:30

Remove duplicate check if cookies are already present

b8efef7

Save and restore cookies to/from preferences

ee5ce0c

So that the user does not have to solve a recaptcha every time he opens the app

Stypox force-pushed the recaptcha branch from bd0d7ce to ee5ce0c Compare April 15, 2020 11:30

Stypox mentioned this pull request Apr 15, 2020

Non Authorized Apps Captcha Verification #3134

Closed

3 tasks

Stypox mentioned this pull request Apr 16, 2020

Incomplete requests, completed reCAPTCHA not persistent #3446

Closed

wb9688 approved these changes Apr 19, 2020

View reviewed changes

wb9688 merged commit 8e9b1b7 into TeamNewPipe:dev Apr 19, 2020

This was referenced Apr 24, 2020

Release 0.19.3 #3478

Merged

Release 0.19.3 #3479

Closed

johnth3doe mentioned this pull request Sep 25, 2020

Unable to load comments + (Re)Captcha issue #4312

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Fix ReCaptcha Activity for another type of recaptcha page #3414

Fix ReCaptcha Activity for another type of recaptcha page #3414

Uh oh!

Stypox commented Apr 10, 2020 •

edited

Loading

Uh oh!

Stypox commented Apr 11, 2020

Uh oh!

Ristovski commented Apr 11, 2020

Uh oh!

TobiGr Apr 11, 2020

Uh oh!

Stypox Apr 15, 2020 •

edited

Loading

Uh oh!

wb9688 Apr 15, 2020

Uh oh!

Stypox Apr 15, 2020

Uh oh!

Uh oh!

Stypox commented Apr 15, 2020

Uh oh!

mehedihasanziku commented Apr 15, 2020 •

edited

Loading

Uh oh!

wb9688 commented Apr 19, 2020

Uh oh!

Stypox commented Apr 19, 2020 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Fix ReCaptcha Activity for another type of recaptcha page #3414

Fix ReCaptcha Activity for another type of recaptcha page #3414

Uh oh!

Conversation

Stypox commented Apr 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is it?

Description of the changes in your PR

Fixes the following issue(s)

Testing apk

Agreement

Uh oh!

Stypox commented Apr 11, 2020

Uh oh!

Ristovski commented Apr 11, 2020

Uh oh!

TobiGr Apr 11, 2020

Choose a reason for hiding this comment

Uh oh!

Stypox Apr 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wb9688 Apr 15, 2020

Choose a reason for hiding this comment

Uh oh!

Stypox Apr 15, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Stypox commented Apr 15, 2020

Uh oh!

mehedihasanziku commented Apr 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wb9688 commented Apr 19, 2020

Uh oh!

Stypox commented Apr 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Stypox commented Apr 10, 2020 •

edited

Loading

Stypox Apr 15, 2020 •

edited

Loading

mehedihasanziku commented Apr 15, 2020 •

edited

Loading

Stypox commented Apr 19, 2020 •

edited

Loading