Custom filter lists sometimes won't update + an exception throwed in a browser while pressing update button in uBO

[kulfoon]

Prerequisites

• I verified that this is not a filter issue
  • Filter issues MUST be reported at filter issue tracker
• This is not a support issue or a question
  • Support issues and questions are handled at /r/uBlockOrigin
• I performed a cursory search of the issue tracker to avoid opening a duplicate issue
  • Your issue may already be reported.
• I tried to reproduce the issue when...
  • uBlock Origin is the only extension
  • uBlock Origin with default lists/settings
  • using a new, unmodified browser profile
• I am running the latest version of uBlock Origin
• I checked the documentation to understand that the issue I report is not a normal behavior

Description

Custom filter lists sometimes won't update, the issue has been happening to me since several months already, but I wasn't sure whether it was uBO's issue, and I was hoping somehow that it was just a temporary issue, which might be fixed somehow in uBO or in a browser as the time passes, but it hasn't happened, and the issue is getting annoying more and more, so I decided to create an issue. I'm still not sure whether it's uBO's, a browser's or an user's issue, but knowing that it occurs in two different browsers (Firefox, Opera) (and in Firefox in two different profiles, which one of them is reasonablly fresh) it seems to be not a browser's issue. Also this guy krystian3w has the same issue, we both have talked in the past about it, I don;t remember exactly, but he was saying something about that to force a successfulludpate of the list in this case is to add some string at the end of the list path name and visit it in a browser, something like a question mark followed by a some number like this ?_=9 which is visible in the uBO logger while updating a filter list, it sometimes works but not always, it seems to not work anymore for me. Unfortunatelly this guy will not reply here in this thread to help us because I had to put him on my blocklist due to his ugly behaviour so he can't reply to my threads anymore.

A specific URL where the issue occurs

An example local list file path:
http://localhost/local_filters_lists_uBO/my_example_list_my_example_list.txt​

Steps to Reproduce

1. Add an example local list with some example filters in it (to do it in Firefox, you need to setup a virtual server first to use local files in browser).
2. Edit the list and add or remove some filters / domains or even a comment.
3. Click the list update icon - a clock icon at the end of the list name.
4. Click the main Update button in uBO.

Expected behavior:

Custom filter lists should be always updated.

Actual behavior:

Custom filter lists won't (SOMETIMES! - hard to reproduce it always! sometimes a filter list will be updated with no problems) update in uBO, despite the fact that the physically txt file has been modified and saved, uBO still showing the old content of the txt file. Repeating the update process: reediting / resaving the txt file and / or clicking update buttons in uBO multiple times or clearing the uBO cache, or hard reloading the page with filter list content bypassing the cache, or restarting a browser or a virtual server in Firefox, gives no result. After some random time, sometimes minutes, sometimes hours, the local filter list will finally update itself with no human input, just doing some random stuff on PC, not always internet related, and then randomly coming back to uBO to see that the list has finally updated itself as the time has passed.

In Firefox, in the browser console I noticed an strange exception which is being throwed exactly at the time I hit main Update button in uBO, not sure whether it's related and whether it's uBO's, a browser's or an user side's issue, (couldn't find a similiar browser's console in Opera, to see what it will say, and a normal Opera's (or Firefox's) console has throwed nothing) (also I don't remember whether the exception was occuring in older versions of uBO and the browser, than the current ones I'm using):

Handler function threw an exception: [Exception... "Component returned failure code: 0x80040111 (NS_ERROR_NOT_AVAILABLE) [nsICacheInfoChannel.isRacing]"  nsresult: "0x80040111 (NS_ERROR_NOT_AVAILABLE)"  location: "JS frame :: resource://devtools/server/actors/network-monitor/network-response-listener.js :: NetworkResponseListener.prototype._getSecurityInfo< :: line 301"  data: no]
Stack: NetworkResponseListener.prototype._getSecurityInfo<@resource://devtools/server/actors/network-monitor/network-response-listener.js:301:26
exports.makeInfallible/<@resource://devtools/shared/ThreadSafeDevToolsUtils.js:109:22
onStartRequest@resource://devtools/server/actors/network-monitor/network-response-listener.js:209:10
Line: 301, column: 0 ThreadSafeDevToolsUtils.js:88:13

Your environment

• uBlock Origin version: uBlock Origin 1.21.5b1 / 1.21.7b0 and earlier versions from last several months
• Browser Name and version: Firefox 68.0.1 64bit / Opera 62.0.3331.72 64bit and earlier versions from last several months
• Operating System and version: Win 7 Pro SP1 64bit

Does not look like issue. uBO downloads list from browser cache. If you wait one hour it should change this ?_=9 number at the end of url, and update fresh from server.

About exception: ThreadSafeDevToolsUtils.js this looks like Developer Tools internal code, not related to uBO or normal browsing without DevTools opened.

Feature :)

https://github.com/gorhill/uBlock/blob/869c63f82115f27ba9f8af939522d12fa3bd82fb/src/js/assets.js#L69

I tried http://0.0.0.0:8080/my-filters.txt with Python server python3 -m http.server 8080 and cannot reproduce. Probably because lack of any caching headers:

[uBlock-user]

Browser is throwing an exception which isn't caught. Similar Firefox bugs -- https://bugzilla.mozilla.org/buglist.cgi?quicksearch=0x80040111&list_id=14820451

@kulfoon What you use as a server? Can you enable XHR in browser console and check what is happening with this network request?

[uBlock-user]

This is how it looks like on Firefox on my end --

On updating the list --

[kulfoon]

gwarser : What you use as a server?

I think that it doesn't matter as the issue occurs on chrome-based browsers too and on the chrome based browsers there is no need to use a server, thus I don't use a server there.

Currently I can't reproduce as it is annoying kind of a random issue, but I'm sure it's a matter of time when I'll reproduce as the issue occurs during the whole last several months, thus when I reproduce I'll tell you then what Network tab shows, maybe it's like in #682 (comment), not an issue but a feature, will see.

[gorhill]

occurs on chrome-based browsers too

What errors do you get in uBO's dev console with Chromium-based browsers?

[kulfoon]

As for "Custom filter lists sometimes won't update" issue I don't get any errors in dev console neither in Firefox nor in Opera, nor I get any errors in uBO logger, only in Firefox's browser console I get the exception, but as it was pointed out it's most probably not related.

I was able to reproduce "Custom filter lists sometimes won't update" issue now and it seems it has to be what gwarser said in #682 (comment), a feature and not an issue.

When the issue occured now first time:

1. the xhr was labeled in "Sent" column as "in cache memory",
2. after I updated again, filter list content got updated and there was no xhr entry,

I changed the content again and updated the filter list and the issue occured the 2nd time:

3. the xhr was labeled in "Sent" column as "in cache memory",
4. after I updated again, filter list content got updated with an xhr entry in columnd "Sent" labeled as "plain"

So it seems a feature and not an issue, I just wonder:

a) after updating the modified filter list second time, in the both reproduced cases above I didn't need to wait an hour to get the list freshly updated (not from cache), I just updated the 2nd time instantly after the 1st time, so what about this "an hour" timer ? On the other hand, also I remember in the past sometimes clicking update list multiple times and it refused to update the list, now it updates just after 2nd try. So what's going on with all these random update time fluctuations ?

b) where is one of the XHR request Nr 2, as seen on the screenshot there are only 3 XHRs, but should be 4, any idea? But never mind.