Could not load nearby places

[nicolas-raoul]

Nearby shows this error toast since a few days.

Most interesting line from the logcat below says Please set a user-agent and respect our robot policy https://w.wiki/4wJS. See also T400119.
This is a recent server-side change: https://phabricator.wikimedia.org/T400119

Are we setting a User Agent?
Marking as High Priority as I believe Nearby is the best feature of our app.

08-29 18:36:01.215  1339 26647 V OkHttp  : --> GET https://query.wikidata.org/sparql?query=SELECT%20%28COUNT%28%3Fitem%29%20AS%20%3FitemCount%29%0AWHERE%20%7B%0A%20%20SERVICE%20wikibase%3Abox%20%7B%0A%20%20%20%20%3Fitem%20wdt%3AP625%20%3Flocation.%0A%20%20%20%20bd%3AserviceParam%20wikibase%3AcornerWest%20%22Point%28139.6927%2035.6456%29%22%5E%5Egeo%3AwktLiteral.%0A%20%20%20%20bd%3AserviceParam%20wikibase%3AcornerEast%20%22Point%28139.7117%2035.6701%29%22%5E%5Egeo%3AwktLiteral.%0A%20%20%7D%0A%7D%0A&format=json
08-29 18:36:01.215  1339 26647 V OkHttp  : --> END GET
08-29 18:36:01.328  1339 26647 V OkHttp  : <-- 403 https://query.wikidata.org/sparql?query=SELECT%20%28COUNT%28%3Fitem%29%20AS%20%3FitemCount%29%0AWHERE%20%7B%0A%20%20SERVICE%20wikibase%3Abox%20%7B%0A%20%20%20%20%3Fitem%20wdt%3AP625%20%3Flocation.%0A%20%20%20%20bd%3AserviceParam%20wikibase%3AcornerWest%20%22Point%28139.6927%2035.6456%29%22%5E%5Egeo%3AwktLiteral.%0A%20%20%20%20bd%3AserviceParam%20wikibase%3AcornerEast%20%22Point%28139.7117%2035.6701%29%22%5E%5Egeo%3AwktLiteral.%0A%20%20%7D%0A%7D%0A&format=json (112ms)
08-29 18:36:01.328  1339 26647 V OkHttp  : content-length: 92
08-29 18:36:01.328  1339 26647 V OkHttp  : content-type: text/plain
08-29 18:36:01.328  1339 26647 V OkHttp  : x-analytics: 
08-29 18:36:01.328  1339 26647 V OkHttp  : server: HAProxy
08-29 18:36:01.329  1339 26647 V OkHttp  : x-cache: cp5020 int
08-29 18:36:01.329  1339 26647 V OkHttp  : x-cache-status: int-tls
08-29 18:36:01.329  1339 26647 V OkHttp  : Please set a user-agent and respect our robot policy https://w.wiki/4wJS. See also T400119.
08-29 18:36:01.329  1339 26647 V OkHttp  : <-- END HTTP (92-byte body)
08-29 18:36:01.331  1339  1339 E NearbyParentFragment$populatePlacesForAnotherLocation: java.lang.Exception: 
08-29 18:36:01.331  1339  1339 E NearbyParentFragment$populatePlacesForAnotherLocation: 	at fr.free.nrw.commons.mwapi.OkHttpJsonApiClient.getNearbyItemCount(OkHttpJsonApiClient.kt:389)
08-29 18:36:01.331  1339  1339 E NearbyParentFragment$populatePlacesForAnotherLocation: 	at fr.free.nrw.commons.nearby.NearbyPlaces.getFromWikidataQuery(NearbyPlaces.java:147)
08-29 18:36:01.331  1339  1339 E NearbyParentFragment$populatePlacesForAnotherLocation: 	at fr.free.nrw.commons.nearby.NearbyController.loadAttractionsFromLocation(NearbyController.java:199)
08-29 18:36:01.331  1339  1339 E NearbyParentFragment$populatePlacesForAnotherLocation: 	at fr.free.nrw.commons.nearby.fragments.NearbyParentFragment.populatePlacesForAnotherLocation$lambda$47(NearbyParentFragment.kt:1629)
08-29 18:36:01.331  1339  1339 E NearbyParentFragment$populatePlacesForAnotherLocation: 	at fr.free.nrw.commons.nearby.fragments.NearbyParentFragment.$r8$lambda$4XHXyGqwsscHJuTq0g0Kbycdm8s(Unknown Source:0)
08-29 18:36:01.331  1339  1339 E NearbyParentFragment$populatePlacesForAnotherLocation: 	at fr.free.nrw.commons.nearby.fragments.NearbyParentFragment$$ExternalSyntheticLambda31.call(D8$$SyntheticClass:0)
08-29 18:36:01.331  1339  1339 E NearbyParentFragment$populatePlacesForAnotherLocation: 	at io.reactivex.internal.operators.observable.ObservableFromCallable.subscribeActual(ObservableFromCallable.java:43)
08-29 18:36:01.331  1339  1339 E NearbyParentFragment$populatePlacesForAnotherLocation: 	at io.reactivex.Observable.subscribe(Observable.java:12267)
08-29 18:36:01.331  1339  1339 E NearbyParentFragment$populatePlacesForAnotherLocation: 	at io.reactivex.internal.operators.observable.ObservableSubscribeOn$SubscribeTask.run(ObservableSubscribeOn.java:96)
08-29 18:36:01.331  1339  1339 E NearbyParentFragment$populatePlacesForAnotherLocation: 	at io.reactivex.Scheduler$DisposeTask.run(Scheduler.java:578)
08-29 18:36:01.331  1339  1339 E NearbyParentFragment$populatePlacesForAnotherLocation: 	at io.reactivex.internal.schedulers.ScheduledRunnable.run(ScheduledRunnable.java:66)
08-29 18:36:01.331  1339  1339 E NearbyParentFragment$populatePlacesForAnotherLocation: 	at io.reactivex.internal.schedulers.ScheduledRunnable.call(ScheduledRunnable.java:57)
08-29 18:36:01.331  1339  1339 E NearbyParentFragment$populatePlacesForAnotherLocation: 	at java.util.concurrent.FutureTask.run(FutureTask.java:317)
08-29 18:36:01.331  1339  1339 E NearbyParentFragment$populatePlacesForAnotherLocation: 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:348)
08-29 18:36:01.331  1339  1339 E NearbyParentFragment$populatePlacesForAnotherLocation: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1156)
08-29 18:36:01.331  1339  1339 E NearbyParentFragment$populatePlacesForAnotherLocation: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:651)
08-29 18:36:01.331  1339  1339 E NearbyParentFragment$populatePlacesForAnotherLocation: 	at java.lang.Thread.run(Thread.java:1119)
08-29 18:36:01.331  1339  1339 E NearbyParentFragment: Could not load nearby places

[rohit9625]

Seems like an urgent one. Are you working on this @RitikaPahwa4444 or should I take up this issue?

[RitikaPahwa4444]

@rohit9625 I have not started looking into this yet, please feel free to pick up if you wish :)

[nicolas-raoul]

I created this branch to ease debugging, it shows HTTP headers: #6414

Seems like our SPARQL requests do not send any header haha:

08-29 23:10:32.380 17367 17471 V OkHttp  : --> GET https://query.wikidata.org/sparql?query=SELECT%20%28COUNT%28%3Fitem%29%20AS%20%3FitemCount%29%0AWHERE%20%7B%0A%20%20SERVICE%20wikibase%3Abox%20%7B%0A%20%20%20%20%3Fitem%20wdt%3AP625%20%3Flocation.%0A%20%20%20%20bd%3AserviceParam%20wikibase%3AcornerWest%20%22Point%28139.6528%2035.6134%29%22%5E%5Egeo%3AwktLiteral.%0A%20%20%20%20bd%3AserviceParam%20wikibase%3AcornerEast%20%22Point%28139.6718%2035.6379%29%22%5E%5Egeo%3AwktLiteral.%0A%20%20%7D%0A%7D%0A&format=json
08-29 23:10:32.380 17367 17471 V OkHttp  : --> END GET
08-29 23:10:32.604 17367 17471 V OkHttp  : <-- 403 https://query.wikidata.org/sparql?query=SELECT%20%28COUNT%28%3Fitem%29%20AS%20%3FitemCount%29%0AWHERE%20%7B%0A%20%20SERVICE%20wikibase%3Abox%20%7B%0A%20%20%20%20%3Fitem%20wdt%3AP625%20%3Flocation.%0A%20%20%20%20bd%3AserviceParam%20wikibase%3AcornerWest%20%22Point%28139.6528%2035.6134%29%22%5E%5Egeo%3AwktLiteral.%0A%20%20%20%20bd%3AserviceParam%20wikibase%3AcornerEast%20%22Point%28139.6718%2035.6379%29%22%5E%5Egeo%3AwktLiteral.%0A%20%20%7D%0A%7D%0A&format=json (222ms)
08-29 23:10:32.604 17367 17471 V OkHttp  : content-length: 92
08-29 23:10:32.605 17367 17471 V OkHttp  : content-type: text/plain
08-29 23:10:32.605 17367 17471 V OkHttp  : x-analytics: 
08-29 23:10:32.605 17367 17471 V OkHttp  : server: HAProxy
08-29 23:10:32.606 17367 17471 V OkHttp  : x-cache: cp5024 int
08-29 23:10:32.606 17367 17471 V OkHttp  : x-cache-status: int-tls
08-29 23:10:32.607 17367 17471 V OkHttp  : Please set a user-agent and respect our robot policy https://w.wiki/4wJS. See also T400119.
08-29 23:10:32.607 17367 17471 V OkHttp  : <-- END HTTP (92-byte body)

It can be fixed using something like this: nicolas-raoul/trilingual-wiki@f289d71

It is late here, I am going to sleep. :-)

[rohit9625]

I created this branch to ease debugging. It shows HTTP headers: #6414

Thanks for the testing branch. Should I push the changes into the same branch or create another?

Seems like our SPARQL requests do not send any header haha:

Yes, you're right, that's exactly the issue.

[rohit9625]

@rohit9625 I have not started looking into this yet, please feel free to pick up if you wish :)

Okay, no problem, I'm on it and it seems like an easy one, will be fixed by tomorrow :)

[rohit9625]

We do send the User-Agent header in our API requests:

--> GET https://commons.wikimedia.org/w/api.php?format=json&action=wbgetentities&props=labels&languagefallback=1&ids=M144362629%7CM145512278%7CM146817940%7CM152702844%7CM152792569
2025-08-29 21:36:43.517 11102-11147 okhttp.OkHttpClient     fr.free.nrw.commons                  I  User-Agent: Commons/6.0.0-debug-main~f810a2d49 (https://mediawiki.org/wiki/Apps/Commons) Android/13
2025-08-29 21:36:43.517 11102-11147 okhttp.OkHttpClient     fr.free.nrw.commons                  I  --> END GET

I think they might have added more restrictions to the UA value. Let me post this in the related Phabricator thread.

[cdanis]

Fixed by sharing the code that sets User-Agent between both of the spots that OkHttpClients are created in the app.