Skip to content

Query fails to collect data for some keywords, which results in StopIteration in the next() #437

@p-veloso

Description

@p-veloso

Describe the bug
Query works with some keywords but triggers StopIteration in the next(search_query) with others even if that keyword works manually on google scholar.

To Reproduce

from scholarly import scholarly, ProxyGenerator

# pg = ProxyGenerator()
# success = pg.FreeProxies()
# scholarly.use_proxy(pg)

test = ["reinforcement", "reinforcement learning"] #it words with "reinforcement" but not with "reinforcement learning"
search_query = scholarly.search_keyword(test[1])

# Retrieve the first result from the iterator
first_author_result = next(search_query) #this line results in the StopIteration  error
scholarly.pprint(first_author_result)

# Retrieve all the details for the author
author = scholarly.fill(first_author_result )
scholarly.pprint(author)

# Take a closer look at the first publication
first_publication = author['publications'][0]
first_publication_filled = scholarly.fill(first_publication)
scholarly.pprint(first_publication_filled)

# Print the titles of the author's publications
publication_titles = [pub['bib']['title'] for pub in author['publications']]
print(publication_titles)

# Which papers cited that publication?
citations = [citation['bib']['title'] for citation in scholarly.citedby(first_publication_filled)]
print(citations)

Expected behavior
I expected the code to work with any keyword and even with combinations of precise keywords with quotation marks as in the manual search on google scholar.

Desktop (please complete the following information):

  • Proxy service: I am new to data crawlers, so I am not sure if I need a service or not
  • python version: 3.9
  • OS: Windows
  • Version 10

Do you plan on contributing?

  • Not now, but maybe in the future when I learn more about crawlers

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions