feat: Add ScrapyWebReader Integration

[hammadbinsajjad]

Description

Scrapy is one of the most popular Python frameworks for web scraping. It is used by many for web scraping at scale.
I recently used LlamaIndex in an Agentic RAG-based app for a hackathon, which ingested data using Scrapy, saved the embeddings, and then utilised LlamaIndex's Function Agent with them. While I was able to use it for my data ingestion, a built-in integration can make this process much more streamlined.

This PR adds integration for Scrapy spider and projects. Spiders can be run to scrape data from sites, which is then converted into documents. Existing projects can also be integrated by providing the project path.

Changes:

• Added Scrapy integration (ScrapyWebReader) in the web readers.
• Implemented the reader to use individual spiders and existing projects with ease.
• Updated web readers' documentation to include ScrapyWebReader class.
• Added unit tests covering both individual spiders and existing project use cases.
• Added usage example in Google Colab

Dependencies:

• Scrapy

New Package?

Did I fill in the tool.llamahub section in the pyproject.toml and provide a detailed README.md for my new integration or package?

• Yes
• No

Version Bump?

Did I bump the version in the pyproject.toml file of the package I am updating? (Except for the llama-index-core package)

• Yes
• No

Type of Change

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

How Has This Been Tested?

Your pull-request will likely not be merged unless it is covered by some form of impactful unit testing.

• I added new unit tests to cover this change
• I believe this change is already covered by existing unit tests

Suggested Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added Google Colab support for the newly added notebooks.
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I ran uv run make format; uv run make lint to appease the lint gods

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB