Pass in args to async_fetch and fetch #54
Description
Have you searched if there an existing issue for this?
- I have searched the existing issues
Python version (python --version)
3.12
Scrapling version (scrapling.version)
0.2.94
Dependencies version (pip3 freeze)
╭──────────────────────────┬──────────────┬─────────────────────────────────────────╮
│ name │ version │ location │
├──────────────────────────┼──────────────┼─────────────────────────────────────────┤
│ aiohappyeyeballs │ 2.4.6 │ │
│ aiohttp │ 3.11.12 │ │
│ aiosignal │ 1.3.2 │ │
│ annotated-types │ 0.7.0 │ │
│ anyio │ 4.8.0 │ │
│ attrs │ 25.1.0 │ │
│ browserforge │ 1.2.3 │ │
│ camoufox │ 0.4.11 │ │
│ certifi │ 2024.12.14 │ │
│ charset-normalizer │ 3.4.1 │ │
│ click │ 8.1.8 │ │
│ cssselect │ 1.2.0 │ │
│ distro │ 1.9.0 │ │
│ fastapi │ 0.115.6 │ │
│ filelock │ 3.17.0 │ │
│ freezegun │ 1.5.1 │ │
│ frozenlist │ 1.5.0 │ │
│ greenlet │ 3.1.1 │ │
│ h11 │ 0.14.0 │ │
│ httpcore │ 1.0.7 │ │
│ httpx │ 0.28.1 │ │
│ idna │ 3.10 │ │
│ iniconfig │ 2.0.0 │ │
│ jiter │ 0.8.2 │ │
│ joblib │ 1.4.2 │ │
│ jsonpatch │ 1.33 │ │
│ jsonpointer │ 3.0.0 │ │
│ langchain │ 0.3.19 │ │
│ langchain-core │ 0.3.37 │ │
│ langchain-openai │ 0.3.6 │ │
│ langchain-text-splitters │ 0.3.6 │ │
│ langserve │ 0.3.1 │ │
│ langsmith │ 0.3.10 │ │
│ language-tags │ 1.2.0 │ │
│ lxml │ 5.3.1 │ │
│ multidict │ 6.1.0 │ │
│ mypy │ 1.14.1 │ │
│ mypy-extensions │ 1.0.0 │ │
│ nltk │ 3.9.1 │ │
│ numpy │ 2.2.3 │ │
│ openai │ 1.64.0 │ │
│ orjson │ 3.10.15 │ │
│ packaging │ 24.2 │ │
│ platformdirs │ 4.3.6 │ │
│ playwright │ 1.50.0 │ │
│ pluggy │ 1.5.0 │ │
│ propcache │ 0.3.0 │ │
│ pydantic │ 2.10.5 │ │
│ pydantic-partial │ 0.7.0 │ │
│ pydantic_core │ 2.27.2 │ │
│ pyee │ 12.0.0 │ │
│ PyJWT │ 2.10.1 │ │
│ PySocks │ 1.7.1 │ │
│ pytest │ 8.3.4 │ │
│ pytest-asyncio │ 0.25.2 │ │
│ python-dateutil │ 2.9.0.post0 │ │
│ python-dotenv │ 1.0.1 │ │
│ python-multipart │ 0.0.20 │ │
│ pytz │ 2025.1 │ │
│ PyYAML │ 6.0.2 │ │
│ rebrowser_playwright │ 1.49.1 │ │
│ regex │ 2024.11.6 │ │
│ requests │ 2.32.3 │ │
│ requests-file │ 2.1.0 │ │
│ requests-toolbelt │ 1.0.0 │ │
│ scrapling │ 0.2.94 │ │
│ screeninfo │ 0.8.1 │ │
│ six │ 1.17.0 │ │
│ sniffio │ 1.3.1 │ │
│ SQLAlchemy │ 2.0.38 │ │
│ sqlalchemy-stubs │ 0.4 │ │
│ sse-starlette │ 1.8.2 │ │
│ starlette │ 0.41.3 │ │
│ stripe │ 11.6.0 │ │
│ tenacity │ 9.0.0 │ │
│ tiktoken │ 0.9.0 │ │
│ tldextract │ 5.1.3 │ │
│ tqdm │ 4.67.1 │ │
│ typing_extensions │ 4.12.2 │ │
│ ua-parser │ 1.0.1 │ │
│ ua-parser-builtins │ 0.18.0.post1 │ │
│ urllib3 │ 2.3.0 │ │
│ uvicorn │ 0.34.0 │ │
│ w3lib │ 2.3.1 │ │
│ yarl │ 1.18.3 │ │
│ zstandard │ 0.23.0 │ │
╰──────────────────────────┴──────────────┴─────────────────────────────────────────╯
What's your operating system?
Windows 11
Are you using a separate virtual environment?
Yes
Expected behavior
I need to be able to pass in a list of args in the async_fetch and fetch of the StealthyFetcher
Even being able to pass in a list of args for the call to with AsyncCamoufox() as browser would be nice, so you could pass
with AsyncCamoufox(...rest of arguments, args=args)
Ex:
async def async_fetch(
self,
url: str,
**kwargs # Catch any additional keyword arguments
) -> Response:
"""Opens up the browser and does your request based on your chosen options.
:param url: Target URL.
:param kwargs: Additional keyword arguments for flexible configuration.
:return: A `Response` object that is the same as `Adaptor` object except it has these added attributes: `status`, `reason`, `cookies`, `headers`, and `request_headers`.
"""
# Default value for 'addons' if not provided in kwargs
addons = [] if self.disable_ads else [DefaultAddons.UBO]
# Store the final response
final_response = None
async def handle_response(finished_response):
nonlocal final_response
if finished_response.request.resource_type == "document" and finished_response.request.is_navigation_request():
final_response = finished_response
camoufox_options = {
'geoip': self.geoip,
'proxy': self.proxy,
'enable_cache': True,
'addons': self.addons,
'exclude_addons': addons,
'headless': self.headless,
'humanize': self.humanize,
'i_know_what_im_doing': True, # To turn warnings off with user configurations
'allow_webgl': self.allow_webgl,
'block_webrtc': self.block_webrtc,
'block_images': self.block_images,
'os': None if self.os_randomize else get_os_name(),
}
camoufox_options.update(kwargs)
async with AsyncCamoufox(**camoufox_options) as browser:
Actual behavior
Can not pass in extra args.
Steps To Reproduce
No response