Crashing while fetching a website with Windows-1256 encoding #80
Description
Have you searched if there an existing issue for this?
- I have searched the existing issues
Python version (python --version)
Python 3.13.0
Scrapling version (scrapling.version)
0.3.1
Dependencies version (pip3 freeze)
aiodns==3.2.0
aiofiles==24.1.0
aiohappyeyeballs==2.4.4
aiohttp==3.11.10
aiohttp-socks==0.4.1
aiosignal==1.3.2
aiosqlite==0.20.0
annotated-types==0.7.0
anyio==4.8.0
APScheduler==3.11.0
asgiref==3.8.1
async-timeout==5.0.1
attrs==24.2.0
beautifulsoup4==4.12.3
behave==1.2.6
blinker==1.9.0
bota==4.0.71
botasaurus==4.0.76
botasaurus_api==4.0.9
botasaurus_driver==4.0.66
botasaurus_proxy_authentication==1.0.16
botasaurus_requests==4.0.38
Brotli==1.1.0
browserforge==1.2.3
build==1.2.2.post1
camoufox==0.4.11
cchardet==2.1.7
certifi==2025.8.3
cffi==1.17.1
cfscrape==2.1.1
chardet==5.2.0
charset-normalizer==3.4.0
click==8.2.1
close_chrome==4.0.40
cloudscraper==1.2.71
colorama==0.4.6
Crawl4AI==0.4.247
cryptography==44.0.0
cssselect==1.3.0
curl_cffi==0.13.0
Cython==3.0.11
dataclasses==0.6
Deprecated==1.2.18
diffusers==0.31.0
distro==1.9.0
elastic-transport==8.15.1
elasticsearch==8.17.0
et_xmlfile==2.0.0
exceptiongroup==1.2.2
execnet==2.1.1
fake-http-header==0.3.5
fake-useragent==2.0.3
fasteners==0.19
feedparser==6.0.11
filelock==3.16.1
Flask==3.1.0
frozenlist==1.5.0
fsspec==2024.10.0
geographiclib==2.0
geoip2==5.1.0
geopy==2.4.1
gevent==24.11.1
geventhttpclient==2.3.3
googletransx==2.4.2
greenlet==3.1.1
gunicorn==23.0.0
h11==0.14.0
httpcore==1.0.7
httpx==0.27.2
huggingface-hub==0.26.2
idna==3.10
importlib_metadata==8.5.0
iniconfig==2.0.0
iso8601==2.1.0
itsdangerous==2.2.0
javascript_fixes==1.1.29
Jinja2==3.1.4
jiter==0.8.2
joblib==1.4.2
jsonschema==4.23.0
jsonschema-specifications==2024.10.1
language-tags==1.2.0
litellm==1.59.1
lxml==6.0.1
markdown-it-py==3.0.0
MarkupSafe==3.0.2
maxminddb==2.8.2
mdurl==0.1.2
mockito==1.5.3
msgspec==0.19.0
multidict==6.1.0
mycdp==1.1.0
nltk==3.9.1
numpy==2.1.3
openai==1.59.9
openpyxl==3.1.5
orjson==3.11.3
outcome==1.3.0.post0
packaging==24.2
pandas==2.2.3
parameterized==0.9.0
parse==1.20.2
parse_type==0.6.4
pdbp==1.6.1
pillow==10.4.0
pip==25.2
platformdirs==4.3.6
playwright==1.55.0
pluggy==1.5.0
propcache==0.2.1
psutil==6.1.1
py==1.11.0
pycares==4.5.0
pycparser==2.22
pydantic==2.10.5
pydantic_core==2.27.2
pyee==13.0.0
Pygments==2.18.0
pynose==1.5.3
pyobjc-core==11.1
pyobjc-framework-Cocoa==11.1
pyOpenSSL==25.0.0
pyotp==2.9.0
pyparsing==3.2.0
pyproject_hooks==1.2.0
pyquery==2.0.1
PySocks==1.7.1
pytest==8.3.3
pytest-html==2.0.1
pytest-metadata==3.1.1
pytest-mockito==0.0.4
pytest-ordering==0.6
pytest-rerunfailures==14.0
pytest-xdist==3.6.1
python-dateutil==2.9.0.post0
python-dotenv==1.0.1
python-socks==2.5.3
pytz==2024.2
PyVirtualDisplay==3.0
PyYAML==6.0.2
rank-bm25==0.2.2
reader==3.15
rebrowser_playwright==1.52.0
referencing==0.36.1
regex==2024.11.6
requests==2.32.3
requests-file==2.1.0
requests-toolbelt==1.0.0
rich==13.9.4
rpds-py==0.22.3
safetensors==0.4.5
sbvirtualdisplay==1.3.0
schedule==1.2.2
scrapling==0.3.1
screeninfo==0.8.1
selenium==4.26.1
seleniumbase==4.32.12
setuptools==80.9.0
sgmllib3k==1.0.0
six==1.16.0
sniffio==1.3.1
snowballstemmer==2.2.0
sortedcontainers==2.4.0
soupsieve==2.6
tabcompleter==1.4.0
tf-playwright-stealth==1.1.0
tiktoken==0.8.0
timer==0.3.0
tldextract==5.3.0
tokenizers==0.21.0
tqdm==4.67.1
trio==0.27.0
trio-websocket==0.11.1
typing_extensions==4.12.2
tzdata==2024.2
tzlocal==5.2
ua-parser==1.0.1
ua-parser-builtins==0.18.0.post1
urllib3==2.2.3
websocket-client==1.8.0
websockets==14.1
Werkzeug==3.1.3
wheel==0.45.0
wrapt==1.17.2
wsproto==1.2.0
xxhash==3.5.0
yarl==1.18.3
zipp==3.21.0
zope.event==5.0
zope.interface==7.2
What's your operating system?
macOS 15.6.1
Are you using a separate virtual environment?
No
Expected behavior
The flow is expected to work fine without any errors
Actual behavior
Crashing while fetching a website
Steps To Reproduce
got an exception error while trying to fetch a website using Windows-1256 encoding
website url: https://marebpress.net
Example:
try: page = Fetcher.get("https://marebpress.net") print(page.status) except Exception as e: print(e)
Output
'utf-8' codec can't decode byte 0xe3 in position 297: invalid continuation byte