HTTPx Slow and hanging on banner in newest versions.

[bogbounty]

Current Behavior:

Hangs on banner randomly, Extremely slow since latest update. Didn't have any issues previous versions. How do i downgrade?

Steps To Reproduce:

For me its random some times it hangs others it doesn't I have tried everything. random-agent, setting threads to low and high. rate limiting to one request per second.

Tried on both VPS and without vps. Same issue.

Version

HTTPX v1.7.0

Results

This is my HTTPx after 10 hours. 20k subdomains. Runs pretty much just as slow with less than 5,000 subdomains.

[dogancanbakir]

Thank you for informing us! I will keep this issue open to see if others are experiencing problems with the latest release. For context, the most recent version has only minor changes; one feature has been improved (screenshot functionality), and dependencies have been updated.

[ogow]

I had a similar problem when probing a list of 100000 resolved hosts, but when I use the -stream flag it got way quicker. Maybe skip dedupe flag helps as well.

-s, -stream                      stream mode - start elaborating input targets without sorting
-sd, -skip-dedupe                disable dedupe input items (only used with stream mode)

I've never used such a big list before so maybe that the problem?

[bogbounty]

I had a similar problem when probing a list of 100000 resolved hosts, but when I use the -stream flag it got way quicker. Maybe skip dedupe flag helps as well.

-s, -stream                      stream mode - start elaborating input targets without sorting
-sd, -skip-dedupe                disable dedupe input items (only used with stream mode)

I've never used such a big list before so maybe that the problem?

Thank you this did actually speed up scans by quite a bit.
Even with a small subdomain list of 1,000 it takes around an hour for it to go through sometimes.

This helped but didn't fix totally.

[ogow]

I agree this is not fixing it completely, I'm using 100 threads and the httpx use less than 2% of my cpu (see image)
This is the command

cat all.subs.dnsx | jq '.host' -r --unbuffered | sort -R | httpx -t 100 -random-agent -stream -skip-dedupe -r 127.0.0.1

[Mzack9999]

@bogbounty @ogow I'm unable to reproduce the issue with a list of 10k items. The scan terminates in ca 3m:

$ time httpx -t 100 -l output.small.txt -random-agent
...

real	3m42.865s
user	3m42.359s
sys	0m29.895s

If the subdomain list has mostly resolvable but unresponsive hosts, you can lower the timeout (-timeout from default 10s to a lower amount, like 2s)