High memory usage on Railway deployment?

[adilanchian]

Hi there!

I have a fairly simple API service deployed on Railway that uses Scrapling and I've noticed that my memory usage on Railways is consistently growing for every request.

I've seen memory usage up to 2GB with no sign of it going down.

I just wanted to see if there was anything I may be missing in Scrapling for cleanup?

Thanks in advance!

[D4Vinci]

Hi,

Can you share the Scrapling code you are using so I can check with you is something from Scrapling or the code can be enhanced?
if it's a memory leak bug in Scrapling I will try to debug and solve it ASAP, of course.

[D4Vinci]

Also, this line in your code is very wrong text_elements = p.find_all("css", "::text")
I think you are trying to get all the text content in the page right? Then you do this: text = page.get_all_text(strip=True)

Sorry my bad I didn't notice it all this time, any text you pass to find_all is treated as tags names and this isn't the case here. If you are trying to select with css then use .css method

[adilanchian]

yooo

really appreciate the info! ya totally get the library has some memory, that's totally fine. just wanting to make sure im doing everything to make sure to keep memory from skyrocketing out of no where.

Also, this line in your code is very wrong text_elements = p.find_all("css", "::text")
I think you are trying to get all the text content in the page right? Then you do this: text = page.get_all_text(strip=True)

this is great haha. thanks so much. will give it a try! not your bad at all homie, thanks for catching it!

really wanted to make your library work for my webscraping jobs. seems super promising. so thank you for this.

[D4Vinci]

I'm glad to help. I'm also working on a new update that saves memory on some operations. I already managed to make querying with CSS selectors using less memory ~145% better than before. I'm also looking into how to optimize caching further.
I will try to publish it soon, probably later today.

[D4Vinci]

Another version was released to optimize the memory usage: https://github.com/D4Vinci/Scrapling/releases/tag/v0.2.98
Please update and tell me how it goes for your case

[adilanchian]

you're insane brotha.

will update today again and let you know as soon as i do!

[adilanchian]

sup man!

just deployed 0.2.98 && made some changes based on your suggestions.

i mean, this is absolutely better 🤘

will give it some time to marinate of course, but feeling great so far.

you rock. thanks again.

[D4Vinci]

Great! Thanks for your feedback dude!

[adilanchian]

last 24h

seems pretty steady with a 9mb rise during this period.

thanks again!