DB failover causes hard lockup but I want to fail reads/writes #6170
-
|
I am using PostgreSQL with Patroni and HA proxy so that traffic is directed to the primary node at all times. During a cluster fail-over I am getting errors like the one below. Unfortunately this causes a hard lock-up of the file system for any active reads or writes. I don't know how to have errors returned to the client in a case like this to abort the operation. I know that mount.cifs has a hard and soft mount option. The soft option will "not hang when the server crashes and will return errors to the user application". Note: localhost below is the HA proxy front-end address. The process attempting a file read is stuck in disk sleep. |
Beta Was this translation helpful? Give feedback.
Replies: 5 comments 18 replies
-
|
The current implementation attempts transactions 50 times, with this value hardcoded in sql.go. If necessary, you can modify it. Pull requests (PRs) are welcome to make this an optional configuration. |
Beta Was this translation helpful? Give feedback.
-
|
Could you please let me know which version you’re using? (The output of juicefs version). I attempted to replicate the issue through code mocking but couldn’t reproduce it. Could you try using the latest version instead? Also, does juicefs.log contain any additional logs? |
Beta Was this translation helpful? Give feedback.
-
On another client I get errors like this. The output is a bit longer but you get the idea. |
Beta Was this translation helpful? Give feedback.
-
|
According to the logs, JuiceFS is still processing requests—50 database requests take over 40 seconds. For read operations, in addition to database request retries, the external reader.go also retries metadata requests, with a default of 10 times. Therefore, a read request could be blocked for up to 10 * 40 seconds （if your db still not restored）. It seems like the operation is hanging. You might want to adjust these two parameters to see if it helps. |
Beta Was this translation helpful? Give feedback.
-
|
It's hanging for longer that 10 * 40 seconds (about 6 minutes). |
Beta Was this translation helpful? Give feedback.
-
|
So I took a harder look at the code and in all cases the traceback starts at Thoughts and feedback? |
Beta Was this translation helpful? Give feedback.
-
|
@mcassaniti |
Beta Was this translation helpful? Give feedback.
-
|
You're right, but this would've only been for a brief period during fail-over. It's likely either the primary went read-only or HA proxy temporarily pointed to the wrong node. In any case it would've resolved reasonably quickly. |
Beta Was this translation helpful? Give feedback.
-
|
After a bit more digging I found something rather interesting. If I run When this situation occurred today I had two nodes that had some locked-up processes. One refreshed its session and the other did not. The one that successfully refreshed its session could have processes killed with I'm going to try changing the heartbeat option to mount from the default of 12 seconds to something higher. This won't yet fix the locked-up processes, but it will mean I don't have to reboot to get things working again. |
Beta Was this translation helpful? Give feedback.
-
|
Another testing update. I put juicefs in debug mode and tried the same file copies I do during a cutover. Below you can see at most 5 retries. I don't believe that changing the SQL retries @jiefenghuang would assist based on what I'm seeing below. Strangely one of my nodes has the reading process stuck in disk sleep (state D) while my other nodes have the same process sleeping (state S). If I send a Reading the code here shows that only the attempt to begin the DB transaction is failing. After the transaction can be successfully started it should succeed and return based on the code flow. |
Beta Was this translation helpful? Give feedback.
-
|
Could you upload the complete logs? Once the number of failures reaches the limit, it will return EIO to FUSE. |
Beta Was this translation helpful? Give feedback.
-
|
2025/06/04 10:06:37.000606 juicefs[3978] : slow operation: getattr (348183): - input/output error <3087.794843> [accesslog.go:83] From your logs above, your client should be able to receive EIO. |
Beta Was this translation helpful? Give feedback.
-
|
I would've thought I'd get an EIO too and I'm worried that I'm not getting anything (failure or success) when the retries seem to succeed again. As for the client side code, below is a cut down version. There's a set of files to transfer and one of them is usually large enough that it can take 30+ seconds to transfer. def transfer_file(
src : pathlib.Path,
dst_dir : pathlib.Path,
target_name : Optional[str] = None
) -> None:
dst = dst_dir / src.name
if target_name:
dst = dst_dir / target_name
with open(src, "rb") as src_fp, open(dst, "wb") as dst_fp:
for chunk in iter(lambda: src_fp.read(4096), b""):
dst_fp.write(chunk) |
Beta Was this translation helpful? Give feedback.
-
|
I tried something different and blocked the SQL traffic (dropping packets) from the JuiceFS client just to see what would show up in the logs. I got a lot of slow operations when I stopped dropping packets but not many failed transactions. I'm guessing that the issue is a hung query that doesn't have a timeout. Does a simple SQL get inside a read-only transaction have a timeout? |
Beta Was this translation helpful? Give feedback.
-
|
Does PostgreSQL support these too? |
Beta Was this translation helpful? Give feedback.
-
|
yes |
Beta Was this translation helpful? Give feedback.
-
|
I needed While one of my nodes was locked-up I decided to restart HA proxy which will forcefully kill off any existing SQL connections. I suddenly got an I/O error back to the process that was stuck. My suspicion about hung SQL queries seems to be correct. I'm about to do some more testing with SQL retries set to 10 and |
Beta Was this translation helpful? Give feedback.
-
|
So, even without the change in SQL retries, the addition of At this point I'll leave the SQL retries alone. Thanks so much for taking the time to work with me on this one. |
Beta Was this translation helpful? Give feedback.
-
Sorry, I got the driver wrong. The parameter for PostgreSQL is connect_timeout. |
Beta Was this translation helpful? Give feedback.
So, even without the change in SQL retries, the addition of
connect_timeout=5has meant that my JuiceFS mounts are no longer getting stuck and causing processes to block.At this point I'll leave the SQL retries alone.
Thanks so much for taking the time to work with me on this one.