Timeout for monitor operation

[kvaps]

Description
When node have problem with stucked operations, it can brake OpenNebula itself, eg it may be broken disk subsutem, disconneted target or some other problem.
OpenNebula runs a lot of /var/lib/one/remotes/tm/<driver>/monitor operations but they are stuck forever.

To Reproduce
Eg right now I have broken LUN and any lvm command is stuck for ages. Try to reproduce that:

• Connect iSCSI target
• Create LVM group
• Create VM in this LVM storage
• Run VM
• Try to disconnect LUN

Now you have broken host, and any lvm command will stuck forever.
Wait for a while, then check ps aux on the opennebula you will se a lots of hanged monitor comands

Expected behavior
OpenNebula will return ERROR on this host monitoring and continue monitoring of the rest hosts.

Details

• Affected Component: Storage Drivers
• Hypervisor: KVM
• Version: 5.6.1

Additional context
Add any other context about the problem here.

Progress Status

• Branch created
• Code committed to development branch
• Testing - QA
• Documentation
• Release notes - resolved issues, compatibility, known issues
• Code committed to upstream release/hotfix branches
• Documentation committed to upstream release/hotfix branches

[kvaps]

The similar problem was described here:
#1702

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. The OpenNebula Dev Team

[stale]

This issue has been automatically closed due to lack of activity/feedback. Please reopen if you have further input or need to bump this. The OpenNebula Dev Team

[kvaps]

BTW, I've just started using linstor_un driver instead of LVM, and everything started working as it should. Linstor is having own timeouts for LVM operations.

[tinova]

this is a sound suggestion, reopening it