Segfault in the vm

[layus]

Hi,

I have reproduced twice a segfault in the vm test /base/vm.oz.
The segfault occurs when running the "exception" test.

The common part is the thread running boost::asio::detail::timer_queue<...>::remove_timer(...).
I will try to reproduce this with a Debug build, but do you have any idea on the root cause of this ?

Program 1:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000055db38985f78 in std::function<long (mozart::VirtualMachine*)>::function(std::function<long (mozart::VirtualMachine*)> const&) ()
[Current thread is 1 (Thread 0x7f48a3371700 (LWP 13043))]
(gdb) bt
#0  0x000055db38985f78 in std::function<long (mozart::VirtualMachine*)>::function(std::function<long (mozart::VirtualMachine*)> const&) ()
#1  0x000055db3898605b in std::_Function_base::_Base_manager<void mozart::PropertyRegistry::registerReadOnlyProp<long>(mozart::VirtualMachine*, char const*, std::function<long (mozart::VirtualMachine*)> const&)::{lambda(mozart::VirtualMachine*)#1}>::_M_manager(std::_Any_data&, std::_Function_base::_Base_manager<void mozart::PropertyRegistry::registerReadOnlyProp<long>(mozart::VirtualMachine*, char const*, std::function<long (mozart::VirtualMachine*)> const&)::{lambda(mozart::VirtualMachine*)#1}> const&, std::_Manager_operation) ()
#2  0x000055db389863c6 in void std::vector<mozart::PropertyRecord, std::allocator<mozart::PropertyRecord> >::_M_emplace_back_aux<std::function<mozart::UnstableNode (mozart::VirtualMachine*)> const&, std::function<void (mozart::VirtualMachine*, mozart::RichNode)> const&>(std::function<mozart::UnstableNode (mozart::VirtualMachine*)> const&, std::function<void (mozart::VirtualMachine*, mozart::RichNode)> const&) ()
#3  0x000055db38986931 in void mozart::PropertyRegistry::registerReadWriteProp<long>(mozart::VirtualMachine*, char const*, long&) ()
#4  0x000055db389848c7 in mozart::PropertyRegistry::registerPredefined(mozart::VirtualMachine*) ()
#5  0x000055db38938cc6 in mozart::boostenv::BoostVM::BoostVM(mozart::boostenv::BoostEnvironment&, long, long, mozart::VirtualMachineOptions, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&&, bool) ()
#6  0x000055db3894c39c in mozart::boostenv::builtins::ModVM::New::call(mozart::VirtualMachine*, mozart::RichNode, mozart::UnstableNode&) ()
#7  0x000055db3894c616 in mozart::builtins::internal::BuiltinEntryPoint<mozart::boostenv::builtins::ModVM::New, 2ul, 2ul, mozart::UnstableNode&, mozart::UnstableNode&>::entryPoint(mozart::VirtualMachine*, mozart::UnstableNode&, mozart::UnstableNode&) ()
#8  0x000055db38974b77 in mozart::Thread::run() ()
#9  0x000055db38981038 in mozart::VirtualMachine::run() ()
#10 0x000055db3893ae1b in mozart::boostenv::BoostVM::run() ()
#11 0x000055db3890541a in main::{lambda(mozart::VirtualMachine*, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, bool)#1}::operator()(mozart::VirtualMachine*, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, bool) const ()
#12 0x000055db38905e14 in std::_Function_handler<bool (mozart::VirtualMachine*, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, bool), main::{lambda(mozart::VirtualMachine*, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, bool)#1}>::_M_invoke(std::_Any_data const&, mozart::VirtualMachine*&&, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&&, bool&&) ()
#13 0x000055db38938182 in mozart::boostenv::BoostVM::start(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, bool) ()
#14 0x00007f48a6604976 in ?? () from /usr/lib/libboost_thread.so.1.61.0
#15 0x00007f48a61c3454 in start_thread () from /usr/lib/libpthread.so.0
#16 0x00007f48a56637df in clone () from /usr/lib/libc.so.6

[Switching to thread 2 (Thread 0x7f48a7074740 (LWP 13042))]
#0  0x000055db38941328 in boost::asio::detail::timer_queue<boost::asio::detail::forwarding_posix_time_traits>::remove_timer(boost::asio::detail::timer_queue<boost::asio::detail::forwarding_posix_time_traits>::per_timer_data&) ()
(gdb) bt
#0  0x000055db38941328 in boost::asio::detail::timer_queue<boost::asio::detail::forwarding_posix_time_traits>::remove_timer(boost::asio::detail::timer_queue<boost::asio::detail::forwarding_posix_time_traits>::per_timer_data&) ()
#1  0x000055db389469c7 in boost::asio::detail::timer_queue<boost::asio::time_traits<boost::posix_time::ptime> >::get_ready_timers(boost::asio::detail::op_queue<boost::asio::detail::task_io_service_operation>&) ()
#2  0x000055db38913f84 in boost::asio::detail::epoll_reactor::run(bool, boost::asio::detail::op_queue<boost::asio::detail::task_io_service_operation>&) ()
#3  0x000055db3891459f in boost::asio::detail::task_io_service::run(boost::system::error_code&) ()
#4  0x000055db388f5f10 in main ()

Dump 2:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000055897e17e33b in boost::asio::detail::timer_queue<boost::asio::detail::forwarding_posix_time_traits>::remove_timer(boost::asio::detail::timer_queue<boost::asio::detail::forwarding_posix_time_traits>::per_timer_data&) ()
[Current thread is 1 (Thread 0x7f026554c740 (LWP 3500))]
(gdb) bt
#0  0x000055897e17e33b in boost::asio::detail::timer_queue<boost::asio::detail::forwarding_posix_time_traits>::remove_timer(boost::asio::detail::timer_queue<boost::asio::detail::forwarding_posix_time_traits>::per_timer_data&) ()
#1  0x000055897e1839c7 in boost::asio::detail::timer_queue<boost::asio::time_traits<boost::posix_time::ptime> >::get_ready_timers(boost::asio::detail::op_queue<boost::asio::detail::task_io_service_operation>&) ()
#2  0x000055897e150f84 in boost::asio::detail::epoll_reactor::run(bool, boost::asio::detail::op_queue<boost::asio::detail::task_io_service_operation>&) ()
#3  0x000055897e15159f in boost::asio::detail::task_io_service::run(boost::system::error_code&) ()
#4  0x000055897e132f10 in main ()

(gdb) thread 2
[Switching to thread 2 (Thread 0x7f0261849700 (LWP 3501))]
#0  0x00007f02646a41cd in read () from /usr/lib/libpthread.so.0
(gdb) bt
#0  0x00007f02646a41cd in read () from /usr/lib/libpthread.so.0
#1  0x00007f0264efa0a3 in boost::random::random_device::operator()() () from /usr/lib/libboost_random.so.1.61.0
#2  0x000055897e180484 in void boost::random::detail::seed_array_int_impl<32, 624ul, boost::random::random_device, unsigned int>(boost::random::random_device&, unsigned int (&) [624ul]) ()
#3  0x000055897e176317 in mozart::boostenv::BoostVM::BoostVM(mozart::boostenv::BoostEnvironment&, long, long, mozart::VirtualMachineOptions, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&&, bool) ()
#4  0x000055897e18939c in mozart::boostenv::builtins::ModVM::New::call(mozart::VirtualMachine*, mozart::RichNode, mozart::UnstableNode&) ()
#5  0x000055897e189616 in mozart::builtins::internal::BuiltinEntryPoint<mozart::boostenv::builtins::ModVM::New, 2ul, 2ul, mozart::UnstableNode&, mozart::UnstableNode&>::entryPoint(mozart::VirtualMachine*, mozart::UnstableNode&, mozart::UnstableNode&) ()
#6  0x000055897e1b1b77 in mozart::Thread::run() ()
#7  0x000055897e1be038 in mozart::VirtualMachine::run() ()
#8  0x000055897e177e1b in mozart::boostenv::BoostVM::run() ()
#9  0x000055897e14241a in main::{lambda(mozart::VirtualMachine*, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, bool)#1}::operator()(mozart::VirtualMachine*, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, bool) const ()
#10 0x000055897e142e14 in std::_Function_handler<bool (mozart::VirtualMachine*, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, bool), main::{lambda(mozart::VirtualMachine*, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, bool)#1}>::_M_invoke(std::_Any_data const&, mozart::VirtualMachine*&&, std::unique_ptr<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::default_delete<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&&, bool&&) ()
#11 0x000055897e175182 in mozart::boostenv::BoostVM::start(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, bool) ()
#12 0x00007f0264adc976 in ?? () from /usr/lib/libboost_thread.so.1.61.0
#13 0x00007f026469b454 in start_thread () from /usr/lib/libpthread.so.0
#14 0x00007f0263b3b7df in clone () from /usr/lib/libc.so.6

[layus]

Pour rappel:

    exception(proc {$}
             functor Fail
             import
               VM
             define
               {VM.getStream}.1 = unit
               raise error(expectedError) end
             end
             Other={VM.new Fail}
             S={VM.getStream}
          in
             {Send {VM.getPort Other} unit}
             S.1 = terminated(Other reason:exception)
          end
          keys:[mvm kill])

@eregon, could it be that the vm is destroyed too early, and some timer left with an invalid pointer ?
Most issues with boost::asio seem related to wrong objects scopes.

[eregon]

@layus Could be, indeed. IIRC I tried in general to access VMs in lambdas and critical code by looking up some identifier and not using the pointer directly.

The first trace is dying during registerReadOnlyProp while creating a new VM, that seems very early.

I'm not sure how remove_timer (in both traces) works and if it composes with other operations, but that's called from the boost asio thread, which we don't control.

[layus]

Even better, with a debug build (soo slow...) I get %** Tell: [1 6] = [1 4 6] in the kill test, so I inserted a {VM.list} = [1] right before Sleeper={VM.new Sleeping} and I now see the amazing, the fabulous, the incredible

%***************************** failure **************************
%**
%** Tell: [1] = [1]
%**
%** Call Stack:
%** procedure 'VM.list' in PC = 140000633039622
%** procedure 'Return.1.2.2.2.2.2.2.2.2.2.1.1' in file "/home/gmaudoux/projets/mozart2/platform-test/base/vm.oz", line 148, column 7, PC = 140000754123098
%** procedure 'ForProc' in file "/home/gmaudoux/projets/mozart2/platform-test/simple_runner.oz", line 85, column 7, PC = 140000753737010
%** procedure 'ForProc' in file "/home/gmaudoux/projets/mozart2/platform-test/simple_runner.oz", line 72, column 4, PC = 140000753739230
%** procedure in file "/home/gmaudoux/projets/mozart2/platform-test/simple_runner.oz", line 22, column 1, PC = 140000753742190
%** procedure 'RootManager,Pickle/fast' in file "/home/gmaudoux/abs/mozart2-git/src/mozart2-git/lib/main/init/Module.oz", line 244, column 7, PC = 140000752552983
%** procedure in file "/home/gmaudoux/abs/mozart2-git/src/mozart2-git/lib/main/base/Base.oz", line 84, column 22, PC = 140000752772180
%**--------------------------------------------------------------

Is there some black magic behind the list that is returned by VM.getList ?

[layus]

Note that I run the test in a loop, and the error only arises when I run some other intensive process at the same time, like a make -j4 of something.

[eregon]

That list seems built in a fairly standard way, but maybe @sjrd can confirm that's OK?

The preemptionTimer callback, BoostVM::onPreemptionTimerExpire keeps a reference to the VM.
But that timer should be canceled before terminating the VM.
Also, I would hope C++ exceptions unlock lock_guards, otherwise all is lost :p

[sjrd]

A better way would be to pass std::move(list) as tail instead of list, similarly to here, but it's fine.