Skip to content

[Core] status Status not OK: ObjectUnknownOwner: An application is trying to access a Ray object whose owner is unknown(3dd79ffe8938fd5bffffffffffffffffffffffff3000000001000000). #58849

@g199209

Description

@g199209

What happened + What you expected to happen

src/ray/core_worker/core_worker.cc:862 (PID: 2135613, TID: 2135731, errno: 11 (Resource temporarily unavailable)): An unexpected system state has occurred. You have likely discovered a bug in Ray. Please report this issue at https://github.com/ray-project/ray/issues and we'll work with you to fix it. Check failed: status Status not OK: ObjectUnknownOwner: An application is trying to access a Ray object whose owner is unknown(3dd79ffe8938fd5bffffffffffffffffffffffff3000000001000000). Please make sure that all Ray objects you are trying to access are part of the current Ray session. Note that object IDs generated randomly (ObjectID.from_random()) or out-of-band (ObjectID.from_binary(...)) cannot be passed as a task argument because Ray does not know which task created them. If this was not how your object ID was generated, please file an issue at https://github.com/ray-project/ray/issues/ 
*** StackTrace Information ***
.../.venv/lib/python3.12/site-packages/ray/_raylet.so(+0x15fd74a) [0x7f1320c4f74a] ray::operator<<()
.../.venv/lib/python3.12/site-packages/ray/_raylet.so(_ZN3ray6RayLogD1Ev+0x488) [0x7f1320c51d98] ray::RayLog::~RayLog()
.../.venv/lib/python3.12/site-packages/ray/_raylet.so(_ZNK3ray4core10CoreWorker20GetOwnerAddressOrDieERKNS_8ObjectIDE+0xf4) [0x7f1320027a64] ray::core::CoreWorker::GetOwnerAddressOrDie()
.../.venv/lib/python3.12/site-packages/ray/_raylet.so(_ZN3ray4core10CoreWorker14PlasmaCallbackERKSt8functionIFvSt10shared_ptrINS_9RayObjectEENS_8ObjectIDEPvEERKS5_S6_S7_+0x2d1) [0x7f1320028db1] ray::core::CoreWorker::PlasmaCallback()
.../.venv/lib/python3.12/site-packages/ray/_raylet.so(+0x9adc1c) [0x7f131ffffc1c] std::_Function_handler<>::_M_invoke()
.../.venv/lib/python3.12/site-packages/ray/_raylet.so(+0xb3c30a) [0x7f132018e30a] std::_Function_handler<>::_M_invoke()
.../.venv/lib/python3.12/site-packages/ray/_raylet.so(+0xf59858) [0x7f13205ab858] EventTracker::RecordExecution()
.../.venv/lib/python3.12/site-packages/ray/_raylet.so(+0xf517c7) [0x7f13205a37c7] std::_Function_handler<>::_M_invoke()
.../.venv/lib/python3.12/site-packages/ray/_raylet.so(+0xb2554b) [0x7f132017754b] boost::asio::detail::executor_op<>::do_complete()
.../.venv/lib/python3.12/site-packages/ray/_raylet.so(+0x15c8beb) [0x7f1320c1abeb] boost::asio::detail::scheduler::do_run_one()
.../.venv/lib/python3.12/site-packages/ray/_raylet.so(+0x15ca569) [0x7f1320c1c569] boost::asio::detail::scheduler::run()
.../.venv/lib/python3.12/site-packages/ray/_raylet.so(+0x15cac72) [0x7f1320c1cc72] boost::asio::io_context::run()
.../.venv/lib/python3.12/site-packages/ray/_raylet.so(+0x9fe544) [0x7f1320050544] ray::core::CoreWorkerProcessImpl::CreateCoreWorker()::{lambda()#1}::operator()()
.../.venv/lib/python3.12/site-packages/ray/_raylet.so(+0xb46770) [0x7f1320198770] thread_proxy
/lib64/libpthread.so.0(+0x81ca) [0x7f132331e1ca] start_thread
/lib64/libc.so.6(clone+0x43) [0x7f1322800e73] clone

Versions / Dependencies

2.51.1

Reproduction script

N/A

Issue Severity

None

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething that is supposed to be working; but isn'tcommunity-backlogcoreIssues that should be addressed in Ray CorestabilitytriageNeeds triage (eg: priority, bug/not-bug, and owning component)

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions