# #43187 \[BC-Insight] Movement Full Node Panics and Crashes Uncleanly on Connection failure with DA Light Node **Submitted on Apr 3rd 2025 at 13:27:40 UTC by @Nirix0x for** [**Attackathon | Movement Labs**](https://immunefi.com/audit-competition/movement-labs-attackathon) * **Report ID:** #43187 * **Report Type:** Blockchain/DLT * **Report severity:** Insight * **Target:** * **Impacts:** * Network not being able to confirm new transactions (total network shutdown) ## Description ## Brief/Intro Unhandled error when DA light node fails or the network connection with DA light node fails, causes fatal panic in Movement full node resulting in unclean crash. ## Vulnerability Details When a network connection between the `movement-full-node` and the `movement-celestia-da-light-node` fails (e.g., TCP Connection Reset, Connection Refused), the `movement-full-node` panics and crashes: 1. An awaited network operation within a core task (e.g. `blocks_from_da.next().await` inside `node/tasks/execute_settle.rs`) returns an `Err(...)`. 2. This `Err` propagates up the call stack via standard Rust error handling (`?` operator) and task management constructs (`try_join!` in `node/partial.rs`). 3. The error eventually reaches the top-level `async fn main` in `src/main.rs`, causing it to terminate and return the `Err`. 4. During shutdown, the blocking thread pool (e.g. used by`DaDB` via `tokio::task::spawn_blocking`), are dropped from within the asynchronous context of a Tokio worker thread. 5. This violates a fundamental safety rule of the Tokio runtime, leading to a fatal, unrecoverable panic with the message: `Cannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context.` Although the overall setup may include a restart for the `movement-full-node` (like the one present in docker config), this only attempts to mitigate the complete outage. The real issue is the abrupt, unclean termination caused by the panic, which should not be the case on another node's failure in a distributed network. ## Impact Details Any transient fault in the DA light node or its network path that results in an abrupt connection error can take down the full node. Even if full node restarts, panic and unclean exit of full node has severe impact - the blocking processes are immediately terminated, e.g. DB write causing potential integrity issues. Even in async tasks, abrupt termination can cause critical issues as it losses all in-memory state (e.g. transactions not yet submitted to DA but marked as committed in mempool).\ In addition, such unclean exit leads to a variety of other issue e.g. incomplete flushing of logs and other diagnostic info. It also significantly impacts the network recovery time by forcing full restart of the node, where a localized retry would have resulted in faster recovery. ## References Included in vulnerability details. ## Proof of Concept ## Proof of Concept 1. Start the entire network using`just movement-full-node docker-compose local` 2. Shutdown/restart light node to simulate failures with light node`docker restart movement-celestia-da-light-node` 3. The full node crashes with following logs`docker logs movement-full-node -f` ``` thread 'tokio-runtime-worker' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.41.1/src/runtime/blocking/shutdown.rs:51:21: Cannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context. stack backtrace: 0: rust_begin_unwind 1: core::panicking::panic_fmt 2: tokio::runtime::blocking::shutdown::Receiver::wait 3: tokio::runtime::blocking::pool::BlockingPool::shutdown 4: core::ptr::drop_in_place 5: as core::future::future::Future>::poll 6: std::panicking::try 7: tokio::runtime::task::harness::Harness::poll 8: tokio::runtime::scheduler::multi_thread::worker::Context::run_task 9: tokio::runtime::context::scoped::Scoped::set 10: tokio::runtime::context::runtime::enter_runtime 11: tokio::runtime::scheduler::multi_thread::worker::run 12: as core::future::future::Future>::poll 13: tokio::runtime::task::core::Core::poll 14: tokio::runtime::task::harness::Harness::poll 15: tokio::runtime::blocking::pool::Inner::run note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. Error: task 62 panicked with message "Cannot drop a runtime in a context where blocking is not allowed. This happens when a runtime is dropped from within an asynchronous context." Stack backtrace: 0: std::backtrace::Backtrace::create 1: anyhow::error:: for anyhow::Error>::from 2: as core::future::future::Future>::poll 3: as core::future::future::Future>::poll 4: std::panicking::try 5: tokio::runtime::task::harness::Harness::poll 6: tokio::runtime::scheduler::multi_thread::worker::Context::run_task 7: tokio::runtime::context::scoped::Scoped::set 8: tokio::runtime::context::runtime::enter_runtime 9: tokio::runtime::scheduler::multi_thread::worker::run 10: as core::future::future::Future>::poll 11: tokio::runtime::task::core::Core::poll 12: tokio::runtime::task::harness::Harness::poll 13: tokio::runtime::blocking::pool::Inner::run 14: std::sys_common::backtrace::__rust_begin_short_backtrace 15: core::ops::function::FnOnce::call_once{{vtable.shim}} 16: std::sys::pal::unix::thread::Thread::new::thread_start 17: start_thread 18: thread_start ``` --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://reports.immunefi.com/movement-labs-attackathon/43187-bc-insight-movement-full-node-panics-and-crashes-uncleanly-on-connection-failure-with-da-light.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.