33813 - [BC - Insight] Double slashing of validators

Submitted on Jul 30th 2024 at 03:08:19 UTC by @Lastc0de for Boost | Shardeum: Core

Report ID: #33813

Report type: Blockchain/DLT

Report severity: Insight

Target: https://github.com/shardeum/shardus-core/tree/dev

Impacts:

• Direct loss of funds

Description

Brief/Intro

Slashing mechanism for POS blockchains is a critical feature. Its implementation must have a strong test suite.

Shardeum has the concept of a penalty, to slash nodes if they have malicious behaviour. For example if a node goes offline while it is in active mode, and stops processing transactions, then other nodes confirm it is a lost node and apply a configurable penalty to that nodes staked tokens. But there is a bug in the shardeum repository which causes the network to apply a penalty 2x of configured value. So when penalty of leaving network early is configured to be 20% of staked value, if a validator leaves it loses 40% of its stake.

I will explain the cause of the bug here and provide a POC after.

Vulnerability Details

One way of getting a penalty is to leave network early, in the shardeum/shardus-core repository if a node removed because it lefts network early, a node-left-early event would be triggered

src/p2p/NodeList.ts

if (isNodeLeftNetworkEarly(node)) {
    const emitParams: Omit<ShardusEvent, 'type'> = {
    nodeId: node.id,
    reason: 'Node left early',
    time: cycle.start,
    publicKey: node.publicKey,
    cycleNumber: cycle.counter,
    }
    emitter.emit('node-left-early', emitParams)
}

this event is handled in src/shardus/index.ts file of same repository

src/shardus/index.ts

Self.emitter.on('node-left-early', ({ ...params }) => {
    try {
    if (!this.stateManager.currentCycleShardData) throw new Error('No current cycle data')
    if (params.publicKey == null) throw new Error('No node publicKey provided for node-left-early event')
    const consensusNodes = this.getConsenusGroupForAccount(params.publicKey)
    for (let node of consensusNodes) {
        if (node.id === Self.id) {
        this.app.eventNotify?.({ type: 'node-left-early', ...params })
        }
    }
    } catch (e) {
    this.mainLogger.error(`Error: while processing node-left-early event stack: ${e.stack}`)
    }
})

and it calls eventNotify function of app which is a method that defined in another repository shardeum/shardeum

src/index.ts

else if (
eventType === 'node-left-early' &&
ShardeumFlags.enableNodeSlashing === true &&
ShardeumFlags.enableLeftNetworkEarlySlashing
) {
    let nodeLostCycle
    let nodeDroppedCycle
    for (let i = 0; i < latestCycles.length; i++) {
        const cycle = latestCycles[i]
        if (cycle == null) continue
        if (cycle.apoptosized.includes(data.nodeId)) {
        nodeDroppedCycle = cycle.counter
        } else if (cycle.lost.includes(data.nodeId)) {
        nodeLostCycle = cycle.counter
        }
    }
    if (nodeLostCycle && nodeDroppedCycle && nodeLostCycle < nodeDroppedCycle) {
        const violationData: LeftNetworkEarlyViolationData = {
        nodeLostCycle,
        nodeDroppedCycle,
        nodeDroppedTime: data.time,
        }
        nestedCountersInstance.countEvent('shardeum-staking', `node-left-early: injectPenaltyTx`)

        await PenaltyTx.injectPenaltyTX(shardus, data, violationData)
    } else {
        nestedCountersInstance.countEvent('shardeum-staking', `node-left-early: event skipped`)
        /* prettier-ignore */ if (logFlags.dapp_verbose) console.log(`Shardeum node-left-early event skipped`, data, nodeLostCycle, nodeDroppedCycle)
    }
}

which calls the injectPenaltyTX function. injectPenaltyTX function creates an internal penalty transactions and puts into shardus by calling shardus.put function

src/tx/penalty/transaction.ts

export async function injectPenaltyTX(
  shardus: Shardus,
  eventData: ShardusTypes.ShardusEvent,
  violationData: LeftNetworkEarlyViolationData | NodeRefutedViolationData | SyncingTimeoutViolationData
): Promise<{
  success: boolean
  reason: string
  status: number
}> {
  let violationType: ViolationType
  if (eventData.type === 'node-left-early') violationType = ViolationType.LeftNetworkEarly
  else if (eventData.type === 'node-refuted') violationType = ViolationType.NodeRefuted
  else if (eventData.type === 'node-sync-timeout') violationType = ViolationType.SyncingTooLong
  const unsignedTx = {
    reportedNodeId: eventData.nodeId,
    reportedNodePublickKey: eventData.publicKey,
    operatorEVMAddress: '',
    timestamp: shardeumGetTime(),
    violationType,
    violationData,
    isInternalTx: true,
    internalTXType: InternalTXType.Penalty,
  }

  const wrapeedNodeAccount: ShardusTypes.WrappedDataFromQueue = await shardus.getLocalOrRemoteAccount(
    unsignedTx.reportedNodePublickKey
  )

  if (!wrapeedNodeAccount) {
    return {
      success: false,
      reason: 'Penalty Node Account not found',
      status: 404,
    }
  }

  if (wrapeedNodeAccount && isNodeAccount2(wrapeedNodeAccount.data)) {
    unsignedTx.operatorEVMAddress = wrapeedNodeAccount.data.nominator
  } else {
    return {
      success: false,
      reason: 'Operator address could not be found for penalty node',
      status: 404,
    }
  }

  // to make sure that differnt nodes all submit an equivalent unsignedTx that is counted as the same unsignedTx,
  // we need to make sure that we have a determinstic timestamp
  const cycleEndTime = eventData.time
  let futureTimestamp = cycleEndTime * 1000
  while (futureTimestamp < shardeumGetTime()) {
    futureTimestamp += 30 * 1000
  }
  unsignedTx.timestamp = futureTimestamp

  const signedTx = shardus.signAsNode(unsignedTx) as PenaltyTX
  const txId = generateTxId(unsignedTx)
  // store the unsignedTx to local map for later use
  recordPenaltyTX(txId, signedTx)

  // Limit the nodes that send this to the <ShardeumFlags.numberOfNodesToInjectPenaltyTx> closest to the node address ( publicKey )
  const closestNodes = shardus.getClosestNodes(
    eventData.publicKey,
    ShardeumFlags.numberOfNodesToInjectPenaltyTx
  )
  const ourId = shardus.getNodeId()
  const isLuckyNode = closestNodes.some((nodeId) => nodeId === ourId)
  if (!isLuckyNode) {
    if (ShardeumFlags.VerboseLogs)
      console.log(`injectPenaltyTX: not lucky node, skipping injection`, signedTx)
    return
  }
  const waitTime = futureTimestamp - shardeumGetTime()
  // since we have to pick a future timestamp, we need to wait until it is time to submit the signedTx
  await sleep(waitTime)

  if (ShardeumFlags.VerboseLogs) {
    console.log(`injectPenaltyTX: tx.timestamp: ${signedTx.timestamp} txid: ${txId}`, signedTx)
  }

  const result = await shardus.put(signedTx)
  /* prettier-ignore */ if (logFlags.dapp_verbose) console.log('INJECTED_PENALTY_TX', result)
  return result
}

shardus.put is defined in the shardeum/shardus-core repository and it validates and sends transactions to queue for execution. After the queue, transactions are passed to the apply function in the shardeum/shardeum repository for execution. apply function would pass the created internal transaction to the applyInternalTx function.

src/index.ts

async apply(timestampedTx: ShardusTypes.OpaqueTransaction, wrappedStates, originalAppData) {
      //@ts-ignore
      const { tx } = timestampedTx
      const txTimestamp = getInjectedOrGeneratedTimestamp(timestampedTx)
      const appData = fixBigIntLiteralsToBigInt(originalAppData)
      // Validate the tx
      const { result, reason } = this.validateTransaction(tx)
      if (result !== 'pass') {
        throw new Error(`invalid transaction, reason: ${reason}. tx: ${Utils.safeStringify(tx)}`)
      }

      if (isInternalTx(tx)) {
        return applyInternalTx(tx, wrappedStates, txTimestamp)
      }

and applyInternalTx function would pass it to applyPenaltyTx function

src/index.ts

async function applyInternalTx(
  tx: InternalTx,
  wrappedStates: WrappedStates,
  txTimestamp: number
): Promise<ShardusTypes.ApplyResponse> {
...
  if (internalTx.internalTXType === InternalTXType.Penalty) {
    const penaltyTx = internalTx as PenaltyTX
    applyPenaltyTX(shardus, penaltyTx, wrappedStates, txId, txTimestamp, applyResponse)
  }
  return applyResponse
}

this applyPenaltyTx function calculated amount of penalty, and calls applyPenalty function to apply penaltyAmount to validator account.

src/tx/penalty/transaction.ts

export async function applyPenaltyTX(
  shardus,
  tx: PenaltyTX,
  wrappedStates: WrappedStates,
  txId: string,
  txTimestamp: number,
  applyResponse: ShardusTypes.ApplyResponse
): Promise<void> {
...
  //TODO should we check if it was already penalized?
  const penaltyAmount = getPenaltyForViolation(tx, nodeAccount.stakeLock)
  applyPenalty(nodeAccount, operatorAccount, penaltyAmount)
  nodeAccount.nodeAccountStats.penaltyHistory.push({
    type: tx.violationType,
    amount: penaltyAmount,
    timestamp: eventTime,
  })
...

applyPenalty function does some calculations and some modifications

src/tx/penalty/penaltyFunctions.ts

export function applyPenalty(
  nodeAccount: NodeAccount2,
  operatorEOA: WrappedEVMAccount,
  penalty: bigint
): boolean {
  /* prettier-ignore */ if (logFlags.dapp_verbose) console.log(`\nApplying Penalty on Node: ${nodeAccount.id} of ${penalty.toString()} SHM`)

  // convert hex value to BN
  operatorEOA.operatorAccountInfo.stake = _base16BNParser(operatorEOA.operatorAccountInfo.stake)
  operatorEOA.operatorAccountInfo.operatorStats.totalNodePenalty = _base16BNParser(
    operatorEOA.operatorAccountInfo.operatorStats.totalNodePenalty
  )
  nodeAccount.stakeLock = _base16BNParser(nodeAccount.stakeLock)
  nodeAccount.penalty = _base16BNParser(nodeAccount.penalty)
  nodeAccount.nodeAccountStats.totalPenalty = _base16BNParser(nodeAccount.nodeAccountStats.totalPenalty)

  if (penalty > nodeAccount.stakeLock) penalty = nodeAccount.stakeLock

  // update operator account
  operatorEOA.operatorAccountInfo.stake -= penalty
  operatorEOA.operatorAccountInfo.operatorStats.totalNodePenalty += penalty

  // update node account
  nodeAccount.stakeLock -= penalty
  nodeAccount.penalty += penalty
  nodeAccount.nodeAccountStats.totalPenalty += penalty
  return true
}

root cause of this bug is because penalty is reduced from stake of validator account here

operatorEOA.operatorAccountInfo.stake -= penalty
...
nodeAccount.stakeLock -= penalty

so if other parts of application wants to do some calculation based on original stake, it is so easy to mistake this stake field by original stake of validator, which is happened in unstake transaction.

Again in apply function of shardeum/shardeum repository, we have below implementation for unstake transaction

src/index.ts

async apply(timestampedTx: ShardusTypes.OpaqueTransaction, wrappedStates, originalAppData) {
...
    if (appData.internalTx && appData.internalTXType === InternalTXType.Unstake) {
        nestedCountersInstance.countEvent('shardeum-unstaking', 'applying unstake transaction')
        if (ShardeumFlags.VerboseLogs) console.log('applying unstake tx', wrappedStates, appData)

        // get unstake tx from appData.internalTx
        const unstakeCoinsTX: UnstakeCoinsTX = appData.internalTx

        // todo: validate tx timestamp, compare timestamp against account's timestamp

        // set stake value, nominee, cert in OperatorAcc (if not set yet)
        const operatorShardusAddress = toShardusAddress(unstakeCoinsTX.nominator, AccountType.Account)
        const nomineeNodeAccount2Address = unstakeCoinsTX.nominee
        // eslint-disable-next-line security/detect-object-injection
        const operatorEVMAccount: WrappedEVMAccount = wrappedStates[operatorShardusAddress]
            .data as WrappedEVMAccount
        operatorEVMAccount.timestamp = txTimestamp

        if (operatorEVMAccount.operatorAccountInfo == null) {
            nestedCountersInstance.countEvent(
            'shardeum-unstaking',
            'unable to apply unstake tx, operator account info does not exist'
            )
            throw new Error(
            `Unable to apply Unstake tx because operator account info does not exist for ${unstakeCoinsTX.nominator}`
            )
        } else {
            operatorEVMAccount.operatorAccountInfo = fixBigIntLiteralsToBigInt(
            operatorEVMAccount.operatorAccountInfo
            )
        }
        fixDeserializedWrappedEVMAccount(operatorEVMAccount)

        if (
            operatorEVMAccount.operatorAccountInfo.certExp > txTimestamp &&
            ShardeumFlags.unstakeCertCheckFix
        ) {
            throw new Error(
            `Unable to apply Unstake tx because stake cert has not yet expired. Expiry timestamp ${operatorEVMAccount.operatorAccountInfo.certExp}`
            )
        }

        // eslint-disable-next-line security/detect-object-injection
        const nodeAccount2: NodeAccount2 = wrappedStates[nomineeNodeAccount2Address].data as NodeAccount2

        const currentBalance = operatorEVMAccount.account.balance
        const stake = BigInt(operatorEVMAccount.operatorAccountInfo.stake)
        let reward = BigInt(nodeAccount2.reward)
        const penalty = BigInt(nodeAccount2.penalty)
        const txFeeUsd = BigInt(ShardeumFlags.constantTxFeeUsd)
        const txFee = scaleByStabilityFactor(txFeeUsd, AccountsStorage.cachedNetworkAccount)
        /* prettier-ignore */ if (logFlags.dapp_verbose) console.log('calculating new balance after unstake', currentBalance, stake, reward, penalty, txFee)
        if (nodeAccount2.rewardEndTime === 0 && nodeAccount2.rewardStartTime > 0) {
            // This block will only be reached if the node is inactive and the force unstake flag has been set
            reward = BigInt(0)

            /* prettier-ignore */ if (logFlags.dapp_verbose) console.log('discarding staking rewards due to zero rewardEndTime')
        }
        const newBalance = currentBalance + stake + reward - penalty - txFee
        operatorEVMAccount.account.balance = newBalance
        operatorEVMAccount.account.nonce = operatorEVMAccount.account.nonce + BigInt(1)
...
    }
...
}

It sets stake variable to current stake value, penalty variable to current penalty value, and when it calculates newBalance it reduces penalty from stake, AGAIN.

Impact Details

This bug could affect direct loss for any validator

References

Add any relevant links to documentation or code

Proof of Concept

To see how the validator penalty applies twice we have to run a network. Almost all of what we do below is from README of different repositories of shardeum. If you have a running network just make sure step 3 and 4 is configured and skip to step 16.

1. clone repositories

git clone --depth=1 [e-mail]:shardeum/shardeum.git
git clone --depth=1 [e-mail]:shardeum/json-rpc-server.git rpc
git clone --depth=1 https://github.com/shardeum/validator-dashboard.git validator01
git clone --depth=1 https://github.com/shardeum/validator-dashboard.git validator02

1. install metamask on your browser and create two accounts validator01, validator02. Copy their address and open shardeum/src/config/genesis.json and append them to the list of genesis accounts.

2. open shardeum/src/config/index.ts and change below values

    cycleDuration: 30,
    baselineNodes: 10, // this is to allow network to process transactions with 10 not 300 node, thanks to Schnilch at discord
    minNodes: 10, // this is to allow network to process transactions with 10 not 300 node, thanks to Schnilch at discord
    forceBogonFilteringOn: false,
    mode: "debug",

1. open shardeum/src/shardeum/shardeumFlags.ts and change below values

    penaltyPercent: 0.2, // it means 20% penalty to stake
    blockProductionRate: 3,
    enableLeftNetworkEarlySlashing: true,
    enableNodeRefutedSlashing: true,

1. go to the shardeum folder and execute npm install && npm prepare

2. install shardus command line tool with npm install -g shardus && npm update @shardus/archiver

3. start a local network by shardus start 10. it spins up a network with 10 nodes and an archiver. run shardus pm2 logs and wait until all 10 nodes are in active mode.

4. Now we have to run our validator and stake some SHM. go to validator01 folder and change docker-compose.yaml to

version: '3.4'

services:
  shardeum-dashboard01:
    container_name: shardeum-dashboard01
    # network_mode: 'host'
    image: ghcr.io/shardeum/server:latest
    working_dir: /home/node/app
    entrypoint: /home/node/app/entrypoint.sh
    volumes:
      - './:/home/node/app:Z'
      - '../validator:/usr/src/app:Z'
    network_mode: host
    env_file: .env

1. create a hashed password with echo password | openssl dgst -sha256 -r (which is what installer.sh does)

2. add a .env file with below content (replace 192.168.1.100 with your computer ip, and replace DASHPASS value with previous step output)

EXT_IP=auto
INT_IP=auto
EXISTING_ARCHIVERS=[{"ip":"192.168.1.100","port":4000,"publicKey":"758b1c119412298802cd28dbfa394cdfeecc4074492d60844cc192d632d84de3"}]
APP_MONITOR=192.168.1.100
DASHPASS=a9d2d1355c90e24d1676e27bfb4b7a37ab8a4041c44f26bfac538a453c46355c
DASHPORT=9030
SERVERIP=192.168.1.100
LOCALLANIP=192.168.1.100
SHMEXT=19001
SHMINT=11001
RPC_SERVER_URL=http://json-rpc-server:18080
RUNDASHBOARD=y

1. replace entrypoint.sh with below text

#!/usr/bin/env bash
ln -s /usr/src/app /home/node/app/validator

echo "Install PM2"

npm i -g pm2

# Pull latest versions of the CLI and GUI

git clone https://github.com/shardeum/validator-cli.git cli

echo "Install the CLI"
cd cli
npm i --silent
npm link
cd ..

git clone https://github.com/shardeum/validator-gui.git gui

echo "Install the GUI"
cd gui
npm i --silent
npm run build
#openssl req -x509 -nodes -days 99999 -newkey rsa:2048 -keyout ./selfsigned.key -out selfsigned.crt -subj "/C=US/ST=Texas/L=Dallas/O=Shardeum/OU=Shardeum/CN=shardeum.org"

# if CA.cnf does not exist, create it
if [ ! -f "CA.cnf" ]; then
    echo "[ req ]
prompt = no
distinguished_name = req_distinguished_name

[ req_distinguished_name ]
C = XX
ST = Localzone
L = localhost
O = Certificate Authority Local Validator Node
OU = Develop
CN = mynode-sphinx.sharedum.local
emailAddress = community@.sharedum.local" > CA.cnf
fi

# if CA.key does not exist, create it
if [ ! -f "CA_key.pem" ]; then
    openssl req -nodes -new -x509 -keyout CA_key.pem -out CA_cert.pem -days 1825 -config CA.cnf
fi

# if selfsigned.cnf does not exist, create it
if [ ! -f "selfsigned.cnf" ]; then
    echo "[ req ]
default_bits  = 4096
distinguished_name = req_distinguished_name
req_extensions = req_ext
x509_extensions = v3_req
prompt = no

[req_distinguished_name]
countryName = XX
stateOrProvinceName = Localzone
localityName = Localhost
organizationName = Shardeum Sphinx 1.x Validator Cert.
commonName = localhost

[req_ext]
subjectAltName = @alt_names

[v3_req]
subjectAltName = @alt_names

[alt_names]
IP.1 = $SERVERIP
IP.2 = $LOCALLANIP
DNS.1 = localhost" > selfsigned.cnf
fi

# if csr file does not exist, create it
if [ ! -f "selfsigned.csr" ]; then
    openssl req -sha256 -nodes -newkey rsa:4096 -keyout selfsigned.key -out selfsigned.csr -config selfsigned.cnf
fi

# if selfsigned.crt does not exist, create it
if [ ! -f "selfsigned_node.crt" ]; then
    openssl x509 -req -days 398 -in selfsigned.csr -CA CA_cert.pem -CAkey CA_key.pem -CAcreateserial -out selfsigned_node.crt -extensions req_ext -extfile selfsigned.cnf
fi
# if selfsigned.crt does not exist, create it
if [ ! -f "selfsigned.crt" ]; then
  cat selfsigned_node.crt CA_cert.pem > selfsigned.crt
fi
cd ../..

# Start GUI if configured to in env file
echo $RUNDASHBOARD
if [ "$RUNDASHBOARD" == "y" ]
then
echo "Starting operator gui"
# Call the CLI command to set the GUI password
operator-cli gui set password -h $DASHPASS
# Call the CLI command to set the GUI port
operator-cli gui set port $DASHPORT
# Call the CLI command to start the GUI
operator-cli gui start
fi

# Deprecated
# operator-cli set external_port $SHMEXT
# operator-cli set internal_port $SHMINT

echo "done";

# Keep container running
tail -f /dev/null

1. replace Dockerfile content with below text

FROM ghcr.io/shardeum/server:latest

ARG RUNDASHBOARD=y
ENV RUNDASHBOARD=${RUNDASHBOARD}

# Copy cli src files as regular user
WORKDIR /home/node/app
COPY --chown=node:node . .

# RUN ln -s /usr/src/app /home/node/app/validator

# Start entrypoint script as regular user
CMD ["./entrypoint.sh"]

1. apply step 8 to 12 in validator02 folder, only in docker-compose.yml file instead of validator01 use validator02

2. Now we have to run a json-rpc-server. To do so go to the rpc folder and open archiverConfig.json and change ip to 192.168.1.100 (or your computer ip address) and run docker compose up. It should run and connect to the archive node without problem.

3. Now go to the validator01 folder and start validator by docker compose up. After a while you would see done output and can open the dashboard in your browser at https://localhost:9030. Then login with your password and go to the Maintenance tab.

4. In the validator01 dashboard, click on Start button to start validator node. After a few seconds in the terminal that you executed shardus pm2 logs there must be a standby node which is this validator01 that we started.

5. Open Metamask and add a new network

Field	Details
Network Name	Shardeum Atomium
New RPC URL	http://localhost:8080
Chain ID	8082
Currency Symbol	SHM
Block Explorer URL (optional)	none or http://localhost:6001/

1. Now change your metamask network to Shardeum Atomium and you must be able to see values that we set for validator01 and validator02 accounts in the genesis.json as balances of these accounts.

2. In the Maintenance tab of validator01 dashboard, click add stake. It opens Metamask, connects validator01 account then set for example 100 as stake value, click on add stake again and confirm transaction in Metamask, when the transaction confirmed node goes to sync mode and after a while, it goes to active!. You can see in Metamask that the account now has a send transaction of 100 and its balance is reduced by ~101 tokens, that 1 is because of the fee. In shardus pm2 logs we can see after a few seconds, the network kicks an old node and resizes its active size to be 10 which is cool. And the removed node would go to standby mode.

3. stop validator01 docker container to simulate an early left node. After a few seconds in shardus pm2 logs you should see active nodes become 9 again.

4. Now go to the validator02 folder and start it like validator01, go to its dashboard, stake some token and wait for the node to go to active mode. We add this seconds validator so network can process our unstake transaction. network size must be at least 10 to process application transactions. Although there is a standby node which started by shardus start 10 but it does not go to active mode again, i think because it does not stake anything. Anyway, now that we have 10 active node again, we can send an unstake transaction to the network.

5. go to validator01 folder and execute this command

RPC_SERVER_URL=http://localhost:8080 node ./cli/build/src/index.js unstake

it asks for a private key, go to Metamask and open validator01 account and copy it's private key, and paste it here.

1. After transaction confirmed you can see in Metamask that instead of 80, 60 tokens are returned to your account. Which is 40% penalty.