Troubleshooting Agora-cl

Select a configuration

Operating system:

Linux, MacOS, Arm64

Windows

Network:

Mainnet

Testnet

Troubleshooting checklist

If you're running into unexpected outputs or errors, use the following checklist to check the status of your configuration. In many cases, this checklist will help you independently resolve your issue. If you still need help after completing this checklist, you can generate a troubleshooting report below and paste it into Telegram when asking for support.

Thank you!

Bosagora is a small but mighty team of engineers who are working hard to support the Agora blockchain. Using this checklist is very helpful because it can reduce the time that it takes to gather information, allowing us to focus on engineering and monitoring. Thank you for helping us!

1. Select a configuration above

If you end up generating a troubleshooting report, your report will include your selected configuration.

3. Execution node sync status

Execution client:

You can check your Nethermind execution node's sync status by navigating to http://localhost:8545/healthchecks-ui or by running curl localhost:8545/health from a separate terminal window. A sync status of false indicates that your node is fully synced.

Nethermind

Operating system:

Linux, MacOS, Arm64

You can check your Besu execution node's sync status by running curl -H 'Content-Type: application/json' -X POST http://localhost:8545 -d '{"jsonrpc":"2.0","method":"eth_syncing","params":[],"id":51}' from a separate terminal window. A sync status of false indicates that your node is fully synced.

Windows

You can check your Besu execution node's sync status by running curl -H "Content-Type: application/json" -X POST http://localhost:8545 -d "{""jsonrpc"":""2.0"",""method"":""eth_syncing"",""params"":[],""id"":51}" from a separate terminal window. A sync status of false indicates that your node is fully synced.

Besu

Operating system:

Linux, MacOS, Arm64

You can check your Geth execution node's sync status by running geth attach and then eth.syncing from a separate terminal window. A sync status of false indicates that your node is fully synced.

Windows

You can check your Geth execution node's sync status by running geth attach ipc:\\.\pipe\geth.ipc and then eth.syncing from a separate terminal window. A sync status of false indicates that your node is fully synced.

4. Execution node peer connectivity

Execution client:

You can check your Nethermind execution node's peer connectivity by navigating to http://localhost:8545/healthchecks-ui or by running curl localhost:8545/health a separate terminal window. A health status of Healthy indicates that your node is connected to peers.

Nethermind

You should periodically see more than a few peers reported through Besu's log output. Refer to Besu's Monitor peer connections documentation for more detailed peer health monitoring guidance.

Besu

You should periodically see more than a few peers reported through Geth's log output. Look for output in the format of peercount=12. Refer to Geth's Connecting To The Network documentation for more detailed peer health monitoring guidance.

5. Execution node version

Execution client:

Use geth version to check Geth's version. See Geth's releases page and join their Discord to stay up to date as we approach Mainnet Merge.

Geth

Review Nethermind's log output to see what version you're using. See Nethermind's releases page and join their Discord to stay up to date as we approach Mainnet Merge.

Nethermind

Review Besu's log output to see what version you're using. See Besu's releases page and join their Discord to stay up to date as we approach Mainnet Merge.

6. Beacon node sync status

You can check your Agora node's sync status by running curl http://localhost:3500/eth/v1/node/syncing | jq from a separate terminal window. When you see "is_syncing":false, your Agora node is fully synchronized with the Agora chain. When you see "is_optimistic":false, your Agora node sees that your execution node is either 1) not yet started, 2) hasn't synced past the merge block or 3) fully synchronized with the execution-layer blockchain.

7. Beacon node peer connectivity

You should periodically see more than a few peers reported through your Agora node's log output. Look for output in the format of peers=12. You can issue curl http://localhost:8080/healthz from a separate terminal window to check connectivity status. If you see currentConnectionError: no contract code at given address, your execution node may still be syncing. Otherwise, if you don't see any errors, your Agora node is connected to peers.

8. Beacon node version

Ensure that you're using the latest stable Agora-cl release. Check Agora-cl's version by issuing the following command: agora-cl.sh beacon-chain --version (Linux) agora-cl.bat beacon-chain --version (Windows).

9. Beacon node ↔ execution node connectivity

Issue curl http://localhost:3500/eth/v1alpha1/node/eth1/connections from a separate terminal window. If you see currentConnectionError: no contract code at given address, your execution node may still be syncing. Otherwise, if you don't see any errors, your Agora node is connected to your execution node. This output can be interpreted as "EN-BN connection is healthy": {"currentAddress":"http://localhost:8551","currentConnectionError":"","addresses":["http://localhost:8551"],"connectionErrors":[]}

10. Fee recipient configuration

Agora-cl will output an error if you attempt to provide an invalid Ethereum wallet address as a fee recipient address. You'll see warnings if a fee recipient address hasn't been provided. See Configure Fee Recipient for more information.

11. Validator status

Network:

Paste your validator's public key (available in your deposit_data-*.json file) into a blockchain explorer like beaconcha.in to check the status of your validator.

Mainnet

Paste your validator's public key (available in your deposit_data-*.json file) into a Testnet blockchain explorer like beaconcha.in to check the status of your validator.

12. Troubleshooting scenarios and solutions

Running into unexpected output, warnings, or errors? Although this is unintuitive, many errors and warnings are expected and have been identified in the below list of troubleshooting scenarios and solutions. We gratefully ask that you review this before asking for support.

13. Troubleshooting report

Issue still not resolved? Generate a troubleshooting report below. Head over to Discord and paste your report for additional troubleshooting assistance.

Troubleshooting scenarios and solutions

Common troubleshooting scenarios and solutions are detailed below.

Troubleshooting your validator node

Scenario	Solution
You see Waiting for keymanager to initialize validator client with web UI...	You'll usually see this message when your Agora node is trying to interact with a validator client instance before the Agora node is fully synced. This is a known limitation. When your Agora node is finished syncing, this message should go away. Visit Check Node and Validator Status to learn how to check the sync status of your Agora node.
Everything seems fine, but your validator balance is going down.	If your validator client is running fine without errors but you're seeing your validator balance decrease, your Agora node may be experiencing issues with connectivity, stability, or synchronization. Check your Agora node logs to see if there are any errors or crashes.
Can't import accounts, stuck in a loop. You see Could not import accounts: could not write accounts: file already exists without proper 0600 permissions	This usually happens when the account you're using doesn't have permission to read and write to the target directory. See this GitHub issue for a workaround.

Troubleshooting your Agora node

Scenario	Solution
Waiting for peers / peer disconnected / no active peers: Waiting for enough suitable peers before syncing... msg="Peer disconnected" active=0	Peers will continuously disconnect and reconnect, so don't worry about Peer disconnected messages. If your Agora node is struggling to find peers: Your Agora node might be suffering from connectivity problems. Visit Improve P2P connectivity for connectivity troubleshooting guidance. Ensure that your firewall isn't restricting any outbound ports for Agora-cl. You may be using an incorrect genesis state or network flag. Every test network requires its own genesis state and network flag. Visit our Quickstart for the latest test network parameters.
Beacon node is stuck during initial sync.	If your node seems stuck (either doing nothing, or stuck in a loop) while syncing, a restart will usually resolve the problem. If you're on Windows, try selecting your console output window and hitting ENTER - this can "unpause" a paused console output stream.
Node is currently optimistic and cannot serve validators: level=error msg="Could not request attestation to sign at slot" error="rpc error: code = Unavailable desc = the node is currently optimistic and cannot serve validators" prefix=validator pubKey=0x01234 slot=65740	This usually means that your execution client isn't yet synchronized. Visit Check Node and Validator Status to learn how to check the sync status of your execution client.
Could not read JWT secret, Could not access JWT secret	You, your terminal window, or the script you're using may not have the permissions required in order to read or write your JWT token. Try elevating privileges or running as Administrator (if you're on Windows).
could not get ancestor state: failed to unmarshal encoding: incorrect size	This usually indicates that your Agora node's data has become corrupt. Try restarting your Agora node with a new or cleared data directory.
could not check configuration values between execution and consensus client error: timeout from http.Client 403 signature invalid Could not connect to execution client endpoint" error="could not make initial request to verify execution chain ID: 401 Unauthorized level=error msg="Could not connect to execution client endpoint" error="could not make initial request to verify execution chain ID: Post "http://localhost:8551/": dial tcp 127.0.0.1:8551: connect: connection refused" prefix=powchain warning msg="Batch is not processed" error="could not process block in batch: timeout from http.Client: received an undefined ee error. warning msg="Batch is not processed" error="could not process block in batch: got an unexpected error in JSON-RPC response: 403 Forbidden: signature is invalid
Agora node doesn't have a parent in db with root...	If you see blocks advancing, then this can usually be ignored. If you don't see blocks advancing, there are likely other warnings and/or errors that will help you troubleshoot.

Troubleshooting your execution node

Scenario	Solution
chain not synced beyond EIP-155	This usually means that your execution client needs more time to "catch up", which you don't need to worry about. If you see that your node is connected to peers and is advancing, your node is healthy.
Fatal: Failed to register the Ethereum service: genesis not found in chain	This can happen when your Geth tries to use an old data directory. Try restarting Geth with a new data directory specified.
Can't find peers on mainnet	Some users have reported peer-to-peer connectivity issues that were caused by old binaries or old data directories. Try using a new data directory, make sure you're using the latest version of your execution client software, and review Configure ports and firewalls for improved network connectivity for port configuration guidance.
I'm running on mainnet, but I see a testnet (Goerli, Prater, etc) specified in my output logs	Your execution client may be using an old binary, or an old database. Try using a new data directory, and make sure you're using the latest version of your execution client software.
the method engine_exchangeTransitionConfigurationV1 does not exist/is not available	Users have resolved this Geth error by 1) updating to the latest version of Geth and 2) ensuring that both Agora-cl and Geth are configured to use JWT (if you're connecting your Agora node to Geth over HTTP). Configuring Geth to use a fresh data directory may also resolve this warning.
ERROR powchain: Unable to process past deposit contract logs, perhaps your execution client is not fully synced error=Receipt not available for 'To' block '14957457'.	This usually means that your execution client needs more time to "catch up", which you don't need to worry about. If you see that your node is connected to peers and is advancing, your node is healthy.
403 signature invalid	This is usually caused by invalid JWT configuration. If you're using HTTP-JWT to connect your EN-BN, ensure that both EN and BN are configured to use the same JWT secret. Different files are OK (eg when your EN and BN are on different machines), but the secret within each JWT file should be the same. See Configure JWT authentication for more information.

Generate troubleshooting report

1. Complete the above troubleshooting checklist
2. Fill in the below form
3. Click Generate troubleshooting report
4. Copy and paste the report into Telegram when asking for support

Execution node startup command

Tip: We recommend redacting wallet addresses, IP addresses, and other personal information as a general operational security best practice.

Beacon node startup command

Validator node startup command

Unexpected output

Tip: Paste the ~100 lines of output before and including the output you're asking about.Generate troubleshooting report

Complete the checklist above before generating...