Use metrics to monitor node performance
To enable the Prometheus monitoring and alerting service to access Hyperledger Besu metrics, use the --metrics-enabled
option. Use Grafana to visualize the collected data. See the sample Besu Full Grafana dashboard.
The Besu example networks have monitoring with Prometheus and Grafana configured.
Use Prometheus to monitor the number of blocks your Besu node is behind the chain head, and to alert you that your node is not keeping up with the chain head.
This recording shows examples of monitoring Hyperledger Besu.
Install Prometheus
To use Prometheus with Besu, install the Prometheus main component. On MacOS, install with Homebrew:
brew install prometheus
You can also install:
- Exporters that send system metrics to Prometheus to monitor non-Besu-specific items such as disk and CPU usage.
- Other Prometheus components, such as the Alert Manager. Additional configuration is not required for these components because Prometheus handles and analyzes data directly from the feed.
Set up and run Prometheus with Besu
To configure Prometheus and run with Besu:
Configure Prometheus to poll Besu. For example, add the following YAML fragment to the
scrape_configs
block of theprometheus.yml
file:- Fragment to insert in prometheus.yml
- Full prometheus.yml example
- job_name: besu
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
static_configs:
- targets:
- localhost:9545global:
scrape_interval: 15s
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
- job_name: besu
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
static_configs:
- targets:
- localhost:9545Prometheus requires 3 MB of space per node per hour for metrics, with a
scrape_interval
of 15 seconds.Start Besu with the
--metrics-enabled
option. To start a single node for testing with metrics enabled, run the following command:- Syntax
- Example
besu --network=dev --miner-enabled --miner-coinbase <COINBASE ADDRESS> --rpc-http-cors-origins="all" --rpc-http-enabled --metrics-enabled
besu --network=dev --miner-enabled --miner-coinbase fe3b557e8fb62b89f4916b721be55ceb828dbd73 --rpc-http-cors-origins="all" --rpc-http-enabled --metrics-enabled
To specify the host and port on which Prometheus accesses Besu, use the
--metrics-host
and--metrics-port
options. The default host and port are 127.0.0.1 (localhost
) and 9545.dangerTo avoid DNS rebinding attacks, if running Prometheus on a different host than your Besu node (any host other than
localhost
), add the hostname that Prometheus uses to--host-allowlist
.For example, if Prometheus is configured to get metrics from
http://besu.local:8008/metrics
, thenbesu.local
has to be in--host-allowlist
.In another terminal, run Prometheus specifying the
prometheus.yml
file:prometheus --config.file=prometheus.yml
View the Prometheus graphical interface.
tipUse a log ingestion tool, such as Logstash, to parse the logs and alert you to configured anomalies.
Run Prometheus with Besu in push mode
The --metrics-enabled
option enables Prometheus polling of Besu, but sometimes metrics are hard to poll (for example, when running inside Docker containers with varying IP addresses). To enable Besu to push metrics to a Prometheus push gateway, use the --metrics-push-enabled
option.
To configure Prometheus and run with Besu pushing to a push gateway:
Configure Prometheus to read from a push gateway. For example, add the following YAML fragment to the
scrape_configs
block of theprometheus.yml
file:- job_name: push-gateway
metrics_path: /metrics
scheme: http
static_configs:
- targets:
- localhost:9091Start the push gateway. You can deploy the push gateway using the Docker image:
docker pull prom/pushgateway
docker run -d -p 9091:9091 prom/pushgatewayStart Besu specifying the
--metrics-push-enabled
option and port of the push gateway:- Syntax
- Example
besu --network=dev --miner-enabled --miner-coinbase <COINBASE ADDRESS> --rpc-http-cors-origins="all" --rpc-http-enabled --metrics-push-enabled --metrics-push-port=9091 --metrics-push-host=127.0.0.1
besu --network=dev --miner-enabled --miner-coinbase fe3b557e8fb62b89f4916b721be55ceb828dbd73 --rpc-http-cors-origins="all" --rpc-http-enabled --metrics-push-enabled --metrics-push-port=9091 --metrics-push-host=127.0.0.1
In another terminal, run Prometheus specifying the
prometheus.yml
file:prometheus --config.file=prometheus.yml
View the Prometheus graphical interface.
View Prometheus graphical interface
Open a Web browser to
http://localhost:9090
to view the Prometheus graphical interface.Choose Graph from the menu bar and click the Console tab below.
From the Insert metric at cursor drop-down, select a metric such as
besu_blockchain_difficulty_total
orethereum_blockchain_height
and click Execute. The values display.Click the Graph tab to view the data as a time-based graph. The query string displays below the graph. For example,
{ethereum_blockchain_height{instance="localhost:9545",job="prometheus"}
.
Metrics list
The following table lists available metrics. Each metric starts with a metric category prefix. Metrics specific to Besu use the besu_
prefix, followed by another metric category. Metric categories can be enabled using the metrics-category
command line option. If a metric has a JSON-RPC equivalent, it is included in the definition column.
Name | Metric type | Definition |
---|---|---|
besu_blockchain_chain_head_gas_limit | Gauge | Block gas limit of the current chain head block |
besu_blockchain_chain_head_gas_used | Gauge | Gas used by the current chain head block |
besu_blockchain_chain_head_ommer_count | Gauge | Number of uncles in the current chain head block (JSON-RPC equivalent: eth_getUncleCountByBlockHash or eth_getUncleCountByBlockNumber ) |
besu_blockchain_chain_head_timestamp | Gauge | Timestamp from the current chain head |
besu_blockchain_chain_head_transaction_count | Gauge | Number of transactions in the current chain head block (JSON-RPC equivalent: eth_getBlockTransactionCountByHash or eth_getBlockTransactionCountByNumber ) |
besu_blockchain_difficulty_total | Gauge | Difficulty of the chain head (JSON-RPC equivalent: difficulty of admin_peers ) |
besu_executors_ethscheduler_computation_active_threads_current | Gauge | Current number of threads executing computation tasks |
besu_executors_ethscheduler_computation_completed_tasks_total | Gauge | Total number of computation tasks executed |
besu_executors_ethscheduler_computation_pool_size_current | Gauge | Current number of threads in the computation thread pool |
besu_executors_ethscheduler_computation_queue_length_current | Gauge | Current number of computation tasks awaiting execution |
besu_executors_ethscheduler_computation_rejected_tasks_total | Counter | Total number of tasks rejected by this computation executor |
besu_executors_ethscheduler_computation_submitted_tasks_total | Gauge | Total number of computation tasks submitted |
besu_executors_ethscheduler_timer_active_threads_current | Gauge | Current number of threads executing timer tasks |
besu_executors_ethscheduler_timer_completed_tasks_total | Gauge | Total number of timer tasks executed |
besu_executors_ethscheduler_timer_pool_size_current | Gauge | Current number of threads in the timer thread pool |
besu_executors_ethscheduler_timer_queue_length_current | Gauge | Current number of timer tasks awaiting execution |
besu_executors_ethscheduler_timer_rejected_tasks_total | Counter | Total number of tasks rejected by this timer executor |
besu_executors_ethscheduler_timer_submitted_tasks_total | Gauge | Total number of timer tasks submitted |
besu_executors_ethscheduler_workers_active_threads_current | Gauge | Current number of threads executing worker tasks |
besu_executors_ethscheduler_workers_completed_tasks_total | Gauge | Total number of worker tasks executed |
besu_executors_ethscheduler_workers_pool_size_current | Gauge | Current number of threads in the worker thread pool |
besu_executors_ethscheduler_workers_queue_length_current | Gauge | Current number of worker tasks awaiting execution |
besu_executors_ethscheduler_workers_rejected_tasks_total | Counter | Total number of tasks rejected by this worker executor |
besu_executors_ethscheduler_workers_submitted_tasks_total | Gauge | Total number of worker tasks submitted |
besu_network_discovery_inflight_interactions_current | Gauge | Current number of inflight discovery interactions |
besu_network_discovery_interaction_count | Counter | Total number of discovery interactions initiated |
besu_network_discovery_interaction_retry_count | Counter | Total number of interaction retries performed |
besu_network_discovery_messages_inbound | Counter | Total number of P2P discovery messages received |
besu_network_discovery_messages_outbound | Counter | Total number of P2P discovery messages sent |
besu_network_netty_boss_pending_tasks | Gauge | Number of pending tasks in Netty boss event loop |
besu_network_netty_workers_pending_tasks | Gauge | Number of pending tasks in Netty workers event loop |
besu_network_p2p_messages_inbound | Counter | Total number of P2P messages received |
besu_network_vertx_eventloop_pending_tasks | Gauge | Number of pending tasks in Vertx event loop |
besu_network_vertx_worker_pool_completed_total | Counter | Total number of tasks completed by Vertx worker pool |
besu_network_vertx_worker_pool_rejected_total | Counter | Total number of tasks rejected by Vertx worker pool |
besu_network_vertx_worker_pool_submitted_total | Counter | Total number of tasks submitted to Vertx worker pool |
besu_peers_connected_total | Counter | Total number of peers connected |
besu_peers_disconnected_total | Counter | Total number of peers disconnected |
besu_peers_pending_peer_requests_current | Gauge | Current number of peer requests pending because peers are busy |
besu_pruner_mark_time_duration | Gauge | Cumulative number of seconds spent marking the state trie across all pruning cycles |
besu_pruner_mark_operations_total | Counter | Total number of mark operations performed |
besu_pruner_marked_nodes_total | Counter | Total number of nodes marked as in use |
besu_pruner_sweep_operations_total | Counter | Total number of sweep operations performed |
besu_pruner_swept_nodes_total | Counter | Total number of unused nodes removed |
besu_stratum_connections | Counter | Number of connections over time |
besu_stratum_difficulty | Gauge | Current mining difficulty |
besu_stratum_disconnections | Counter | Number of disconnections over time |
besu_stratum_miners | Gauge | Number of connected miners |
besu_synchronizer_chain_download_pipeline_processed_total | Counter | Number of entries processed by each chain download pipeline stage |
besu_synchronizer_chain_download_pipeline_restarts | Counter | Number of times chain download pipeline has been restarted |
besu_synchronizer_fast_sync_pivot_block_current | Gauge | The current fast sync pivot block |
besu_synchronizer_fast_sync_pivot_block_selected_count | Counter | Number of times a fast sync pivot block has been selected |
besu_synchronizer_fast_sync_validation_mode | Counter | Number of blocks validated using light vs full validation during fast sync |
besu_synchronizer_in_sync | Gauge | Whether or not the local node has caught up to the best known peer (1 or 0) |
besu_synchronizer_task | Summary | Internal processing tasks |
besu_synchronizer_world_state_completed_requests_total | Counter | Total number of node data requests completed as part of fast sync world state download |
besu_synchronizer_world_state_existing_nodes_total | Counter | Total number of node data requests completed using existing data |
besu_synchronizer_world_state_inflight_requests_current | Gauge | Number of in progress requests for world state data |
besu_synchronizer_world_state_node_requests_since_last_progress_current | Gauge | Number of world state requests made since the last time new data was returned |
besu_synchronizer_world_state_pending_requests_cache_size | Gauge | Pending request cache size for fast sync world state download |
besu_synchronizer_world_state_pending_requests_current | Gauge | Number of pending requests for fast sync world state download |
besu_synchronizer_world_state_pipeline_processed_total | Counter | Number of entries processed by each world state download pipeline stage |
besu_synchronizer_world_state_retried_requests_total | Counter | Total number of node data requests repeated as part of fast sync world state download |
besu_transaction_pool_pending_transactions_messages_skipped_total | Counter | Total number of pending transactions messages skipped by the processor |
besu_transaction_pool_transactions | Gauge | Current size of the transaction pool (JSON-RPC equivalent: result number of txpool_besuTransactions ) |
besu_transaction_pool_transactions_added_total | Counter | Count of transactions added to the transaction pool |
besu_transaction_pool_transactions_messages_skipped_total | Counter | Total number of transactions messages skipped by the processor. |
ethereum_best_known_block_number | Gauge | Estimated highest block available (JSON-RPC equivalent: highestBlock of eth_syncing , or eth_blockNumber if not syncing) |
ethereum_blockchain_height | Gauge | Current height of the canonical chain (JSON-RPC equivalent: eth_blockNumber ) |
ethereum_peer_count | Gauge | Current number of peers connected (JSON-RPC equivalent: net_peerCount ) |
ethereum_peer_limit | Gauge | Maximum number of peers this node allows to connect |
jvm_buffer_pool_capacity_bytes | Gauge | Bytes capacity of a given JVM buffer pool |
jvm_buffer_pool_used_buffers | Gauge | Used buffers of a given JVM buffer pool |
jvm_buffer_pool_used_bytes | Gauge | Used bytes of a given JVM buffer pool |
jvm_classes_loaded | Gauge | Current number of classes loaded in the JVM |
jvm_classes_loaded_total | Counter | Total number of classes loaded since the JVM started execution |
jvm_classes_unloaded_total | Counter | Total number of classes unloaded since the JVM started execution |
jvm_gc_collection_seconds | Summary | Seconds spent in a given JVM garbage collector |
jvm_memory_bytes_committed | Gauge | Committed bytes of a given JVM memory area |
jvm_memory_bytes_init | Gauge | Initial bytes of a given JVM memory area |
jvm_memory_bytes_max | Gauge | Maximum bytes of a given JVM memory area |
jvm_memory_bytes_used | Gauge | Used bytes of a given JVM memory area |
jvm_memory_pool_bytes_committed | Gauge | Committed bytes of a given JVM memory pool |
jvm_memory_pool_bytes_init | Gauge | Initial bytes of a given JVM memory pool |
jvm_memory_pool_bytes_max | Gauge | Maximum bytes of a given JVM memory pool |
jvm_memory_pool_bytes_used | Gauge | Used bytes of a given JVM memory pool |
jvm_threads_current | Gauge | Current thread count of a JVM |
jvm_threads_daemon | Gauge | Daemon thread count of a JVM |
jvm_threads_deadlocked | Gauge | Cycles of JVM threads in deadlock waiting to acquire object monitors or ownable synchronizers |
jvm_threads_deadlocked_monitor | Gauge | Cycles of JVM threads in deadlock waiting to acquire object monitors |
jvm_threads_peak | Gauge | Peak thread count of a JVM |
jvm_threads_started_total | Counter | Started thread count of a JVM |
jvm_threads_state | Gauge | Current count of threads by state |
process_cpu_seconds_total | Counter | Total user and system CPU time spent in seconds |
process_max_fds | Gauge | Maximum number of open file descriptors |
process_open_fds | Gauge | Number of open file descriptors |
process_start_time_seconds | Gauge | Start time of the process since Unix epoch in seconds |
- The
ethereum_best_known_block_number
metric always has a value. When theeth_syncing
JSON-RPC method returns false, the current chain height displays. - Although the
ethereum_peer_limit
metric does not have a JSON-RPC equivalent, themax peers
command line option sets the maximum number of P2P connections that can be established.