Use OpenTelemetry
You can use the OpenTelemetry monitoring and tracing service to gather node metrics and traces. To enable OpenTelemetry to access Hyperledger Besu, use the --metrics-enabled
and --metrics-protocol=opentelemetry
options. Use Splunk to visualize the collected data. A Besu Sync example is available.
Use OpenTelemetry to monitor the sync time of your Besu node and show where time is spent internally and over the JSON-RPC interface.
This office hours recording shows examples of monitoring Hyperledger Besu.
Install OpenTelemetry Collector
Download and install the OpenTelemetry Collector.
You can also install exporters that send system metrics to OpenTelemetry to monitor non-Besu-specific items such as disk and CPU usage. The OpenTelemetry Collector can connect to additional applications, and may be deployed in Kubernetes environments as a daemonset.
Setting up and running OpenTelemetry with Besu
Configure OpenTelemetry to accept data from Besu. For example, use the following configuration for your
otel-collector-config.yml
file, and send data to Splunk and Splunk APM:otel-collector-config.ymlreceivers:
otlp:
protocols:
grpc:
http:
exporters:
splunk_hec/traces:
# Splunk HTTP Event Collector token.
token: "11111111-1111-1111-1111-1111111111113"
# URL to a Splunk instance to send data to.
endpoint: "https://<SPLUNK INSTANCE>:8088/services/collector"
# Optional Splunk source: https://docs.splunk.com/Splexicon:Source
source: "besu:traces"
# Optional Splunk source type: https://docs.splunk.com/Splexicon:Sourcetype
sourcetype: "otlp"
# Splunk index, optional name of the Splunk index targeted.
index: "traces"
# Maximum HTTP connections to use simultaneously when sending data. Defaults to 100.
max_connections: 20
# Whether to disable gzip compression over HTTP. Defaults to false.
disable_compression: false
# HTTP timeout when sending data. Defaults to 10s.
timeout: 10s
# Whether to skip checking the certificate of the HEC endpoint when sending data over HTTPS. Defaults to false.
# For this demo, we use a self-signed certificate on the Splunk docker instance, so this flag is set to true.
insecure_skip_verify: true
splunk_hec/metrics:
# Splunk HTTP Event Collector token.
token: "11111111-1111-1111-1111-1111111111113"
# URL to a Splunk instance to send data to.
endpoint: "https://<SPLUNK INSTANCE>:8088/services/collector"
# Optional Splunk source: https://docs.splunk.com/Splexicon:Source
source: "besu:metrics"
# Optional Splunk source type: https://docs.splunk.com/Splexicon:Sourcetype
sourcetype: "prometheus"
# Splunk index, optional name of the Splunk index targeted.
index: "metrics"
# Maximum HTTP connections to use simultaneously when sending data. Defaults to 100.
max_connections: 20
# Whether to disable gzip compression over HTTP. Defaults to false.
disable_compression: false
# HTTP timeout when sending data. Defaults to 10s.
timeout: 10s
# Whether to skip checking the certificate of the HEC endpoint when sending data over HTTPS. Defaults to false.
# For this demo, we use a self-signed certificate on the Splunk docker instance, so this flag is set to true.
insecure_skip_verify: true
# Traces
sapm:
access_token: "${SPLUNK_ACCESS_TOKEN}"
endpoint: "https://ingest.${SPLUNK_REALM}.signalfx.com/v2/trace"
# Metrics + Events
signalfx:
access_token: "${SPLUNK_ACCESS_TOKEN}"
realm: "${SPLUNK_REALM}"
processors:
batch:
extensions:
health_check:
pprof:
zpages:
service:
extensions: [pprof, zpages, health_check]
pipelines:
traces:
receivers: [otlp]
exporters: [splunk_hec/traces, sapm]
processors: [batch]
metrics:
receivers: [otlp]
exporters: [splunk_hec/metrics, signalfx]
processors: [batch]It is easiest to run the OpenTelemetry collector with Docker with the following command:
- Syntax
- Example
docker run -d \
-v ./otel-collector-config.yml:/etc/otel/config.yaml \
-e SPLUNK_ACCESS_TOKEN=<access token> \
-e SPLUNK_REALM=<realm> \
-p 4317:4317 \
otel/opentelemetry-collector-contrib:latestdocker run -d \
-v ./otel-collector-config.yml:/etc/otel/config.yaml \
-e SPLUNK_ACCESS_TOKEN=abcdefg654 \
-e SPLUNK_REALM=us1 \
-p 4317:4317 \
otel/opentelemetry-collector-contrib:latestYou can also refer to this Docker-compose example.
Start Besu with the
--metrics-enabled
and--metrics-protocol=opentelemetry
options. For example, run the following command to start a single node:- Syntax
- Example
OTEL_EXPORTER_OTLP_ENDPOINT=https://<host>:<port> besu --network=dev --miner-enabled --miner-coinbase <COINBASE ADDRESS> --rpc-http-cors-origins="all" --rpc-http-enabled --metrics-enabled --metrics-protocol=opentelemetry
OTEL_EXPORTER_OTLP_ENDPOINT=https://localhost:4317 besu --network=dev --miner-enabled --miner-coinbase fe3b557e8fb62b89f4916b721be55ceb828dbd73 --rpc-http-cors-origins="all" --rpc-http-enabled --metrics-enabled --metrics-protocol=opentelemetry
The OpenTelemetry SDK mandates how to configure the OpenTelemetry gRPC client, so data flows to the collector from Besu.
You can use the following environment variables:
Name Description Required OTEL_EXPORTER_OTLP_ENDPOINT OpenTelemetry Collector endpoint, of the form https://host:port
. The default value ishttps://localhost:4317
Yes OTEL_EXPORTER_OTLP_INSECURE Whether to allow insecure connections for OpenTelemetry data. False by default. No
*[APM]: Application Performance Monitoring