Skip to main content

Use OpenTelemetry

You can use the OpenTelemetry monitoring and tracing service to gather node metrics and traces. To enable OpenTelemetry to access Hyperledger Besu, use the --metrics-enabled and --metrics-protocol=opentelemetry options. Use Splunk to visualize the collected data. A Besu Sync example is available.

tip

Use OpenTelemetry to monitor the sync time of your Besu node and show where time is spent internally and over the JSON-RPC interface.

This office hours recording shows examples of monitoring Hyperledger Besu.

Install OpenTelemetry Collector

Download and install the OpenTelemetry Collector.

tip

You can also install exporters that send system metrics to OpenTelemetry to monitor non-Besu-specific items such as disk and CPU usage. The OpenTelemetry Collector can connect to additional applications, and may be deployed in Kubernetes environments as a daemonset.

Setting up and running OpenTelemetry with Besu

  1. Configure OpenTelemetry to accept data from Besu. For example, use the following configuration for your otel-collector-config.yml file, and send data to Splunk and Splunk APM:

    otel-collector-config.yml
    receivers:
    otlp:
    protocols:
    grpc:
    http:

    exporters:
    splunk_hec/traces:
    # Splunk HTTP Event Collector token.
    token: "11111111-1111-1111-1111-1111111111113"
    # URL to a Splunk instance to send data to.
    endpoint: "https://<SPLUNK INSTANCE>:8088/services/collector"
    # Optional Splunk source: https://docs.splunk.com/Splexicon:Source
    source: "besu:traces"
    # Optional Splunk source type: https://docs.splunk.com/Splexicon:Sourcetype
    sourcetype: "otlp"
    # Splunk index, optional name of the Splunk index targeted.
    index: "traces"
    # Maximum HTTP connections to use simultaneously when sending data. Defaults to 100.
    max_connections: 20
    # Whether to disable gzip compression over HTTP. Defaults to false.
    disable_compression: false
    # HTTP timeout when sending data. Defaults to 10s.
    timeout: 10s
    # Whether to skip checking the certificate of the HEC endpoint when sending data over HTTPS. Defaults to false.
    # For this demo, we use a self-signed certificate on the Splunk docker instance, so this flag is set to true.
    insecure_skip_verify: true
    splunk_hec/metrics:
    # Splunk HTTP Event Collector token.
    token: "11111111-1111-1111-1111-1111111111113"
    # URL to a Splunk instance to send data to.
    endpoint: "https://<SPLUNK INSTANCE>:8088/services/collector"
    # Optional Splunk source: https://docs.splunk.com/Splexicon:Source
    source: "besu:metrics"
    # Optional Splunk source type: https://docs.splunk.com/Splexicon:Sourcetype
    sourcetype: "prometheus"
    # Splunk index, optional name of the Splunk index targeted.
    index: "metrics"
    # Maximum HTTP connections to use simultaneously when sending data. Defaults to 100.
    max_connections: 20
    # Whether to disable gzip compression over HTTP. Defaults to false.
    disable_compression: false
    # HTTP timeout when sending data. Defaults to 10s.
    timeout: 10s
    # Whether to skip checking the certificate of the HEC endpoint when sending data over HTTPS. Defaults to false.
    # For this demo, we use a self-signed certificate on the Splunk docker instance, so this flag is set to true.
    insecure_skip_verify: true
    # Traces
    sapm:
    access_token: "${SPLUNK_ACCESS_TOKEN}"
    endpoint: "https://ingest.${SPLUNK_REALM}.signalfx.com/v2/trace"
    # Metrics + Events
    signalfx:
    access_token: "${SPLUNK_ACCESS_TOKEN}"
    realm: "${SPLUNK_REALM}"

    processors:
    batch:

    extensions:
    health_check:
    pprof:
    zpages:

    service:
    extensions: [pprof, zpages, health_check]
    pipelines:
    traces:
    receivers: [otlp]
    exporters: [splunk_hec/traces, sapm]
    processors: [batch]
    metrics:
    receivers: [otlp]
    exporters: [splunk_hec/metrics, signalfx]
    processors: [batch]

    It is easiest to run the OpenTelemetry collector with Docker with the following command:

    docker run -d \
    -v ./otel-collector-config.yml:/etc/otel/config.yaml \
    -e SPLUNK_ACCESS_TOKEN=<access token> \
    -e SPLUNK_REALM=<realm> \
    -p 4317:4317 \
    otel/opentelemetry-collector-contrib:latest

    You can also refer to this Docker-compose example.

  2. Start Besu with the --metrics-enabled and --metrics-protocol=opentelemetry options. For example, run the following command to start a single node:

    OTEL_EXPORTER_OTLP_ENDPOINT=https://<host>:<port> besu --network=dev --miner-enabled --miner-coinbase <COINBASE ADDRESS> --rpc-http-cors-origins="all" --rpc-http-enabled --metrics-enabled --metrics-protocol=opentelemetry

    The OpenTelemetry SDK mandates how to configure the OpenTelemetry gRPC client, so data flows to the collector from Besu.

    You can use the following environment variables:

    NameDescriptionRequired
    OTEL_EXPORTER_OTLP_ENDPOINTOpenTelemetry Collector endpoint, of the form https://host:port. The default value is https://localhost:4317Yes
    OTEL_EXPORTER_OTLP_INSECUREWhether to allow insecure connections for OpenTelemetry data. False by default.No

*[APM]: Application Performance Monitoring