Create a Kafka Event Stream

This tutorial provides instructions to create a Kafka event stream to stream Goerli Testnet events.

Prerequisites

To run this tutorial, you must have the following installed:

1. Create Folders

The node requires a data directory for the blockchain data. Create the directories for the node data:

Goerli-Network/
├── data

2. Start a Besu Node

In the Goerli-Network directory start the Besu node and enable the Kafka plugin.

besu --data-path=data --network=goerli --logging=DEBUG --plugin-kafka-enable=true --plugin-kafka-stream=pegasys-test-stream      

The command line specifies:

Note

The tutorial uses the default URL of the Kafka server (127.0.0.1:9092). Use the --plugin-kafka-url option to specify an alternate URL.

3. Start the Apache Kafka Server

TBD

4. Confirm that the Event Stream Works

View the Pegasys Plus logs to confirm that records are sent correctly to the pegasys-test-stream. For example:

Example

2019-09-25 09:28:08.970+10:00 | kafka-producer-network-thread | producer-1 | DEBUG | KafkaPublisher | Record sent in topic pegasys-test-stream to partition 0 with offset 1427.

In the Kafka UI view the record using the offset specified in the logs.

Kafka Event Stream

5. Stop the Node

When finished using the node, stop it using Ctrl+C in the terminal window.

6. Stop the Docker Containers

To stop the Docker containers, run docker ps to list all running containers, then run docker stop to stop each container.