Mirroring a Topic Between 2 Oracle Event Hub (Kafka) Clusters in 20 Minutes

Oracle Event Hub Cloud Service (OEHCS) is a Kafka managed PaaS cloud service. In a few minutes you can provision a full Kafka cluster ready for creating topics for sending and consuming messages.

In this post we will configure a feature called ‘mirroring’ that allows you to replicate the messages that are sent to a topic in a source cluster to another topic in a destination cluster.

FIRST

Create 2 OEHCS instances, the names and sizes you want. See this post for help (coming soon).

Create a topic in the origin cluster, name it R1.

Create a topic in the destination cluster, name it R1R1.

SECOND

Grab the public IP of one of the brokers in the destination cluster. ssh to that server and follow what is stated here (except for the point number 10)

Configuration files examples:

mkdir -p /u01/oehpcs/confluent/etc/mirror-maker

vi sourceClusterConsumer.config

bootstrap.servers=130.61.36.87:6667,130.61.86.29:6667
group.id=replica-consumer
exclude.internal.topics=true
auto.offset.reset=earliest
partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor

vi targetClusterProducer.config

bootstrap.servers=130.61.46.183:6667,130.61.80.227:6667
acks=-1
max.in.flight.requests.per.connection=1
compression.type=none

Message for hackers: The IP’s were real one day but they don’t exist any more 😉

THIRD

Execute this command:

/u01/oehpcs/confluent/bin/kafka-mirror-maker.sh --consumer.config /u01/oehpcs/confluent/etc/mirror-maker/sourceClusterConsumer.config --producer.config /u01/oehpcs/confluent/etc/mirror-maker/targetClusterProducer.config --num.streams 2 --whitelist "R.*" --message.handler kafka.tools.OehcsTopicSuffixMirrorMakerHandler --message.handler.args R1

FOURTH

Produce messages to R1 topic in origin cluster the way you can. See this post for help.

Please note that a yellow band appears in the Throughput graph indicating that bytes are  going out the topic, who is extracting messages?

FIFTH

Notice that the R1R1 topic in destination cluster is receiving messages.

So far so good, we have replicated the topic in another cluster thousand miles away…

Stop the mirroring process, add partitions to R1R1 topic up to 20 and start the mirroring again with this new command:

/u01/oehpcs/confluent/bin/kafka-mirror-maker.sh --consumer.config /u01/oehpcs/confluent/etc/mirror-maker/sourceClusterConsumer.config --producer.config /u01/oehpcs/confluent/etc/mirror-maker/targetClusterProducer.config --num.streams 20 --whitelist "R.*" --message.handler kafka.tools.OehcsTopicSuffixMirrorMakerHandler --message.handler.args R1

As you can see now the mirroring runs faster!

That’s all folks!

Enjoy 😉