Mirroring a Topic Between 2 Oracle Event Hub (Kafka) Clusters in 20 Minutes

Oracle Event Hub Cloud Service (OEHCS) is a Kafka managed PaaS cloud service. In a few minutes you can provision a full Kafka cluster ready for creating topics for sending and consuming messages.

mirror

In this post we will configure a feature called ‘mirroring’ that allows you to replicate the messages that are sent to a topic in a source cluster to another topic in a destination cluster.

FIRST

Create 2 OEHCS instances, the names and sizes you want. See this post for help (coming soon).

oehcs1

Create a topic in the origin cluster, name it R1.

oehcs2

Create a topic in the destination cluster, name it R1R1.

oehcs3

SECOND

Grab the public IP of one of the brokers in the destination cluster. ssh to that server and follow what is stated here (except for the point number 10)

oehcs5

Configuration files examples:

mkdir -p /u01/oehpcs/confluent/etc/mirror-maker

vi sourceClusterConsumer.config

bootstrap.servers=130.61.36.87:6667,130.61.86.29:6667
group.id=replica-consumer
exclude.internal.topics=true
auto.offset.reset=earliest
partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor

vi targetClusterProducer.config

bootstrap.servers=130.61.46.183:6667,130.61.80.227:6667
acks=-1
max.in.flight.requests.per.connection=1
compression.type=none

Message for hackers: The IP’s were real one day but they don’t exist any more 😉

THIRD

Execute this command:

/u01/oehpcs/confluent/bin/kafka-mirror-maker.sh --consumer.config /u01/oehpcs/confluent/etc/mirror-maker/sourceClusterConsumer.config --producer.config /u01/oehpcs/confluent/etc/mirror-maker/targetClusterProducer.config --num.streams 2 --whitelist "R.*" --message.handler kafka.tools.OehcsTopicSuffixMirrorMakerHandler --message.handler.args R1

FOURTH

Produce messages to R1 topic in origin cluster the way you can. See this post for help.

oehcs6

Please note that a yellow band appears in the Throughput graph indicating that bytes are going out the topic, who is extracting messages?

FIFTH

Notice that the R1R1 topic in destination cluster is receiving messages.

oehcs7

So far so good, we have replicated the topic in another cluster thousand miles away…

Stop the mirroring process, add partitions to R1R1 topic up to 20 and start the mirroring again with this new command:

/u01/oehpcs/confluent/bin/kafka-mirror-maker.sh --consumer.config /u01/oehpcs/confluent/etc/mirror-maker/sourceClusterConsumer.config --producer.config /u01/oehpcs/confluent/etc/mirror-maker/targetClusterProducer.config --num.streams 20 --whitelist "R.*" --message.handler kafka.tools.OehcsTopicSuffixMirrorMakerHandler --message.handler.args R1

As you can see now the mirroring runs faster!

That’s all folks!

Enjoy 😉

Marcación Proa