Oracle Event Hub Cloud Service (OEHCS) is a Kafka managed PaaS cloud service. In a few minutes you can provision a full Kafka cluster ready for creating topics for sending and consuming messages.
In this post we will configure a feature called ‘mirroring’ that allows you to replicate the messages that are sent to a topic in a source cluster to another topic in a destination cluster.
FIRST
Create 2 OEHCS instances, the names and sizes you want. See this post for help (coming soon).
Create a topic in the origin cluster, name it R1.
Create a topic in the destination cluster, name it R1R1.
SECOND
Grab the public IP of one of the brokers in the destination cluster. ssh to that server and follow what is stated here (except for the point number 10)
Configuration files examples:
mkdir -p /u01/oehpcs/confluent/etc/mirror-maker vi sourceClusterConsumer.config bootstrap.servers=130.61.36.87:6667,130.61.86.29:6667 group.id=replica-consumer exclude.internal.topics=true auto.offset.reset=earliest partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor vi targetClusterProducer.config bootstrap.servers=130.61.46.183:6667,130.61.80.227:6667 acks=-1 max.in.flight.requests.per.connection=1 compression.type=none
Message for hackers: The IP’s were real one day but they don’t exist any more 😉
THIRD
Execute this command:
/u01/oehpcs/confluent/bin/kafka-mirror-maker.sh --consumer.config /u01/oehpcs/confluent/etc/mirror-maker/sourceClusterConsumer.config --producer.config /u01/oehpcs/confluent/etc/mirror-maker/targetClusterProducer.config --num.streams 2 --whitelist "R.*" --message.handler kafka.tools.OehcsTopicSuffixMirrorMakerHandler --message.handler.args R1
FOURTH
Produce messages to R1 topic in origin cluster the way you can. See this post for help.
Please note that a yellow band appears in the Throughput graph indicating that bytes are going out the topic, who is extracting messages?
FIFTH
Notice that the R1R1 topic in destination cluster is receiving messages.
So far so good, we have replicated the topic in another cluster thousand miles away…
Stop the mirroring process, add partitions to R1R1 topic up to 20 and start the mirroring again with this new command:
/u01/oehpcs/confluent/bin/kafka-mirror-maker.sh --consumer.config /u01/oehpcs/confluent/etc/mirror-maker/sourceClusterConsumer.config --producer.config /u01/oehpcs/confluent/etc/mirror-maker/targetClusterProducer.config --num.streams 20 --whitelist "R.*" --message.handler kafka.tools.OehcsTopicSuffixMirrorMakerHandler --message.handler.args R1
As you can see now the mirroring runs faster!
That’s all folks!
Enjoy 😉
