Advertisements

Creating a Fast&Simple Container for Sending Messages to a Topic in Oracle Event Hub Cloud Service (aka OEHCS, which is a Kafka cluster) and Deploying it to Kubernetes Cluster


The container uses 4 environment variables, you can find a container already built for you here

SOURCE CODE OF THE PRODUCER

var sleep = require('system-sleep');

const oehcs_connect_url = process.env.OEHCS_CONNECTURL

const topic_name = process.env.TOPIC_NAME

const num_partitions = process.env.NUM_PARTITIONS

const message = process.env.MESSAGE

var kafka = require('kafka-node'),

HighLevelProducer = kafka.HighLevelProducer,

client = new kafka.KafkaClient({kafkaHost: oehcs_connect_url}),

producer = new HighLevelProducer(client);

var i = 0;

while (i >= 0 ){

var payloads = [{ topic: topic_name, messages: message , partition: i}];

//producer.on('ready', function () {

producer.send(payloads, function (err, data) {

console.log(data);

});

// });

i = i + 1;

if (i > num_partitions -1){

i = 0;

sleep( 1 );

}

}

THE DOCKERFILE

FROM oraclelinux:7-slim
WORKDIR /app
ADD . /app
RUN curl --silent --location https://rpm.nodesource.com/setup_10.x | bash -
RUN yum -y install nodejs npm
CMD ["node","producer-direct.js"]

THE YAML FOR K8S DEPLOYMENT

apiVersion: apps/v1

kind: Deployment

metadata:

name: oehcsnodeproducer-direct

labels:

app: oehcsnodeproducer-direct

spec:

replicas: 1

selector:

matchLabels:

app: oehcsnodeproducer-direct

strategy:

type: Recreate

template:

metadata:

labels:

app: oehcsnodeproducer-direct

spec:

containers:

- image: javiermugueta/oehcsnodeproducer-direct

env:

- name: OEHCS_CONNECTURL

value: "<ip1>:6667,<ip2>:6667,..."

- name: TOPIC_NAME

value: "R1"

- name: NUM_PARTITIONS

value: "10"

- name: MESSAGE

value: "{'put here what you want'}"

name: oehcsnodeproducer-direct

TEST IT AND SEE WHAT HAPPENS

Create the deployment and after 10 minutes take a look to production messages ratio:
kubectl apply -f my.yaml
oehcs-scale1.png
More or less 400/second…
Scale the deployment and take a look to new production ratios:
kubectl scale deployment oehcsnodeproducer-direct --replicas=2
Around 8000 messages/second!
Now add 9 partitions to the topic and take a look to new ratios:
add partitions
With 2 pods running and 10 partitions we are producing around 10K messages per second! As you can see partitioning improves performance!
10partitions.png
Let’s double the number of pods and see new ratios:
kubectl scale deployment oehcsnodeproducer-direct --replicas=4
And now 18K/second!
18k
That’s all folks!
Enjoy 😉

IT

Advertisements

javier mugueta View All →

Me and my circumstance

1 Comment Leave a comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: