Creating a high available multi-region datalake with 2 Oracle databases and 2 replicated object storage buckets for purposes such as reporting, ML or AI | Part two (final)

Let’s continue with the work we started in previous post

In this post we are creating two databases in 2 different OCI regions and 2 object storage buckets for storing json data coming from a streaming source such as Kafka.

STORY

The DIABOLIC MESSAGES PRODUCER sends data to OCI Streaming topic, it is coded in Go and uses the OCI SDK. Then, a configured SERVICE CONNECTOR gets messages from topic and stores in object storage bucket. A BUCKET REPLICATION POLICY copies datafiles to bucket in region2. Databases 1 and 2 has got an external table and a view created as a select on the table. Finally the views are exposed as REST services.

A couple of node.js microservices are deployed on OKE and GKE. Each one consumes the REST endpoint exposed on each database.

A global load balancer configured as a Traffic Management Steering Policy balances the services between Google and OCI.

step 10: Create GKE CLUSTER

We are provisioning a gke witn dcloud shell as follows:

gcloud beta container --project "<yourproject>" clusters create-auto "autopilot-cluster-1" --region "<google region>" --release-channel "regular" --network "projects/<project>/global/networks/default" --subnetwork "projects/<project>/regions/<region>/subnetworks/default" --cluster-ipv4-cidr "/17" --services-ipv4-cidr "/22"

STEP 11: deploy microservice in GKE

Clone the repo, modify viewer2.yaml to point to the REST endpoint created in steps 5,6,7 in previous post

git clone https://github.com/javiermugueta/moskispub.git
cd moskispub
kubectl apply -f viewer2.yaml

STEP 13: GRAB THE PUBLIC IP OF THE LOADBALANCER CREATED IN GKE

Wait for the loadbalancer to be created and get the IP for later use

STEP 14: Deploy microservice in OKE

Clone repo, modify viewer1.yaml, deploy, wait for resources to be created and grab the public IP of the loadbalencer in OCI

git clone https://github.com/javiermugueta/moskispub.git
cd moskispub
kubectl apply -f viewer1.yaml

STEP 15: CREATE A PUBLIC DNS ZONE IN OCI

oci dns zone create -c ocid1.compartment.oc1..a...2q --name mymoskis.org --zone-type PRIMARY

Grab the public DNS servers associated to the zone.

STEP 15: REGISTER A PUBLIC DNS IN YOUR REGISTRAR

Adquire a public DNS in ypur preferred registrar

STEP 16: MODIFY DNS SERVERS ASSOCIATED TO THE DOMAIN

In your registrar portal, change the default DNS’s to the ones obtained in step 14

STEP 17: CREATE “A” RECORDS IN ZONE

Create a couple of A records in the Zone, each one pointing to the public IP of the loadbalancer created in steps 13 and 14

At the end of this step, microservices should be working by its names:

STEP 18: CREATE A HEALTHCHECK

Create a healthcheck that scans the availability of the 2 microservices. Note “vantage points” run outside OCI in other cloud providers

STEP 19: CREATE FAILOVER POLICY

Select TTL, create a couple of pools, one for oci and other for google:

Select pool priority first serve from google, then from oci:

Add attached domain to failover policy, this creates a subdomain with the name such as signals.moskis.org

STEP 20: TEST IT!

Verify http://failover.moskis.org/signals/?offset=1&limit=1 is working:

Delete deployment in GKE:

View the health-check status:

Request again http://failover.moskis.org/signals/?offset=1&limit=1, after a few seconds the response should be served from OCI:

Redeploy again in GKE:

Wait until heath-checks are all in green again:

Now, delete deployment in OKE:

Request again and see how the service is retrieved again from google with no loose of service:

That’s all for the story, hope it helps! 🙂