Near Real Time Data Driven SaaS Integration with Streaming | Part 1: Overview

File based approaches

File-based data exchange comfort a large percentage of the integrations between SaaS and other ERP solutions in the past and today. The approach is robust and allows a large number of transactions to be executed in batch without affecting the online systems, but it has an inconvenience: the information is not updated at the last moment.

Streaming Solutions

Streaming solutions (kafka, JMS, …) allow weakly coupled systems, which provide everything necessary to have information updated almost in real time without overloading the source and target systems.

Indeed, the use of streaming allows the source system to publish data to an intermediate flow where data is stored during a sufficient time so that the target systems have the chance to consume the data according to the load fluctuations to which they are subjected and the source system publishes data in the same way.

The benefits are, among others:
– Destination systems are not overloaded
– Data arrives at the destination much earlier than in batch mode
– No data is lost

Oracle OCI Streaming

OCI Streaming is a Kafka compatible, secure, no lock-in, pay as you use, scalable, cheap streaming solution that allows the purpose mentioned with very low effort and ease to develop and deploy.

Source systems

Source systems, such as Oracle SaaS, can be queried by an intermediate publisher with the REST API’s, get the data and put as messages in topics utilising the streaming API included in OCI SDK’s provided.

Target Systems

Destination systems, receive messages by means of an intermediate publisher that get messages from the topics and send to the target using their own API.

Containerisation and “K8szation”

One good approach for this purpose is creating a containerised programs to be deployed and run in a Kubernetes Cluster.

Next Post: Near Real Time Data Driven SaaS Integration with Streaming | Part 2: Extracting/Publishing Data Changes

Hope it helps! 🙂