Near Real Time Data Driven SaaS Integration with Streaming | Part 2: Extracting/Publishing Data Changes to Streaming Topics

As we mentioned in a previous post, the idea here is to develop a construct that seeks for changes in the SaaS data system and publishes those changes to a stream for later consumption from other systems over there.

Indeed, as we see in the diagram, there is a block in which we are executing a program that run in loops of “get all” requests to a list of API endpoints that have been configured with the Administration User interface. The logic is:

  • wake up every x seconds
  • for each endpoint registered
  • get parameters such as conditions, last successful execution time, …
  • execute GET requests in loops with chunks of N records per call and get the data until there is no more data retrieved
  • put the data in the topic with the streaming API

As we mentioned in this post, the SaaS REST API’s have a common pattern so its easy to create a program that executes the logic mentioned and put it in an image container to be deployed in K8s, we’ll show an example in a new post sooner.

And that’s all for today, hope it helps! 🙂

One Comment

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.