As we mentioned in a previous post, the idea here is to develop a construct that seeks for changes in the SaaS data system and publishes those changes to a stream for later consumption from other systems over there.
Indeed, as we see in the diagram, there is a block in which we are executing a program that run in loops of “get all” requests to a list of API endpoints that have been configured with the Administration User interface. The logic is:
- wake up every x seconds
- for each endpoint registered
- get parameters such as conditions, last successful execution time, …
- execute GET requests in loops with chunks of N records per call and get the data until there is no more data retrieved
- put the data in the topic with the streaming API
As we mentioned in this post, the SaaS REST API’s have a common pattern so its easy to create a program that executes the logic mentioned and put it in an image container to be deployed in K8s, we’ll show an example in a new post sooner.
And that’s all for today, hope it helps! 🙂
approaching off the bow 
One Comment