KANINI set out to build an automated data ingestion and processing framework with intelligent reporting capabilities on Databricks on AWS using the medallion data architecture, connecting data pipelines to source databases such as MariaDB and MySQL.
Confluent’s data ingestion platform was used for ingesting large data volumes in real-time from the company’s diverse systems of records, including third-party product vendors, and saving it in S3 Buckets.
A reporting platform was developed to generate various reports across multiple business areas including credit card and embedded finance based on various business conditions from sources such as MariaDB and inputs from other partners located in S3. These automated reports were securely transferred to the partner bank’s S3 location using SFTP.
All in all, the Databricks-powered data management platform served as an end-to-end data engineering solution for ingesting, transforming, processing, organizing, and delivering data efficiently.