Case Study

An AI-powered Data Ingestion and Reporting Platform Built on Databricks

A Silicon Valley-based lending company builds a high-speed real-time data ingestion solution on Databricks to streamline reporting

An esteemed loans and credit cards company in the US leverages Databricks to build a robust AI-powered data platform that ingests petabytes of data at high speed and accelerates business processes for enhanced customer satisfaction and revenues.
Download PDF
Industry & Region: Banking and Financial Services, USA
Tech Stack & Techniques:
AWS Cloud Stack: AWS MSK, AWS Glue, AWS RDS, Data Platform: Confluent, Databricks, Apache Spark, Databricks Delta Live Tables, Platform for Change Data Capture: Debezium, Data Warehouse: Amazon Redshift, Programming Language: Python, Shell Script, RDBMS: MariaDB, MySQL
Client Overview
Our client is a loans and credit cards company based out of Silicon Valley in the US, delivering specialized financial services to customers through strategic business partnerships with lending banks across the US.
Business Challenge
The company’s business expansion resulted in a significant surge in the volume of data inflow – all of which had to be processed fast and accurately to prepare and submit complex financial reports for each lending partner on time. Their current data processes were largely manual, time-intensive, and error-prone. To make the partnerships with lending banks more sustainable, the company needed a modern technology solution that would process such large volumes of data accurately on a near real-time basis in a cost-effective manner.
Solution Offered
KANINI set out to build an automated data ingestion and processing framework with intelligent reporting capabilities on Databricks on AWS using the medallion data architecture, connecting data pipelines to source databases such as MariaDB and MySQL.
The new solution was integrated with Debezium to drive change data capture mechanisms and automate the process of identifying and tracking changes made to data sources over time. This allowed the team to spend minimum time on change management tasks.
Confluent’s data ingestion platform was used for ingesting large data volumes in real-time from the company’s diverse systems of records, including third-party product vendors, and saving it in S3 Buckets.
Databricks was leveraged to extract the data from S3 using Delta Live Tables to deliver it to the company’s data warehouse in Redshift.
A reporting platform was developed to generate various reports across multiple business areas including credit card and embedded finance based on various business conditions from sources such as MariaDB and inputs from other partners located in S3. These automated reports were securely transferred to the partner bank’s S3 location using SFTP.
All in all, the Databricks-powered data management platform served as an end-to-end data engineering solution for ingesting, transforming, processing, organizing, and delivering data efficiently.
Value Delivered
  • Strengthened the client’s business relationships with issuer banks through fast and accurate report generation.
  • Significantly transformed customer experiences by accelerating the company’s lending processes.
  • Empowered the client to harness the full potential of Databricks Lakehouse architecture – its numerous handy tools and features, including the advanced security analysis tool.
  • Enabled high-speed data ingestion, single view of data, high flexibility and scalability, real-time insights, and increased opportunity for collaboration.
  • Allowed easy integration with advanced BI tools such as Tableau and Microsoft Power BI, delivering meaningful insights through interactive data visualization.
Want to know how Databricks can create value in your business by streamlining end-to-end data processing?
Talk to us
© 2024 KANINI Software Solutions | All Rights Reserved | Privacy Policy

AI-powered Data Ingestion and Reporting Platform Built on Databricks