Customer expectations are constantly rising, and organizations must deliver quick and efficient services. They must also deal with regulatory compliance intricacies, process complexities, and many other such challenges to streamline service delivery and performance. Legacy systems that feature outdated technologies and inefficient manual processes slow down responsiveness to evolving business challenges and industry trends. Modern, Cloud-native solutions emerge as an effective response to legacy ordeals – improving efficiency and customer experience by streamlining processes.
Table of Contents
Existing Monolithic System
A long-running worker process, developed using the legacy .NET Framework 4.5.2, was operating on a high-end AWS EC2 virtual machine. This process was scheduled to trigger at specific intervals, particularly during weekends or off-peak hours. It was a monolithic application that retrieved millions of records from a massive PostgreSQL database, consolidating them into meaningful reports for end customers.
Upon reviewing the application code, it became clear that scaling the process was impossible without rewriting the module from scratch. Running a legacy .NET Framework application on an expensive AWS EC2 Windows Server instance was not a cost-effective solution.
After successfully generating the reports, the worker process exported them as multiple PDF files and uploaded them to an S3 bucket. These reports were then accessed by various applications according to business requirements.
Challenges Encountered
- The legacy worker process was designed to run on a single thread, causing data retrieval and computation to occur sequentially, resulting in poor throughput.
- Since the worker process was built using the .NET Framework, it was tightly coupled with the Windows environment.
- The .NET application relied on an outdated third-party component for PDF generation, which lacked many features needed to meet current business demands.
- Although the application was scheduled to start during off-peak times and weekends, the virtual machine ran continuously throughout the day, leading to unnecessary operational costs. It was discovered that the EC2 server instance was underutilized.
- The report data, available in JSON format, was stored in a relational PostgreSQL database. This led to the database growing to terabytes in size, significantly increasing costs. It cost them about $10,000 per month.
- Lastly, for each report generation, the data had to be processed to extract the required consolidated information. This meant the same data processing occurred repeatedly and was stored in temporary tables in the database. This preliminary data processing consumed most of the computation time for report generation.
Simple, yet powerful Cloud-Native Solution
The new report engine flow:
- Instead of storing the report data within a costly relational database, the new design utilizes affordable AWS S3 storage. This approach not only reduces costs but also efficiently manages data growth.
- When new report data is added to the source S3 bucket, AWS Lambda is triggered to begin preliminary data processing. The processed data is then stored in AWS DynamoDB for further processing.
- Additionally, some relational data is stored in an AWS Aurora MySQL instance. This decision is based on the structure of the processed data and the need for backward compatibility.
- Unlike the previous approach, intermediate report data is retained for subsequent report generation, significantly optimizing data processing costs.
- The report data includes information about the report generation request. Once this request is added to the source S3 bucket, the Lambda function validates the report parameters and initiates the report generation process by starting an ECS Fargate Task.
- The ECS container generates the report using the preprocessed data and uploads the final report as a PDF file to the target S3 bucket.
- The ECS container utilizes a revamped report engine built with .NET 6.0 and the latest version of a third-party PDF generator.
- The consolidated report is available within 20 minutes, a significant improvement from the previous 6 to 8 hours. The new report engine uses preprocessed data from DynamoDB and MySQL databases to streamline data processing.
- Most importantly, the operating cost of the report generation module is reduced to $500 from $10,000 approximately.
Building Future-ready Processes with Cloud
Adapting to future business demands and market competition necessitates agile innovation. Modern cloud solutions not only offer scalability but also cost-effectiveness. Embracing cloud-native design enhances business operations while minimizing operating expenses.
KANINI empowers organizations to plan and execute their journey in the cloud. Our end-to-end cloud enablement services include enterprise cloud strategy, cloud migration, cloud application services, cloud optimization, and cloud security & monitoring.
Want to learn more about cloud modernization and how it can optimize processes within your enterprise? Connect with us.
Author
