Challenge: Legacy Database with Limited Scalability

Quantum GmbH, a leading energy service provider, manages large volumes of transaction data to calculate various key figures for internal visualization tools.
The previous processes ran on an Oracle database, which, however, reached its scalability and performance limits.
In particular, the existing SOAP interface, which extracted XML data from the Oracle database, was not optimal for modern data-driven analyses and machine learning.
The goal was to build a modern, scalable, and high-performance data warehouse infrastructure to enable more efficient analyses and reporting.

Solution: Migration from Oracle to Databricks with Optimized ETL Pipelines

To provide Quantum with a future-proof and scalable solution, the entire data infrastructure was migrated to Databricks.

The key steps of the migration:

• Design and construction of a scalable data warehouse:
• Development and optimization of ETL pipelines with PySpark:
• Implementation of CI/CD processes with GitHub Actions:
• Data modeling for key figure calculations:
• Structured storage of transaction data using optimized data models for high-performance queries.
• Processing of XML files from the SOAP interface using powerful PySpark pipelines that are efficient and scalable.
• Automated deployment processes for the ETL pipelines to ensure continuous improvements and versioning.
• Calculation of important key figures from the transaction data to provide the data for internal tools.

Result: Scalability, Performance & Automation

Thanks to the migration to Databricks, Quantum GmbH now benefits from a highly scalable, high-performance, and automated data infrastructure.
Improved Data Processing: The new ETL pipelines can efficiently process large volumes of XML data, which significantly enhances the overall performance of data delivery.
Automated Processes: The introduction of CI/CD with GitHub Actions enables changes to be deployed faster and more securely, thereby facilitating maintenance and further development.
Better Analysis & Reporting: With optimized data models and fast queries, various key figures for internal applications can be calculated efficiently, enabling more precise and faster decision-making.
Future-Proof & Flexible: By switching from Oracle to Databricks, Quantum is now excellently positioned for future scaling and expansion of the data platform.