Kaluza - amarti

Introduction

Kaluza is an intelligent energy platform that introduces new flexibility into the energy system by optimising individual devices.

Amarti worked within data engineering on the Analytical Platform Team. Kaluza’s objective was to build a robust and highly available analytical platform, to facilitate data warehousing, analysis, and data science use cases. With a focus on data-driven product management and the implementation of best practices, Kaluza aimed to leverage amarti’s expertise to enhance their platform’s capabilities and streamline their data workflows.

Objective

Our primary objective was to enable Kaluza to establish a robust data governance framework, improve data integration processes, and enhance data quality. Our team aimed to centralise data efforts, decentralise mapping processes, and introduce new technologies and methodologies to optimise the data platform.

Methods

Analytical Platform Development:

The amarti team integrated with Kaluza to form a strong team of 4 data engineers, 2 data analysts, 1 SEM specialist, and 1 Product Owner to develop the analytical platform.

Applied Test-Driven Development (TDD), DevOps, and Infrastructure as Code (IAC) best practices to ensure a highly available and scalable platform.Utilised Google Cloud, Terraform, GCS, BigQuery, Cloud Build, Cloud Run, Secret Manager, Composer (Managed Airflow), GCR, and Workflows in the tech stack for platform development.

Data Ingestion and Migration:

Ingested energy domain data and 3rd party data sources such as GreenHouse, Workday, Jira, etc., into the platform.

Successfully migrated OEA’s data to the new data platform, ensuring seamless integration and data integrity.

Security and Vulnerability Monitoring:

Designed and implemented a security-first approach for both the applications and the platform. Leveraged Snyk for monitoring vulnerabilities in applications, ensuring proactive identification and mitigation.

Analytics Framework Upgrade:

Collaborated with the team to build and deploy a major new version of the analytics framework.. The upgraded framework offered modular and easily reproducible environments for testing, UAT, and sendbacks.

Automated Data Quality Checks and Alerting:

Set up automated data quality checks and alerting mechanisms using Datadog and Dataform. Proactively identified and addressed data issues, ensuring data integrity and catching issues before clients were aware of them.

Data-driven Agile Journey:

Initiated a data-driven Agile journey within the team, implementing daily stand-ups, retrospectives, planning sessions, epic refinement, and story mapping. Enabled data-driven decision-making processes and iterative development cycles.

Customer Feedback Mechanism:

Implemented a customer feedback mechanism to gather qualitative and quantitative insights for data-driven product management.

Empowered the team to make informed decisions based on customer feedback.

Results and Impact

The collaboration between Kaluza and amarti resulted in numerous positive outcomes and benefits, including:

Implemented Analytical Platform- Development of a robust and highly available analytical platform, meeting the organisation’s data warehousing and analysis needs.
Improved Data Ingestion Processes – Allowing integration of energy domain data and 3rd party sources seamlessly.
Successful Data Migration – Successful migration of OEA’s data to the new data platform, ensuring data integrity and smooth transition.
Optimised Analytical Platform – Implementation of TDD, DevOps, and IAC best practices, resulting in a scalable, reliable, and efficient analytical platform.
Boosted Security – Enhanced security measures with a security-first approach and continuous vulnerability monitoring using Snyk, ensuring the platform’s resilience against potential vulnerabilities.
Analytics Framework Upgraded – Major upgrade of the analytics framework, Supernova, providing modular and easily reproducible environments for testing, UAT, and sendbacks. This improved the team’s agility and accelerated the development and deployment processes.
Automated Data Quality Checks – Implementation of automated data quality checks and alerting mechanisms using Datadog and Dataform resulted in proactive issue detection. As a result, data issues were resolved before clients were even aware of them, leading to improved data reliability and client satisfaction.
Optimised Collaboration & Decision-Making Process – Introduction of a data-driven Agile journey brought significant progress in the team’s collaboration and decision-making processes. Daily stand-ups, retrospectives, planning sessions, and other Agile practices facilitated efficient project management and enhanced transparency.
Enhanced Customer Feedback Mechanism for Insights – The establishment of a customer feedback mechanism allowed Kaluza to collect qualitative and quantitative insights for data-driven product management. This approach enabled the team to make informed decisions based on customer preferences, leading to improved product development and customer satisfaction.
Adoption of Comprehensive Tech Stack – The adoption of a comprehensive tech stack, including Google Cloud, Terraform, GCS, BigQuery, and others, empowered the team to build a highly available, scalable, and secure analytical platform. This infrastructure supported seamless data workflows, efficient data analysis, and enhanced collaboration among team members.

Overall, the collaboration between Kaluza and amarti resulted in a positive impact on the organisation. The enhanced analytical platform, data-driven decision-making processes, automated data quality checks, and customer feedback mechanisms contributed to improved data reliability, agility, and customer satisfaction. Kaluza is now equipped with a robust platform that supports its data-driven goals and enables them to stay at the forefront of the energy domain.