Mallappa Malagi
Bangalore
Summary
Experienced Australian Permanent Resident with over 12 years of IT expertise in data engineering, ETL development, and cloud-based data solutions. Specializes in designing scalable Big Data pipelines using Apache Spark, Azure Databricks, Pyspark, SQL, Kafka, and NiFi. Skilled in constructing real-time and batch data integration workflows to enhance data quality, governance, and performance. Demonstrated success in delivering high-quality solutions for BFSI, telecommunications, superannuation, and hospitality industries. Versatile professional with a proven ability to work effectively both onsite in Australia and offshore. Thrives in Agile environments, prioritizing cost optimization and system efficiency.
Overview
12
12
years of professional experience
1
1
Certification
Work History
Senior Specialist – Data/AI Engineering
AT&T Communication Services India Pvt Ltd
02.2021 - Current
- Spearheaded migration of legacy Hive and NiFi workflows to Azure Databricks using Apache Spark, enhancing scalability and maintainability.
- Engineered, coded, and debugged applications using Spark SQL, DataFrames, PySpark, Datasets, and custom Python functions.
- Proficient in developing complex SQL queries and utilising Spark SQL and DataFrames for efficient and scalable data transformations.
- Reduced Databricks Unit (DBU) usage from 560 to 25 daily, saving approx. 200,000 DBUs annually.
- Integrated Kafka streaming into Delta Lake architecture for real-time insights and downstream consumption.
- Automated CI/CD pipelines via Azure DevOps for production deployment from GitHub.
- Delivered Spark job designs aligned with customer expectations through detailed HLDs and LLDs.
- Applied Delta Lake UniForm format to convert tables to Apache Iceberg for cross-engine compatibility.
- Leveraged Azure Databricks Copilot and GitHub Copilot to boost code quality, streamline development, and reduce debugging time.
Big Data Developer
LTIMindtree (Client: Marriott International)
01.2018 - 08.2019
- Migrated traditional data warehousing systems to modern AWS data lake stack.
- Developed scalable ingestion and transformation pipelines using Apache NiFi, Spark (Scala), Kafka.
- Implemented batch and real-time processing jobs on AWS EMR and S3.
- Built custom ingestion pipelines using NiFi to pull data from MQ, FTP, S3 and push to Kafka topics.
- Optimised Spark workloads to enhance performance and resource utilisation.
Developer – Data Engineering
Mindtree Ltd (Client: Link Market Services)
08.2013 - 12.2017
- Worked directly onsite in Sydney with superannuation and managed funds teams.
- Delivered reporting and analytics frameworks for regulatory compliance and investor reporting.
- Developed stored procedures, performed performance tuning and supported ETL reporting cycles.
- Collaborated with cross-functional Australian teams on data migration and enhancement projects.
- Supported both India and Australia-based clients with tailored SQL-based data solutions.
Education
Bachelor of Engineering (B.E.) - Information Science
B.V. Bhoomaraddi College of Engineering And Technology
Hubli, India01.2012
Skills
- Big Data: Apache Spark (SQL, DataFrames, Streaming), PySpark, Scala, Apache Kafka, Apache NiFi, Delta Lake
- Cloud Platforms: Azure Databricks, Azure Data Lake, AWS (EMR, S3, Lambda)
- ETL & Data Integration: Complex SQL, real-time and batch pipelines, data quality
- DevOps Tools: Azure DevOps, GitHub, CI/CD automation
- Databases: MS SQL Server, Oracle, Azure Delta Lake
- Agile & Delivery: Scrum methodology, cross-functional collaboration
- Tools: Copilot (Azure/GitHub), JIRA, Confluence
Certification
- Databricks Certified Associate Developer for Apache Spark
- Microsoft Certified: Azure Fundamentals (AZ-900)
Awards
“You Deserve” Award – AT&T, March 2024, “Mindtree Pillar” Award – 2017–2018, “Outstanding Performer” – 2016–2017, 2017–2018, “Unstoppable” Award – Mindtree, April 2017
Additional Details
- Work Rights: Australian Permanent Resident – immediately available for local roles
- Location Preference: Open to relocation across Australia
- Availability: 2 weeks’ notice
Timeline
Senior Specialist – Data/AI Engineering
AT&T Communication Services India Pvt Ltd
02.2021 - Current
Big Data Developer
LTIMindtree (Client: Marriott International)
01.2018 - 08.2019
Developer – Data Engineering
Mindtree Ltd (Client: Link Market Services)
08.2013 - 12.2017
Bachelor of Engineering (B.E.) - Information Science
B.V. Bhoomaraddi College of Engineering And Technology