Mallappa Malagi
Bangalore
Summary
Experienced Australian Permanent Resident with over 12 years of IT expertise in data engineering, ETL development, and cloud-based data solutions. Specializes in designing scalable Big Data pipelines using Apache Spark, Azure Databricks, Pyspark, SQL, Kafka, and NiFi. Skilled in constructing real-time and batch data integration workflows to enhance data quality, governance, and performance. Demonstrated success in delivering high-quality solutions for BFSI, telecommunications, superannuation, and hospitality industries. Versatile professional with a proven ability to work effectively both onsite in Australia and offshore. Thrives in Agile environments, prioritizing cost optimization and system efficiency.
Overview
12
12
years of professional experience
1
1
Certification
Work History
Senior Specialist – Data/AI Engineering
AT&T Communication Services India Pvt Ltd
02.2021 - Current
- Spearheaded migration of legacy Hive and NiFi workflows to Azure Databricks using Apache Spark, enhancing scalability and maintainability.
- Engineered, coded, and debugged applications using Spark SQL, DataFrames, PySpark, Datasets, and custom Python functions.
- Proficient in developing complex SQL queries and utilising Spark SQL and DataFrames for efficient and scalable data transformations.
- Reduced Databricks Unit (DBU) usage from 560 to 25 daily, saving approx. 200,000 DBUs annually.
- Integrated Kafka streaming into Delta Lake architecture for real-time insights and downstream consumption.
- Automated CI/CD pipelines via Azure DevOps for production deployment from GitHub.
- Delivered Spark job designs aligned with customer expectations through detailed HLDs and LLDs.
- Applied Delta Lake UniForm format to convert tables to Apache Iceberg for cross-engine compatibility.
- Leveraged Azure Databricks Copilot and GitHub Copilot to boost code quality, streamline development, and reduce debugging time.
Big Data Developer
LTIMindtree (Client: Marriott International)
01.2018 - 08.2019
- Migrated traditional data warehousing systems to modern AWS data lake stack.
- Developed scalable ingestion and transformation pipelines using Apache NiFi, Spark (Scala), Kafka.
- Implemented batch and real-time processing jobs on AWS EMR and S3.
- Built custom ingestion pipelines using NiFi to pull data from MQ, FTP, S3 and push to Kafka topics.
- Optimised Spark workloads to enhance performance and resource utilisation.
Developer – Data Engineering
Mindtree Ltd (Client: Link Market Services)
08.2013 - 12.2017
- Worked directly onsite in Sydney with superannuation and managed funds teams.
- Delivered reporting and analytics frameworks for regulatory compliance and investor reporting.
- Developed stored procedures, performed performance tuning and supported ETL reporting cycles.
- Collaborated with cross-functional Australian teams on data migration and enhancement projects.
- Supported both India and Australia-based clients with tailored SQL-based data solutions.
Education
Bachelor of Engineering (B.E.) - Information Science
B.V. Bhoomaraddi College of Engineering And Technology
Hubli, India01.2012
Skills
- Big Data: Apache Spark (SQL, DataFrames, Streaming), PySpark, Scala, Apache Kafka, Apache NiFi, Delta Lake
- Cloud Platforms: Azure Databricks, Azure Data Lake, AWS (EMR, S3, Lambda)
- ETL & Data Integration: Complex SQL, real-time and batch pipelines, data quality
- DevOps Tools: Azure DevOps, GitHub, CI/CD automation
- Databases: MS SQL Server, Oracle, Azure Delta Lake
- Agile & Delivery: Scrum methodology, cross-functional collaboration
- Tools: Copilot (Azure/GitHub), JIRA, Confluence
Certification
- Databricks Certified Associate Developer for Apache Spark
- Microsoft Certified: Azure Fundamentals (AZ-900)
Awards
“You Deserve” Award – AT&T, March 2024, “Mindtree Pillar” Award – 2017–2018, “Outstanding Performer” – 2016–2017, 2017–2018, “Unstoppable” Award – Mindtree, April 2017
Additional Details
- Work Rights: Australian Permanent Resident – immediately available for local roles
- Location Preference: Open to relocation across Australia
- Availability: 2 weeks’ notice
Timeline
Senior Specialist – Data/AI Engineering
AT&T Communication Services India Pvt Ltd
02.2021 - Current
Big Data Developer
LTIMindtree (Client: Marriott International)
01.2018 - 08.2019
Developer – Data Engineering
Mindtree Ltd (Client: Link Market Services)
08.2013 - 12.2017
Bachelor of Engineering (B.E.) - Information Science
B.V. Bhoomaraddi College of Engineering And Technology
Similar Profiles
Amanjot KaurAmanjot Kaur
Customer Support Specialist at Wireless communication services india pvt ltdCustomer Support Specialist at Wireless communication services india pvt ltd
Anitha JoshiAnitha Joshi
Lead Data Analyst at AT&T Communication Services India Pvt. Ltd.Lead Data Analyst at AT&T Communication Services India Pvt. Ltd.
Samir KumarSamir Kumar
Senior Manager Country Operations at Wireless Communication Services India Pvt Ltd.Senior Manager Country Operations at Wireless Communication Services India Pvt Ltd.
Raghu Vamsi MRaghu Vamsi M
Associate Manager- Finance at Reverie Language Technologies LimitedAssociate Manager- Finance at Reverie Language Technologies Limited
Shwetha AgarwalShwetha Agarwal
Artist Manager at Nineone IncArtist Manager at Nineone Inc