GIT
Andrew Deni X G
Senior Data Engineer
Bangalore
Summary
Innovative and results-oriented data engineer with over 8 years of experience and a strong background in building high-performance data pipelines and REST API frameworks. Expert in PySpark, Databricks, and binary CAN data extraction from vehicles. Demonstrated success in implementing event-based data logging approaches and enhancing data processing systems for efficient use in digital twin projects. Accustomed to working closely with system architects, software architects and design analysts to understand business or industry requirements to develop scalable data pipeline applications.
Overview
2025
2025
years of professional experience
4
4
years of post-secondary education
2
2
Languages
Work History
Senior Data Engineer
Volvo
3 2021 - Current
- Developed and implemented serverless event-driven data integration in Azure by building real-time data pipeline to expose charging/faults to customers.
- Led development of high-capacity data pipeline using PySpark and Azure Databricks, focused on extracting binary CAN data from vehicles, thereby significantly boosting processing efficiency.
- Built a data orchestration pipeline to extract terabytes of data using Airflow, S3, Spark and Iceberg. Implemented strategic partitions and bucketing to automate signal segregation from 4000 plus parameters reducing more than 50% time in data preparation for analysts.
- Collaborated with system engineers enabling data logging approach by introducing event-based trigger system and created an automated extraction and testing framework using Azure, Databricks and PySpark
- Created a framework to create Data Catalog together with iceberg and python's profiling capabilities to profile the data and merge it with data catalog tool such as OpenMatadata to enable GENAI tasks
- Created a Automated CAN data quality framework using Spark to check value thresholds, nulls, data integrity etc.. Created interactive dashboards with Grafana to display quality metrics and an instant overview on data anomalies which reduced bugs at the early stage of product development.
- Integrated and maintained continuous integration (CI) systems using Jenkins and Azure DEVOPS to automate testing and deployment processes, significantly improving development speed and quality.
- Participated in Stakeholder meetings to create more awareness towards data driven development and align data priorities together with hardware/software development
Data Engineer
Accenture
11.2016 - 03.2021
- Orchestrated setup and maintenance of Hadoop cluster in Windows environment, integrating Hive, Spark, MongoDB, Postgres, Angular, and Python with focus on robust version control.
- Gained hands-on experience with AWS services, especially AWS Kinesis Data Stream and AWS Lambda, including development of scalable Python modules for Cloud Watch metrics integration with Splunk.
- Pioneered HAdeaP (Healthcare ETL Engine) project, patented application (U.S. Patent Application No.: 16/008,602) developed from scratch using Hadoop ecosystem components (HDFS, Spark, Scala, Python, Hive) and D3JS for data visualization.
- Developed and maintained Python Flask-based REST API framework for efficient data fetching and transformation from various sources, including databases and flat files, into Extraction Framework.
- Enhanced data processing capabilities by replacing existing Informatica Framework with Python-based ETL Framework, leading to 30% increase in performance, focusing on MongoDB to SQL Server data transformation.
- Co-developed and partly owned patented Digital Data Assistant (chatbot) using Natural Language Processing and deep learning techniques.
- Modified existing AWS Glue Tables to Delta Tables for optimized merge operations in Hive.
Education
B-Tech - Electronics And Communication Engineering
Karunya Institute of Technology And Sciences
Coimbatore 06.2012 - 05.2016
Skills
Data Engineering
PySpark
Databricks
Event-Based Data Logging
Binary CAN Data Processing
REST API Development (FastAPI)
Version Control (eg, Git)
Continuous Integration (CI)
Team Collaboration and Workflow Optimization
Azure
Real time data processing
Accomplishments
- Received ACE(Accenture Celebrates Excellence) highest honor in accenture for Automation(Spark, Python, Angular and oozie)
- Patent Received for being the sole developer for Digital Data Assistant( as a part of HAdeaP)
Software
Cloudera
Spark 3, DataBricks
Python Flask/Fast Api
Oozie
AWS(EMR, Lambda, Kinesis, Cloudwatch), Azure
Splunk, Tableau
Hive
Mongo DB
Airflow
Azure
Iceberg
Trino
Open Metadata
Grafana
Timeline
Data Engineer
Accenture
11.2016 - 03.2021
B-Tech - Electronics And Communication Engineering
Karunya Institute of Technology And Sciences
06.2012 - 05.2016
Senior Data Engineer
Volvo
3 2021 - Current
Languages
English
Tamil