Lingareddy Madhan
Senior Data Engineer
Bengaluru,KA
Summary
Data Engineer with 8 years of experience working on various types of data, different ETL and BI tools. Technically strong in Python, Apache Spark (PySpark) and SQL. Experience across Payments, Finance, Construction and Retail domains. Worked on cloud platforms like AWS and GCP. Responsive expert experienced in monitoring database performance, troubleshooting issues and optimizing database environment. Possesses strong analytical skills and deep understanding of database technologies and systems. Equally confident working independently and collaboratively as needed.
Overview
7
7
years of professional experience
2
2
Certifications
Skills
- Python
- SQL
- Data Modeling
- Streaming
- BI Development
- AWS
- GCP
- Databricks
- Airflow
- Apache Kafka
- PySpark
- Amazon Redshift
- Snowflake
- BigQuery
- AWS Glue
- AWS Lake Formation
- Tableau
- Quicksight
- Data Analaysis
- Incorta
- Serverless
- Git
Work History
Senior Data Engineer
Procore Technologies Inc.
06.2024 - Current
- Built low latency data streaming workflows by leveraging Databricks and Apache Kafka to build a Delta lakehouse.
- Reengineered existing ETL workflows to improve performance by identifying bottlenecks and optimizing code accordingly.
- Working on designing data quality framework to validate data at various stages of the ingestion process.
Sr Data Engineer
Pasarpolis
Bangalore
4 2022 - 05.2024
- Designed and implemented ETL architecture using Cloud Composer on GCP and documented the process.
- Created Airflow Workflows for ETL and generating reports
- Architected a Data archival system and authored base scripts for the python repository.
- Migrated BigQuery Datasets across regions successfully for Compliance reasons.
- Setup a real-time data streaming pipeline using managed services like Data Streams and BigQuery.
- Designed warehouse table for Tableau Dashboard that displays SLA Quartiles and Customer Lifetime values.
- Working on POC for a cost effective real time data system, focusing majorly on tools offered by AWS and Open Source.
- Updated ETL logic for Huge tables to bring down the load times by 20 %.
Sr Data Engineer
ADF Data Science
Chennai
04.2021 - 03.2022
- Orchestrated data pipelines for migrating data from MySQL to AWS Redshift using AWS DMS
- Authored a script using Python and PySpark for building Historical data from change data capture (CDC) files in AWS S3
- Authored an ETL workflow in Pentaho to detect daily Schema changes
- Designed metadata and calendar dimension as per business requirements
- Installed an open source Data Cataloging tool called Amundsen for data governance
- Drafted a Python code to validate the counts by parsing Bin Log files of MySQL
- Enabled data lineage tracking for Spark workloads using Spline which is also an open source tool
- Designed and Developed data pipeline using Python and Spark on AWS Glue for building Warehouse Tables
- Have setup monitoring and alerts using AWS CloudWatch to notify on Slack in case of errors.
- Optimized Analytic queries to bring down the runtime from 2 Hrs to 2 Mins by taking advantage of Redshift data distribution and query tuning techniques.
- Built a scalable Data Lake on AWS S3 using AWS Glue, Apache Hudi and Lake Formation, which enables easier Data Shares with clients across different AWS accounts without the need for any ETL.
- Built numerous dashboards on Quicksight and taken care of Admin activities
BI Developer
Wipro
Bangalore
02.2018 - 03.2021
- Worked as a BI Developer and Data Engineer for Broadcom Inc., Collected and analyzed data requirements from clients
- Built Data models and ETL workflows using Incorta
- Data Enrichment using python pandas and Rest APIs
- Optimized code using Multithreading and Multiprocessing whilst using pandas framework to reduce execution times from over 6Hrs to a couple of hours
- Developed numerous dashboards on Incorta
- Authored Python Scripts to automate Dashboard delivery using Incorta's REST API and pandas for data cleaning
- Developed PTD dashboards on Incorta, which were previously running on Oracle Publisher
- Optimized these queries for a huge jump in performance
- Designed and developed a series of dashboards for tracking employee performance at various levels.
Education
Bachelor of Technology - Electronics And Communications Engineering
IIIT, RGUKT
04.2017
Awards
Incorta 4 Fundamentals For Admins And Developers : 2019
Python Intermediate, HackerRank : 2020
AWS CDA, Udemy : 2020
Customer Charmer Award, Wipro : 2021
Personal Information
Title: Senior Cloud Data Engineer
Email: madhanreddy575@gmail.com
Phone: +91-8886893512
Languages
- English
- Telugu
- Hindi
Timeline
Senior Data Engineer
Procore Technologies Inc.
06.2024 - Current
Sr Data Engineer
ADF Data Science
04.2021 - 03.2022
BI Developer
Wipro
02.2018 - 03.2021
Bachelor of Technology - Electronics And Communications Engineering
IIIT, RGUKT
Sr Data Engineer
Pasarpolis
4 2022 - 05.2024
Similar Profiles
Garrett AckardGarrett Ackard
Senior Account Executive at Procore Technologies Inc.Senior Account Executive at Procore Technologies Inc.
Houston MerckHouston Merck
Solutions Specialist at Procore Technologies Inc.Solutions Specialist at Procore Technologies Inc.
Isaiah ZavalaIsaiah Zavala
Commercial Customer Success Engineer at Procore Technologies Inc.Commercial Customer Success Engineer at Procore Technologies Inc.
Jeff KemperJeff Kemper
Senior Director, Customer Success at Procore Technologies Inc.Senior Director, Customer Success at Procore Technologies Inc.
Neha DasguptaNeha Dasgupta
Team Lead – Inside Sales & Growth Strategy at LivspaceTeam Lead – Inside Sales & Growth Strategy at Livspace