Summary
Overview
Skills
Work History
Education
Awards
Personal Information
Languages
Timeline
Generic
Lingareddy Madhan

Lingareddy Madhan

Senior Data Engineer
Bengaluru,KA

Summary

Data Engineer with 8 years of experience working on various types of data, different ETL and BI tools. Technically strong in Python, Apache Spark (PySpark) and SQL. Experience across Payments, Finance, Construction and Retail domains. Worked on cloud platforms like AWS and GCP. Responsive expert experienced in monitoring database performance, troubleshooting issues and optimizing database environment. Possesses strong analytical skills and deep understanding of database technologies and systems. Equally confident working independently and collaboratively as needed.

Overview

7
7
years of professional experience
2
2

Certifications

Skills

  • Python
  • SQL
  • Data Modeling
  • Streaming
  • BI Development
  • AWS
  • GCP
  • Databricks
  • Airflow
  • Apache Kafka
  • PySpark
  • Amazon Redshift
  • Snowflake
  • BigQuery
  • AWS Glue
  • AWS Lake Formation
  • Tableau
  • Quicksight
  • Data Analaysis
  • Incorta
  • Serverless
  • Git

Work History

Senior Data Engineer

Procore Technologies Inc.
06.2024 - Current
  • Built low latency data streaming workflows by leveraging Databricks and Apache Kafka to build a Delta lakehouse.
  • Reengineered existing ETL workflows to improve performance by identifying bottlenecks and optimizing code accordingly.
  • Working on designing data quality framework to validate data at various stages of the ingestion process.

Sr Data Engineer

Pasarpolis
Bangalore
4 2022 - 05.2024
  • Designed and implemented ETL architecture using Cloud Composer on GCP and documented the process.
  • Created Airflow Workflows for ETL and generating reports
  • Architected a Data archival system and authored base scripts for the python repository.
  • Migrated BigQuery Datasets across regions successfully for Compliance reasons.
  • Setup a real-time data streaming pipeline using managed services like Data Streams and BigQuery.
  • Designed warehouse table for Tableau Dashboard that displays SLA Quartiles and Customer Lifetime values.
  • Working on POC for a cost effective real time data system, focusing majorly on tools offered by AWS and Open Source.
  • Updated ETL logic for Huge tables to bring down the load times by 20 %.

Sr Data Engineer

ADF Data Science
Chennai
04.2021 - 03.2022
  • Orchestrated data pipelines for migrating data from MySQL to AWS Redshift using AWS DMS
  • Authored a script using Python and PySpark for building Historical data from change data capture (CDC) files in AWS S3
  • Authored an ETL workflow in Pentaho to detect daily Schema changes
  • Designed metadata and calendar dimension as per business requirements
  • Installed an open source Data Cataloging tool called Amundsen for data governance
  • Drafted a Python code to validate the counts by parsing Bin Log files of MySQL
  • Enabled data lineage tracking for Spark workloads using Spline which is also an open source tool
  • Designed and Developed data pipeline using Python and Spark on AWS Glue for building Warehouse Tables
  • Have setup monitoring and alerts using AWS CloudWatch to notify on Slack in case of errors.
  • Optimized Analytic queries to bring down the runtime from 2 Hrs to 2 Mins by taking advantage of Redshift data distribution and query tuning techniques.
  • Built a scalable Data Lake on AWS S3 using AWS Glue, Apache Hudi and Lake Formation, which enables easier Data Shares with clients across different AWS accounts without the need for any ETL.
  • Built numerous dashboards on Quicksight and taken care of Admin activities

BI Developer

Wipro
Bangalore
02.2018 - 03.2021
  • Worked as a BI Developer and Data Engineer for Broadcom Inc., Collected and analyzed data requirements from clients
  • Built Data models and ETL workflows using Incorta
  • Data Enrichment using python pandas and Rest APIs
  • Optimized code using Multithreading and Multiprocessing whilst using pandas framework to reduce execution times from over 6Hrs to a couple of hours
  • Developed numerous dashboards on Incorta
  • Authored Python Scripts to automate Dashboard delivery using Incorta's REST API and pandas for data cleaning
  • Developed PTD dashboards on Incorta, which were previously running on Oracle Publisher
  • Optimized these queries for a huge jump in performance
  • Designed and developed a series of dashboards for tracking employee performance at various levels.

Education

Bachelor of Technology - Electronics And Communications Engineering

IIIT, RGUKT
04.2017

Awards

Incorta 4 Fundamentals For Admins And Developers : 2019 

Python Intermediate, HackerRank : 2020 

AWS CDA, Udemy : 2020 

Customer Charmer Award, Wipro : 2021

Personal Information

Title: Senior Cloud Data Engineer

Email: madhanreddy575@gmail.com

Phone: +91-8886893512

Languages

  • English
  • Telugu
  • Hindi

Timeline

Senior Data Engineer

Procore Technologies Inc.
06.2024 - Current

Sr Data Engineer

ADF Data Science
04.2021 - 03.2022

BI Developer

Wipro
02.2018 - 03.2021

Bachelor of Technology - Electronics And Communications Engineering

IIIT, RGUKT

Sr Data Engineer

Pasarpolis
4 2022 - 05.2024
Lingareddy MadhanSenior Data Engineer