Summary
Overview
Work History
Education
Skills
Awards
Timeline
Generic
Lingareddy Madhan

Lingareddy Madhan

Senior Cloud Data Engineer

Summary

Data Engineer with 6 years of experience working on various types of data, different ETL and BI tools. Technically strong in Python, Apache Spark (PySpark) and SQL. Experience across Payments, Finance and Retail domains. Worked on cloud platforms like AWS and GCP. Responsive expert experienced in monitoring database performance, troubleshooting issues and optimizing database environment. Possesses strong analytical skills, excellent problem-solving abilities, and deep understanding of database technologies and systems. Equally confident working independently and collaboratively as needed and utilizing excellent communication skills.

Overview

6
6
years of professional experience
3
3
Languages

Work History

Sr Data Engineer

Pasarpolis
Bangalore, IN
Apr.2022 - Current
  • Designed and implemented the ETL architecture using Cloud Composer on GCP and documented the process.
  • Created Airflow Workflows for ETL and generating reports
  • Implemented CI/CD using GitLab.
  • Architected a Data archival system and authored the base scripts for the python repository
  • Migrated BigQuery Datasets across regions successfully due to compliance reasons.
  • Setup a real-time data streaming pipeline using managed services like DataStreams and BigQuery
  • Designed the warehouse table for Tableau Dashboard that displays SLA Quartiles and Customer Life time values.
  • Working on a POC for a cost effective real time data system, focusing majorly on tools offered by AWS and Open Source.
  • Updated ETL logic for Huge tables to bring down the load times by 20 %.

Sr Data Engineer

ADF Data Science
Chennai, IN
Apr.2021 - Mar.2022
  • Orchestrated data pipelines for migrating data from MySQL to AWS Redshift using AWS DMS
  • Authored a script using Python and PySpark for building Historical data from the change data capture (CDC) files in AWS S3
  • Authored an ETL workflow in Pentaho to detect the Schema changes
  • Designed metadata and calendar dimension as per the business requirements
  • Installed an open source Data Cataloging tool called Amundsen for data governance
  • Drafted a Python code to validate the counts by parsing Bin Log files of MySQL
  • Enabled data lineage tracking for Spark workloads using Spline which is also an open source
  • Designed and Developed data pipeline using Python and Spark on AWS Glue for building Warehouse Tables
  • Have setup monitoring and alerts using AWS CloudWatch to notify on Slack in case of errors.
  • Optimized the Analytic queries to bring down the runtime from 2 Hrs to 2 Mins by taking advantage of Redshift data distribution and query tuning techniques.
  • Built a scalable Data Lake on AWS S3 using AWS Glue, Apache Hudi and Lake Formation, which enables easier Data Shares with clients across different AWS accounts without the need for any ETL.
  • Built numerous dashboards on Quicksight and taken care of Admin activities
  • Lead a team of 2, mentored and monitored daily activities

BI Developer

Wipro
Bangalore, IN
Feb.2018 - Mar.2021
  • Worked as a BI Developer and Data Engineer for Broadcom Inc., Collected and analyzed data requirements from clients
  • Built Data models and ETL workflows using Incorta
  • Data Enrichment using python pandas and Rest APIs
  • Optimized code using Multithreading and Multiprocessing whilst using the pandas framework
  • Developed numerous dashboards on Incorta
  • Authored Python Scripts to automate Dashboard delivery using Incorta's REST API and pandas for data cleaning
  • Developed PTD dashboards on Incorta, which were previously running on Oracle Publisher
  • Optimized these queries for a huge jump in performance
  • Designed and developed a series of dashboards for tracking employee performance at various levels.

Education

Bachelor of Technology - Electronics And Communications Engineering

IIIT, RGUKT
India
Jul.2013 - Apr.2017

Skills

Pythonundefined

Awards

Incorta 4 Fundamentals For Admins And Developers : 2019 

Python Intermediate, HackerRank : 2020 

AWS CDA, Udemy : 2020 

Customer Charmer Award, Wipro : 2021

Timeline

Sr Data Engineer

Pasarpolis
Apr.2022 - Current

Sr Data Engineer

ADF Data Science
Apr.2021 - Mar.2022

BI Developer

Wipro
Feb.2018 - Mar.2021

Bachelor of Technology - Electronics And Communications Engineering

IIIT, RGUKT
Jul.2013 - Apr.2017

Similar Profiles

CREATE PROFILE
Lingareddy MadhanSenior Cloud Data Engineer