Shanmugavel Murugaperumal
Senior Data Engineer
Tirunelveli
Summary
Senior Data Engineer with 8+ years of expertise in AWS and Databricks, specializing in designing scalable data architectures, developing ETL operations, managing data lakes, and ensuring data quality. Proficient in data warehousing, analytics, and orchestrating complex data workflows, with a strong focus on data governance and real-time streaming analytics. Experienced in crafting and executing high-performance ETL pipelines and supporting data-driven strategies through innovative data engineering practices.
Overview
8
8
years of professional experience
5
5
Certifications
Work History
Senior Data Engineer
Koantek Cloud and AI Services
04.2024 - Current
- Managing the transition of data from AWS DMS to Databricks for the Instant Cash Loans platform, facilitating smooth data operations and effective communication between various technical teams
- Leveraged Delta Lake, Unity Catalog, and Medallion Architecture to effectively organize and manage incremental data ingestion within the Lakehouse Platform, ensuring a robust and scalable data infrastructure
- Utilized Databricks Autoloader to efficiently load incremental data into the database, automating data ingestion and ensuring seamless updates for a dynamic data architecture
- Deployed Change Data Capture – CDC strategies to track and process incremental data changes, enhancing data responsiveness and maintaining up-to-date data across systems
- Designed and scheduled Databricks workflows to automate and orchestrate data loading processes, ensuring smooth and reliable data integration and processing across the platform
- Client- Adani
Data Engineering Consultant
Accenture Solutions Private Limited
07.2021 - 04.2024
- Worked within the realm of Investment Banking, tasked with aggregating collaborative data from diverse sources into the data warehousing platform, aiming to produce reports that facilitate informed and strategic business decision-making
- Construct a Data Lake in Amazon S3 within an account and structure the lake utilizing AWS Lake Formation, enabling seamless data consumption across various consumer AWS accounts
- Utilized Terraform to build the necessary infrastructures and developed a Jenkins-based CI/CD pipeline for enhanced usability and efficiency
- Enhanced the efficiency of Spark Jobs by implementing optimizations for seamless execution within the cluster
- Client- JP Morgan & Chase
Data Engineer
VueData Technologies
05.2020 - 06.2021
- Worked within a retail-oriented domain, conducting in-depth analysis of product sales performance to extract actionable insights aimed at improving overarching sales strategies
- Contributed to AWS pipeline creation and streamlined workflows through ETL assignments, ensuring the efficient migration of data from MySQL to Amazon RedShift via AWS Glue
- Utilized the Spark framework with Python to implement transformations and actions on the data source, subsequently storing the outcomes in Amazon RedShift
- Client- Relief Factor
Systems Engineer
Tata Consultancy Services
12.2017 - 05.2020
- Contributed expertise within a retailer-focused domain, facilitating the migration of data from an on-premises database to the AWS data warehouse
- Conducted comprehensive analysis on extensive datasets hosted in Amazon Web Services(AWS) utilizing Redshift
- Developed a PySpark script to facilitate on-demand sharing of sales data analysis with users in an ad-hoc manner
- Client- Woolworths
Education
Bachelor Of Technology -
Rajalakshmi Engineering College
Chennai05-2017
Skills
AWS - S3, Redshift, Glue, Athena, Lambda, Step Functions,IAM
Databricks - Auto Loader, Delta Lake, Unity Catalog, Databricks Jobs, Cluster Management
Big Data
SQL
Hadoop
Hive
Apache Spark
Python
Terraform
BitBucket
Git
Jenkins
Data Governance - Immuta
Certification
Databricks Certified Data Engineer Professional
AWARDS
- Honored with the prestigious Ace Award for 2023 at Accenture, recognizing exceptional performance and outstanding contributions to innovative projects and team success.,
- Awarded the Star Employee of the Year in 2020 at TCS for exemplary achievements, demonstrating exceptional skills and dedication to driving key initiatives and delivering impactful results.
Timeline
Senior Data Engineer
Koantek Cloud and AI Services
04.2024 - Current
Data Engineering Consultant
Accenture Solutions Private Limited
07.2021 - 04.2024
Data Engineer
VueData Technologies
05.2020 - 06.2021
Systems Engineer
Tata Consultancy Services
12.2017 - 05.2020
Bachelor Of Technology -
Rajalakshmi Engineering College