Senior Data Engineer with 9+ years of experience as a Business Intelligence Developer, Data Engineer, Cloud Migration Specialist and Solution Architect on prem, AWS, GCP,Data Warehouse concepts with different data modelling architecture. Ability to handle priorities, lead and manage time effectively in a high-pressure environment on tight deadlines without compromising on quality.
Overview
10
10
years of professional experience
2016
2016
years of post-secondary education
3
3
Certifications
Work History
Associate Architect
Valuelabs
12.2023 - Current
Provided technical support during sales calls and delivered solution presentations focused on data integration, processing, and analytics on GCP and AWS.
Developed an automated solution to extract and standardize files (XLSX, XLS, CSV, TXT) attached to Jira issues via REST API, storing them in GCS and processing them for list matching and analysis.
Built centralized marketing data pipelines by ingesting campaign data from diverse sources (APIs, databases, flat files, SFTP) into BigQuery, using Apache Airflow for orchestration and implementing email verification workflows.
Fixed critical data bugs and ensured data integrity across complex cloud-based pipelines, significantly enhancing data reliability.
Improved performance by implementing BigQuery partitioning and clustering, resulting in faster query times and reduced compute costs.
Led cloud migration initiatives, moving analytics workloads from Amazon Redshift to BigQuery and dashboards from Looker to Grafana.
Delivered a POC using Databricks to demonstrate scalable data processing and showcasing capabilities in PySpark, Delta Lake, and MLflow for advanced data transformation and machine learning pipelines.
Data Engineering Specialist
Accenture Strategy and Consulting UK and Ireland
08.2022 - 11.2023
Design the cloud solution for migration like data pipeline, schema handling, historical data load.
Identify best methods of migration (manual and automated) and define cloud services, architecture, and environments for hosting the Solution
Implemented new design model for existing data warehouse and saves cost by 20% by improving the performance in Redshift.
Taken ownership of the IAM requirements across the platform to set up personas in GCP.
Automated the data catalog for policy tagging to the columns and user role using Terraform.
Provided technical leadership and delivered innovative ideas for data modernization on AWS using Glue, Glue Brew, Crawler and Event Bridge, Cloud formation, Redshift and Lambda
Tech Lead
Harman Connected Service
04.2022 - 08.2022
Migration from Informatica to AWS cloud using different services.
Developed automated script to handle good and bad files, data pipeline from various sources using spark, Redshift, s3, lambda, ECS, cloudwatch, Glue, python orchestrated through step function.
AWS Data Engineer and Solution Architect
Deloitte Touche Tohmatsu India LLP
12.2020 - 04.2022
Implemented spark framework using Glue and EMR
Set up IAM policies and roles for users and services Have done POC for AWS services like Glue Brew, Data Lakehouse
Build an asset to handle different file format in S3 to store meta data information of the table using glue crawler and data catalog, validating the data using Athena
Estimated the cost for the client using AWS Migration Acceleration Program funding for 5 years which helped them getting insight on their budget.
Have worked on multiple RFPs to support company for new project proposals.
Software Engineer
Tech Mahindra
10.2015 - 12.2020
Supporting on-prem legacy data warehouse in informatica and helped client to migrate to AWS cloud using AWS Data Migration Service and Elastic Container services from oracle 9g and 11g in cloudera
Implemented data pipeline to analyse network data using s3, Athena, Lamba and Quicksight for 3/4G data
Have performed data cleansing and data transformation using Dataiku
Performed POCs on the migration of on-prem oracle DB to AWS using AWS SCT agent and Redshift
Maintained old warehouse in Informatica and SAP BO.
Single-handedly done POC's on reporting tools like Superset, Looker, Tableau, Qliksense based on the performance, self service and cloud integration using Redshift, Athena and S3.