Abdul Kalam
Bengaluru
Summary
Results-driven AWS Data Engineer with expertise in Python and a proven track record at Eli Lilly Services India. Successfully optimized AWS S3 storage, achieving 20% cost savings, while leading the implementation of Infrastructure as Code. Adept at API integration and fostering collaboration, enhancing operational efficiency across teams.
Overview
6
6
years of professional experience
1
1
Certification
Work History
Senior Data Engineer II
Eli Lilly Services India Pvt Ltd
Bengaluru
02.2023 - Current
- AWS S3 Lifecycle Management and Cost Optimization: Implemented automated AWS S3 lifecycle policies for data lifecycle management, transitioning objects from S3 Standard to Glacier (90 days) to Deep Archive (365 days), optimizing archival workflows, and reducing storage costs by approximately 20–35% annually (approximately $5K to $10K savings).
- Agent – ServiceNow Ticket Automation: Built a Retrieval-Augmented Generation (RAG)-based agent integrated with ServiceNow APIs to automatically create and categorize service tickets from natural language queries.
- Framework Development for Asynchronous Job Monitoring: Designed and developed a comprehensive framework to capture the status of asynchronous jobs using API Gateway, AWS Step Functions, Lambda, and DynamoDB, enabling real-time monitoring and streamlined processing.
- Team Leadership in AWS CDK Implementation: Led the team in designing and implementing an AWS CDK framework to provision critical services, including Lambda, Glue, and Redshift. IAM, Cognito, and API Gateway. In alignment with Infrastructure as Code (IaC) best practices.
- Unified AWS Account Framework: Developed an innovative framework that consolidated three separate AWS accounts into a unified repository. This initiative significantly improved management and deployment processes, boosting collaboration, security, and operational consistency across the organization.
- API Security Enhancement: Engineered a robust API that retrieves data from third-party sources, and implemented a Cognito user pool using AWS CDK to validate JWT tokens. Thereby strengthening APL security.
- Infrastructure as Code Implementation: Led the setup of AWS deployment infrastructure within a GitHub repository, ensuring streamlined and repeatable deployments.
- Data Transformation and Ingestion: Developed PySpark scripts to read data from data lakes. Perform complex transformations, and ingest the processed data into Amazon Redshift, aligned with star schema modeling.
- Orchestration Automation: Developed AWS Step Functions to orchestrate complex workflows, ensuring efficient and reliable process automation.
- Role and Permission Management: Created OlDC roles, federated roles, and service roles, meticulously defining and assigning the necessary permissions to ensure secure and compliant access control.
- Automated Input File Quality Control: Spearheaded the development of an automated QC process for input files, which reduced manual effort by approximately 50% by eliminating the need to load files into Hive and manually execute SQL queries.
Data Engineer
Deloitte (Lincoln Financials)
Bangalore
02.2021 - 06.2021
- Spearheaded migration from Informatica BDM to AWS Cloud Services, developing ETL scripts for Amazon Redshift.
- Created data profiling tool that assessed source file integrity and quality.
- Developed automation scripts that reduced manual validation efforts for QA team by 70%.
- Collaborated with onshore DevOps team to gather project requirements, ensuring alignment with organizational standards.
- Built ETL transformation logic using PySpark in Glue to optimize data ingestion processes.
- Proposed adoption of AWS services like Glue, Secret Manager, and Lambda for enhanced operations.
- Implemented modular Python scripts to streamline data processing from various sources into datamart.
- Automated API-based ingestion process, reducing manual effort by 80% through conversion to Python script.
Big Data Engineer
Cognizant (Royal Bank of Canada)
Chennai
12.2019 - 02.2021
- Developed SQL logic to validate data processing accuracy.
- Ingested bulk data into IDP tool, leveraging architectural models for efficient processing.
- Conducted unit testing to ensure comprehensive application functionality using DICE Framework.
- Debugged and processed jobs to meet specified output requirements.
Education
B.E - Electronics and Commnication Engineering
PSG COLLEGE OF TECHONOLOGY
Coimbatore01.2019
Skills
- Python
- SQL
- PySpark
- AWS
- S3
- Kafka
- Iceberg
- Generative AI
- Glue
- Redshift
- DynamoDB
- Lambda
- API Gateway
- ETL
- Data Modeling
- Data Pipeline
- Data Profiling
- Business Intelligence
- Star Schema
- Data Warehouse
- ETL Pipeline
- Infrastructure as Code
- AWS CDK
- CFT
- Pandas
- PostgreSQL
- MySQL
- NoSQL
- API integration
- Infrastructure as code
Certification
- Databricks Generative AI Associate
- AWS Cloud Certified Practitioner
- Databricks - Gen AI Fundamentals
- Data Analysis and Visualization: Pandas, Matplotlib
Timeline
Senior Data Engineer II
Eli Lilly Services India Pvt Ltd
02.2023 - Current
Data Engineer
Deloitte (Lincoln Financials)
02.2021 - 06.2021
Big Data Engineer
Cognizant (Royal Bank of Canada)
12.2019 - 02.2021
B.E - Electronics and Commnication Engineering
PSG COLLEGE OF TECHONOLOGY