Osama Shahid
SENIOR DATA ENGINEER | AWS AZURE DATABRICKS AI-READY DATA PLATFORMS
New Delhi
Summary
Results-driven Cloud Data Engineer with over 3 years of experience in designing scalable, fault-tolerant data platforms on AWS and Azure. Expertise includes distributed Spark processing, metadata-driven ETL/ELT pipelines, serverless architectures, and CI/CD automation. Proven track record in creating production-grade data solutions that enhance reliability and performance, utilizing tools such as AWS Glue, Lambda, Azure Databricks, and Airflow.
Overview
4
4
years of professional experience
2
2
Certifications
Work History
Senior Data Engineer
Tata Consultancy Services (TCS) Ltd.
11.2022 - Current
- Architected and migrated 15+ TB of biomedical and commercial data from on-prem SQL Server to AWS Aurora PostgreSQL using CDC-based ETL/ELT pipelines with AWS Glue, DMS, and Step Functions, achieving less than 5 minutes downtime.
- Built distributed Spark and AWS Glue pipelines processing millions of records daily, improving throughput by ~35% through partition optimization, query pushdowns, and parallelism tuning.
- Developed near-real-time ingestion and AWS Lambda-based automation workflows for Lilly Data Marketplace platform using EventBridge-driven orchestration and metadata-based routing, enhancing data accessibility and responsiveness.
- Designed reusable schema evolution, metadata-driven ingestion, and enterprise data quality frameworks, reducing schema-related production incidents by ~90%, ensuring high reliability of downstream data.
- Implemented CI/CD automation using GitHub Actions and CloudFormation across multi-account AWS environments, reducing deployment cycles from days to under 30 minutes.
- Developed Azure Databricks notebook-based processing workflows using Medallion Architecture and Unity Catalog, orchestrated through Azure Data Factory (ADF) for scalable enterprise analytics.
- Led and mentored team of 4 engineers, collaborating with enterprise stakeholders to implement scalable data platform enhancements for Eli Lilly, fostering team growth and project success.
Education
Bachelor of Technology - Petroleum Engineering
Graphic Era University
Dehradun, Uttarakhand06-2022
Skills
- Languages: Python, PySpark, JavaScript, SQL
- Distributed data processing
- Cloud data platforms
- Python programming
- Data pipeline design
- ETL development
- Data Warehousing
- Workflow automation
- Data engineering
Advanced SQL
API development
Performance tuning
Git version control
Data quality assurance
Spark development
Certification
Microsoft Certified: DP-104 – Microsoft Azure Database Administrator
Timeline
Senior Data Engineer
Tata Consultancy Services (TCS) Ltd.
11.2022 - Current
Bachelor of Technology - Petroleum Engineering
Graphic Era University
Core Skills
- Core Frameworks: DQ
- Metadata
- Schema Evo