Hi, I’m
Atul Sharma
Seasoned Data Engineering Manager & Architect
Noida,Uttar Pradesh
Summary
Senior Data Engineering Manager with 12+ years of expertise, specializing in architecting high-impact data platforms and leading large-scale teams. I led the design and implementation of one of Asia's largest and most scalable Data Lakes, transforming raw data into critical business assets. I am a hands-on leader skilled in building and mentoring high-performing engineering teams from the ground up, ensuring operational excellence and continuous innovation. Currently, I am driving cloud-native data strategy across AWS, GCP, and Azure (including hybrid Cloudera migration) while pioneering the integration of AI/ML and Data Observability platforms. I turn data strategy into cost-efficient, production-ready solutions
Overview
12
years of professional experience
2
Languages
Work History
Nielsen Media
Senior Data Engineering Manager / Architect
09.2024 - Current
Job overview
- Key Projects – Audience Measurement - Television(National & Local), Streaming Platform, Fulfilment Center, Universe Estimates
- Strategic Leadership (Data Mesh) - Led a 40+ team of Mid-to-Principal engineers, defining the data strategy and architecting a Data Mesh-inspired platform for the critical US Audience Measurement product to establish domain ownership and data-as-a-product.
- Cloud Architecture & DevOps (IaC) - Architected the org's scalable data platform (EMR, S3, Glue, Redshift) by implementing Infrastructure as Code (Terraform) and CI/CD pipelines, automating 90% of infrastructure provisioning and enforcing architectural consistency.
- Cost Optimization & Cloud-Native Ops (EKS) - Drove a 50% AWS cost reduction ($75k/month) for Local TAM by leading the strategic migration of Spark workloads to EMR on EKS, optimizing pod-level resource allocation and auto-scaling.
- Pipeline Intelligence & Observability (AI) - Engineered resilient real-time (Kafka, Spark Streaming) and batch (Airflow) pipelines processing 50 TB/day; slashed data downtime by 40% by implementing a Data Observability platform (Monte Carlo/Soda) to proactively detect data quality anomalies.
- Stakeholder & MLOps Enablement (AI) - Partnered with Data Science stakeholders to build the company's first MLOps framework, architecting reusable data pipelines and deploying a Databricks Feature Store to accelerate model training and deployment.
Jio Platforms Limited
Senior Data Engineering Manager / Architect
03.2019 - 09.2024
Job overview
- Key Projects – Real Time Event & Analytics Platform, JioCinema – IPL Analytics, Customer 360, Reliance Hospital Analytics, Jio Financial Services Analytics, On premise&Cloud Migrations for 1000+ ETL Jobs and Dashboards
- Strategic Leadership & Team Building: Built and scaled a high-performing team of 30+ Mid-to-Senior engineers from scratch, owning the end-to-end design, development, and maintenance of all data pipelines, infrastructure, and delivery.
- Data Mesh Architecture: Architected and implemented a decentralized, large-scale Data Mesh platform to drive domain-oriented data ownership, leveraging Apache Spark for compute, Kafka for real-time ingestion, and Grafana for platform-wide observability.
- High-Volume, Real-Time Processing (Cost & Performance): Engineered and managed a high-throughput data platform for the IPL Cricket League, processing 2.5 million TPS (30 TB/day).
- Cloud & Pipeline Optimization: Achieved a 30% reduction in data processing time and a 70% cloud cost reduction on the IPL platform through aggressive Spark tuning, strategic EMR/Dataproc architecture choices, and optimized resource management.
ZS Associates
Associate Consultant
01.2018 - 02.2019
Job overview
- Architected the central ingestion framework for the enterprise Data Lakehouse (Delta Lake/Iceberg)
- Leading a 10-engineer team to integrate 20+ disparate data sources (Kafka, APIs, RDBMS). Standardized deployments using Terraform (IaC) and Airflow, enabling self-service onboarding for new data domains in alignment with our Data Mesh strategy.
Wunderman Thompson(a Salmon Company Before)
Associate System Analyst
02.2016 - 01.2018
Job overview
- Executed the end-to-end strategic migration of legacy on-premise Hadoop/Cloudera clusters to a cloud-native AWS platform (EMR, S3, Redshift). This program automated 100% of the new infrastructure using Terraform (IaC), resulting in a 40% reduction in TCO and simultaneously upskilling the entire engineering team on modern cloud data services.
Cognizant
Programmer Analyst
02.2014 - 01.2016
Job overview
- Development on Enterprise Data Lake on the Hadoop ecosystem (Hortonworks), enabling large-scale advanced analytics. This platform standardized Apache Spark (Scala/Python) and Hive for batch processing, Kafka for real-time ingestion, and Oozie for workflow orchestration, replacing legacy ETL systems.
Education
Uttar Pradesh Technical University(UPTU)Lucknow
Bachelor of Technology from CSE
07-2013
Skills
Cloud Platforms & Data Ecosystems (Strategic and Server-less)
AWS: EMR(EC2/EKS), Redshift, S3, Glue, Kinesis, MSK, Lambda, Cloudwatch
GCP: BigQuery, Dataflow, DataProc, Pub/Sub, Vertex AI, GCS, Cloud Composer
Azure: Synapse, Databricks, Data Factory, Event Hub
Lakehouse/Warehouse: Databricks, Snowflake, Delta Lake, Iceberg
On Premise: Apache Ecosystem with CDP and HDP
Data Architecture & Modelling (Foundational Design)
Architecture Patterns: Data Mesh, Data Fabric, Data Lakehouse, Data Warehousing
Data Modeling: Dimensional Modeling (Star/Snowflake), Data Vault, Conceptual, Logical and Physical Data Modelling
Database Design: SCD, Indexing, Partitioning, Query Optimisation
Databases: PostgreSQL, MySQL, Hive, NoSQL(MongoDB, CassandraDB, HBase, CosmoDB)
BigData Engineering & Core Compute
Timeline
Senior Data Engineering Manager / Architect
Nielsen Media
09.2024 - Current
Senior Data Engineering Manager / Architect
Jio Platforms Limited
03.2019 - 09.2024
Associate Consultant
ZS Associates
01.2018 - 02.2019
Associate System Analyst
Wunderman Thompson(a Salmon Company Before)
02.2016 - 01.2018
Programmer Analyst
Cognizant
02.2014 - 01.2016
Uttar Pradesh Technical University(UPTU)
Bachelor of Technology from CSE