Senior Data Engineer
Santhosh Kumar
Cloud Destinations
Ooty
4
years of professional experience
1
Certification
- Senior Data Engineer and Databricks Certified Data Engineer Professional with nearly four years of experience delivering cloud data solutions using Databricks, PySpark, SQL, and Delta Lake.
- Experienced in developing batch, incremental, and streaming ETL/ELT pipelines using Medallion Architecture and Delta Live Tables.
- Strong foundation in data modeling and business transformations, including Delta upsert, SCD handling, latest-record deduplication, and partitioning.
- Delivered $420K+ in annual Amazon S3 savings, reducing daily storage costs by ~48% through ZSTD compression and Delta Lake optimization.
- Built reusable, configuration-driven ingestion frameworks for APIs, JSON, CSV, Parquet, and campaign-event spreadsheets.
- Hands-on experience with Unity Catalog, Delta Sharing, and secrets management, alongside encryption.
- Automated Databricks Workflows deployments using Asset Bundles, GitHub Actions, and Terraform, with scheduling, production monitoring, and Slack alerts.
- Developed validation and reconciliation frameworks, reducing manual verification effort by ~60% and supporting zero-downtime legacy-to-Databricks cutovers.
- Led a three-person engineering team through an Astronomer Hybrid-to-Hosted migration covering 7 workspaces and 45 production deployments.
- Built conversion agents supporting ongoing migrations of 600+ Airflow pipelines to Amazon MWAA.
- Migrated Legacy system to Databricks, reducing runtimes and query-processing costs by 60%.
- Partnered with client stakeholders to deliver production solutions, including user–campaign Gold-layer models and validated daily data for Marigold Sailthru.
Work History
Senior Data Engineer
7 Months- Delivered over $420K in annual S3 savings, reducing daily storage costs by approximately 48% through ZSTD compression, Delta Lake OPTIMIZE, VACUUM, and Z-ORDER.
- Designed and implemented migration and transformation logic for legacy-to-Databricks pipelines, achieving zero downtime cutovers with reconciliation and rollback controls.
- Provisioned Databricks clusters using Terraform, and automated deployments through Databricks Asset Bundles and GitHub Actions CI/CD.
- Administered Unity Catalog governance, and managed Delta Sharing and Databricks secrets for secure data access.
- Led a team through an Astronomer Hybrid-to-Hosted migration, covering seven workspaces and 45 production deployments.
- Built a conversion agent supporting the migration of over 600 Airflow pipelines to MWAA, delivering converted code for team deployment and testing.
- Collaborated with client stakeholders on solution design, production releases, and incident resolution, addressing IAM, and cross-account access issues through Terraform changes.
- Led design of scalable ETL pipelines using Spark, Python, and SQL for reliable data delivery
- Developed cloud data solutions across AWS services, optimizing storage, processing, and integration workflows
Data Engineer
11 Months- Developed Databricks ETL/ELT pipelines using PySpark, SQL, and Delta Lake, implementing Bronze, Silver, and Gold layers within a Medallion Architecture.
- Worked with streaming pipelines and Delta Live Tables (DLT) for Databricks data processing.
- Implemented Delta upserts, SCD handling, latest-record deduplication, and partitioning for reliable historical and incremental processing.
- Designed user–campaign gold-layer models, delivering validated data to Marigold Sailthru for event campaigns.
- Built reusable ingestion frameworks for APIs, JSON, CSV, Parquet, and campaign-event spreadsheets, incorporating configurable business transformations.
- Used Databricks Git folders for notebook version control, and configured Jobs/Workflows schedules, success/failure notifications, and Slack alerts.
- Developed reusable schema validation, reconciliation, and end-to-end testing frameworks, reducing manual verification effort by ~60%.
- Implemented AES encryption and PII anonymization, and supported production troubleshooting, Delta recovery, and data reloads.
- Built scalable data pipelines for cloud and on-premises sources using SQL, Python, and ETL frameworks.
- Validated data quality with profiling checks, transformation rules, and exception handling routines.
- Fine-tuned query performance and optimized database structures for faster, more accurate data retrieval and reporting.
Associate Data Engineer
2 Years 2 Months- Developed AWS Glue ETL jobs using PySpark to transform raw S3 data into partitioned Parquet datasets.
- Configured Glue crawlers and Data Catalog tables to maintain schemas and partition metadata for Athena analytics.
- Contributed to the Snowflake-to-BigQuery migration, implementing GCS ingestion, SQL stored procedures, CDC logic, and timestamp-based incremental loading.
- Optimized selected SQL workloads from approximately 60 seconds to 3–5 seconds, reducing associated query-processing costs by 60%.
- Performed source-to-target validation, investigated data discrepancies, and documented pipeline logic, mappings, and troubleshooting procedures.
- Built and maintained ETL pipelines for reliable ingestion and transformation across cloud data platforms.
- Automated repeatable workflows using Python and orchestration tools to improve processing consistency.
Education
Master of Science - Computer Science with Data Analytics
Bachelor of Science - Computer Technology
Skills
Databricks: Azure Databricks
Delta Lake
Delta Live Tables (DLT)
Jobs/Workflows
Git folders
Genie
Languages: Python
SQL
Data Processing: PySpark
Apache Spark
Spark SQL
pandas
streaming pipelines
ETL & Data Engineering: ETL/ELT
Medallion Architecture
data modeling
CDC
SCD
incremental ingestion
AWS Glue
Cloud & Storage: AWS (S3
EC2
Athena
EMR)
Azure Data Lake Storage
Google Cloud Storage
Data Warehousing: BigQuery
Snowflake
Amazon Redshift
Orchestration: Apache Airflow
Astronomer
Amazon MWAA
Governance & Security: Unity Catalog
Delta Sharing
Databricks secrets management
CI/CD & Infrastructure: Databricks Asset Bundles
Terraform
GitHub Actions
Git (GitHub
GitLab)
Monitoring & Alerting: Amazon CloudWatch
Databricks job monitoring
success/failure alerts
Slack notifications.
Accomplishments
- Delivered $420K+ in annual Amazon S3 cost savings, reducing daily storage costs by approximately 48% through ZSTD compression and Delta Lake optimization.
- Reduced manual data verification effort by approximately 60% by developing a reusable framework for schema validation, reconciliation, and recovery.
- Executed zero-downtime migrations of legacy pipelines to Databricks across multiple regions, with integrity checks and rollback readiness.
- Reduced SQL execution time from approximately 60 seconds to 5 seconds and query-processing costs by 60% through incremental loading and SQL optimization.
- Led a three-person engineering team through an Astronomer Hybrid-to-Hosted migration covering 7 workspaces and 45 production deployments.
Certification
Databricks Certified Data Engineer Professional
Certifications
10-2026
Timeline
Senior Data Engineer
Cloud Destinations
04.2026 - CurrentRead More
Data Engineer
Cloud Destinations
04.2025 - 03.2026Read More
Associate Data Engineer
Cloud Destinations
01.2023 - 03.2025Read More
Dr. N.G.P. Arts And Science College
Master of Science from Computer Science with Data Analytics
Read More
Dr. N.G.P. Arts And Science College
Bachelor of Science from Computer Technology
Read More