Anurag Gautam

Project: Incred Data Platform

• Engineered a resilient data pipeline using AWS DMS , AWS SQS, Python & AWS Lambda handling over around 1 million records daily, Seamlessly ingested user transactions & third party data into delta lake using databricks.
• Build a near real time data pipeline using databricks autoloader, improving data refresh rates from 1 hour batches to ~2-minute intervals significantly increasing data availability.
• Automated hourly data pipeline for sourcing data from the third-party LeadSquared CRM, ensuring data accuracy with stringent data quality checks and metadata management
• Migrated over 100+ TB of historical data, incremental jobs, views, job clusters from Hive Metastore to Databricks Unity Catalogue, enhancing data lineage and implementing improved access controls.
• Developed Due Diligence reports for the finance team during audits and improving audit accuracy by 30% . Designed a dedicated data mart for finance team, enabling stakeholders and investors to make more informed decisions.
• Implemented automated backfills and outage-recovery scripts to maintain data platform stability.
• Contributed to cost optimization initiatives for efficient data platform operations.

Project: Client - Zee 5

• Built a PySpark pipeline to remove PII data from ~17 TB of historical datasets stored in AWS S3. Processed data at scale using Apache Spark and automated execution using Apache Airflow.
• Built a PySpark ETL pipeline to ingest and process incremental OTT subscription data into staging and master tables.
• Managed metadata with AWS Glue Data Catalog and enabled analytics using Hive and Amazon Athena.
• Automated daily batch workflows using Apache Airflow with monitoring and failure recovery

Project: Client - Arrow Electronics

• Supported Teradata to Azure Synapse migration with a focus on data extraction and validation. Ensured data accuracy through schema, count, and data quality checks.