Pawan Rahwani

Data Engineer / Database Engineer

• Designed, developed, and maintained end-to-end ETL pipelines to extract data from multiple source systems and load it into centralized analytics platforms.
• Built and optimized data models using 3rd Normal Form (3NF) and dimensional modeling techniques to support enterprise reporting and BI use cases.
• Developed and maintained DBT Cloud models and transformations, enabling modular, version-controlled, and automated data pipelines.
• Optimized SQL queries and ETL scripts on Amazon Redshift, improving query performance and scalability for large datasets.
• Analyzed complex data issues, performed root-cause analysis, and implemented corrective actions to improve data accuracy and consistency.
• Monitored ETL workflows and data platform performance, proactively identifying and resolving failures and performance bottlenecks.
• Collaborated with data architects, analysts, and business stakeholders to translate business requirements into scalable data solutions.
• Implemented data security controls, backup and recovery strategies, and compliance-aligned data handling processes.
• Created and maintained technical documentation for ETL pipelines, data models, and transformation logic.
• Designed and implemented a GDPR-compliant data pipeline to perform hard deletes across bronze and silver layers, ensuring complete removal of sensitive data for all entities.
• Developed and validated comprehensive test cases to ensure data deletion accuracy, pipeline reliability, and regulatory compliance.
• Conducted in-depth analysis of Apache Hudi write mechanisms, cleanup policies, and configuration parameters to optimize data writing and data lifecycle management on Amazon S3.
• Analyzed AWS Glue job performance for high-volume data ingestion from Kafka (MSK) and implemented repartitioning strategies to optimize data writing throughput and processing efficiency.

BIG DATA & ANALYTICS

• Designed and optimized PySpark applications on Azure Databricks to process large-scale datasets and support advanced analytics.
• Improved PySpark job performance through tuning and optimization, achieving up to a 30% reduction in runtime.
• Built automated data quality checks and validation frameworks, reducing manual effort and improving reliability.
• Developed and implemented a COVID risk and human mobility analytics model for the NSW (Australia) government, achieving 92% accuracy while reducing compute resource usage by 85%; later adapted the solution for the New Zealand government to monitor tourist movement.
• Designed and implemented data quality evaluation processes using metrics such as ping density and tile density to assess external partner data, reducing manual validation efforts by 60%.
• Developed data processing solutions for analytics use cases including mobility analysis, audience segmentation, and campaign reporting.