Proficient in data integration and ETL processes, specializing in Azure Data Factory and PySpark. Committed to delivering efficient data solutions through effective problem-solving and technical proficiency.
Executed data extraction and integration from diverse sources into Azure Data Lake using ETL pipelines.
Migrated data from on-premises systems to Azure utilizing Azure Data Factory for seamless transitions.
Developed and tuned ADF copy activities and transformations with Mapping Data Flows for enhanced efficiency.
Converted data into optimal formats, ensuring effective reads, memory usage, and calculated key metrics.
Implemented Spark in Databricks with Python to improve processing speed using Data Frames and Spark SQL API.
Enhanced performance of existing algorithms in Hadoop by employing Spark Context, Spark SQL, and Data Frames.
Created dynamic pipelines via parameterization and control tables to meet project requirements effectively.
Applied business logic with PySpark in Databricks and ADF, delivering tailored data solutions.