Results-driven Azure Data Engineer with proven expertise at Atos Eviden, specializing in ETL processes and data governance. Successfully built ingestion pipelines using Azure Data Factory and optimized performance with Pyspark. Adept at leveraging SQL for data manipulation, demonstrating strong analytical skills and a commitment to data integrity.
• Built ingestion pipelines using Azure Data Factory and scheduled using the same.
• Used Azure Databricks to perform the transformations on the data.
• Used Unity Catalog for the data governance.
• Used PySpark intensively for the performance optimization.
• Used Python to create and modify trigger scripts.
• Used SQL for data manipulation and to build databases.
• Used ADLS for data storage.
• Worked intensively with Delta tables.
Manulife - Data Ingestion, Transformations and enhancements
I was responsible for ingesting data from global financial institutions using Azure Cloud Services, ensuring seamless integration of various financial data sources. My role involved curating this data to meet business requirements, focusing on risk management within the BFSI (Banking, Financial Services, and Insurance) domain. The project aimed to streamline the management of financial data, improve data quality, and enhance the accuracy of risk analysis models for business-critical applications.
SunTrust Bank, 04/01/18 - 02/28/21, PySpark, Python, Hive, Shell Scripting, Impala, TWS, Oozie, Created Hive tables and worked on them using Hive QL, which will automatically invoke and run MapReduce jobs in the backend., Managing and scheduling batch Jobs on a Hadoop Cluster using Oozie., Experienced in loading and transforming large sets of structured, semi-structured and unstructured data.,
Bank of America (BOFA), 03/01/16 - 03/31/18, Spark, Hive, Shell Scripting, Impala, TWS, Oozie, Responsible for building distributed data solutions using Hadoop. Experienced in loading and transforming large sets of structured, semi-structured and unstructured data Hadoop concepts.,