Anjali P
Bengalore
Summary
Experienced Azure Data Engineer with over 4.4 years in the IT industry, seamlessly integrating a robust background in software engineering with specialized proficiency in leading Azure technologies. Demonstrates expertise in Azure Services, including Azure Data Factory, Azure Databricks, Azure SQL Database, Azure Stream Analytics, Azure Cosmos DB, Azure Synapse Analytics, and Azure Data Lake Storage. Additionally skilled in various storage solutions, such as Key Vault, and well-versed in Python, SQL, PySpark, and Shell scripting. Possesses an in-depth understanding of Spark Architecture, Spark Core, Spark SQL, Data Frames, and related components crucial for effective Azure Data Engineering. Proven track record in designing and developing robust Data Engineering solutions, encompassing Data warehousing, ETL (Extract, Transform, Load), and data modelling and architecture. Successfully handles diverse datasets, demonstrating hands-on experience with various file formats like JSON, Parquet, CSV, and ORC within the dynamic Databricks environment. Engineered Azure Data Factory pipelines to facilitate seamless data transformation from Blob Storage to Azure Data Lake. Solid knowledge base in Azure cloud platforms, coupled with practical experience in CI/CD pipelines and Agile methodologies. As an Azure Data Engineer, brings a wealth of experience in key areas, including SQL Server Integration Services (SSIS), ensuring the successful execution of Azure Data Engineering projects.
Overview
5
5
years of professional experience
1
1
Certification
Work History
Software Consultant
EPERGNE SOLUTIONS
Remote
04.2024 - Current
- Worked as Data Engineer with BRITISH PETROLEUM (BP)
- Served as primary liaison between software customers and development team, relating feedback and concerns for future patch cycles.
- Presented product demonstrations to clients, showcasing features and benefits that met their specific needs.
- Designed, tested, installed and monitored new systems.
- Provided quality assurance testing for pre-release software through alpha and beta cycle development channels.
Azure Data Engineer
Quess Corp Limited
Bengalore, India
01.2023 - Current
- This is the ingestion, migration and data transformation Project
- Used PySpark and Azure Ecosystem like (Databricks, Data factory, Key-vault, ADLS etc.) Tools: Azure Databricks, Azure Data factory, Azure Key-Vault, Workflows, Python, PySpark, SQL
- Worked with the program of Instakart data ingestion from offshore and working closely with stakeholders for design, development and supporting the application
- Extract Transform and Load Data from Azure Data Lake (ADLS) and processing the data in Azure Databricks
- Worked on multiple file formats for Analyzing & transforming the data to uncover insights into the customer usage patterns
- Experience in developing Spark applications using Spark-SQL in Databricks for data extraction, transformation, and aggregation
- Hands on experience on developing Spark- SQL queries in creating temporary tables
- Implemented business logic as per the requirement using Databricks with PySpark and Spark
- Inside the notebooks we can write transformation logic by using PySpark and schedule that notebooks for execution
- Worked on data movement to different layers in Data Lake storage
- Implemented a data pipeline to ingest the data incrementally for relational sources
- Created trigger for different sources and different pipelines like event based, scheduled trigger In Azure Data Factory
- Worked on design and implementation of PySpark based framework
- Created trigger for different sources and different pipelines like event based, scheduled trigger In Azure Data Factory
- Worked on creating NSG rules, azure key vaults and deploying applications using azure Devops
- Worked according to confluence pages for documentation.
Azure Data Engineer
Shadowfax Technologies Private Limited
Bengalore, India
09.2022 - 01.2023
- This is the ingestion, migration and data transformation Project
- Used PySpark and Azure Ecosystem like (Databricks, Data factory, Key-vault, ADLS etc.) Tools: Azure Databricks, Azure Data factory, Azure Key-Vault, Workflows, Python, PySpark, SQL
- Worked as Data Engineer to store, process and manage huge amount of data collected from various sources
- Worked with internal and external team to understand, design and develop the required solutions
- Implemented spark based multiple joins framework to join multiple facts and dimensions table based on join type and join condition
- Implemented business logic as per the requirement using Databricks with PySpark and Spark
- Developed the spark code to process the parquet files and dumped delta table to delta lake
- Worked on using spark Data Frame API to complete Data manipulation within spark session
- Job Scheduling by using Databricks Workflows
- Developed PySpark scripts for data load and data transformations
- Created trigger for different sources and different pipelines like event based, scheduled trigger In Azure Data Factory
- Worked on design and implementation of PySpark based framework
- Resolved defects/bugs, environmental issues during QA testing, pre-production, and production
- Involved in CI/CD pipeline using tools git.
Data Engineer
Radiiuscard Solutions Private Limited
Bengalore, India
09.2019 - 09.2022
- Implemented spark based multiple joins framework to join multiple facts and dimensions table based on join type and join condition
- Implemented business logic as per the requirement using Databricks with PySpark and Spark
- Developed the spark code to process the parquet files and dumped delta table to delta lake
- Troubleshoot and resolve database issues: Monitor database performance and identify potential problems
- Troubleshoot and resolve database issues, such as performance bottlenecks and data corruption
- Implement solutions to prevent future issues
- Performance Optimization: Conduct performance tuning and optimization of spark and sql queries to enhance database responsiveness
- Implement and manage indexing strategies to improve query performance.
Education
Bachelor of Arts -
Sree Vinayaka Degree College
01.2017
Skills
- PyCharm
- Jupiter Notebook
- VsCode
- Notepad
- Windows
- Linux
- MySQL
- Oracle
- SQL
- Python
- PySpark
- JIRA
- Git
- Azure DevOps
Certification
- DP-900: Microsoft Azure Data Fundamentals
- DATABRICKS: Lakehouse Fundamentals
Timeline
Software Consultant
EPERGNE SOLUTIONS
04.2024 - Current
Azure Data Engineer
Quess Corp Limited
01.2023 - Current
Azure Data Engineer
Shadowfax Technologies Private Limited
09.2022 - 01.2023
Data Engineer
Radiiuscard Solutions Private Limited
09.2019 - 09.2022
Bachelor of Arts -
Sree Vinayaka Degree College