Sowmiya T C
Madurai
Summary
Results-driven Azure Data Engineer with over 5.7 years of experience, including more than 3.5 years specializing in Azure cloud technologies. Expertise in designing, developing, and maintaining scalable data pipelines and ETL solutions using tools such as Azure Data Factory, Azure Databricks, PySpark, and SQL. Proficient in managing structured and semi-structured data from diverse sources into Azure Data Lake and Snowflake to enhance analytics and reporting capabilities. Strong understanding of data warehousing principles, performance optimization techniques, and innovative cloud-based data engineering solutions.
Overview
2
2
Certification
6
6
years of professional experience
Work History
Senior Data Engineer
Cognizant Technologies Solutions
Kochi, India
10.2020 - Current
- Developed scalable ETL/ELT pipelines using Azure Data Factory and Azure Databricks to process large-scale pharmaceutical domain datasets.
- Built PySpark notebooks for data cleansing, transformation, aggregation, and loading into curated layers.
- Implemented Medallion Architecture (Bronze, Silver, Gold) using Delta Lake in Azure Data Lake Storage.
- Created incremental and SCD Type 1/Type 2 data pipelines using Delta tables.
- Integrated data from APIs, FTP, SharePoint, SQL Server, and flat files into Azure cloud storage.
- Developed ADF pipelines to extract data from on-premises SQL Server using Self-Hosted Integration Runtime.
- Migrated historical data from SQL Server into Azure Data Lake Storage Gen2 in Parquet and Delta formats.
- Optimized Spark jobs and improved pipeline performance by partitioning, caching, and efficient joins. Improved pipeline execution performance by 40% through Spark optimization techniques.
- Configured parameterized pipelines, triggers, linked services, datasets, and monitoring in ADF.
- Worked with external tables, stages, and file formats in Snowflake.
- Implemented CI/CD deployment using Azure DevOps and ARM templates.
- Monitored data pipelines and resolved production issues with minimal downtime.
- Worked closely with cross-functional teams, including business analysts, data architects, and QA teams, to gather requirements, perform impact analysis, and deliver solutions that met business needs.
- Providing support to team members on the technical side to achieve the project timeline.
Education
BE - Electronics and Communication Engineering
PSNA College of Engineering And Technology
Dindigul, TamilNadu05-2020
Skills
- Azure Data Factory
- Azure Databricks
- Pyspark
- SQL
- Python
- Snowflake
- Unix Scripting
- Oracle
- SQL Server
- Power BI
- ETL development
- Data pipeline design
Accomplishments
- Designed and implemented end‑to‑end data pipelines using Azure Data Factory, ingesting data from SQL Server, APIs, and flat files, improving data availability by 40%.
- Built incremental and full-load ETL pipelines with ADF, reducing data processing time by 30% through optimized triggers and partitioning.
- Automated data ingestion workflows with event-based triggers, eliminating manual intervention and reducing failures by 25%.
Certification
- DP-900: Microsoft Azure Data Fundamentals
- Databricks Certified Data Engineer Associate
Languages
English
Advanced (C1)
Tamil
Bilingual or Proficient (C2)
Timeline
Senior Data Engineer
Cognizant Technologies Solutions
10.2020 - Current
BE - Electronics and Communication Engineering
PSNA College of Engineering And Technology