SOMASEKAR K

Client - crescent petroleum

• Designed, developed, and optimized reusable PySpark and SQL scripts using Azure Databricks to improve data processing performance and scalability.
• Designed, developed, and maintained end-to-end ETL/ELT pipelines using Azure Data Factory (ADF) and Azure Databricks, ensuring reliable data ingestion, transformation, and high data quality.
• Integrated structured and unstructured data from multiple source systems into Azure Data Lake for enterprise analytics and reporting.
• Implemented Medallion Architecture (Bronze, Silver, Gold) using Delta Lake to build scalable, reliable, and analytics-ready data pipelines.
• Designed and optimized Delta Lake tables, improving query performance and supporting high-volume data processing workloads.
• Performed data validation, schema validation, duplicate handling, and source-to-target reconciliation to ensure data integrity and consistency.
• Collaborated with business stakeholders, SMEs, and cross-functional teams to translate business requirements into scalable technical solutions.
• Monitored, optimized, and troubleshot Azure Databricks jobs and Azure Data Factory pipelines, ensuring reliable production execution and performance optimization.
• Supported CI/CD deployments using Azure DevOps and managed work items through JIRA in an Agile/Scrum environment.

• Wrote basic SQL queries for data extraction and analysis.
• Assisted in developing data processing scripts using PySpark and Python.
• Supported ETL processes for data loading and transformation.
• Performed data validation and data quality checks.
• Collaborated with the team on data engineering and reporting tasks.