Ramesh Maliki

● Designed and implemented ETL pipelines using ADF, Azure Databricks, and PySpark to process large-scale structured and unstructured data into ADLS.

● Built Delta Lake architecture for batch and streaming data processing with PySpark and SQL for advanced transformations and aggregations.

● Developed and optimized PySpark SQL queries for data validation, transformations, and integration with downstream systems.

● Optimized pipeline performance by implementing partitioning, caching, and parallelism in PySpark jobs.

● Ensured data security with Azure Key Vault, encryption for data at rest and in transit.

• Developed data pipelines using ADF to automate data movement between various sources and sinks
• Configured data transformations and data cleansing within pipelines
• Scheduling and monitoring pipeline execution
• Perform complex data transformations and aggregations on large datasets
• Utilized Pyspark for cleaning, transforming, and analyzing datasets
• Developed automated data pipelines using Pyspark for data ingestion, transformation, and loading
• Optimized Spark jobs for performance and scalability.
• Collaborated on ETL (Extract, Transform, Load) tasks, maintaining data integrity and verifying pipeline stability.
• Fine-tuned query performance and optimized database structures for faster, more accurate data retrieval and reporting.

• Involved in Business requirement document walk through to understand functionality
• Extracted, transformed, and loaded data from source systems to Azure Data Storage services
• Created pipelines in Azure Data Factory for batch job automation
• Ingested data to Azure Services like Azure Data Lake, Azure SQL, Azure Synapse using Azure Data bricks
• Used Spark-SQL for faster data processing and testing
• Interacted with client teams to gain better understanding of their deployment needs.

• Worked with Microsoft Azure Components like Data factory, to move data from on premises database to Azure database via azure portal
• Requirement gathering from Client and provide better solutions
• Map source system data elements to target system and develop, test, and support extraction, transformation, and load processes
• Developed Pipelines in Azure Data factory using Multiple Activities
• Integrated data from on-premises like ORACLE, and BLOB Source to Azure Data Lake Store (ADLS)
• Created Pipelines, Datasets using activities for automatic data integration in Azure Data factory
• Created Scheduled based, Event based, Tumbling window triggers on demand basis
• Performed Data Validation and Data Quality checks after completion of Data ingestion

• Participated in business process walk through to understand functionality
• Analyzed Transformation rules using mapping rule document
• Understanding business requirement and functional requirements specification and documenting test scenarios, test cases, test execution, defect management, status reporting
• Coordinating with different teams for doing end to end testing
• Running workflows and shell scripts to validate results
• Involved in ETL validations like Structure Validations, Count Validation, Constraint Validations, Duplicate check validations and data validations

• Participated in business process walk through to understand functionality
• Analyzed Transformation rules using mapping rule document
• Understanding business requirement and functional requirements specification and documenting test scenarios, test cases, test execution, defect management, status reporting
• Coordinating with different teams for doing end to end testing
• Running workflows and shell scripts to validate results
• Involved in ETL validations like Structure Validations, Count Validation, Constraint Validations, Duplicate check validations and data validations
• Attending weekly dev sync up calls with development teams for project discussions and participating triage (defect) calls.