Rupesh Shinde

Azure DataBricks

• Description: Leveraged Azure DataBricks for big data processing and advanced analytics, enabling efficient handling of large datasets and real-time data analysis.
• Tools: Azure DataBricks, PySpark, SQL, Azure Storage
• Activities:Developed and optimized PySpark scripts for data ingestion, transformation, and aggregation.
  Implemented machine learning models within Azure DataBricks for predictive analytics.
  Integrated Azure DataBricks with Azure Data Lake and Azure SQL Data Warehouse for seamless data flow.
  Created notebooks in Azure DataBricks for data exploration, visualization, and reporting.
  Conducted performance tuning and optimization of Spark jobs to improve processing speed and resource utilization.
  

Azure Data Factory

• Description: Designed and implemented Azure Data Factory pipelines to automate and orchestrate complex data workflows, ensuring efficient data movement and transformation.
• Tools: Azure Data Factory, Azure Storage, SQL, Azure DataBricks, Azure Functions
• Activities:Created and managed data pipelines for ETL processes, including data ingestion, transformation, and loading from various sources to target destinations.
  Developed custom activities and data flows using Azure Data Factory to handle complex data transformation logic.
  Integrated Azure Data Factory with various data sources such as SQL databases, Azure Blob Storage, and REST APIs.
  Implemented monitoring and alerting mechanisms to ensure the reliability and performance of data pipelines.
  Automated the scheduling and execution of data pipelines, reducing manual intervention and improving efficiency.
  Collaborated with cross-functional teams to gather requirements and design data integration solutions tailored to business needs.
  

Talend ETL Development

• Description: Designed and developed ETL jobs to load data from legacy systems into Redshift Datawarehouse.
• Tools: Talend Data Integration, Talend Management Console, Redshift, SQL Workbench
• Activities:Built efficient Talend jobs for processing fact and dimension tables with complex transforms and Type 1/Type 2 changes based on ETL specification documents.
  Deployed all jobs to Talend Management Console (TMC).
  Automated data pipeline executions using TMC by scheduling jobs and adding dependencies between pipelines using the webhook mechanism.
  Provided production support, addressing any issues in the production code.
  Performed data analysis and validation by writing SQL queries.
  

Dell Boomi Integration

• Description: Designed various integration processes using Boomi.
• Tools: Dell Boomi, Oracle, MySQL, PostgreSQL, Aqua Data Studio
• Activities:Participated in requirements gathering, integration design, development, configuration, administration, and documentation.
  Coordinated with offshore teams to address technical or business requirement queries.
  Configured environments in AtomSphere.
  Developed Atoms/Molecules on DEV/Test/Prod servers.

SSIS Development

• Description: High-level design of ETL SSIS packages for integrating data using OLE DB connection from heterogeneous sources (Oracle, DB2, CSV, flat file, text format data) by using multiple transformations provided by SSIS.
• Tools: SSIS, SQL Server Agent, Oracle, MS-SQL, Visual Studio
• Activities:Understood the BRD document and created ETL processes to populate data for SQL.
  Built efficient SSIS packages for processing fact and dimension tables with complex transforms and Type 1/Type 2 changes.
  Gathered requirements and designed data warehouse and data mart entities.
  Provided production support, addressing any issues in the production code.
  Performed data analysis and validation by writing SQL queries.