Microsoft SQL Server
DATA ENGINEER
IBM
Bengaluru
05.2021 - Current
- Responsible for Creating SSIS packages to perform ETL Operations on Data from different sources and loading transformed data into DIM & FACT tables
- Implementing ETL logic using SSIS to include best practices in scheduling and error/exception handling with email notification
- Experience in Creating SQL Agent jobs, Scheduling, Monitoring and Deploying in production
- Experience in creating Pipelines, Data Flows, Datasets, Linked Services, triggers to perform ETL and ingesting data using Azure Data Factory
- Performing ETL Operations Using PySpark and Spark SQL in Azure Databricks
- Manipulating and transforming large datasets using PySpark's Data Frame and SQL API in Azure Databricks
- Scheduling Databricks notebooks using ADF
- Experience in monitoring and error handling of ADF pipelines
- Experience in T-SQL development to write complex queries by joining multiple tables, great ability and maintain stored procedures, triggers, user defined functions
- Experience in creating and updating Clustered and Non-Clustered indexes to keep up the SQL server performance
- Experience in Performance tuning of query and Procedure
- Generating reports from Dim & Fact Tables Using MS SQL
- Writing python script to manipulate/copy/move/Zip/Unzip files from one folder to another folder using python
- Fetching Data from SQL Server using python and loading data to csv files and manipulating them using python and pandas.
