Data Engineer with 4.5 years of experience in designing, implementing scalable data platforms for both on-prem and cloud solutions. Hands-on experience in designing and eveloping complex Report and Taskflows using SSIS, ADF and Databricks and Have hands-on experience in working on cloud platforms such as Snowflake for datawarehouse and snowpark. Proven experience in proactive management of the project, client communication.
• Collaborated with business architects to facilitate the migration of on-premises Data Warehouse to the cloud, organizing existing warehouse tables into individual subject areas as per business stakeholder requirements.
• Designed ETL pipeline for data warehouse tables, created Databricks notebook for on-premises data warehouse migration.
• Analyzed PL/SQL procedures and SSIS workflow, processed their business functions, and transformed them into Databricks notebooks.
• Utilized Databricks Autoloader for ingesting JSON data from AWS S3 into Delta tables, enabling structured data streaming into staging zones for the creation of fact and dimension tables in Snowflake.
• Deployed Databricks notebooks in QA and Production environments, testing validation notebooks to ensure near real-time data quality assurance.
• Managed overall project using Jira and confluence for sprint planning and DevOps using GitLab.
• Designed and implemented the other way sync framework to sync the new point of sale on going data back to the oracle on-prem database to stabilize the data in both point of sale.
• Responsible for gathering requirements from the client, for analyzing the requirements and creating the design.
• Based on the design, created complex database objects like Stored Procedures, Functions, Packages and triggers using PL/SQL and SQL.
• Designed and Developed PLSQL Procedures and migrated it to target via ETL tool (SSIS).
• Created various automation processes using stored procedures to eliminate manual works.
• Implemented purging and vacuum technique to improve the performance of the control table.
• Delivering the data in a useful and appealing way to users giving the user the ability to interact with the data.
• Involved in the Deployment activities and provided support to monitor the jobs after deployment.
• Making large and/or complex data more accessible, understandable and usable Transforming, improving and integrating data, depending on the business requirements.
• Extracted data from several flat files and transformed it in accordance with the business logic mentioned by client.
• RSEG’s refinery software used to extract directional drilling survey data .
• Database/historian development, data extraction/transformation/loading, data filtering and quality control, exploratory data analysis and visualization, and programming with an emphasis on scripting languages.
• Extracted data from several flat files and transformed imported to database. After completing survey,I make changes in Excel Sheet and report to client in JIRA board.
Technology: Ms Excel, Oracle 12c