Tushara Sammiti

Project : Red DWH (Real Estate Data)

Responsibilities :

• Designing and developing Azure Data Factory (ADF) Extensively for ingesting data from different source systems like relational and non-relational to meet business functional Requirements.
• Designed and Developed event driven architectures using blob triggers and data factory.
• Creating pipelines, data flows and complex data transformations and manipulations using ADF and PySpark with Databricks.
• Automated jobs using different triggers like Events, Schedules, and Tumbling in ADF
• Created, provisioned different Databricks clusters, notebooks, jobs, autoscaling.
• Ingested huge volume and variety of data from Disparate source systems into Azure Data Lake Gen2 using Azure Data Factory V2.
• Created several Databricks Spark jobs with Pyspark to perform several tables to table operations.
• Performed data flow transformation using the data flow activity.
• Implemented Azure, self-hosted integration runtime in ADF.
• Developed streaming pipelines using Apache Spark with Python.
• Created, provisioned multiple Databricks clusters needed for batch and continuous streaming data processing and installed the required libraries for the clusters.
• Ø Improved performance by optimizing computing time to process the streaming data and saved cost to company by optimizing the cluster run time.
• Perform ongoing monitoring, automation, and refinement of data engineering solutions.
• Created Linked service to land the data from SFTP location to Azure Data Lake.
• Extensively used SQL Server Import and Export Data tool.
• Working with complex SQL views, Stored Procedures, Triggers, and packages in large databases from various servers.
• Experience in working on both Agile and waterfall methods in a fast pace manner.
• Generating alerts on the Daily metrics of the events to the product people.
• Extensively used SQL Queries to verify and validate the Database Updates.
• Ø Suggest fixes to complex issues by doing a thorough analysis of root cause and impact of the defect.
• Provided 24/7 On-call Production Support for various applications and provided resolution for night-time production job, attend conference calls with business operations, system managers for resolution of issues.
• Design and Development of ADF Pipelines to transform data from heterogenous sources to Azure SQL
• DW/Synapse.
• Worked with various data moment, control, and transformation activities
• Good experience in creation of various Linked services and data sets for ADF pipelines
• Implemented error handling, logging, configurations, and email notifications using Logic apps
• Deploying ADF pipelines into different environments like QA, UAT and PROD using Azure DevOps pipelines
• Extensively worked with various triggers and implanted as per the requirements
• Created Pipelines for data migration from On premise SQL, Yellow Brick DB, file systems to Azure Data lake
• store.
• Worked with Azure key vault, Logic apps and Automation account
• Writing Complex queries, stored procedures, Views, Cursors, SQL Joins.
• Worked on performance tuning for existing stored procedures, Functions.
• Imported data from Azure Data Lake and Blob storages.
• Deployment and monitoring of Pipelines & Scheduling and executing SSIS packages in ADF
• Assisted in fine-tuning transformations for increased performance and efficiency of ssis packages.
• Involved in daily scrum calls and weekly sprit calls.
• On call support during weekend installations.
• Handled production job failures and given permanent fixes
• Developed artifacts that are consumed by the data engineering team such as source to target mappings, data quality rules, and data transformation rules, Joins etc.
• Wrote Python scripts to load data from Web APIs to staging DB.
• Collaborated with the project team to outline project requirements in JRD meetings.
• Facilitated the maintenance of the project by performance tuning Python apps.
• Identified the dimensions along with the measures and fact on the top of OLTP source.
• Implemented exploratory data analysis by utilizing simple machine learning algorithm.
• Enhanced and optimized Spark/Scala/ pyspark jobs to aggregate, group and run data mining tasks using the Spark framework.
• Involved in the naming standards which incorporated the enterprise data modelling.
• Defined multiple constraints in logical phase of the data modelling life cycle.
• Imported the data from various formats like JSON, ORC and Parquet to HDFS cluster with compressed for optimization.
• Deployed Pyspark applications and developed in Databricks cluster.
• Well versed with REST APIs and CRUD operations to access and transform analytic data.
• Importing and exporting data into HDFS and hive using Sqoop and Kafka with batch and streaming.
• Involved in complete Big Data flow of the application data ingestion from upstream to HDFS, processing the data in HDFS and analyzing the data using several tools.
• Initiated the data modelling sessions, to design and build/append appropriate data mart models to support the reporting needs of applications.
• Created complex stored procedures to perform index maintenance and data profiling for loading data marts and generating datasets for reports.
• Used Hive to join multiple tables of a source system and load them to Elastic search tables.
• Imported the data from various formats like JSON, Sequential, Text, CSV, AVRO and Parquet to HDFS cluster
• with compressed for optimization.
• Scripted new measures and calculated tables in Power BI Desktop utilizing DAX
• Used Power BI gateway for implementing DAX functions to automate the data refresh of Power BI datasets.
• Utilized Azure Data Factory to perform Azure-SSIS integration and deployed SSIS packages to cloud.
• Handled structured and unstructured datasets.
• Built high quality, reliable, and consistent sound systems that are aligned and scale with our data business needs.
• Recreated existing application logic and functionality in the Azure Data Factory, Azure SQL Database and Azure
• SQL Data Warehouse environment.
• Performed evaluation of on-premises SQL database to Azure SQL.
• Implemented Azure Data Factory operations and deployed into Azure for moving data from on-premise into cloud.
• Designed and implemented end-to-end data solutions (storage, integration, processing, visualization) in Azure.
• Created CI/CD Pipelines in Azure DevOps environments by providing their dependencies and tasks
• Architected and implemented ETL and data movement solutions using Azure Data Factory.
• Built a data pipeline in Azure Data Factory to fetch the data from on-premises database to Azure SQL database.
• Environment: Azure Data Factory (ADF v2), Azure SQL Database, Azure Data Lake, BLOB Storage, SQL server, Data bricks, Python, ADLS Gen 2, Azure Cosmos DB, Azure Event Hub, Azure Machine Learning.