Sudeep Behera

Genpact

02.2020 - Current

Worked on several complex Queries and Joining Methods in SQL to fetch the relevant Insights from the Data.
Extensively used PySpark Dataframe API,Python pandas API to create new datasets, modified existing datasets logic and onboarded datasets to production.
Desiged multiple automated Pyspark and Spark SQL codes for several purpose like MRD instance wise counts,logic building,dynamic validations and dataset mutation.
Designed and developed Spark applications using Pyspark, Spark-SQL for data extraction, transformation and aggregation from multiple file formats for analyzing and transforming the data to uncover insights into customer usage patterns using Databricks.
Developed end to end ETL framework for migrating the data from on-premise database(SQL Server) into Azure Datalake.
Developed pipelines which handle all sort of transformations/cleaning in to performed on each table and load the transformed data into next layer using Azure Data Factory.
Good understanding on Spark architecture including Spark Core,SparkSQL,Dtaframes,Driver Node,Worker Node,Stages,Executors and Tasks.
Implemented several spark optimization techniques for better performance of jobs.
Performed Data quality check processes .Checked consistency,Metadata, schema match,NULL check, numerical analysis between production data and first line source.

Similar Profiles