Soumyakanta Rath

• Worked on creating solid framework for Data Quality issues between Events(streaming data) and Summary data. Finding the data leakage, data quality issues.
• Comparing and building framework for event based data with the batch data to ensure quality of information is intact.
• Ensured data quality through rigorous testing, validation, and monitoring of all data assets, minimizing inaccuracies and inconsistencies.
• Enhanced system performance by designing and implementing scalable data solutions for high-traffic applications.
• Delivered exceptional results under tight deadlines, consistently prioritizing tasks effectively to meet project timelines without compromising quality or accuracy.
• Work in a team of data engineers and data scientists and analysts to understand their data needs and develop data solutions that meet those needs.
• Worked with AI engineers to facilitate data and Api building to implement Amazon Bedrock LLM to text to sql models for data analysis.
• Design, build, and maintain scalable data platforms and pipelines to support the needs of the business.

Tech Stacks:- Databricks, Delta lake, Unity Catalog,Flink, Kafka, DynamoDB, Python

• Design, build and optimize the data architecture using suitable design patterns and effective Data Model to data pipelines to make them accessible for Business Data Analysts, Data Scientists and Business users to enable data-driven decision making and to create single source of truth. Developed database architectural strategies at modeling, design and implementation stages to address business or industry requirements.
• Build serverless data lake architecture using AWS tech stacks:-S3, Glue, Redshift, Athena, Step function, lambda, cloudwatch .. for daily, weekly and incremental load.
  Build data pipeline on robust Data profiling and Data quality with pluggable code during stages of data pipeline and Data Quality is in desired range, schema valid or not, data format errors
• Gathered, defined and refined requirements, led project design and oversaw implementation.
  Designed data models for complex analysis needs.
  Designed and created Audit/Control log table and Master Data management for transparent view of status of data loading. Optimize data pipelines, algorithms and data storage to improve performance and scalability.
• Work closely with Analysts to Productionize and scale value-creating abilities including data integrations and transformations, model features and statistical and ML models(ARIMA models, Random Forest, Regression). Wrote Complex Redshift sql views for Data Analyst and Data Scientist to leverage models on top of it.
  Brought best-practice to proactively and continuously build data related practices within the team.

• Creating Spark jobs, tasks in Databricks on TBS of data with optimized code and scheduled it using Databricks workflow.
• Bulk loading from external stage(AWS S3), internal stage to Snowflake cloud using COPY command.
  Used Snowpipe for continuous data ingestion from S3.
• Created and orchestrated Airflow Dags for various data pipelines and ad-hoc jobs.
• Work with analytics partners to deploy scalable data pipelines for analytical needs in Databricks.
• Created Spark/Scala UDF's for ML Team on geohash to decode and haversine distance.
• Build and expose metadata catalog for the Data Lake for easy exploration, profiling as well as lineage requirements.
• Enable Data Science teams to test and productionize various ML models, including propensity, risk and fraud models to better understand, serve and protect our customers.
• Proved successful working within tight deadlines and a fast-paced environment.
• Tech Stacks :- AWS S3, Databricks, Airflow, Snowflake, Scala Spark, Python

• Worked for US largest Retail chain build Data Pipeline for South America Cost of goods and supply chain management for Data Science Team on stochastic, classification models.
• Developed, implemented and maintained data analytics protocols, standards, and documentation.
• Worked in creating decoupled and scalable Serverless Datalake architecture using AWS cloud environment (AWS S3, EMR, Redshift, AWS lambda, AWS Step function ) and No SQL databases for one of Retail giant. Extensively used Spark as processing engine with optimization to achieve optimal performance for data pipeline.
• Build scalable ETL pipeline design/implementation with large distributed data on top of Hive and Spark.
  Worked in variety of Data Ingestion process from source system(RDBMS,SFTP & API Endpoints).
• Expertise in using Spark SQL with various data sources like JSON, CSV, Parquet and Hive.
• Demonstrated experience in designing and developing reusable code Frameworks, libraries, and components.
• Worked in Dynamic Environment with Scrum Master, Product Owner, Data Architect, Data Scientist and DevOps.
• Collaborated with Business Analyst, Architect Team to maintain data integrity and verifying stability of ETL pipeline.
• Analyzed complex data and identified anomalies, trends, and risks to provide useful insights to improve internal controls.
• Collaborated with system architects, design analysts and others to understand business and industry requirements.
• Managed large scaled TBs of data and billions of data in Spark jobs.

• Involved in building data pipeline from collecting terabytes of log information, storing, cleansing and extracting meaningful information that will be loaded to NoSQL DB(HBase/Cassandra) for further analysis or to run ML models on it in Apache Spark Ecosystem with Scala.
• Involved in transitioning from traditional Datawarehouse to Hive through the query language. Develop efficient hive scripts with joins in dataset using various techniques.
• Expertise in creating Hive tables(External/Managed), partitioning, bucketing,loading, aggregating of table and creating UDF's in Hive.
• Imported and Exported large sets of RDBMS data into HDFS and vice versa using Sqoop.
• Experiencing in processing large amounts of structured and unstructured data. Played a role in Data pipeline building with Relational and NoSQL DB.
• Implementing design enhancements and change requested by Business/Project requirement.

• IRI is an American market research company which provides clients with consumer data, shopper, and retail market intelligence data and Analytic Solutions focused on the consumer packaged goods (CPG) industry. IRI’s clients include 95 percent of the Fortune Global 500 CPG, retail and healthcare companies.
• Developed, tested and maintaining the ETL process necessary to load data into a data warehouse, ensuring data quality and reconciliation between RDBMS.
• Developed database applications using Oracle SQL/ PlSQL and Python on Unix/Linux environments,Extensive exposure to Python modules to interact with data.
• Managing ETL loads and Jobs in Netezza and Oracle Warehouse.
• Assisted in designing and implementation of data marts and data warehousing applications as per business needs.
• Designed data extraction processes to meet business application requirements and data sources,Provided technical expertise and guidance for data management, quality and reporting functions.
• Suggested improvements in processes for managing data integrity and quality for overall enterprise.

• Played a key-role in identifying, build, developing automation features and provide maintenance to existing tools in order to eliminate costs, improve turnaround time and overall quality.
• Troubleshooting/ Root cause analysis of any data related issues.
• Developed reports and perform ad-hoc data analysis using SQL queries and provide data extracts to users when requested.

• Created shell script in loading data to Data warehouse concepts like Star schema, Snowflake schema.
• Primary focus in automating manual process, status notification and simplifying process.
• Troubleshooting/ Root cause analysis of any data related issues.
• Developed reports and perform ad-hoc data analysis using SQL queries and provide data extracts to users when requested.
• Identified root causes of recurring problems to implement effective solutions.