Rohan Srivastava

Project 1 :

• Helped a client to move data from different sources (SDTP/FTP/Db2 etc) to ADLS Gen2 using ADF pipeline .
• Basic transformation was done in pyspark using databricks activity
• End to End pipeline was automated and we used Azure Devops Pipeline for CICD .

Project 2 :

• Helped a client to move 14 use cases originally build in spark scala to migrate from Hadoop to Azure Databricks .
• The data was ingested using Qlik in ADLS and which act as a source of all the 14 use cases .
• Jenkins was used as CICD tool which migrated the project to uat & prod .

• Worked upon development of Spark based logics for aggregating,designing,cleansing, transforming terabytes of structured & semi structured data available in HDFS .
• Parametrized shell scripting code was written which was responsible for automating and triggering spark & hive scripts.
• Experience in Ingesting the data from different RDBMS using spark framework and converting the same into appropriate views and tables .
• Developed a framework to perform monitoring over different Data Science models and report them in case the model performance degrades .
• Decompressed non supportable formats of data around 4 TB in hdfs and made it handy to use for Data Science team using various Big data technologies

• Hadoop was used for ETL to ingest, aggregate and parse the data from about 20 data sources .
• Developed Sqoop scripts in order to make the interaction between Hive and SQL Database .
• Built hive logics which has been further used in creating predictive model variables .
• Converted GBM class file into Pojo models using Java .GBM files are the one generated using H2O model (ML algorithm).
• Aggregated Pojo model and logistic regression code with Hive logics using shell scripting .
• End to End automation were done and it was made sure that the script daily delivers reports to clients in correct format .
• Scheduled scripts daily using TWS jobs

• Developed analytics at the point of Business Sales in a Pharmacy Based chain .
• The Daily sales were pulled using Sqoop from DB2 server and saved as hive tables .
• Business Logics were converted using hive .The Daily sales after transformation was visualized using Tableau .
• The final tables were optimized and saved into appropriate formats using partitioning and bucketing .