Adarsh Pallav

• Worked on design and development of an api solutions to exchange intelligence amongst business units ,which is churned out of various models.
• Worked on flask streaming and file based data transfers to different vendors.
• Worked on performance tuning on api's to reduce the overall response time.
• Worked on ETL for bulk data loading from hive to mongoDB
• Worked on designing and managing documents based no sql DB (mongoDB) and using is to store and send data via API and pyspark jobs.
• Worked on creating a framework to flatten bulk nested json data to flat csv files.
• Working on hive to aws migrations.
• Working on CI/CD to deploy jobs in various environments using various tools like bitbucket, docker,jenkins etc.

Data Engineer

• Built a complex data pipeline involving multiple sources including DB2,Datastage,Hadoop as source and BIGQUERY as sink using Python,SQL,HQL,Bash Scripting,Cloud Composer, Apache Airflow,Apache sqoop which ran successfully on the production environment without major fixes.
• Worked On Hadoop to GCP Migration which involved writing the existing data pipeline to GCP compatible data pipeline. It revolved majorly around GCS Bucket Storage, Bigquery, Apache Sqoop, ETL, Composer,Airflow,Python ,Shell Scripts with exceptions of few stories requiring Dataflow .
• Worked in a team to implement a real time data ingestion to Bigquery with a ingestion time frequency of 3 minutes, The architecture for the same was KAFKA -> FLUME->GCS->BigQuery.
• Worked on implementing various use cases of Airflow Dags like branching,subdags,xcom's,variables,task dependencies ,wrapper dag to call child dags etc
• Worked on Datastage master sequences ,sequences, and pxJobs to implement end to end data pipleline for loading the data into BigQuery and scheduling the jobs using Cloud Composer.
• Worked in deployment for all the above mentioned projects and help stabilize the same .The CI/CD tools used in this were GIT,BitBuckrt,jenkins and Control-M job scheduler.

Product Support and Integration ,Worked on various products to resolve the existing issues using SQL and shell scripting