Kartik Gupta

Overall, 6+ years of experience in IT industry having 3 years of professional experience as a Data Engineer Hands on experience on major components of Hadoop Ecosystem like HIVE, Hadoop Distributed File Systems (HDFS), Scala, Spark and PySpark. Worked with various file formats like json, parquet, avro, csv, xml etc. Developed reusable transformations to load data from Flat files and other data sources to the Data Warehouse. Imported data from separate sources into Spark RDD for processing.

1. IC Reporting

Responsibilities 

· Preparing the HLD and LLD documents based on the business requirement.

· Understanding the requirement scope and building the business logic using Spark.

· Developed SFTP jobs to pull data from source servers

· Developed Data ingestion jobs to validate and load data into Hive tables

· Developed Spark jobs to create compiled tables from raw data and stored into Hive and Mysql Tables

· Developed spark jobs to create final reports from compiled data.

· Integrated HBase with Spark to store data in HBase whenever OLTP operations required

· Giving project walkthrough to the PSS team for their approval for production deployment.

· Creating the Change Related Documents like MOP, Code Review Report and Implementation Plan  to submit the Change Address Board.

· Deploying the extractions in production in a given time window and providing support to the downstream business users during deployment.

· Getting the UAT signoff from clients post implementation.

2.  BI Reporting Upgrade 

Responsibilties

· Preparing the HLD and LLD based on the business requirement.

· Understanding the requirement scope and building the business logic using Spark.

· Data Validation in Spark after data ingestion to HDFS

· Consuming the data from AWS S3 as one of the data source

· Used EMR whenever transformations to be done in cloud environment

· Extracting the hive table data using spark/ pyspark and performing field level aggregation.

· Creating managed table and store all transformed data in Functional layer.

· Providing project walkthrough to the stakeholders for their approval for production deployment.

· Creating the data Mapping documents. 

· Getting the UAT signoff from clients post implementation.

3.  BI Reporting

Responsibilties

· Involved in Requirement Analysis, Design, and Development

· Involved in code/design analysis, strategy development and project planning.

· Created Hive Warehouse tables using Hive.

· Joining large datasets using Hive

· Transformed the data using Hive.

· Integrated HBase with Hive to store data in HBase whenever OLTP operations required

· Develop multiple Sqoop jobs for data import and export.

· Write the HIVEQL to perform transformation.

· Written Hive queries for data analysis to meet the business requirements.

· Load the data into Hive partitioned tables for business team analytics