Sarjeet Sharma
Lead Cloud Engineer
Male,HR
Summary
Lead Cloud Consultant with strong business acumen and technical experience in building scalable data lakes using modern data platfoms on Cloud and On Prem set up. Self-motivated professional with strong communication skills and ability to learn new things quickly and work as a team.
Overview
12
12
years of professional experience
6
6
years of post-secondary education
7
7
Certificates
Work History
Lead Data Engineer
Incedo Inc
Gurgaon, Haryana
12.2017 - Current
- Project: Clinical Studies Analytics
- Team Size : 10
- Developed, implemented and maintained data analytics pipelines, standards, and documentation.
- Architecture involves Nifi For fetching data from Web API/XML Feed/Cloud PostgreeSQL DB for multiple data Formats(Json/XML/Text) storing raw files in Azure blob Containers(Azure Data Lake Storage)
- Azure Databricks is used for processing the raw data and storing them in the Azure Blob Storage
- Involved in Gathering business requirements and building
- Scala with Jackson Json4s is used for parsing the multi Nested Json files and breaking them into structured format
- Designed and Build the complete Nifi for Ingestion purpose which is used in multiple projects for API Data fetching
- ADF (Azure Data Factory) is used for triggering the Databricks Jobs on Daily basis
- Responsible for designing and developing the complete pipeline using ADF and NIfi
- Hive is used for building Golden Layer Aggregates Tables which are used by downstream apps for further Data analysis and Dashboard building
- Implemented agile development practices for complete project implementation
- Project: Pricing/Sales Analytics
- Team Size: 5
- Architecture involves Nifi for fetching CSV files from SFTP location to HDFS Storage, Shell script for preprocessing validation and Apache Spark for Data validation & storage in Hortonworks Hadoop Storage.
- Responsible for building the Spark jobs and scheduling it via Oozie Scheduler
- Projects involves migration of traditional RDBMS data validation Flow to Spark Based Processing Framework with Hadoop Environment
- Automated the complete ETL process and Build it from Scratch to end
- The process make sure if any files are not following the validation rules will get rejected and Business will be informed on the same
- Responsible for conversion of all the Hql code to Dataframe API fomat for Silver Level Staging Tables
Senior Big Data Engineer
NCR Corporation
Gurgaon, Haryana
02.2017 - 12.2017
- Project: Supply Chain Spend Analytics
- Team Size: 5
- Architecture involves Kafka integrated with Flume and Kafka High Level Consumer API for streaming xml messages into HDFS and Local File System, Hive for ETL, Python for Parsing, Shell Scripting for Job Execution, CTRL+M for Job Scheduling Tableau for Reporting built on HDP 2.3 Hadoop Distribution
- Involved in all the development phases (Analysis, Design, Build, Documentation, Testing and Deploy)
- Written python script for parsing the source xml messages to csv and moving the data to HDFS
- implemented kafka aith flume for realtime delivery of log messages to HDFA
- Written Shell script for Daily Data insertion and Data Transformation in different hive schemas
- Have done data modelling for complete Database schema
- Involved in designing the Data Lake structure for complete enterprise
- Used Hive for performing ETL Processing (Data Cleaning, Incremental update, Final Load)
- Involved in Gathering business requirements and converting them to Detail design Document
- Implemented agile development practices for complete project implementation
- Build a POC on Real time stream processing using Spark Streaming, Spark Core API with Scala .
Senior Software Engineer
Accenture Services Private Limited
Hyderabad, Telangana
06.2015 - 06.2017
- Project: Easy Tools Reporting
- Team Size: 5
- Lient: IHG
- Architecture involves Kafka for streaming the near real time Json messages in HDFS
- Hive for data warehousing, Syncsort as an ETL tool in HortonWorks Hadoop Distribution
- Developed data pipeline using Kafka with Flume to ingest Raw data in Json format into HDFS for data archival and analysis
- Writing Shell Script for executing Syncsort Job’s for parsing the Json data into Flat files and moving to HDFS location
- Implemented Kafka high Level Consumer API in Java for consuming the Real time Json Messages into ETL Tool - Syncsort for ETL Purpose
- Written Hive queries for creating external tables, storing and processing the data for further analysis on Visualization tools
- Created design document for the complete data flow and giving walkthrough on the same to Client
- Worked on Spark Sql for Building reports for Revenue Management Team via Zeppellin Visualization Interface
- Coordinating with the business teams for understanding requirements during Design Period
- Responsible for presenting Weekly metrics to end clients and Attending Weekly Call with OnShore Clients for Over All progress.
Senior Software Engineer
Accenture Services Private Limited
10.2014 - 10.2015
- Project: Equipment Health Management
- Eam Size: 6
- Lient: Confidential - Major US Heavy Equipment Manufacturer’s
- Product health data (Temp, Pressure, Oil level etc.) are captured by the sensor’s and received on the data center end with the help of hardware’s
- At data center the sensor data is cleansed and performed analysis to predict the machine failure or service requirement for future
- Architecture involves Mysql Server for receiving the raw sensor data, ETL is being performed by using Sqoop for data migration from Myql to HDFS, Hive for analysis and Oozie for Workflow Creation
- Tableau for Data Visualization in Cloudera CDH4.7
- Creating data integration for ETL using Oozie Coordinator on HUE
- Installed and configured Hive and written Hive-QL scripts
- Writing Sqoop scripts for data migration in batches from MySQL Server to HDFS
- Writing Hive queries for creating external tables, storing and processing the data for further analysis on Visualization tools
- Co-coordinating with the business teams for understating requirements and converting them into detail design document
- Imparting guidance to the associates on all development areas.
Software Engineer
Accenture Services Private Limited
04.2014 - 09.2014
- Ad Campaign Data Analysis
- Eam Size: 3
- Lient:, The Ad Data Analysis is intended to provide a full range of services from understanding User behavior, visualization and predictive analytical capabilities with big data, to dash boarding/reporting of data post extraction and transformation
- Ad data from different Ad networks is used to Perform ETL and providing the below analysis to the client
- Revenue analysis for the ad campaign period, Predicative analytics for future ad campaign costing and Business recommendation for improving the sales by analyzing the client browsing pattern
- Installed and configured Pig and written PigLatin scripts
- Developed Pig Latin scripts to cleanse the web server log files and uploading the data into HDFS
- Written Oozie Coordinator Jobs to transfer data from RDBMS to HDFS using Sqoop
- Involved in creating Hive tables, loading with data and writing hive queries
- Used HBase for Storing the Recommendation related data
- Worked with business teams and created Hive queries for ad hoc access
- Performance tuning of the queries by implemented different performance guidelines
- Creating Test scripts and deployment plans for production code changes.
Software Engineer
Accenture Services Private Limited
03.2012 - 04.2014
- Application Support and Web Server Administration
- Team Size: 6
- Client: Confidential - American OnG Production and Transmission Corp
- Analyzing and resolving critical issues of web application
- Coordinating with end users and fixing critical L2 issues within stipulated time
- Managing Server administration task’s and responsible for full cluster Performance monitoring by using PSM- Dynatrace Performance Monitoring Tool
- Communicating with the end user and customizing the solution per their requirement
- Accommodated Adhoc customization requests using core java
- Performed Data changes, Data Set up, Bug Fixes, report extraction, Master Data set up and Audit requests
- Displayed know how of overall functionality in Windchill PLM Web Application
- Mastered In export and administration module of GTM (Global Trade Management) SCM
- Handling the different process related to Incident’s management
- Off-shore POC for Change Management and Problem Management Process
- Connecting with the change board every week for all changes reported by the Team
- Performing code changes for the bugs reported by the business team
- Creating UAT’s for the change and communicating with clients for UAT Approval
- Writing complex ad-hoc queries in SQL as per the requirement submitted by business team.
Associate Software Engineer
Accenture Services Private Limited
09.2011 - 02.2012
- ERP Baan Administrator
- Team Size: 3
- Lient: American OnG Production and Transmission Corp
- Worked as an administrator for Baan ERP, GTM (SCM) and Windchill (PLM) application’s which involved taking care of the User administration including assigning Roles, Deletion of invalid accounts and modification of profiles as per the client Request
- Worked as a support analyst for performing the Auditing on whole Baan production, Dev and Test environment monthly to make sure all environments are in sync
- Written Script in Python for removal of special characters in the production NSR Data files
- Worked and coordinated with Dev team for performing the Code promotion from Dev to Test to production as a part of administration activities
- Creating Daily Status report team for the full team and sending it to business leads.
Education
Master of Computer Application - Computer Science And Programming
U P Technical University
UP 04.2007 - 06.2010
Bachelor of Science - Chemistry
Delhi University
Delhi 03.2003 - 04.2006
Skills
Scala, HIveQL, SQL, Python, Shell Scripting, Core Java,GitLab
Hive, Sqoop,Apache Nifi, Kafka, Flume, ZooKeeper, Hue, Oozie, Apache Ambari, Zeppelin, Apache Spark 2x, 3x, Map-Reduce,Spark Streaming
Apache Spark 2x-3x,Apache Hadoop 2x, Hortonworks (23, 25), Cloudera CDH 47
Azure ADF, ADLS, Databricks, Cosmos DB, HD Insight
Certification
Azure Fundamentals Certified - AZ 900
Timeline
Databricks Gen AI Fundamentals
07-2023
Databricks Certified Engineering Associate
11-2022
Azure Data Engineer Certified Associate
08-2021
Azure Architect Technologies – AZ 303
12-2020
SAFE Agile Scrum Practitioner Certified
12-2020
Azure Fundamentals Certified - AZ 900
07-2020
Lead Data Engineer
Incedo Inc
12.2017 - Current
Senior Big Data Engineer
NCR Corporation
02.2017 - 12.2017
Senior Software Engineer
Accenture Services Private Limited
06.2015 - 06.2017
Senior Software Engineer
Accenture Services Private Limited
10.2014 - 10.2015
Software Engineer
Accenture Services Private Limited
04.2014 - 09.2014
Software Engineer
Accenture Services Private Limited
03.2012 - 04.2014
ISEB BCS Analyst Certified
01-2012
Associate Software Engineer
Accenture Services Private Limited
09.2011 - 02.2012
Master of Computer Application - Computer Science And Programming
U P Technical University
04.2007 - 06.2010
Bachelor of Science - Chemistry
Delhi University
03.2003 - 04.2006