Pawan Negi
Data Engineer
New Delhi
Summary
A business professional having more than 3 years of experience in Information Technology with expertise in Big Data development working on various Big Data tools such as HIVE, SQOOP, SPARK(using Python), AWS with good understanding of HDFS.
Overview
3
3
years of professional experience
4
4
years of post-secondary education
Work History
Senior Consultant
Birlasoft
New Delhi
04.2022 - 09.2022
- Designing and configuring end to end Data Pipeline from On-premise servers and cloud databases as well
- Developing and unit testing ETL Python transformations at various layers in AWS Glue using pyspark, pandas, numpy libraries
- Deploying, managing and configuring AWS RDS instances and Redshift clusters
- Establishing and configuring connections through different types of APIs with Lambda functions in AWS and ingested data into S3
- Configuring and managing EC2 instances in public and private subnets.
- Developed Step functions to orchestrate the AWS glue jobs in a sequential order and glue triggers, cloudwatch events if needed to execute at a scheduled interval.
- Implemented and configured CI/CD pipelines across different environments using AWS CodeCommit, CodeBuild, CloudFormation and CodePipeline.
- Implemented email notification mechanism on failure/succession of a AWS Glue job or a lambda function using AWS SNS and AWS EventBridge rules.
Application Development Analyst
Accenture Solutions Pvt Ltd
New Delhi
09.2019 - 03.2022
- Developed ETL processes in AWS Glue to migrate data from on-premise SQL servers into AWS environment
- Developed AWS Glue JDBC connections to ingest data from Oracle/SQL servers into AWS S3
- Unit testing of transformations developed via SQL on AWS Athena
- Configuration of CICD pipeline in AWS among different environments
- Deploying and managing EC2 instances and resolving security vulnerabilities over them.
- Used Sqoop to import large data from traditional RDMS to HDFS and also handled incremental Sqoop.
Education
Bachelor Of Technology - Electronics And Communications Engineering
Maharaja Agrasen Institute Of Technology, Guru Gobind Singh Indraprastha University
New Delhi,India 08.2015 - 07.2019
Skills
Python
Timeline
Senior Consultant
Birlasoft
04.2022 - 09.2022
Application Development Analyst
Accenture Solutions Pvt Ltd
09.2019 - 03.2022
Bachelor Of Technology - Electronics And Communications Engineering
Maharaja Agrasen Institute Of Technology, Guru Gobind Singh Indraprastha University
08.2015 - 07.2019