Dynamic Sr. Data Engineer at One97 Communication Ltd (Paytm) with expertise in designing robust data pipelines and optimizing performance. Proven track record in real-time data processing and proactive problem-solving, leveraging skills in PySpark and cloud storage to enhance operational efficiency and drive impactful data-driven decisions.
Hi, I’m
Praful Saxena
Senior Data Engineer
Noida,U.P
Summary
Overview
8
years of professional experience
3
Certification
Work History
One97 Communication Ltd (Paytm)Noida
Sr. Data Engineer
03.2023 - Current
Job overview
- Collaborated closely on Consumer and Merchant tech tasks to align features with business needs.
- Developed and updated customer features while debugging issues for optimal functionality.
- Managed data processing and storage infrastructure across cloud platforms like AWS S3.
- Designed and implemented data pipelines for efficient extraction, transformation, and loading into data warehouse.
- Maintained 99% uptime for data pipelines ingesting streaming and transactional data using PySpark.
- Conducted performance tuning of database systems and queries to enhance efficiency.
- Installed and configured Apache Airflow, creating DAGs and migrating Azkaban flows to Airflow.
- Created Python scripts for alert notifications on Slack regarding On-Call incidents.
- Developed and integrated AI tools on the MCP server to automate feature development and deployment. Transitioned from manual coding to prompt-based workflows, enabling the AI system to auto-generate, deploy, and activate new features, significantly streamlining and accelerating the release process.
- Designed and managed real-time data pipelines by consuming payloads from Kafka topics via bootstrap servers, storing and processing data in subject topics, and orchestrating downstream delivery through adapter streaming integrated with GET, PUT, and POST APIs for seamless event-driven operations.
- Developed an automated ETL pipeline that maps customers to phone numbers, featuring a TSX-based frontend dashboard, enabling business teams to self-serve. Orchestrated backend processing with Scala jobs running on Azkaban, allowing business users to trigger mapping operations from the UI, and delivering mapped CSVs to email via presigned URLs for secure access.
- Optimized complex ETL pipelines, achieving a 40% reduction in costs, and decreasing the
runtime of a job by an average of 35% through performance tuning and resource
management. - Analyzed and designed a migration strategy from a traditional data warehouse to Lakehouse
architecture, evaluating performance benchmarks, cost optimization opportunities, and data
governance frameworks for large-scale feature engineering pipelines. - Built a near real-time customer location enrichment pipeline using Kafka. Previously, location updates were
processed through scheduled batch jobs every 30 minutes, where customer location data was transferred via
SFTP and updated periodically.Enhanced the architecture to a fully real-time streaming pipeline by consuming
events directly from Kafka, enriching and processing the data in-stream, and deriving GeoHash values from
latitude and longitude coordinates. The processed data is then used to update customer city and state information
in real time on CleverTap.Implemented a Kafka-based processing flow where: Incoming events are consumed
from Kafka topics.Processed and enriched data is published to consumer sink topics.External Topic Adaptor
consumes the sink topic data.External Enricher service performs real-time updates on CleverTap customer
profiles.
NCC LimitedLucknow
Jr. Engineer
04.2018 - 03.2023
Job overview
- Provide Analytical and Statistical solutions to the team by performing data analysis of machinery and plant performance, production and categorizing them in orders, which in turn helps to find the maximum usage of the machinery and plants. Hence, surge the efficiency.
- Reviewing and recommending improvements of all SOP procedures. Reviewing datasets related to machineries, plants over ERP (UCIIMS) and Tappet-Box. Acquiring data from primary or secondary data sources and maintaining databases.
- Work with Relational databases such as MySQL, SQL Server. Manage SQL Server databases through multiple product lifecycle environments.
- Extracting data from source Tappet-Box and Max-Pro. Performing DDL, DML operations as per client requirement and providing the insights. Documented the data such as ER and attributes, file descriptions and definitions.
- Ensured Data recovery, Maintenance and Data Integrity and space requirements for database. Developed reports, views and triggers for data analysis.
- Worked with join patterns, implemented Map side joins, and reduce side joins using Map-Reduce. Developed multiple map-Reduce jobs to perform data cleaning and pre-processing.
- Implemented and optimized Hadoop/Map-Reduce algorithms for Big-data analytics. Implementing pre-defined operators in spark such as map, flat map, filter, reduceBykey, groupby, aggregateBykey etc.
Education
Imarticus Learning InstituteLucknow
from Machine Learning
12-2022
University Overview
U.P.T.ULucknow
Bachelors of Technology from Mechanical Engineering
06-2014
University Overview
State Board of U.PLucknow
Senior Secondary
University Overview
State Board of U.PLucknow
Secondary
University Overview
Skills
- Apache Kafka
- Python
- Data pipeline design
- Cloud data storage
- PySpark development
- Real-time data processing
- Airflow orchestration
- Data analysis tools
- SQL database management
- Big data analytics
- Performance tuning
- Data modeling
- Machine learning
- ETL development
- Hadoop ecosystem
- Real-time analytics
- Python programming
- SQL and databases
- Data modeling techniques
- SQL
- Natural language processing
- Deep learning
- Time series analysis
- Tableau software
- Machine learning algorithms
- Big data technologies
- Proactive problem solving
- Big data processing
- Data warehousing
- Deep learning
- Time series analysis
- Tableau software
- Machine learning algorithms
- Big data technologies
- Proactive problem solving
- Kafka streaming
- Big data processing
- Data warehousing
Certification
- Completed Python course from Udemy (Advanced)
- Completed Machine-learning course from Internshala
- Completed Six Sigma Course from IIBM
Projects
Projects
E-Commerce Shipping Delivery Dataset (Advanced ML), 06/01/22 - 07/31/22, In order to check whether the product will deliver on time or not, since delivery of the product is based on mode of transport, location and availability of the product. Therefore, it is very urgent to perform the EDA and pre-processing because it will refine our data to perform ML algorithms., NumPy, Pandas, PCA, Machine Learning Age-Gender-Ethnicity Face Detection by Using CNN, 10/01/22 - 11/30/22, I took a dataset of real time UTK age, gender, ethnicity face dataset in this I have performed Data Distribution work, Image Pre-Processing work and Convolutional Neural Network to make prediction of the ethnicity, age and gender. Use of CNN on this dataset and motto is to identify a person face by his ethnicity because in near future this AI will be applicable in government agencies., OpenCV, Keras/TensorFlow, Deep Learning
Achievements Extracurricular Activities
Achievements Extracurricular Activities
- Secured 7th rank in KPMG Hackathon Skillenza Competition
- Achieved 2nd position in Inter College Football Competition
- Secured top 500 rank in Kaggle Competition Tabular Playground Series
- Worked on cancer Aid Society during National Cancer Control Program and awarded for awareness
- Enrolled as Volunteer at BHUMI NGO working for orphanage
Work Preference
Job Search Status
Open to workWork Type
Full TimeLocation Preference
HybridRemoteSalary Range
₹1800000/yr - ₹3500000/yrCoursework
Coursework
- Mathematics- Linear Algebra, Basic Calculus
- Machine Learning- Linear Regression, Classification, Ensemble Technique, Random Forest, KNN, SVM, PCA, Naïve Bayes
- Deep Learning- ANN (Artificial Neural Network), CNN (Convolutional Neural Network), RNN, NLP (Natural Language Processing), LSTM, Open-CV, Clustering
Timeline
Sr. Data Engineer
One97 Communication Ltd (Paytm)
03.2023 - Current
Jr. Engineer
NCC Limited
04.2018 - 03.2023
Imarticus Learning Institute
from Machine Learning
U.P.T.U
Bachelors of Technology from Mechanical Engineering
State Board of U.P
Senior Secondary
State Board of U.P
Secondary