Pankaj Kumar
Bengaluru
Summary
Results-driven Senior Engineer with over 11 years in IT and 5 years specializing in Big Data and cloud development using PySpark and Azure. Expertise in designing scalable data pipelines and migrating on-premise systems to Azure. Proficient in Spark Core, Spark SQL, DataFrames, Hadoop, Hive, Azure Data Factory, and Databricks, with solid experience in data warehousing and BI reporting.
Overview
12
12
years of professional experience
Work History
Senior Engineer
Cigniti Technologies,Bright Horizons
08.2025 - 02.2026
- Designed and implemented large-scale ETL pipelines for data ingestion, transformation, and aggregation using PySpark, HQL, Python, and Azure, facilitating efficient data processing and analysis.
- Built automated data validation framework using PySpark, enhancing data accuracy and reducing manual effort by 50%.
- Utilized PySpark, DataFrames to perform data preprocessing tasks such as filtering, de-duplication, null value handling and business rule enforcement to ensure data accuracy and consistency, reducing errors by 35%.
- Executed data validation with SQL and PySpark scripts in cloud-based environments, ensuring data integrity and reliability.
- Now a coforge company
Senior Engineer
Apexon, CRUX
08.2019 - 09.2024
- Developed and maintained scalable data pipelines using PySpark in Databricks to ingest, transform, and process high-volume stock market and financial market datasets from multiple data sources.
- Processed daily market data, including trade, pricing, security master, and reference data, ensuring accuracy and timely availability for downstream analytics to support informed decision-making.
- Designed and implemented reusable PySpark modules for data cleansing, transformation, de-duplication, and business rule enforcement on large-scale financial datasets.
- Built an automated PySpark-based data validation framework to verify market data completeness, schema consistency, and business rule compliance, reducing manual validation effort by approximately 50%.
- Developed reusable Python scripts to automate data ingestion, validation, and exception handling, enhancing operational monitoring and streamlining data workflows.
- Wrote and optimized complex SQL and HiveQL queries for data extraction, transformation, reconciliation, and reporting across large financial datasets.
- Worked with the Hadoop (HDFS) ecosystem to store and process high-volume historical market data, utilizing Hive for efficient querying and data analysis.
- Optimized PySpark jobs using partitioning, caching, broadcast joins, and efficient DataFrame transformations, reducing processing time by nearly 30%.
- Implemented data quality checks, including schema validation and duplicate detection, ensuring reliability of financial attributes such as ticker symbols and trade dates for accurate analytics.
QA Engineer
Imagine Technology and Services Pvt Ltd,LandMark Healthcare
11.2018 - 07.2019
- Validated SSIS ETL workflows, ensuring correct data extraction, transformation, and loading from source to target tables.
- Created and executed SQL test cases to compare source and target data after SSIS package runs.
- Tested data loaded from application to EDW tables on MS SQL server, ensuring data integrity and reliability.
- Verified successful execution and error handling in SSIS package logs, contributing to error resolution and data quality assurance.
- Ensured accuracy and layout consistency of SSRS reports, confirming correct aggregation logic for reliable data insights.
Project Engineer
Wipro, Allied Irish Bank
08.2017 - 10.2018
- Executed ETL test cases using Teradata SQL, enhancing data integrity through advanced test queries.
- Used many SQL commands like Qualify, Rank, Partition By etc to make test simple.
- Developed diverse testing scenarios and documented test cases to ensure comprehensive coverage.
- Engaged daily with clients to provide updates and address concerns, fostering strong client relationships.
Project Engineer
Wipro,Canara Bank
11.2016 - 08.2017
- Developed and validated ETL pipelines in SAS DI Studio to ensure data integrity and accessibility.
- Tested SAS OLAP cubes and validated data hierarchies to guarantee accuracy of SAS WRS reports.
- Managed project schedules to ensure timely delivery of engineering solutions.
- Coordinated cross-functional teams to facilitate effective communication and collaboration.
- Developed technical specifications for project requirements and deliverables.
Project Engineer
Wipro,ESIC
04.2014 - 10.2016
- Led client demos for HIS BI reports and dashboards, showcasing functionality and facilitating user feedback.
- Developed BI reports and dashboards for modules like HIS, Finance, and Insurance.
- Created and scheduled ETL jobs in SAS DI Studio to fetch data from multiple applications into landing zone, transforming and loading into Data Warehouse to support data access.
- Performed SQL-based ETL testing to validate data quality and integrity.
- Validated reports down to data level using MySQL.
- Defined dimension hierarchies using SAS Cube Studio to enable report drill-down functionality.
- Scheduled DI jobs and cubes by creating flows in SAS SMC Schedule Manager, and triggered execution through LSF.
- Authored High-Level Design (HLD), Low-Level Design (LLD), and Test Design Documents, and collaborated with offshore teams and managers for reviews.
- Addressed data and reporting requests from stakeholders, delivering results on time to meet project deadlines.
Education
Bachelor Of Engineering - CSE
Haldia Institute of Technology
Haldia, West Bengal, India07-2013
Skills
- Pyspark
- Azure
- Databricks
- Hadoop
- Hive
- SparkSQL
- SQL
- Python
- Data Modeling
- Data Warehousing
- Data Quality
- Apache Airflow
- SAS DI Studio
- SAS BI
- SSRS
- API Development
- API Testing Tools
- JIRA
- Shell Script
Strengths And Achievements
- Commitment to work and ability to learn new technologies on my own.
- A dedicated team player, committed to providing high-quality support.
- Flexible and versatile to adapt to any new environment and work on any project.
- Quick learner and ability to meet tight deadlines and work under pressure.
- Got 'On the spot' award for Aug & Oct Financial Year (2015-2016).
- Got Special recognition Award.
- Got Certificate of Achievement for the contribution of ESIC Program.
Timeline
Senior Engineer
Cigniti Technologies,Bright Horizons
08.2025 - 02.2026
Senior Engineer
Apexon, CRUX
08.2019 - 09.2024
QA Engineer
Imagine Technology and Services Pvt Ltd,LandMark Healthcare
11.2018 - 07.2019
Project Engineer
Wipro, Allied Irish Bank
08.2017 - 10.2018
Project Engineer
Wipro,Canara Bank
11.2016 - 08.2017
Project Engineer
Wipro,ESIC
04.2014 - 10.2016
Bachelor Of Engineering - CSE
Haldia Institute of Technology