Debayan Pal
Hyderabad
Summary
Dynamic Data Engineer with a proven track record at IBM India Pvt Ltd, specializing in data lake enhancements and API integration. Excelled in transforming data warehouses and leveraging AWS services, demonstrating exceptional Python and collaborative skills. Achieved significant efficiency improvements in data management and exposure, ensuring robust, scalable solutions.
Overview
11
11
years of professional experience
1
1
Certification
Work History
Data Engineer
IBM India Pvt Ltd
Hyderabad
06.2022 - Current
- The objective of the project is to provide a method of data exposure (mainly supporting the synchronous request-response serving pattern on demand) that enables the consumer to rapidly utilize the data from the data lake without the need to develop their own data exposure pattern.
- It enhances the data lake by offering a framework that facilitates the exposure of data through an API pull pattern.
- It enables the delivery teams to mainly focus on transforming and transferring data to the insight zone (DynamoDB) without spending any effort to build a new backend API service.
- Work with the business analyst team to gain a deep understanding of the requirements, and help in building the mapping document.
- Prepare the DAB document, which contains the details of high-level architecture design, security design of the data lake, sequence diagram, cost of each AWS component, logging, and monitoring design, etc.
- Development of the end-to-end application code assigned to me, and overseeing the development activity of teammates.
IT Analyst
TATA Consultancy Services Ltd.
12.2020 - 06.2022
- Migrate the data warehouse from Ab Initio (ETL tool) to an on-premise Hadoop cluster using PySpark, Hive, Python, and MongoDB.
- Gather and analyze client requirements, translating them into Spark-based implementation designs.
- Conduct end-to-end development, unit testing using the Pytest framework for ensuring stable and efficient production code.
Senior Software Engineer
ENQUERO GLOBAL LLP / TAVANT TECHNOLOGIES
08.2019 - 12.2020
- This project is to provide a datalake solution Cloud to the client for their data management which was previously on premise
- Project is to build the entire Datalake in Cloud which will give a time-efficient and better manageable solution to the client to deal with their ongoing reporting problem on premise
- Involved in Data Ingestion from MySQL database to s3 using DMS
- Invoking lambda to create tables in Athena
- Created spark code for transforming data applying business logic
- Created tables in Athena for all the layers
- Created glue job for all the transformations related tasks
- Converting the sql queries provided by the client to AWS glue jobs
Accenture Solutions Pvt. Ltd.
07.2017 - 07.2019
- This project is to provide a solution to the client for maintaining their customer information, moving the historical data into HDFS and transforming this huge data into relevant information useful for the client
- Project is built to fulfill the various requirement of the client such as Reporting, Marketing analysis and providing new offers to the customers, sim swap campaigning, generating billing reports
- Involved in Data Ingestion from MySQL database to HDFS using Sqoop
- Created spark code for transforming data applying business logic
- Created hive tables like- managed tables, external tables, partitioned tables as per the need
- Created hive scripts that help market analysts spot emerging trends by comparing fresh data with historical data
- Involved in Import and export Data from Hive to MySQL, HDFS to Hive, and HDFS to MySQL Using Sqoop
- Working on processing data for ETL activity by applying spark transformation and actions on Spark RDD and Dataframe like map, filter, group by, join, saveAsTextFile etc
Software Developer
Teamware Solutions Ltd.
02.2017 - 07.2017
- The objective of this project is to implement a Reconciliation process using Spark
- Receiving files from the upstream system, this recon process checks the summary count and the data count for each entity and loads the status into Oracle database
- Developed Spark framework to implement Reconciliation process
- This process creates the report of missing transactions in the feed and provides a monthly report to the customer informing the reason of missing transactions in the feed
- Involved in Design, Code review and Unit Test case review
Software Developer
Tech Mahindra Ltd.
01.2014 - 12.2016
- The aim of the project is to provide analytics to the users for the prediction of fraudulent on cards life cycle, whereas the fraudulent includes the birth of card to the closure of it
- Requirement Analysis on customer requirements and Technical Design documentation
- Maintaining SDLC process throughout the project implementation by creating LLD and DLD and HLD at each level of process
- Converting customer requirements into Abinitio graphs in an efficient manner
- Reviewing the code once the graph has been designed and ready for implementation
- Testing in a QA environment before implementing
- Preparing Document of Understanding (DOU) for the project release with proper sign-offs with all the required documents
Education
Bachelor of Technology - Information Technology
INSTITUE OF ENGINEERING AND MANAGEMENT
Kolkata06-2013
Skills
- Python
- Unix shell scripting
- Azure ADLS
- Apache Spark
- AWS s3,Glue,Lambda,Athena,Redshift
- SQL
- Hive
- HBase
- Data modelling
- CI/CD
- Git
- Bitbucket
- Jira
- Performance tuning
Certification
- CCA Spark and Hadoop Developer, Cloudera, 05/04/21
- AWS Cloud Practitioner, AWS, 11/03/26
Disclaimer
I hereby declare that all the information mentioned above is true to the best of my knowledge.
Accomplishments
- Instrumental in enabling Business Lending Lob to roll out the Business Lending Foundational Data Product as planned.Despite many obstacles and ensured data product being delivered is of the highest possible quality.
Websites
Timeline
Data Engineer
IBM India Pvt Ltd
06.2022 - Current
IT Analyst
TATA Consultancy Services Ltd.
12.2020 - 06.2022
Senior Software Engineer
ENQUERO GLOBAL LLP / TAVANT TECHNOLOGIES
08.2019 - 12.2020
Accenture Solutions Pvt. Ltd.
07.2017 - 07.2019
Software Developer
Teamware Solutions Ltd.
02.2017 - 07.2017
Software Developer
Tech Mahindra Ltd.
01.2014 - 12.2016
Bachelor of Technology - Information Technology
INSTITUE OF ENGINEERING AND MANAGEMENT
Similar Profiles
MUKESH KUMARMUKESH KUMAR
Automation Lead at Kyndryl Solutions Pvt. Ltd, IBM India Pvt. Ltd, Client: Airtel IndiaAutomation Lead at Kyndryl Solutions Pvt. Ltd, IBM India Pvt. Ltd, Client: Airtel India
Sameer P. MulgundSameer P. Mulgund
Network Reliability Engineer IBM Cloud at IBM, IBM India Pvt LtdNetwork Reliability Engineer IBM Cloud at IBM, IBM India Pvt Ltd
Anjan BhaumikAnjan Bhaumik
Scrum Master/Team Lead/workload Manager/Security Delivery Specialist at Kyndryl Solutions Pvt Ltd (Previously IBM India Pvt Ltd)Scrum Master/Team Lead/workload Manager/Security Delivery Specialist at Kyndryl Solutions Pvt Ltd (Previously IBM India Pvt Ltd)