Databricks
Sayan Kumar Nandy
Azure Data Engineer
Kolkata
Summary
With over 9.5+ years of experience in the data engineering and SAP ecosystem, I have developed a strong expertise in building, managing, and optimizing end-to-end data pipelines across various platforms and technologies. I started my career as an SAP BW and BODS developer, where I honed my skills in data modeling, extraction, transformation, and loading (ETL) processes.
Over the last 3 years, I have transitioned to a Data Engineer role, primarily working with Azure Databricks, Azure Data Factory (ADF), and SQL Data Warehousing.
Overview
10
10
years of professional experience
4
4
years of post-secondary education
4
4
Certifications
3
3
Languages
Work History
Senior Data Engineer
Lexmark International
04.2022 - Current
- Spearheaded migration process of multi-layer ETL jobs from BigDecisions to Azure Data Factory and Azure Databricks, transforming UI-based solutions into scalable, reusable codebases using PySpark, improving report consistency and achieving 50% increase in data processing time.
- Integrated and processed data from multiple sources like AWS S3, AWS RedShift, D365, SqlServer, Oracle, handling various file formats like CSV, Parquet, Json, ORC and txt, using Azure Data Factory.
- Developed generic notebook to automate data processing of files with multiple data formats, leveraging Databricks' AutoLoader
capabilities, reducing manual code deployment and enhancing operational efficiency. - Developed Ingestion process to automate data load from SharePoint to Azure Databricks, using Azure Data Factory, completely removing manual intervention.
- Recreated functions and stored procedures of SqlServer in Databricks, enabling advanced reporting capabilities and decreasing report generation time by 50%, utilizing delta table optimization technique like Z-Ordering, Optimize and Vacuum and Databricks Workflow.
- Integrated multiple notebooks to streamline data processing and governance using Databricks Workflow, following Lakehouse Medallion Architecture, to process batch datasets of size 1TB, approximately containing 2 billion records daily.
- Planned and executed real-time streaming solution to support dynamic data processing, using Azure EventHub, Spark Structured Streaming and Databricks Workflow, efficiently managing 2 million records per hour.
- Successfully Completed Application Migration from Datastage to SAP BODS within tight timeline during SAP S4 HANA version upgradation.
Senior Associate Consultant
Infosys Ltd.
08.2018 - 04.2022
- Designed, Developed and Supported complex ETL jobs using Data Integrator Scripts, Workflows, Dataflow, Data stores, Transforms and Functions
- Implemented BODS job which will Work along with FIORI App as per Client Requirements
- Experience in Error handling and debugging coding issues and troubleshoot production problems
Application Development Analyst
Accenture
11.2014 - 08.2018
- Worked on SAP BODS and BW projects for data migration and reporting.
- Developed ETL jobs, InfoProviders, and queries for business users.
- Led data integration efforts using LSMW and supported production environments.
Education
B-Tech - Computer Science & Engineering
Meghnad Saha Institute Of Technology
Kolkata, India 06.2010 - 05.2014
Skills
Python programming
undefinedCertification
Certified SAFe® Practitioner
Affiliations
- Won Manager Appreciation Award For key Contribution in Structured Streaming Project- Lexmark.
- Won Quarterly Appreciation Award for successful Application Migration (Datastage to BODS)- Lexmark.
- Received AMS Peer Award for AMS Reporting Support as an appreciation for key team player in the project- Accenture.
- Received Insta award for contribution towards Business during Pandemic - Infosys
- Peer Award – Infosys
Work Preference
Work Type
Full TimeWork Location
RemoteImportant To Me
Work-life balanceCompany CultureFlexible work hoursPersonal development programsWork from home optionSoftware
Azure Data Factory
Pyspark
SAP BODS
Languages
English
Advanced (C1)
Hindi
Advanced (C1)
Bengali
Advanced (C1)
Timeline
Certified SAFe® Practitioner
04-2025
Databricks Certified Data engineer associate
03-2023
Senior Data Engineer
Lexmark International
04.2022 - Current
AZ-900(Microsoft Azure Fundamentals)
03-2022
CCA175 Cloudera Spark and Hadoop Developer
12-2020
Senior Associate Consultant
Infosys Ltd.
08.2018 - 04.2022
Application Development Analyst
Accenture
11.2014 - 08.2018
B-Tech - Computer Science & Engineering
Meghnad Saha Institute Of Technology
06.2010 - 05.2014
Similar Profiles
BEDANTADEEP DUTTABEDANTADEEP DUTTA
Agile Scrum Master at Lexmark InternationalAgile Scrum Master at Lexmark International
SAYAN SINGHASAYAN SINGHA
Senior UX Designer at Lexmark InternationalSenior UX Designer at Lexmark International
Jeffrey EhlertJeffrey Ehlert
Global Sourcing Process Improvement and Strategy Manager, Investor Relations Manager at Lexmark International, Inc.Global Sourcing Process Improvement and Strategy Manager, Investor Relations Manager at Lexmark International, Inc.
Ankita Kumari RajakAnkita Kumari Rajak
Associate Engineer, Fibre and Network Delivery at BT ESERV INDIA PVT LTDAssociate Engineer, Fibre and Network Delivery at BT ESERV INDIA PVT LTD
Swati Parvatham GhoshSwati Parvatham Ghosh
Marketing Lead – SEO, Content & Social Media at No Nirvana DigitalMarketing Lead – SEO, Content & Social Media at No Nirvana Digital