Divesh Harisinghani
Data Engineer
Gurugram
Summary
Experienced Data Engineer with 7.4 years of expertise in crafting, implementing, and refining robust ETL solutions within the Azure ecosystem. Proficient in leveraging Azure Data Lake, Azure Databricks, Azure Data Factory, Azure Synapse Analytics, Azure Blob Storage, and Azure SQL Database to develop efficient data pipelines. Demonstrated success in driving substantial performance enhancements, evidenced by a remarkable 40% reduction in query execution time through adept optimization of Hive and Spark jobs. Proven ability to fortify data security frameworks, resulting in a notable 20% decrease in security incidents . Skilled in query tuning and optimization, particularly adept within Databricks and Spark environments. Passionate about harnessing technology to drive data-driven insights and streamline operations.
Overview
8
8
years of professional experience
4
4
years of post-secondary education
1
1
Certification
Work History
Staff Engineer
Nagarro
08.2022 - Current
- Created ETL pipelines tailored for customer facing products such as scorecard and Vendor Partner Plus.
- Extracted data from diverse sources including DB2, mysql and Teradata channeling into Apache Druid for optimal low latency data retrieval.
- Achieved significant optimization in big data processing, notably 40 % decrease in query execution time by optimizing Hive and Spark jobs.
- Engineered a comprehensive framework to facilitate the seamless migration of oozie workloads to airflow.
- Utilized Jenkins release pipeline for deployment of code in both QA and
- Provided crucial Production and UAT support ensuring the reliability and
Data Engineer
Natwest Group
06.2021 - 07.2022
- Demonstrated strong organizational and time management skills while managing multiple projects.
• Developed ETL pipelines for Data products like Feature Bank using pyspark. Building Data lake platform on Azure cloud using services like ADLS gen2 ,Azure Databricks, Synapse analytics .
• Data Management, Data Access, Data Governance and Integration , Security, and Operations performed by using Cloudera Platform.
• Supported the tenants applications in all the areas starting from provisioning their environments, access provisioning ,job optimization , pipeline building , fixing applications or platform related issues.
• Optimizing Hive , Spark and Impala workloads for tenant applications.
Associate
Cognizant Technologies
04.2020 - 06.2021
- Managed 40+ production hadoop server (2 instances of server) for cloudera distribution in banking domain.
- Developed reconciliation scripts for Data Quality and Integrity checks using pyspark.
- Troubleshooting issues for end users in both the environment which was managed by ServiceNow Portal and JIRA tickets within the specified SLAs.
- Deployed databases for different zones in hadoop server like Data Discovery, Data Analytics Store, Enterprise Analytics Zones and also creating new sentry roles, Ad groups and mapping accordingly.
- Troubleshooting Oozie, Yarn, Spark issues.Running and creating BDR jobs for hive and hdfs replication.
Software Engineer
Global Logic Technologies
04.2018 - 03.2020
- Hands on experience in Hadoop ecosystem including HDFS, Spark, Hive, Sqoop, Oozie, MapReduce , YARN, Sentry, Kerberos, Kafka .
- Wrote Sqoop jobs to import the data into HDFS.Involved in creating Hive tables, loading with data and writing Hive Queries that will run internally in MapReduce way.
- Use Hive optimization techniques during joins and best practices in writing Hive scripts using Hive.
- Involved in converting Hive Queries into Spark transformations using Spark RDD and Python.
Software Engineer
Newgen Software Technologies
08.2016 - 12.2017
- Interacting with clients' business user and process owner groups to gather the business requirements effectively to enable the development/configuration team to take the project further and ensure deliveries with no issues/gaps.
- Implement Log4j as method for debugging and testing purposes.
- Integration with ambit which is Core Banking Solution .
- Worked on JSP, Servlets as a technology to implement various CRs in the project.
- Developed a SOAP based Webservice for third party to provide a service for Document upload in OmniDocs(DMS).
Education
Bachelor of Technology - Computer Science And Engineering
The NorthCap University
Gurgaon, India 05.2012 - 06.2016
Skills
Azure Databricks
Certification
Azure Data Engineer Associate
Timeline
Staff Engineer
Nagarro
08.2022 - Current
Azure Data Engineer Associate
02-2022
Data Engineer
Natwest Group
06.2021 - 07.2022
Associate
Cognizant Technologies
04.2020 - 06.2021
Software Engineer
Global Logic Technologies
04.2018 - 03.2020
Software Engineer
Newgen Software Technologies
08.2016 - 12.2017
Bachelor of Technology - Computer Science And Engineering
The NorthCap University
05.2012 - 06.2016
Similar Profiles
Yuvika SinghYuvika Singh
Associate Director at NagarroAssociate Director at Nagarro
ANUJ KUMARANUJ KUMAR
Delivery / Program Manager at NagarroDelivery / Program Manager at Nagarro
Rajyashree ChaturvediRajyashree Chaturvedi
Associate Staff Engineer at NagarroAssociate Staff Engineer at Nagarro
Anas KhanAnas Khan
Senior Team Leader at NagarroSenior Team Leader at Nagarro
Amanjot Kaur LambaAmanjot Kaur Lamba
Workday Integration Consultant at AccentureWorkday Integration Consultant at Accenture