Selvamani M
Bengaluru
Summary
Dynamic Data Engineer with a proven track record at HP, specializing in designing efficient data pipelines and optimizing systems. Skilled in Talend Big Data and SQL, I excel in problem-solving and delivering high-quality data solutions that drive business insights. Passionate about mentoring and fostering collaboration across teams.
Overview
7
7
years of professional experience
Work History
Data Engineer
HP
Bengaluru
08.2022 - Current
- Designed data pipelines to manage large-scale data flows, improving data accessibility for analytics.
- Optimized existing data systems to enhance performance, resulting in reduced latency for data retrieval.
- Analyzed complex datasets to provide insights for business intelligence initiatives.
- Collaborated with cross-functional teams to integrate data solutions, enabling cohesive data strategy.
- Mentored junior engineers on best practices in data engineering methodologies.
Associate Consultant
Capgemini Technology Services India Limited
02.2021 - Current
- Onboarded data from sources like SAP, Oracle, Vertica, and flat files into Hadoop environment to enhance data accessibility.
- Prepared HiveQL scripts for complex transformations, aligning data processing with business requirements.
- Managed tables in EDL layer for organized data storage.
- Established verification processes to maintain data quality and accuracy.
- Transferred transformed data to Azure SQL DB using ADF Pipeline, ensuring seamless integration for analytics.
- Talend BigData (V 7.3.1) is used for all orchestration and data movement.
Project Engineer
Wipro Limited
05.2019 - 01.2021
- Developed ETL jobs to extract data from diverse sources, including SQL, Oracle, Vertica, SAP Hana, and flat files.
- Create a ADF Pipeline to load the data from Azure Blob to Azure SQL Database.
- Designed and developed frameworks for onboarding and integrating data from various platforms into Big Data environment.
- Preparing HQL Scripts to do the forecasting in Supply Chain Management.
- Facilitated discussions with business stakeholders to define requirements and clarify client expectations.
Education
Bachelor’s - Electronics and Communication
Karpagam College of Engineering
01-2019
Skills
- SQL
- SQL programming
- Azure ADF
- Talend Big Data
- Hive
- Spark framework
- Scala programming
- Advanced Python
- Data integration
- Pipeline design
- Data governance
- Query optimization
- Problem solving
Disclaimer
I, SELVAMANI M, do hereby confirm that the information given above is true to the best of my knowledge.
Scope
Project Scope is to onboard data from different sources like Sap, Oracle, Vertica and also from flat files to Hadoop Environment. Prepare HiveQL Scripts to do complex transformations to meet the business requirement. All the tables are maintained in the EDL Layer. There are multiple checks and balances to make sure the data quality and accuracy. Once all the transformation is done, data will be loaded to Azure SQL DB through ADF Pipeline. Talend BigData (V 7.3.1) is used for all orchestration and data movement.
Roles And Responsibilities
- Designing and developing frameworks for onboarding and integrating data from different platforms to Big Data environment.
- Preparing HQL Scripts to do the forecasting in Supply Chain Management.
- Developing ETL Jobs to extract the data from different source that comprise databases like SQL, Oracle, Vertica, SAP Hana and flat files.
- Create a ADF Pipeline to load the data from Azure Blob to Azure SQL Database.
- Communicating with Business to gather the requirements and meet the expectations and managing the client expectations as a indicator of Quality.
Timeline
Data Engineer
HP
08.2022 - Current
Associate Consultant
Capgemini Technology Services India Limited
02.2021 - Current
Project Engineer
Wipro Limited
05.2019 - 01.2021
Bachelor’s - Electronics and Communication
Karpagam College of Engineering