Anup Tiwari
Lead Data Engineer
Gurugram
Summary
Lead Data Engineer with 8 years relative experience and overall 10 years in BFSI domain. Skilled in data architect, understand business requirement and convert into technical, developing End to End pipeline, well versed with Azure data engineering related services, Databricks.
Overview
3
3
years of professional experience
1
1
Certification
Work History
Lead Data Engineer
Prudential Assurance
02.2023 - Current
- Optimized data processing by implementing cutting-edge data engineering techniques and tools.
- Aligned business objectives with technical requirements through close collaboration with product owners providing insights based on available data sources.
- Developed many complex and critical data applications.
- Optimized existing data pipelines.
- Developed scalable infrastructure capable of handling vast amounts of structured and unstructured data, improving overall system performance.
- Designed complex data processing applications.
Data Engineering Lead
UnitedHealth Group
07.2022 - 02.2023
- Collaborated with cross-functional teams to define requirements and develop robust data solutions.
- Developed efficient data processing application.
- Streamlined communication between Data Engineering Lead team members by hosting weekly meetings that encouraged open discussion of ongoing projects.
- Migrated legacy systems to modern big data platforms, ensuring seamless business continuity during transitions.
- Improved data quality by implementing ETL processes and data cleansing techniques.
Specialist Data Engineer
L&T InfoTech
10.2021 - Current
- Worked for Citi Bank in the data platform team
- Developed pipline to process the huge amount of data
- Optimized the piepline
- Closely working with business team to understand their data requirement.
Senior Software Engineer
Bank Of America
10.2016 - 9 2021
- Built a Data-pipeline to consume the data and dump into Hive table using Zaloni- Sqoop framework (Raw Data)
- Developed spark-scala job which do read the data from multiple bronze tables and process the data as per the business requirement
- Insert processed data into gold hive table
- These data will be further moved to oracle table and Beatle landing zone that will be used by business team
- Involved in bug fixing and query optimization.
Education
Master of Computer Application - undefined
RGPV, Bhopal
07.2008 - 2011.05
Skills
- Spark
- Data Warehousing
- Data Modelling
- ETL
- CI/CD
- Azure Cloud
- Databricks
- DLT
- Hive
- Kafka
- Core Java
- Python
- Scala
- SQL
- Spark
Certification
Databricks Certified Associate for Apache Spark 3.0, 10/20/2021
Languages
English
Hindi
Timeline
Lead Data Engineer
Prudential Assurance
02.2023 - Current
Data Engineering Lead
UnitedHealth Group
07.2022 - 02.2023
Specialist Data Engineer
L&T InfoTech
10.2021 - Current
Senior Software Engineer
Bank Of America
10.2016 - 9 2021
Master of Computer Application - undefined
RGPV, Bhopal
07.2008 - 2011.05
Similar Profiles
Jackline MaaseJackline Maase
Agency Manager at Prudential AssuranceAgency Manager at Prudential Assurance
Thajmeel AhamedThajmeel Ahamed
Prudential Financial Advisor at Prudential Assurance Company Singapore (Pte) LtdPrudential Financial Advisor at Prudential Assurance Company Singapore (Pte) Ltd
NOR AZLAN BIN NASIRNOR AZLAN BIN NASIR
Financial Consultant at Prudential Assurance Company SingaporeFinancial Consultant at Prudential Assurance Company Singapore