Aman Kumar
Data Engineer - Big Data Analytics
Noida,
Summary
Having more than 10 years of experience in IT Industry and with can-do attitude, phenomenal time management skills and a strong user focus, constantly striving to learn new technologies and look for new ways to better myself in this rapidly changing industry.
Overview
11
11
years of professional experience
6
6
years of post-secondary education
Work History
Senior Technical Lead
HCL Technologies Ltd.
Noida
02.2020 - Current
- Developing & Managing Big Data Platform and serving real time Platforms for end customers for money transfer which involve complex pipelines with strict SLA’s.
- Migration from big data platform to Snowflake and AWS.
- Worked on development of critical reporting framework that helped WU to achieve on-time reporting for regulators.
- Worked on multiple automations for projects to mitigate the manual effort and make the monitoring automated for SLA critical components.
Assistant Consultant
Tata Consultancy Services Ltd.
Noida
10.2012 - 01.2020
- Worked on multiple data projects in big data technologies which helped end users to gain insight on financial and royalty on iTunes data.
- Served as team lead to have direct interaction with client and performed Techno functional lead role.
- Took initiatives to perform first time data ingestion for new tasks and worked with multiple teams.
- RFP/SOW submission for new projects.
- Handing responsibilities for projects starting from requirement gathering to production deployment.
Education
Master Of Computer Applications (M.C.A) - Computer Applications
Motilal Nehru National Institute of Technology
Allahabad 07.2009 - 06.2012
Bachelor of Science - Mathematics
Lucknow University
Lucknow 07.2005 - 06.2008
Skills
Hadoop Distribution: Cloudera, Hortonworks
Programming Language : Python, Unix, Pyspark, Spark With Scala
Data Processing: Spark, Hive, Python
Data Ingestion: Sqoop, Matillion, Spark Streaming, Talend
Database/ Cloud DW : Snowflake, Cassandra, Teradata, Oracle
Orchestrator: Airflow, Oozie, Autosys, Rundeck
Public Cloud:AWS
Projects
Hadoop to Snowflake Migration
Lead Developer/ Technical Specialist
- Project aimed at migrating the complex compliance jobs from Hadoop eco system to snowflake DWH.
- While migrating, queries were optimized, and un-necessary code has been removed.
- Pipelines were developed in Matillion to execute from snowflake.
- Instead of hive UDF, snowflake inbuilt functions were used.
- Used clustering keys in large tables for speed up the queries.
- Used snowflake connector for spark to move history data from hive to snowflake and comparing the data between hive and snowflake post migration.
- Used JavaScript based UDFs and UDTFs in case functionality was not achievable with snowflake inbuilt functions.
- Used snowsql, snowpipe, snowflake COPY, variant data handling/flattening,Time travel etc.
Timeline
Senior Technical Lead
HCL Technologies Ltd.
02.2020 - Current
Assistant Consultant
Tata Consultancy Services Ltd.
10.2012 - 01.2020
Master Of Computer Applications (M.C.A) - Computer Applications
Motilal Nehru National Institute of Technology
07.2009 - 06.2012
Bachelor of Science - Mathematics
Lucknow University
07.2005 - 06.2008