Satyam Agarwal

Copenhagen

Summary

Data Engineer having 5+ years of experience in Big Data analytics and Data Warehousing using Hadoop Eco system tools, SQOOP, Hive, Spark, Scala, Unix Shell Scripting, SQL, Azure and Oracle. to enable a better data integration ,aggregation and reporting process.

Proficient in programming languages such as Python, Scala and SQL with strong understanding of database design and management. Skilled in data modeling, ETL processes, and designing/implementing data lakes and data warehouses.
Experience in implementing partitioning, bucketing, vectorization, cost based optimization, indexing and various optimization techniques to improve the performance of Hive queries.
Working experience in Agile and SAFe environments. Having a decent knowledge in Machine Learning and experience on several algorithms like Regression, Decision Trees and Neural Networks
Responsive expert experienced in monitoring database performance, troubleshooting issues and optimizing database environment.
Complex problem-solver with analytical and driven mindset.

Overview

years of professional experience

Certification

Work History

Tata Consultancy Services

Data Engineer

06.2019 - Current

Job overview

With expertise in building Big Data and Analytics, Responsible for design, development and optimization of scalable data pipelines for various structured and unstructured data sources, building API Integrations and accelerating performance of ETL processes using Hadoop Ecosystem tools /Apache Spark
Created SPARK programs with use of dynamic memory allocation and in memory computation to ensure robust transformations.
Experience in implementing partitioning, bucketing, vectorization, cost based optimization, indexing and various optimization techniques to improve the performance of Hive queries.
Experience in writing shell scripts and automating the spark and hive workloads using UC4 Automic Scheduler in the production system Experience with different file formats and handling the structured and nonstructural data using HIVE.
Responsible for gathering Business Requirements from Product Owners and creation of rules and transformations in SPARK and Scala with respect to them in order to receive targeted data requirements
Responsible for building unified Power BI dashboard for entire group of teams
Collaborated on ETL (Extract, Transform, Load) tasks, maintaining data integrity and verifying pipeline stability.
Developed and delivered business information solutions for more then 500 different aggregation rules for several invoicing services.
Gathered, defined and refined requirements, led project design and oversaw implementation.
Contributed to internal activities for overall process improvements, efficiencies and innovation.

Education

IIIT

Post Graduate Diploma from Machine Learning And Artificial Intelligence

09.2021

VIT

Bachelor of Technology from Computer Science And Engineering

06.2019

Skills

Big Data Technologies: Hadoop, HDFS, YARN, Apache Hive, SQOOP, Apache Spark
Service Provider: Hortonworks, Cloudera
Programming Languages: Scala, Python
Databases: MySQL, Oracle
Data Visualization: Microsoft POWER BI
Version Control: Bitbucket and Anaconda

Operating System: Linux/Unix, Windows
Atlassian Suite: Jira, Confluence
Machine Learning: Regression, Classification, Decision Trees, Clustering,
Small file handling in HIVE and Spark
Performance optimization in joins and queries
Handling different big data file formats : like Parquet, ORC and Avro

Certification

Machine Learning Course by Stanford University (Coursera)
Certified SAFe® 5 Practitioner Scaled Agile

Languages

English

Proficient

German

Beginner

Hindi

Bilingual or Proficient (C2)

Timeline