Balaji K G
Senior Data Engineer
Chennai,TN
Summary
Engineer with 6 years experience in Data Engineering domain. Interested in complex problem solving, real-time analytics, and reporting solutions that involve large-scale data warehousing.
Overview
6
6
years of professional experience
3
3
years of post-secondary education
Work History
Senior Data Engineer
Fourkites Private Limited
Chennai, Tamil Nadu
02.2018 - Current
- Designed and developed recommendation engine for notifying truck drivers for better route, weather and traffic prediction, theft zone alerts etc, with help of data science models.
- Processing >20M records per day to give better recommendation and analyzing over the history of million records to provide quality suggestions to the end users.
- Designed Custom ETL pipeline for one of dynamic yard product to capture CDC data using Debezium and integrated with Kafka for transforming and moving to DWH ( using spectrum external table concept)
- Designed and developed a new custom ETL application using Hadoop & Hbase to handle large-scale volume data processing into DWH.
- Custom ETL applications can process around 6L records per hour and 15 Million records per day, which is 5X improvement to existing ETL applications.
- Designed and implemented Custom ETL framework for other internal teams to interact with any source systems like Kafka, RDS and pushed into any systems like Kafka, DWH, no-sql data store like Dynamo / Hbase.
Software Engineer - Data
Systech Solutions Private Limited
Chennai, Tamil Nadu
02.2017 - 02.2018
- Designed and implemented multiple real time applications using spark streaming to pull data from multiple third party vendors.
- Effectively monitored real time applications using grafana.
- Have done most of admin activities to monitor and maintained HDP clusters.
- Have created and published few internal reports with help of tableau.
Data Engineer
ADF Data science Private Limited
Chennai, Tamil Nadu
11.2015 - 01.2017
- Designed and Developed Warehouse from scratch with custom python based pipeline.
- Have done POC for exploring open source Pentaho ETL tool and with positive results, migrated legacy ETL pipeline to pentaho for better monitoring, scheduling and maintenance.
Education
Bachelor of Engineering - Electrical Engineering
KLN College of Engineering
Madurai 10.2011 - 02.2015
Skills
Programming : Python, Scala
SQL: Redshift, Snowflake & Postgres
NOSQL: HBase & DynamoDB, Basics of Cassandra
AWS : EMR , S3, RDS, DynamoDB, EC2, SQS, Redshift
Azure : HDInsights Cluster, Azure Cosmos, Azure blob storage, Key vault, VM, Azure databases for postgres
Logging & Monitoring : Logentries, Grafana, Prometheus , Cloudwatch, Log analytics workspace in azure
Workflow : Apache Airflow
Bigdata Frameworks : Hadoop, HDFS, Hive, Spark (Streaming), Kafka, Zookeeper, Sqoop, Hbase - Phoenix,
Agile Tools : Jira, Scrum, Kanban
CI/CD : Jenkins
CDC Tool : Debezium
Timeline
Senior Data Engineer
Fourkites Private Limited
02.2018 - Current
Software Engineer - Data
Systech Solutions Private Limited
02.2017 - 02.2018
Data Engineer
ADF Data science Private Limited
11.2015 - 01.2017
Bachelor of Engineering - Electrical Engineering
KLN College of Engineering
10.2011 - 02.2015