MANIKANTHA NEKKALAPUDI
Senior Data Engineer
Hyderabad
Summary
Detail-oriented data engineer designs, develops and maintains highly scalable, secure and reliable data engineering infrastructure. Accustomed to working closely with data and system architects and data analysts to understand business to develop comprehensive data processing systems.
Overview
9
9
years of professional experience
2
2
Certifications
4
4
Languages
Work History
Senior Data Engineer
TTEC Digital
11.2021 - Current
- Built self-healing ingestion pipelines to obtain hourly data from Salesforce, Cisco and SFTP sources in ADF for Sales CDP
- Curated raw layer data and update the reporting tables with incremental data using SCD Type 2 method
- Built an intermediate layer for data intensive table with only latest records per unique id at ingestion step to reduce the processing time for reporting layer tables and datasets
- Optimized ADF pipelines to perform parallel copy activity for Salesforce objects
- Optimized PySpark code and ADF parallel copy activity to cut down runtime by 50% for hourly loads
- Implemented an insurance escalation process for contact center agents to get all the historical information in the warehouse for a customer within 30 mins with minimal input information
- Implemented a call directive creation/cancellation framework for agents in salesforce using the Salesforce API and scaled it for multiple process across business units
- Created process for data enrichment for incoming data for various sources for analytics datasets
- Created and debugged existing reporting Stored Procedures in Synapse that produces datasets for business teams
- Worked with stakeholders from BI and business teams to debug the data issues and inconsistencies in the reporting datasets generated from Stored Procedures in Synapse.
Senior Data Engineer
Affine
03.2021 - 11.2021
- Improved CDC framework in bash to ingest data from source SAP views and ingest into HDFS using Cloudera HDP for a commodities client
- Created a producer and consumer processes to ingest data and create reporting datasets independently for efficiency of the incremental runs and to complete within 30 mins SLA
- Built reporting datasets using Impala SQL and Apache Kudi tables for business teams
- Created an improved methodology for weekly deployments with GitHub release tags.
Data Engineer
Accenture
12.2017 - 03.2021
- Built a Step function that converts files like txt, csv, and other file formats to parquet file format based on S3 event triggers and Lambda functions for Step function invocation
- Created Glue jobs to process the files to parquet file format within the Step Function
- Created Spark-based ingestion scripts for ingesting data using SCD Type 2 on schedule from various insurance vendors using AWS EMR
- Worked with CloudFormation templates to create ephemeral AWS EMR clusters using configuration YAML files on S3 for test clusters
- Used CloudFormation to create deploy resources like S3 buckets, IAM roles, Lambda functions and glue jobs for production environment
- Converted reporting SQL scripts in Teradata scripts into Spark SQL for performance gains with Spark
- Converted the ingestion scripts to Databricks version of PySpark for platform migration and used native workflow for creating jobs and workflows
- Using Databricks jobs API trigger the jobs on schedule and send alerts based on failure to the stakeholders.
Test Automation Engineer
Accenture
11.2015 - 12.2017
- Designed a test automation framework with Java-Selenium framework with TestNG
- Executing the regression test suite on the Selenium grid and publishing the results to the product stakeholders
- Used Pandas and NumPy libraries in Python to build python scripts for validating data displayed in the dashboards.
Education
B.E. (Automation and Robotics) -
B.V.Bhoomaraddi College of Engineering and Technology, Hubli
Skills
IaC
Terraform
Athena
Step Function
Lambda
Glue
AWS EMR
AWS S3
Azure Storage
Azure Synapse
Azure Data Factory
PySpark
SQL
Python
Blog
https://maninekkalapudi.io
Certification
AWS Developer Associate
Work Availability
monday
tuesday
wednesday
thursday
friday
saturday
sunday
morning
afternoon
evening
swipe to browse
Timeline
Senior Data Engineer
TTEC Digital
11.2021 - Current
Senior Data Engineer
Affine
03.2021 - 11.2021
Data Engineer
Accenture
12.2017 - 03.2021
Test Automation Engineer
Accenture
11.2015 - 12.2017
B.E. (Automation and Robotics) -
B.V.Bhoomaraddi College of Engineering and Technology, Hubli