Narendra T
Azure Data Engineer
Hyderabad,Telangana
Summary
Seasoned IT professional with 6.6 years of expertise, including 5.5 years as an Azure Data Engineer proficient in Pyspark, ETL, Python, Databricks, and Azure SQL. Experienced in Airflow, adept at driving innovative solutions for complex business challenges. Specialized in JavaScript-based Squish automation for CT-Scan applications for 1.1 years.
Overview
7
7
years of professional experience
3
3
years of post-secondary education
3
3
Languages
Work History
Big Data Engineer
Massmutual
Hyderabad
10.2023 - Current
- Developed and maintained Apache Airflow DAGs for job automation and orchestration.
- Led the Life Underwriting QA Project, extracting API data from MaestroQA and implementing Slowly Changing Dimension (SCD) Type 1 for fact and relation tables.
- Collaborated with business teams to resolve Life and DI insurance data issues, identifying gaps and implementing effective solutions.
- Designed and implemented an incremental data load process, improving data processing efficiency.
- Investigated and resolved agent_name data discrepancies by working with the MaestroQA team, leading to the integration of a new API for accurate data retrieval.
- Currently developing Airflow DAGs for a project where the ServiceNow team manually triggers DAGs with parameters like policy_number and ticket_number.
- Engineered a dynamic SQL query generator that constructs WHERE clauses based on input parameters and executes backend queries.
- Automated data extraction and transformation, converting SQL query results into TXT files and storing them securely in Amazon S3.
- Integrated extracted data with MFT processes, enabling seamless handoff to mainframe systems for PDF generation to support business operations.
Big Data Engineer
Cognizant Technology Solutions
Hyderabad
10.2022 - 10.2023
- Experience in migrating Impala SQL scripts to Azure SQL in Databricks and connecting Azure Databricks tables to Power BI and Ensure output matches with existing report
- Transforming data and writing output tables into Azure Data Lake Storage (ADLS) using Pyspark code
- Migrated Scala code to PySpark while maintaining the transformation logic
- Experience in migrating SSIS packages to Azure Databricks using PySpark and Ensure the functionality of SSIS packages is maintained
- Hands-on experience in application development within Azure Databricks using PySpark and Azure Storage (ADLS Gen2)
- Hands-on experience with Databricks Workspace User Interface, managing notebooks, and Delta Lake with Python and Spark SQL
- Proficient in implementing data validation processes utilizing both Azure services and Cloudera Data Platform (CDP) for ensuring data accuracy and quality
- Experience in creating workflows/jobs in Databricks
Data Engineer
TCS
Hyderabad
11.2021 - 10.2022
- Migrated Certificate Monitor project from a virtual machine (VM) to Azure Databricks, leveraging PySpark
- Proficiently designed, developed, and executed data processing tasks within the Azure Databricks environment, enhancing efficiency and scalability
- Development of daily table check project for Product Quality Analytics Team using PySpark
- It performs regular checks on tables to ensure data quality and implemented data validation and quality checks
- Development of Primary Suction Project to limit harm to the environment by extracting flue gases
- Implementation of PySpark code on Databricks for transformation in staging and master layers
- Developing Spark applications using Python in Azure Databricks and Leveraging capabilities of Azure Databricks for data processing and analysis
Data Engineer
Capgemini Technology Services India Limited
Bangalore
07.2019 - 10.2021
- Connected Azure SQL tables to Power BI and ensured report consistency
- Established connectivity between Azure SQL tables and Power BI, enabling real-time data visualization and analysis
- Ensured that Power BI reports matches with existing reports and maintain consistency in data presentation and insights
- Created validation scripts for L2 tables to ensure data consistency
- Developed validation scripts to compare data in Azure L2 tables with Power BI template
- Ensured that data in Azure L2 tables aligned with expected values, maintaining data accuracy and reliability
- Wrote data into Azure SQL server using Azure Databricks
- Created workflows in Databricks for scheduling and data processing
- Developed workflows and jobs within Databricks to automate and schedule data processing tasks
- Orchestrated execution of data pipelines, ensuring timely and efficient data processing and transformation
Squish Automation Engineer
Capgemini Technology Services India Limited
Bangalore
06.2018 - 07.2019
- Expertise in Automation testing for standalone application
- Expertise in using Agile Testing Methodology
- Well-versed in defect tracking and bug reporting using Rally
- Actively involved in daily stand-up and kick-off meetings
Education
Bachelor of Technology - EEE
CBIT, JNTU-A University
Proddatur 05.2014 - 04.2017
Skills
Apache Spark
Python
Azure Databricks
Spark SQL
Airflow
API Integration & Data Extraction
AWS S3
ADLS Gen2
ETL and Data Transformation
Data Warehousing
Performance Optimization
Data Quality and Testing
Integration with Azure Services
Azure DevOps
PowerBI
Git and Version Control
Accomplishments
- Connecting Azure SQL tables to Power BI and ensuring the visuals are matching.
- Experience in migrating Scala code to PySpark in Databricks.
- Transforming data and writing output tables into Azure Data Lake Storage (ADLS) using PySpark code.
- Migrating the Scala code to PySpark while maintaining the transformation logic.
- Experience in migrating SSIS packages to Azure Databricks using PySpark.
- Ensuring the functionality of the SSIS packages is maintained in the PySpark migration.
- Hands-on experience in application development within Azure Databricks using PySpark and Azure Storage (ADLS Gen2).
- Hands-on experience with Databricks Workspace User Interface, managing notebooks, and Delta Lake with Python and Spark SQL.
- Experience in creating workflows/jobs in Databricks.
Timeline
Big Data Engineer
Massmutual
10.2023 - Current
Big Data Engineer
Cognizant Technology Solutions
10.2022 - 10.2023
Data Engineer
TCS
11.2021 - 10.2022
Data Engineer
Capgemini Technology Services India Limited
07.2019 - 10.2021
Squish Automation Engineer
Capgemini Technology Services India Limited
06.2018 - 07.2019
Bachelor of Technology - EEE
CBIT, JNTU-A University
05.2014 - 04.2017