NAVYA ADAPA
Chennai
Summary
Innovative and results-driven Data Engineer with over 5+ years of IT experience specializing in scalable cloud data architectures, automated ETL/ELT pipelines, and GenAI integrations. Proven expertise in transitioning static data architectures into intelligent, self-correcting logic using AWS Bedrock foundation models and Snowflake CoCo. Proficient in building robust metadata-driven data frameworks, optimizing high-volume ingestion streams via Python, PySpark, and Snowflake Stored Procedures, and orchestrating serverless workflows across the AWS ecosystem. Adept at leveraging advanced RAG mechanisms, designing dimensionally modeled structures (SCD Type 2), and deploying self-healing monitoring workflows to ensure high data availability and strict SLA compliance.
Overview
4
4
Languages
1
1
Certification
5
5
years of professional experience
Work History
Senior Consultant C1/POC: Agentic AI Data Pipeline
Capgemini
chennai
04.2025 - Current
- Deployed AWS Bedrock foundation models for unstructured document parsing and semantic analysis, transforming architecture from static ETL to intelligent, self-correcting logic that improves data accuracy. ·
- Developed autonomous data pipeline using AWS Bedrock and Snowflake CoCo, automating end-to-end data processing, metadata creation, and code generation tasks to enhance data handling efficiency. ·
- Automated generation of unified schemas, target layouts, and DDL scripts, creating a self-healing monitoring workflow to autonomously identify and propose real-time fixes for ingestion failures, reducing downtime. ·
- Utilized Snowflake CoCo CLI and Snowsight capabilities to enable natural language driven data exploration and autonomous dynamic table configurations. ·
Senior Consultant (Cloud Platform Engineer / C1)
Capgemini
Chennai
04.2025 - Current
- Design and maintain robust data ingestion frameworks to migrate high-volume data from diverse source databases to Snowflake using SQL and Stored Procedures.
- · Managed and enhanced a metadata-driven data pipeline framework (Data Foundation Framework) deployed across AWS and Snowflake. ·
- Assumed core ownership of the framework as the primary support engineer and spearheaded the development of new automation processes. ·
- Processed structured raw files and API data ingestion layers, applying strict data quality, validation, and SLA rules into Snowflake target tables. ·
- Designed and built dimensionally modelled data structures with Slowly Changing Dimension (SCD) Type 2 logic using Snowflake Stored Procedures. ·
- Enabled downstream analytics and business intelligence reporting by delivering optimised API endpoints, secure database views, and automated file exports.
- Implement and manage automated job scheduling using Cron utilities to optimize pipeline execution and meet strict delivery schedules.
- Oversee SLA monitoring to ensure files arrive within scheduled windows, maintaining high data availability for downstream consumers.
- Develop automated error-handling mechanisms where file transfer or ingestion failures are logged into dedicated Snowflake audit tables for tracking and troubleshooting.
- Manage secure file transfers and directory synchronisation using WinSCP to ensure seamless data movement from source systems.
Data Engineer
Infosys
Chennai
10.2024 - 12.2024
- Utilized Putty to interact with Hive, Hadoop, SQL, Shell Scripting, Python, and PySpark for efficient data processing and analysis.
- Developed and executed shell scripts to automate file triggers and data ingestion processes.
- Generated DCU queries to extract and transform data from the Gold layer to the Palladium layer.
- Employed PySpark to perform complex data analysis and generate meaningful insights.
- Conducted data analysis to identify trends, anomalies, and opportunities for improvement.
- To further enhance the data pipeline's scalability and performance, migrated the existing Hadoop and Hive workloads to Azure Databricks.
- Leveraged Databricks' unified analytics platform to streamline data ingestion, transformation, and analysis processes.
- Developed and executed PySpark scripts on Databricks to perform complex data transformations and aggregations.
- Sampradaa Software Technologies-Payroll
Data Engineer
NetConnect Global
Chennai
03.2024 - 07.2024
- As a Data Engineer, my role is focused on harnessing data to deliver insightful responses through advanced querying and natural language processing.
- I specialize in Python programming and orchestrate data workflows using AWS services such as Lambda for serverless computing.
- Cloud9 for integrated development environments, CloudWatch for monitoring, and SNS/SQS for messaging and event-driven architecture.
- Integral to my responsibilities is the implementation of cutting-edge AI technologies like Retrieval-Augmented Generation (RAG) and Large Language Models (LLM).
- These frameworks combine information retrieval techniques with generative models, ensuring that responses are not only accurate but also contextually relevant and comprehensive.
- I design and deploy robust data pipelines using AWS Glue for ETL (Extract, Transform, Load) processes and orchestrate intricate workflows using AWS Step Functions.
- Additionally, I leverage specialized modules and libraries such as Textract, PyMuPDF, PDFPlumber, Spire.PDF, Camelot, and TabulaPy to extract and analyze data from diverse document formats, thereby enhancing our data ingestion and processing capabilities.
- By leveraging LLM and other advanced technologies, I enable stakeholders to derive actionable insights crucial for strategic decision-making and business growth.
Data Engineer
Agilisium Consulting Services
Chennai
02.2023 - 03.2024
- Project: PySpark, Spark SQL, AWS Services (S3, Glue, Lambda, SNS, Step Function, SQS, DynamoDB, CloudWatch), Azure Databricks.
- Attending Planning meeting to understand the tasks for the sprint.
- Attending the DSM (Daily Standup Meetings) to update the task status.
- Developed Spark applications using PySpark and Spark SQL for data extraction, transformation, and aggregation from multiple file formats, enhancing data analysis and transformation processes.
- Automated various activities using Python, significantly improving efficiency and reducing manual intervention.
- Created and optimized SQL scripts for automation purposes, ensuring seamless data handling and workflow automation.
- Converted the csv files to parquet files using PySpark in Azure Databricks.
- Performed data quality checks using PySpark in Databricks notebooks.
- Created Azure Data Factory V2 Linked Services to connect to Azure Databricks.
- Created the Azure Data Factory V2 pipelines to invoke the Azure Databricks notebooks.
- Created the Azure Data Factory V2 pipelines for daily, weekly, and monthly full load.
- Created triggers in Azure Data Factory V2 to run pipelines on scheduled time.
- Created the stored procedure to update the audit logs of the ADFV2 copy activity.
- Created the dynamic pipelines to copy multiple tables using dynamic parameters.
- Attending the sprint retrospective meetings to review the sprint flaws and improvements.
Data Engineer
Clayfin Technologies
Chennai
08.2022 - 11.2022
- Developed Spark applications using PySpark and Spark SQL for data extraction, transformation, and aggregation from multiple file formats, enabling efficient data analysis and transformation.
- Automated various tasks using Python to streamline processes and enhance productivity, and developed SQL scripts for automation purposes to ensure robust and efficient data handling.
- Extracted data from sources, including customer details of products such as snacks and cold beverages, ingesting it into the data lake and processing it using AWS Lambda and AWS Glue to facilitate seamless data management and transformation.
Data Engineer
Market Simplified
Chennai
01.2022 - 07.2022
- This project is mainly focused on data extraction and aggregation from diverse folders within Amazon S3.
- This initiative involves leveraging Python and AWS services to read and extract JSON data using access keys and secret access keys for proof-of-concept purposes.
- The extracted data is transformed and consolidated into Parquet files utilizing windowing operations such as lead, lag, rank, and dense rank, alongside aggregate operations including sum, average, and count.
- These Parquet files are then stored back in S3, providing a structured dataset that is readily accessible for data scientists.
- This project not only showcases my proficiency in cloud-based data processing and Python programming but also highlights my ability to facilitate data-driven insights through efficient data engineering practices.
Data Engineer
Sutherland Global Services
Chennai
02.2021 - 12.2021
- I possess extensive experience in managing projects focused on extracting data from client-specific APIs, transforming it, and loading it into S3 buckets in Parquet format.
- My expertise spans AWS services, where I adeptly utilize Python and PySpark for efficient data ingestion and transformation tasks.
- I am responsible for orchestrating data extraction from APIs and ensuring its storage in S3 in Parquet format, while also monitoring job performance.
- I excel in building and deploying solutions on AWS, leveraging tools like Boto3 and requests for automation.
- I have implemented robust AWS solutions integrating S3 and AWS Lambda for streamlined data workflows.
- Additionally, I am proficient in writing Python-based Spark applications and utilizing Spark SQL for processing and analyzing Parquet datasets stored in S3.
- My proficiency extends to using PyCharm and Bolos for effective data management and ETL job execution on AWS Lambda.
Education
Bachelor of Science - Electronics & Communication Engineering
Vasireddy Venkatadri Institute of Technology
Guntur, AP01-2020
Diploma of Higher Education - Mathematics, Physics
Sri Chaitanya junior College
Guntur, AP01-2014
Skills
- ETL development
- PYTHON
- Web-based software engineering
- Data structure development
- API design
- SQL
- PYSPARK
- AWS
Certification
- AWS Certified Solutions Architect - Associate
- Azure Databricks Data Engineer - Associate(DP-750)
Languages
Telugu
Proficient
C2
Hindi
Beginner
A1
English
Advanced
C1
Tamil
Elementary
A2
Interests
- Swimming
- Reading Novels
Timeline
Senior Consultant C1/POC: Agentic AI Data Pipeline
Capgemini
04.2025 - Current
Senior Consultant (Cloud Platform Engineer / C1)
Capgemini
04.2025 - Current
Data Engineer
Infosys
10.2024 - 12.2024
Data Engineer
NetConnect Global
03.2024 - 07.2024
Data Engineer
Agilisium Consulting Services
02.2023 - 03.2024
Data Engineer
Clayfin Technologies
08.2022 - 11.2022
Data Engineer
Market Simplified
01.2022 - 07.2022
Data Engineer
Sutherland Global Services
02.2021 - 12.2021
Bachelor of Science - Electronics & Communication Engineering
Vasireddy Venkatadri Institute of Technology
Diploma of Higher Education - Mathematics, Physics
Sri Chaitanya junior College