Venkata Ramesh Pampana

Nottingham,UK

Summary

I am a highly skilled Data Engineer with 2+ years of experience, with overall 10+ years of work experience specializing in various data technologies including Snowflake, Azure Databricks, Pyspark, and Spark. My expertise lies in data transformation, data ingestion using Snowpipe, SnowSQL, making me proficient in data sharing, analysis, and reporting. I have a strong background in ETL/ELT processes, data Processing, Agile methodologies, and debugging, delivering robust solutions. My deep understanding of Snowflake's core features including Time Travel, Zero copy cloning, Stream & Tasks , star schema and snowflake schema, allows me to design scalable ETL/ELT processes. My problem-solving skills, strong communication, and multitasking abilities enable effective performance tuning, data governance, and translating business needs into technical solutions. Proficient in SQL, Python, SQL Server, Oracle, data warehouse and data pipeline infrastructure, I ensure seamless data processing and data management in the Big Data landscape and have understanding of big data technologies like Hadoop and Spark.

Overview

years of professional experience

Work History

Data Engineer

Cognizant Technology Solutions India Private Ltd

10.2017 - Current

Designed and implemented end-to-end ETL processes using PySpark, AWS Glue, and Snowpipe for seamless data ingestion, transformation, and loading into the Snowflake data warehouse
Provide clean & transformed data to be consumed by analytics and data science teams
Conducted error handling, debugging, and troubleshooting to identify and resolve issues in data pipelines, maintaining data quality and integrity
Optimized data pipelines and queries, reduction in query execution time and improving overall system performance
Perform tasks such as writing scripts, calling APIs, write SQL queries, etc
Stayed updated with the latest industry trends and best practices, continuously improving data engineering processes and contributing to the organization's data-driven success
Collaborated with cross-functional teams to understand and address data requirements, translating business needs into technical solutions.

GIS Engineer

RMSI Pvt Ltd

10.2016 - 08.2017

Implement Data Validation, Quality Checks, Profiling
Involved in import data from various RDBMS into HDFS using Spark which includes Incremental Load to populate Hive External Table and vice-versa
Involved in import data from various RDBMS into Hive Tables which includes Queries using Spark
Designed both Managed and External Hive Tables and Defined static and dynamic partitions as per requirement for optimized performance on production datasets.

Education

Master of Science - Computer Science

Nottingham Trent University

Nottingham, UK

04.2001 -

Skills

Snowflake
ADF
ETL
Azure Databricks
Python
SQL
SparkSQL
Spark

PySpark
EC2
Linux
ADLS GEN2
Big Data
Hadoop
Hive
Terraform

Accomplishments

Reliable Sprint Engagement: Consistently participated in all sprints, contributing to seamless project progression and fostering effective teamwork.

Projects

Migration Analytics in Retail Domain

Developed an ETL pipeline to extract data stored in AWS S3 for further processing using PySpark., Data Transformation and Cleaning: Developed PySpark and AWS Glue jobs for efficient data transformation and cleansing of data., Job Monitoring and Maintenance: Actively monitored and maintained Glue jobs, including debugging and resolving issues to ensure job stability and accuracy., Cross-Functional Collaboration: Collaborated closely with Data Scientists, Analysts, and team members to comprehend data requirements, guaranteeing that processed data aligned with their needs., Thorough Documentation: Created comprehensive documentation for PySpark scripts, SQL queries, and data transformations, fostering transparency and facilitating collaboration., SQL Query Execution: Executed SQL queries in Snowflake data warehouse to fulfil client requirements and support business KPIs effectively.

Data Ingestion in Media & Entertainment Domain

Data Retrieval: Fetched API data using Python, applying data cleaning, transformation, and extraction., Cloud Deployment: Utilized Terraform to deploy AWS Lambda, automating data retrieval and processing., Data Integration: Designed Lambda function to fetch, process, and store data in S3 for analysis., Monitoring and Debugging: Set up AWS CloudWatch for real-time monitoring and issue resolution., Data Accessibility: Ensured processed data availability in S3, well organized for downstream use.

Timeline