Swapnil patil
AWS data engineer
Pune,Maharashtra
Summary
Dynamic Data Engineer with 3 Years of Experience in managing large-scale data infrastructure. Proficient in SQL ,data modeling ,ETL pipelines , and python scripting. Skilled in AWS big data technologies. Dedicated to driving operational excellence and delivering high-quality results.
Overview
3
3
years of professional experience
Work History
AWS Data Engineer
Tata Consultancy Services
11.2021 - Current
- Working on media domain ETL project to process large amount of data and generate rating using technologies like Python , Pyspark , SQL , Hadoop and managing the data pipelines using the Orchestration tool Airflow .
- Engineered and executed automation system for data infrastructure, boosting scalability by 40% to meet organizations analytics demand, resulting in a 20% reduction in data processing timer.
- Developed a scalable ETL framework to handle fluctuating media volumes, accommodating a peak load of 2TB per day, which ensured uninterrupted service and enhanced the overall efficiency of data processing operations.
- Automated data quality checks, reducing data processing errors by 20 % and ensuring high quality data for analysis.
- Using AWS EMR clusters to Run and manage the Spark jobs which helps to optimizing performance and scalability for processing large data in a big data environment.
- Fixing production issues and code logics.
- Wrote and maintained custom scripts to increase system efficiency and performance time.
- Monitored and tested application performance to identify potential bottlenecks, develop solutions and collaborate with developers on solution implementation.
- Technology Stack : Python, Scala, Spark, PySpark, SQL, Airflow, AWS S3, AWS EMR, AWS Lambda
Education
Bachelor of Engineering - Electronics and telecommunication Engineering
SSBT College of Engineering And Technology
Jalgaon, India08.2021
Skills
- Python
- SQL
- Spark
- Airflow
- AWS EMR
- AWS S3
- AWS Lambda
- ETL pipeline
- big data processing
- Jira Board
- Git Version Control
- Data Pipeline Design
Achievements
Received on-the-spot award for optimizing the ETL pipeline and developing Airflow DAGs to automate data validation, enhancing efficiency and reliability in the data processing workflow
Work Availability
monday
tuesday
wednesday
thursday
friday
saturday
sunday
morning
afternoon
evening
swipe to browse
Timeline
AWS Data Engineer
Tata Consultancy Services
11.2021 - Current
Bachelor of Engineering - Electronics and telecommunication Engineering
SSBT College of Engineering And Technology
Similar Profiles
Vrunda ChindhadeVrunda Chindhade
HR Manager at Tata Consultancy Services, TCS, Tata Consultancy Services LimitedHR Manager at Tata Consultancy Services, TCS, Tata Consultancy Services Limited
Sreya BasuSreya Basu
Procare Analyst at Tata Steel(Tata Consultancy Services)Procare Analyst at Tata Steel(Tata Consultancy Services)
Arun VArun V
Program Manager at Tata consultancy services | Financial Ombudsman ServicesProgram Manager at Tata consultancy services | Financial Ombudsman Services
Naimish AgnihotriNaimish Agnihotri
Assistant Delivery Manager at Tata Consultancy Services, Global Shared ServicesAssistant Delivery Manager at Tata Consultancy Services, Global Shared Services
Bhushan AkkarBhushan Akkar
Senior Buyer at JBT MarelSenior Buyer at JBT Marel