Vishnu Annavarapu
Hyderabad
Summary
Experienced IT professional with a strong background in various technologies, including Hadoop, Scala, Python, AWS, Databricks, MongoDB, Spark, Airflow, and MySQL. Demonstrated ability to design scalable, configurable, and maintainable solutions for complex business challenges. Strong communication skills and a commitment to staying current with industry trends and best practices. Seeking a challenging role to leverage skills and experience and contribute to a dynamic and innovative team. Passionate about machine learning, deep learning, natural language processing, and statistics.
Overview
5
5
years of professional experience
1
1
Certification
Work History
Data Engineer
S&P Global
Hyderabad
04.2022 - Current
- Developed and implemented Spark applications using and Scala.
- Designed and developed data pipelines to ingest data into delta tables and Kafka cluster using Apache Spark.
- Optimized Spark jobs for improved performance, scalability, and reliability.
- Integrated Kafka with Spark Streaming for real-time analytics on streaming data sets.
- Utilized Scala to call API services and efficiently handle XML inputs, enabling seamless data retrieval and processing.
- Designed and implemented effective strategies for loading and analyzing data in delta tables, identifying valuable patterns and providing actionable insights for data-driven decision-making.
- Leveraged Python to create a Py script for scheduling Databricks jobs, automating data processing tasks and improving workflow efficiency.
Data Engineer
Capgemini
Hyderabad
09.2021 - 04.2022
- Data Transformation with Spark: Employ Spark for data transformation tasks, including processing and refining data stored in MongoDB. This involves operations like aggregation, filtering, and joining datasets for analysis.
- Perform MongoDB Operations: Utilize Python to execute read and write operations on MongoDB databases, enabling efficient data manipulation such as querying, inserting, updating, and deleting records.
- Pipeline Development: Create automated data pipelines using Spark, Python, and MongoDB to streamline data processing tasks, including transferring data from MongoDB to SQL databases, ensuring smooth and efficient data flow for analysis and reporting purposes.
- Designed MongoDB database schemas to store application data.
AWS Data Engineer
Tata Consultancy Services
Hyderabad
01.2019 - 09.2021
- Developed and maintained data pipelines to ingest, store, process and analyze large datasets in AWS S3 buckets.
- Implemented automated pipeline using Glue, Step and Lambda functions.
- Conducted data transformation and cleansing using Pyspark and SQL to maintain data quality.
- monitoring and alerting through Cloud Watch for issue resolution.
- Collaborated with cross-functional teams to ensure data integrity and accuracy.
Education
Bachelor of Technology - ECE
Qis College of Engineering & Technolgy
Ongole04-2017
Skills
- Data Migration
- SQL
- Scala
- Python
- MongoDB
- AWS
- Spark
- Databricks
- NLP
- Recurrent Neural Network
- Deep Learning
Certification
- AZ 900
- AWS Cloud Practitioner
Timeline
Data Engineer
S&P Global
04.2022 - Current
Data Engineer
Capgemini
09.2021 - 04.2022
AWS Data Engineer
Tata Consultancy Services
01.2019 - 09.2021
Bachelor of Technology - ECE
Qis College of Engineering & Technolgy
Similar Profiles
SUSHANTH REDDY MyakaSUSHANTH REDDY Myaka
Senior Data Quality Researcher at S&P GlobalSenior Data Quality Researcher at S&P Global
Ismail MohammedIsmail Mohammed
Data Specialist at S&P GlobalData Specialist at S&P Global
Suraj ShettySuraj Shetty
Principal Analyst – Automotive | S&P Mobility at S&P GlobalPrincipal Analyst – Automotive | S&P Mobility at S&P Global
Komal Komal null
Research Associate II at S&P GlobalResearch Associate II at S&P Global
Chemicala Harshitha ReddyChemicala Harshitha Reddy
Associate Software Developer at SID Global SolutionsAssociate Software Developer at SID Global Solutions