Vishal Eravachi

Data Engineering Role

• Worked on designing and developing medallion architecture for streaming data in AWS and Azure.
• Developed multiple pyspark scripts for data transformations in each layer in Databricks.
• Experience in End to end Deployment using CICD github actions,Azure devops.
• Mentored junior consultants, helping them enhance their skills and contribute more effectively to projects

GenAI Developer Role(Lead Team of 4)

• Developed end to end pipeline for GEN AI architecture using RAG approach using azure open AI and LLM models such as GPT-3.5,GPT-4,claude,llama etc
• Developed end to end pipeline for developing chatbot by scraping data from confluence,support documents excel/ppt/pdfs .
• Rich knowledge in Databricks Vector Search,opensearch,chromaDB,FAISS etc.
• Proven expertise in prompt engineering and fine tuning large language models.
• Developed multiple ML pipelines on databricks which included data preprocessing,model training,registering model and model serving.

• Implemented near-real tume solution for ingesting data everyday to snowflake using AWS & Kafka.
• Developed Snowflake dashboards to provide real-time insights of prod status.
• Developed automation process to parse millions of XML & JSON files and load to SF.

• Creating glue jobs in aws glue studio as per business requirement
• Cloud orchestration using stepfunctions,kubernetes
• API development using flask,django framework
• Creating multiple lambda functions and serverless API end points for data ingestion into dynamodb
• Complete architecture and design of dynamodb tables
• Automation of pipeline using boto3 boto3 SDK for python as well as tableau automation
• Creating snowpipes to load data from s3
• Using databricks along with AWS for data transformations using pyspark
• Experience in ingestion of streaming data via aws kinesis data stream,firehose.

• Created multiple python scripts to automate the process of data comparison between On-premise(Oracle DB) and AWS(S3) and saved almost 4 hours of manual task every day
• Created python utilities such as search scripts,data consolidation in excel to automate manual tasks thereby saving 40% of total efforts in a sprint
• Created GUI in python using tkinter to develop an interface for all the scripts created and presented to client
• Created aws glue jobs for data transformation and processing
• Automated test management tool Zephyr and maintained the test cases,acceptance criteria etc.

• Work extensively on Hadoop Distribution File System Command
• Work with the Sqoop to load data from the external database to HDFS
• Creation and data loading in the hive tables and analysis of the data over them
• Performing full code deployments on unix servers
• Using Sql to update customer details in the backend oracle DB
• Developed basic shell scripts to automate the deployment process.