Nikhil Devihosur

Stock Market Real-Time Data Analysis:

• Implemented Python scripts for data extraction, transformation, and loading (ETL), optimizing performance and reliability.
• Orchestrated the setup and configuration of Kafka cluster for efficient real-time data ingestion and distribution across the system.
• Developed robust data consumption pipeline using AWS S3, ensuring high availability and fault tolerance.
• Configured AWS Glue to automatically crawl the stored data, maintaining an up-to-date data catalog for enhanced data management and accessibility.
• Leveraged AWS Athena for ad-hoc SQL-based data analysis, empowering stakeholders with actionable insights for informed decision-making.

Spotify Data Analysis:

• Created an end-to-end data engineering pipeline for extracting, transforming, and loading (ETL) Spotify data.
• Pipeline integrated with the Spotify API, utilized AWS Lambda for data extraction, employed transformation functions, and stored the processed data in Snowflake.
• The automation of data extraction and loading processes enhanced efficiency and enabled seamless analytics.

• Enhanced research accuracy by implementing rigorous data validation techniques.
• Streamlined data analysis processes for improved efficiency and report generation.
• Conducted comprehensive market research to identify key industry trends and inform strategic decision-making.
• Leveraged expertise in Microsoft Excel to manipulate data sets, perform calculations, and create informative charts and graphs.

Data Pipeline Development:

• Designed and implemented scalable data pipelines, ensuring efficient and reliable data processing from source to destination
• Utilized Big Data applications like Apache Oozie to orchestrate and automate data workflows, reducing processing time.

Database Management:

• Managed and optimized the Enterprise Data Lake to enhance data retrieval speed and overall system performance
• Implemented indexing strategies resulted in a 15% improvement in query performance

Big Data Technologies:

• Applied Apache Spark to process and analyze large datasets efficiently
• Implemented optimizations for distributed computing, resulting in faster data processing for big data workloads

ETL (Extract, Transform, Load) Processes:

• Developed and maintained ETL processes using Talend to transform raw data and load it to the Data Lake
• Implemented data extraction mechanisms, reducing data processing errors

Collaboration and Documentation:

• Collaborated with cross-functional teams to understand data requirements and deliver tailored solutions
• Maintained comprehensive documentation of data engineering processes, facilitating knowledge transfer and onboarding.