NAGIREDDY RAVIKUMAR

• Collaborated with data modeling teams, stakeholders, and data analysts to comprehend data requirements and translate them into technical specifications and structured data representations.
• Developed Spark applications in Scala for performing data cleansing, event enrichment, data aggregation, and data preparation to meet business requirements.
• Implemented data quality checks and validation processes to ensure accuracy, consistency, and completeness of data.
• Worked on various data formats like AVRO, Sequence File, JSON, Parquet and XML.
• Worked on fine-tuning spark applications to improve the overall processing time for the pipelines.
• Created Hive tables, loaded with data, and wrote Hive queries to process the data. Created Partitions and used Bucketing on Hive tables and used required parameters to improve performance.
• Debugged common issues with Spark RDDs and Data Frames, resolved production issues, and ensured seamless data processing in production environments.
• As per the business requirement storing the spark processed data in HDFS/S3 with appropriate file formats.
• Performed Import and Export of data into HDFS and Hive using Sqoop and managed data within the environment.
• Created EC2 instances and EMR clusters for Spark Code development and testing.
• Performed step execution in EMR clusters for the job deployment as per requirements.
• Used Agile Scrum methodology/ Scrum Alliance for development.

Environment: Apache Spark, Spark SQL, Scala, Hive, Hadoop, Sqoop, HDFS, AWS.

• Involved in loading the data into HDFS from different Data sources like SQL Server, AWS S3 using Sqoop and load into Hive tables.
• Involved in creating Hive tables, loading data from different data sources, HDFS locations and other hive tables.
• Created SQOOP jobs and Scheduled them to handle incremental loads from RDBMS into HDFS and applied Spark transformations.
• Created Hive external tables to perform ETL on data that is generated on daily basis.
• Developed Spark code in Scala and Python(Pyspark) and deployed it in AWS EMR.
• Was responsible for Optimizing Spark SQL and HIVE queries that helped in saving Cost to the project.
• Worked in monitoring, managing, and troubleshooting the Hadoop and Spark Log files.
• Worked on Hadoop within Cloudera Data Platform and running services through Cloudera manager.
• Involved in Agile methodologies, daily Scrum meetings, Sprint planning.

Environment: Apache Spark, Spark SQL, Scala, Hive, Hadoop, Sqoop, HDFS, AWS, Python, Pyspark.