ARUNA MURUGESAN

Senior Data Engineer

Virtusa software services

Bengaluru

03.2016 - Current

• Developed Pig Latin scripts in the areas where extensive coding needs to be reduced to analyze large data sets.
• Involved in extracting huge volumes of data from relational databases to HDFS using Apache Sqoop.
• Experienced in creating Hive schema, external tables and dynamic partitions.
• Using various performance tuning technologies to handle very large set of data.
• Worked on other tools like WinSCP for transferring the files from Windows to UNIX and vice-versa, Putty for connecting to UNIX and run the scripts.
• Used several optimization techniques to efficiently manage and consume cluster space.
• Handling customer escalation in day-to-day basis and giving suggestions to the support team for better resolution.
• Involved in the complete lifecycle (SDLC) of the project i.e. Design, Development, Implementation, Unit testing and Support.
• Used data ingestion techniques to integrated millions of data to the repositories in daily basis.
• Worked on loading and transformation of large sets of structured data into Hadoop system.
• Mentored and Managed team of four to develop in Hadoop applications
• Design, develop and implement data processing and analytics solutions using Apache Spark
• Converting SQL queries, MapReduce programs into Spark transformations
• Using Oozie for workflow management and crontab for scheduling jobs.
• Implemented data validation checks resulting in less errors and enhance data reliability
• Analyze large datasets using PySpark’s DataFrame transformations and actions. This includes filtering, aggregating, joining, and performing complex data transformations.

Similar Profiles