Skilled Data Engineering Manager with a demonstrated history of working in the finance industry. Expert in designing and developing complex Data Lake to Big data implementations & led different projects and implemented them successfully. Solid understanding of Bigdata technologies such as Hadoop and Spark along with AWS cloud. Highly proficient in creating/delivering on process improvement initiatives, project commitments and excellent service delivery. Strong communication skills and excellent in documentation. Great team player, confident leader & forever a student eager to learn new technologies.
• Developed complete end to end Big–data processing in hadoop eco system.
• Developed Spark framework to facilitate setup of datalakes and supporting master projects.
• Developed Spark code using Scala.
• Imported data from different sources like HDFS to Spark RDD.
• Handled adhoc requests to meet the business requirements using Spark Context,SparkSQL,Data Frames,Hive.
• Written extensive Hive queries to do transformations on the data to be used by downstream models.
• Created hive schemas using performance techniques like partitioning.
• Loaded and extracted the data using Sqoop from Oracle into HDFS.
• Used SFTP to transfer and receive the files from various upstream and downstream systems.
• Involved in scheduling the jobs in Automic UC4.
• Used Jira for project tracking, Bug tracking and Project Management.
• Importing of data from various data sources into Hadoop and transform data in flexible ways by using Apache Flume,Kafka.
• Involved in some product business and functional requirements and documentation in Confluence.
• Big Data batch job monitoring.
• Big Data application support.
• Big Data L2 support.
• Big Data application development.
• Coordination with IT operations, architects for Big Data support.
• Transition for Big Data projects towards support.
ITIL v4 certified