Hariharasudhan Sabapathi

Sharp and talented Lead Data Engineer with 12 years of product development history. Expertise in driving projects and leading cross-functional teams to consistently meet key program deliverables, targeting Lead level assignments in Big Data/Data Analytics with an organization of reputation.

Spark Tech Upgrade : Client - Marriott Led Spark and EMR Upgrade Initiative Directed the successful upgrade of Apache Spark and Amazon EMR versions, ensuring seamless migration of all Spark jobs with full backward compatibility. Coordinated cross-functional efforts to minimize downtime and maintain data pipeline integrity throughout the transition. 

AOS (Ad-server Operative System) Data Mapping Spark Developer, Existing on-prem Operative (o1) migrating to cloud operative system. Extracting the source from different topics from datalake and apply transformation to map the datasets and load the results into S3 target bucket for the downstream teams., Spark Scala, AWS S3, Terraform, concourse CI/CD, Databricks.

AOS DataStream To DataLake Big data Developer, Spark Streaming application ingest configured topics from AOS kafka broker to S3 datalake and store them as 'delta' format files. Application uses AOS Kafka Schema registry to deserialize AVRO payloads from AOS datastream. AWS (s3, cloudwatch, elastic search, athena, spark scala), Terraform, Grafana, Kafka, DynamoDB, Concourse CI/CD, Databricks Delta lake and data versioning. Zoho CRM Data Migration from various sources Spark Developer, To get the Co-ordinates (latitude & longitude) of a record based on the address information and update/insert the Co-ordinates values in those corresponding records. Using these geocode info, sales persons can find the entities located near him. We used BigData technologies and Zoho Maps api to achieve our goal. For streaming data we used Kafka., Cloudera, Sqoop, Spark with Scala, Hive.

EffecTv Data Portal NodeJS, Aws(glue api, ECS, cloud watch, event rule, glue job, lambda), Spark with Scala, Angular, Postges Aurora Serverless, Internal portal/tool for Comcast-EffecTV customers where they can visit this website (dataportal.comcast.com) and ask for access of tables and buckets and view how many roles/users accessed the table/bucket on any day. Admin/Managers can approve or deny the bucket access request.

• Spark
• Apache Hadoop
• Hive
• Sqoop
• Databricks
• Terraform
• Concourse CI/CD
• AWS Services (S3, EMR, Lambda, ECS, RDS, CloudWatch, SNS, SES)
• MonteCarlo

• Mysql
• PostgresSQL
• NoSQL

• Apache Maven
• SBT

• Ensuring the successful delivery of emerging big data solutions
• Defining strategy, principles, vision, and standards
• Providing advisory services around data frameworks, governance, and quality
• Simulating, designing, developing & deploying computationally complex and practical data.
• Leading the development of enterprise technology standards, governance processes, and performance metrics
• Analyzing & reviewing business, functional, and high-level technical requirements