Ratnesh Mishra

Working in dataplane team for (Apache) Hive under open data analytics org of AWS EMR.

• Hive Upgrade: Leading the upgrade of Hive on EMR to version 4, integrating 5,000+ new commits while ensuring compatibility with existing Hive features in EMR. Oversaw a major Hive Metastore upgrade, a critical component for Spark, Flink, Hudi, and other EMR services. Also contributed to upgrading Hive 4 support for Hudi, Flink etc enabling seamless ecosystem interoperability.
• Hive Performance Improvement :Worked on Hive performance improvements as part of the TPC-DS 3TB performance enhancement initiative, targeting read performance for ORC and Parquet formats on EMR. Delivered 10+ optimizations—including non-ORC file listing, Tez relaxed locality, and ORC split computation enhancements—resulting in a 1.59x total runtime improvement on EMR-7.6.0 vs. EMR-7.0.0, surpassing our original 1.5x goal.
• Glue data Catalog Views : Worked with Athena , Spark and Glue/lakeformation teams for adding Glue data catalog views (single common view across AWS services like Amazon Athena and Amazon Redshift). Developed APIs for enabling creation of glue data catalog views via lake formation which involved end to end development of system for providing on the fly query compilation and validation for spark and Athena.
• FGAC for hive using LakeFormation : Worked on adding support for fine grained access control in Hive on EMR with AWS lake formation using runtime IAM roles.

User Profiles - Designed and developed scalable grpc APIs for various aggregate profiles like Behavior , transaction , content profiles powering features like user personalization and targeted ads. Currently operating under a load about 40k rpm .

User Segmentation Framework- Contributed to the development of in-house developed user segmentation framework designed for facilitating features like user bucketing and funneling.

Developed query engine over elastic for transforming different logical combination of business funnels into elastic queries .

Data pipeline - Acting as a key player in the team responsible for adding various features , improving ,maintaining Moonfrog's data pipeline cluster handling ~40M/ minute unique data events , ~3.2TB/day data by volume,
Major Feature contributed to include

Design and development of autoscaling capability for the stateful, distributed data pipeline .which helped to handle traffic surge efficiently when pipeline traffic suddenly became nearly 4X during Lockdown (from serving 15B events/day to ~58B events/day)

Leagues Service as a Platform - Developed leagues service(a feature for increasing user engagement ) as a platform and also integrated and released for TPG game , currently serving ~4M DAU (Daily Active Users)

Migration of Stat server to Kubernetes - Centralised stat server cluster (handling ~ 20k request per sec) containerised and deployed on Kubernetes using AWS EKS , helpful for saving redundant maintenance effort and cost.

Data Pipeline - Responsible for adding various features , improving ,maintaining Moonfrog's high scale data pipeline cluster handling ~40M/ minute unique data events , ~3.2TB/day data by volume, maintained in redshift backed in s3.

Data lake query Capability - Migrated existing CSV data to parquet using AWS EMR and added direct query capability from data lake(s3) using AWS Athena.

SDKs and Dashboards - Developed various SDKs like Stats client SDK , League's SDK , stat server SDK , RTS SDK(for tracking game concurrents ) etc and in house Dashboards(for tracking different business metrics)

Worked in Preorder team(responsible for everything backend till order placement), in a Microservice environment as a sole owner of major business units or services.

Major projects developed or contributed to :-

Backend services for Apps- Worked on various services responsible for serving initial configs, articles, handling push notifications etc for 1MG app.

Payments - Worked on payments , 1MG wallet, involving third party wallet integrations and payment handling on 1MG app and website.

Microservice Framework- Enhanced in-house developed Microservice framework Vyked by adding various features like graceful service restarts , improved logging, timeouts etc.

Catalog - Worked on 1Mg catalog, service responsible for serving OTC categories and products also developed portal for adding and modifying categories and products for the category managers.

• Worked in the ENCC team working for T Mobile US.