Siddharth Mathur

Data Engineering

• Optimized 30+ ETL pipelines using Airflow and Spark, reducing runtime by 40% and saving $12K/year through partition optimization and broadcast joins.
• Migrated legacy pipelines to AWS serverless (Lambda, Glue), scaling to handle 50GB/day with 30% lower operational overhead.
• Integrated with an existing Kafka-based system, reading 50GB+/day of trading data and updating downstream sources with 99.99% reliability on AWS EC2/Kubernetes.

Backend Development

• Built Go/Python microservices (Gin, FastAPI) for financial reporting ("My-Account") and payout processing ("Nubra"), handling 5K RPS with
• Revamped "Funds Service" to consolidate $500M+ AUM data across 10+ sources, improving query speed by 60% using PostgreSQL materialized views.
• Developed an influencer onboarding dashboard using FastAPI, enabling influencers to add leads and track performance metrics, integrated with Zoho CRM and PostgreSQL hosted on AWS.
• Designed and implemented models for an internal tool using Django, automating manual tasks and providing the support team with client-related data insights. Utilized MySQL and PostgreSQL as primary databases.

Data Quality & Governance

• Slashed data errors by 70% via PySpark validation frameworks (Great Expectations) and automated anomaly detection which saved a lot of engineering hours/month.
• You implemented a scalable and automated solution to pseudonymize 10M+ user records using AWS Glue and PySpark, ensuring compliance with India’s DPDP Act and global data privacy regulations. The encrypted data was integrated into a data warehouse for analytics.

Collaboration & Impact

• Partnered with cross-functional teams (CCT, Product) to resolve root-cause issues, reducing user complaints by 45% via Jira-driven sprint planning.

Scalability

• Migrated legacy pipelines to AWS serverless (Lambda, Glue), scaling data processing capacity to handle 50GB/day with improved efficiency, reducing operational overhead, and enabling seamless future scalability.

Scalability & Performance Optimization (Khelyaar): Feature Development (Classic Rummy): Process Automation: Deployment & CI/CD:

• Led the development and implementation of scalable features for Khelyaar, including smart contests, registration queues, and auto-scaling servers, enabling the app to scale from 1k to 1M users.
• Reduced API failure percentage during peak hours from 60% to 5% by optimizing backend infrastructure, implementing caching strategies, and improving database performance.
• Designed and developed new features for Classic Rummy, such as private contests, player-category-based contests, tournaments, and geo-location-based user blocking, enhancing user engagement and security.
• Automated data request handling for business teams, streamlining processes and improving operational efficiency.

Backend & Infrastructure Management (Khelyaar):

• Managed all backend services on AWS Cloud, utilizing EC2 and Lambda for scalable and serverless computing.
• Implemented RabbitMQ for queue management and Redis for inter-service communication, ensuring reliable and efficient data flow.
• Transitioned from a single database instance to a master-slave configuration, improving system reliability and scalability.
• Integrated Ronuza as a live match data source, transitioning from API-based to WebSocket-based communication for real-time updates.
• Worked with Cassandra, MySQL, and Redis databases to ensure efficient data storage, retrieval, and caching.

Testing & Collaboration:

• Developed a mock testing environment to simulate inputs and conduct comprehensive testing, ensuring robust feature deployment.
• Collaborated with third-party vendors for data source integration and new feature discussions, ensuring seamless integrations and innovative solutions.

• Initially deployed applications using build scripts, later transitioning to Docker for containerization and Jenkins for CI/CD pipelines, improving deployment efficiency and consistency.
• Streamlined the deployment process, reducing manual errors and enabling faster release cycles.