Sankit Acharya

• Designed and maintained a real-time ETL pipeline using Kafka and microservices; later led the migration to Databricks to support scalable batch and streaming workflows, reducing operational costs by approximately 30%.
• Developed a configurable request orchestration component for vendor-specific ingestion logic in Kafka pipelines, supporting multiple query types and dynamic date range handling. Used a modified Klee’s algorithm to eliminate redundant keyword-based requests across overlapping time windows.
• Developed a dynamic priority queue algorithm to merge and optimize vendor queries under a 280-query limit, handling variable coverage percentages dependent on data volume, thereby maximizing ingestion within download constraints.
• Migrated persistent storage from MongoDB to Delta Lake, significantly improving performance, query reliability, and scalability for data pipelines ingesting millions of records per day.
• Developed and maintained the Bronze and Silver layers in a Medallion architecture: onboarded new data sources, applied normalization and transformation logic using PySpark, and contributed to the Gold layer by implementing Spark higher-order functions to structure model output for Elasticsearch aggregation.
• Provisioned and managed Databricks infrastructure using Terraform, with multi-cloud support across AWS and GCP; led the migration to GCP to leverage credits and reduce infrastructure spend.
• Built and maintained CI/CD pipelines using GitHub Actions to automate deployments for both ETL workflows and Terraform-based infrastructure, ensuring consistent, peer-reviewed updates across environments.
• Maintained a monolithic codebase with TDD-based unit tests and integration testing, enabling deployments to shift from multi-day efforts to sub-24-hour release cycles.
• Implemented observability with OpenTelemetry and the ELK stack, and integrated Slack alerting for real-time visibility, issue detection, and operational response.

• Led the backend design and development of a modular, horizontally scalable platform for building data preparation and machine learning workflows via a drag-and-drop interface.
• Built a dynamic execution engine using directional graphs, topological sorting, multiprocessing, and hashing to optimize execution flow and cache results, reducing processing time by 65%.
• Worked independently on backend components while collaborating with frontend and ML teams to integrate UI features and model functionality.
• Platform adopted by eight departments and deployed across multiple global offices.
• Tech stack: FastAPI, Gunicorn, Nginx, Redis, MongoDB.

Data Visualization Application

• Developed a Utility tool to extract data from various databases and structure it, reducing 85% of manual effort per issue to a total saving of 15 hours per week
• Implemented Pandas data frame and ChartJS for a complete visualization
• Built an RPC server to integrate Backend: Python and Frontend: Electron(React)

API Schedule Sync Application

• Developed a script to extract and check for discrepancies across multiple platforms resulting in better sync and avoiding crisis management
• Reduced the execution time to 80% using Python multiprocessing pool to handle CPU-bound work and AsyncIO to handle I/O-bound work and network control using request sessions
• Led a group to the successful completion of the project

• Designed and developed company website increasing audience engagement and improving brand visibility.
• Produced event management presentation providing a detail view of company services.
• Contributed ideas for support, accounting and strategic planning improving customer experience.