Bhargavi Mekala

Technology Stack: AWS Airflow, S3, EMR, EC2, Lambda, SNS, Snowflake, Databricks, Pyspark, PostgreSQL, Palantir Foundry, Talend

Responsibilities:

• Designed and implemented scalable data pipelines to ingest, transform, and load data from multiple sources into Snowflake and S3, handling (200+) datasets up to 1 TB.
• Developed and implemented PySpark scripts to apply complex business logic for transforming large datasets, ensuring data readiness for downstream processes and machine learning models.
• Developed and maintained Airflow DAGs for automating workflows, ensuring seamless integration with other tools like AWS lambda, Snowflake, S3, PostgreSQL, Talend, EMR, Oracle, SQL server.
• Worked extensively with AWS services (S3, EMR, EC2, Lambda, SNS, Airflow) to support data orchestration, storage, and processing.
• Utilized Databricks workflows with CI/CD pipelines, leveraging Databricks asset bundles to automate and execute scripts for model training and scoring, ensuring efficient and scalable machine learning processes.
• Worked on EMR Cost Optimizations which helped us to track and to complete the jobs with less resources, less time and in efficient way.
• Developed and optimized SQL scripts to generate analytical reports, enabling data-driven business decisions and improving operational efficiency.
• Worked in integrating Palantir Foundry for visualization and user-driven decision-making, enabling insights into sales, forecasts, and adjustments.
• Coordinated with cross-functional teams, including data scientists and business analysts, to ensure data readiness for model training and scoring.
• Used Git and Bitbucket for version control and CI/CD pipelines, ensuring smooth deployment and integration of workflows and scripts across environments.

Technology Stack: AWS S3, Redshift, Lambda, Apache Airflow, Databricks, Pyspark

Responsibilities:

• Optimized data processing by implementing efficient ETL pipelines and streamlining database design.
• Proficiently managed DAGs, both new and existing, leveraging Apache Airflow to optimize workflow orchestration.
• Used Databricks for ETL processes, implementing the medallion architecture (bronze, silver, gold layers) to organize and optimize data workflows.
• Analysed the data,and created generic logics to reduce redundancy in data.
• Successfully established a continuous data load mechanism using Snowpipe, ensuring real-time data ingestion and reducing latency for timely insights.
• Developed purposeful views based on business needs. Translated business propositions into quantitative queries to collect/clean the necessary data.
• Translated business propositions into quantitative queries to collect/clean the necessary data.