Swathi K

• Built an ETL pipeline using PySpark on Azure Databricks to process JSON claim data from Azure Data Lake Storage (ADLS).
• Used Azure Data Factory (ADF) for orchestration and workflow automation.
• Created Hive-compatible external tables on ADLS for efficient querying.
• Implemented fraud detection logic using PySpark, business rules, and tagging.
• Enabled schema evolution handling using the Avro file format.
• Loaded clean data into Azure Synapse Analytics for reporting and analytics.
• Configured alerts and monitoring using Azure Monitor and Log Analytics.
• Secured data access using Azure RBAC, and stored secrets in Key Vault.

Azure Data Lake, Azure Databricks, Azure Data Factory, PySpark, Spark SQL, Avro, Hive Tables, JSON, Azure Synapse, ADF Pipelines, Azure Monitor.

Pharmacy Temperature Monitoring – AWS

• Built an automated data pipeline on AWS to track pharmacy fridge temperatures using sensor data.
• Ingested raw sensor files into Amazon S3 with Hive-style partitioning by pharmacy and date.
• Used EventBridge to trigger Step Functions, which ran PySpark jobs on EMR for data cleaning and validation.
• Stored processed data in S3, and loaded it into Amazon Redshift using the COPY command for reporting.
• Sent success/failure alerts via AWS Glue and SNS to notify the monitoring team.
• Implemented IAM roles and policies for secure access across all AWS services.
• Helped detect temperature breaches early, reducing the risk of medicine loss, and supporting compliance.

Key Skills

AWS S3, EMR, PySpark, Redshift, Step Functions, EventBridge, Glue, SNS, Hive, SQL, IAM, Data Pipeline Automation.