Bushra Shaik

• Implemented Medallion Architecture to structure data ingestion (Bronze), transformation, and enrichment (Silver), and curation for analytics (Gold).
• Developed parameterized and metadata-driven pipelines using Azure Data Factory (ADF), with Jinja templates, enabling dynamic and reusable ETL processes.
• Ingested data from multiple on-premises and cloud sources into Azure Data Lake Storage (ADLS), and managed the data lifecycle efficiently.
• Utilized Azure Databricks for scalable data transformation and enrichment using PySpark and Delta Live Tables (DLT) for real-time and incremental data processing.
• Implemented Slowly Changing Dimensions (SCD Type 1 and Type 2) to maintain historical data accuracy and track changes over time.
• Integrated Unity Catalog for centralized governance, access control, and data lineage tracking across the lakehouse environment.
• Loaded curated data into Azure Synapse Analytics (Data Warehouse) for advanced analytics and business intelligence reporting.
• Automated workflows and alerts using Azure Logic Apps, ensuring seamless orchestration and monitoring of data pipelines.
• Ensured data quality, consistency, and compliance across all stages through validation, auditing, and metadata management.

Tech Stack: Azure Data Factory.|Azure Databricks.|Delta Live Tables.|Unity Catalog.|Azure Data Lake.|Azure Synapse Analytics.|Logic Apps.| PySpark | Jinja |SCD Types 1 and 2.| Medallion Architecture

• Designed and implemented data migration workflows with Sqoop for on-premise to Hadoop transitions.
• Built custom Sqoop connectors to integrate with proprietary data sources, and performed data cleansing and transformation during import.
• Created a centralized data lake on HDFS, integrating data from databases, Excel files, and server logs.
• Exported curated Hadoop data to external databases using Sqoop for business intelligence.
• Refined schema design to boost efficiency of Hive queries.
• Diagnosed and resolved Hive performance bottlenecks, including data skew, inefficient joins, and slow query execution.
• Managed schema evolution and compatibility in Hive for evolving datasets.

Tech Stack: Hadoop, Sqoop, Hive, HDFS, Linux, Shell Scripting, Data Governance, and Performance Tuning.

• Led end-to-end data migration projects, ensuring seamless transition from legacy ERP systems to on-premises SQL Server databases.
• Implemented SQL-based validation scripts, enhancing post-migration data accuracy by 30%.
• Automated reporting and dashboard creation in Excel, streamlining metric workflows for enhanced decision-making.
• Created and maintained detailed data mapping documentation and audit trails for compliance and scalability.
• Collaborated with cross-functional teams to align data architecture with business objectives and analytics requirements.