Pratik Patil

Project #1

Project Title: ABM Development

Environment: AWS S3, AWS RDS, Qlik Replicate, Databricks, Pyspark, Spark-SQL, Postgres.

Description: Project Involves data migrations from Aurora Postgres to AWS S3 using a tool Qlik Relicate and making the data available in Databricks Unity Catalog using structured streaming with readStream and writeStream. Developed custom primary and foreign keys, enabling seamless data migration through Bronze, Silver, and Gold layers. Implemented complex transformations, merge conditions, optimizations, and data reconciliation to ensure data integrity and performance for reporting and analytics.

Roles and Responsibilities:

• Designed and implemented data pipelines to migrate data from Aurora Postgres to AWS S3.
• Apply Transformations and lookups in Qlik to develop custom primary and foreign keys for seamless integration and consistent data relationships across tables.
• Managed the transition of data from S3 to Bronze, Silver, and Gold layers with optimized ETL workflows.
• Optimized data pipelines and queries for efficient storage and faster processing in Databricks.
• Collaborated with cross-functional teams to define requirements and resolve data-related challenges.
• Documented processes, transformations, and pipeline configurations for future reference and scalability.

Project #2

Project Title: Operational Data Store

Environment: AWS RDS, S3, AWS DMS, SQL, Python, SNS, etc.

Description: Migrated legacy data from mainframe systems to a modern relational database using AWS DMS. The project involved applying data transformations during migration, optimizing data structures, and creating SQL views and materialized views to support frontend application requirements. Focused on complex joins and relationships to ensure seamless data accessibility and performance.

Roles and Responsibilities:

• Designed and implemented AWS DMS architecture for real-time data migration and replication across heterogeneous and homogeneous databases.
• Configured and managed DMS replication tasks for full-load, ongoing replication, and change data capture (CDC).
• Performed data transformations using AWS DMS to map and modify schema structures during migration processes.
• Developed and maintained SQL triggers, functions, stored procedures and views to ensure data integrity and automate business logic.
• Performed query optimization and troubleshooting to enhance database performance and reduce latency.

Project Title: Inventory and Pricing Reporting

Environment: AWS S3, AWS Glue, Snowflake

Roles and Responsibilities:

• Designed and implemented the ingestion pipeline to process JSON data from an S3 bucket.
• Configured S3 triggers and managed data storage structures.
• Developed AWS Glue jobs to transform JSON data into the desired format.
• Implemented data cleansing, schema validation, and conversion to CSV format.
• Configured Snowpipe to automate the data loading process from the S3 landing bucket to Snowflake tables
• Set up event notifications to trigger Snowpipe workflows.
• Optimized Glue jobs and Snowpipe configurations to ensure efficient data processing and loading.
• Monitored pipeline performance and resolved bottlenecks.

Roles and Responsibilities:

• Develop and optimize database schemas, tables, views, and stored procedures.
• Create conceptual, logical, and physical data models to support product features and business logic.
• Define relationships, constraints, and indexing strategies for optimal performance.
• Analyze and optimize database queries, indexing, and partitioning to ensure high performance.
• Resolve bottlenecks and improve database response times.
• Implement database security best practices, including encryption, role-based access control, and audit trails.
• Perform regular backups, recovery testing, and system upgrades.
• Document database designs, configurations, and processes