Karan Bajaj

Results-driven Data Engineer with experience at Enquero, adept in designing robust data pipelines and leveraging AWS Glue for ETL processes. Proficient in Python and PySpark, I excel in troubleshooting complex issues while collaborating effectively with clients. Committed to delivering valuable insights through data analysis and pipeline optimization.

• Team Reward - INTEGRITY (Performance with exceptional integrity for outstanding contribution.
• Peer to Peer award.

Big Data ETL Pipelines on AWS (Batch), Databricks, AWS Batch, MWAA, Athena, Redshift, 

• Created ETL batch pipelines (SCD1) to ingest, transform and implement business logic as part of retail business domain.
• Developed data processing code using Databricks, Py-Spark, and Spark SQL to load data into Delta Tables. Orchestrated the pipeline using Airflow.
• Implemented DQ checks between source and transformed data and loaded data under delta tables which were eventually pushed to Redshift using COPY operation.

Source Data Migration Project, AWS S3, Databricks, Py-spark, Airflow

• Analyzed existing source tables using older data and format.
• Identified the need for migration due to changes in the AWS S3 location.
• Compare and Performed validations on old vs new Data.
• Created new tables to accommodate the updated data.
• Enhanced the Py-Spark code to read and Process data from the new location in Parquet format.
• Modified the Airflow DAG (Directed Acyclic Graph) to efficiently process the new data.

Pipeline Development For MongoDB to Redshift Integration 

• Pyspark, Airflow, AWS, Databricks, MongoDB, Extract data from a specific MongoDB collection.
• Build Ingress and Load Pipeline to ingress raw data and used Pyspark and SQL to transform the data.
• Implement incremental updates or changes to the data by creating delta Tables and loaded cleansed data to redshift tables. Designed and implemented Airflow Directed Acyclic Graphs (DAGs) to automate the ETL process.
• Scheduled data ingestion, transformation, and loading tasks.
• Performed end-to-end testing to check proper functioning of pipeline.

• Microsoft AZ 900 certification
• Databricks Associate Data Engineer (Ongoing)