Confident and results-driven Data Engineer with 2+ years of experience accomplished in compiling, transforming and analyzing complex information. Proficient at Data Engineering, DataOps, Data Orchestration, Data Pipelines, Data Integration, Data Reliability, Data Migration and building Dashboards in Cloud Platforms.
MAX-NG [APR 2023 - Present]
Built reports for various KPI's and created leaderboard Dashboard where stakeholders can identify top and low performers based on all filtration metrics at each metric level
Created a cloud function to pick each file from a generic bucket and check the documentStatus and based on it move it to a specific GCS bucket and load it to a specific BQ table.
Built Monetary Collection reports for each of the agents Collection performance for timeline intervals which helped to improve the agents collection analysis.
DrivenBrands [DEC 2021 - APR 2023]
The source data available from each Brand is integrated into Business needed data by Streaming from the source, ETL process and populated it into the BQ table with Airflow through CI/CD Pipeline.
The various sites in the SONNY'S CARWASH CONTROLS are pulled from the API endpoint and loaded into Temp Table. Created a DBT model for BQ tables for each category and ingested into the destination table and did DataOps work from the retrieved data and created a view.
Upgraded Apache Airflow 1.10.10 to Cloud Composer Airflow 2.2.5 version. I have Collaborated with the team on changing the configuration and DAG changes with respect to the Cloud Composer Environment.
Explored the Google Ads 360 API and gathered the information of generating the OAuth Token. Developed a DAG to ingest the missing data dump into the BQ table.
Built a DBT model and ingested in the Piperr, for data ingestion from postgres to BQ on daily basis with Airflow DAG developed from Piperr.
Language, Frameworks and Tools
Python, TensorFlow, PyTorch, CNN, OpenCV, Cloudera Workbench for Data Science Cloud Platform, Docker, Jupyter Notebook, Kaggle, Google Colab, Git.
Languages
Data Engineering
Cloud Platforms
Database
Frameworks and Tools
Version Control
Data Visualization
Agile Project Management
CI/CD