ASEER JESAN A

1) Spearheading the development and management of ETL pipelines in Google Cloud Environment.

• Developed Pipelines in Vertex AI, combining both Vertex AI clusters and BigQuery's Compute to execute pipelines for efficient cost management and performance.
• Managing movement of data sets through multiple services on each layer of pipeline from Cloud SQL to BigQuery to Google Cloud Storage for different use cases.
• Followed Effective storage techniques in BigQuery (Partitions and Physical / logical Bytes) and Google Cloud Storage.
• Ensured Data Quality of Raw datasets using Dataplex's wide range of options.

2) Developed multiple modules in Existing pipeline in Azure Data Factory to cater needs of Tool/Dashboard aimed at effective monitoring and management of pipelines.

• Developed automated workflow to live feed Ganglia Metrics from spark cluster to Blob Storage on pipeline trigger for effective monitoring on cluster health.
• Created workflow to organize pipeline output files by dynamically creating folders named after each pipeline run ID, ensuring structured and traceable data storage.

• Designed and Built Data Pipelines handing Data Ingestion, Data Quality, Data Transformations, Mapping Management, using Azure Services such as Azure Data Factory, Azure Databricks and Azure Blob Storage.
• Incorporation of Machine Learning Models in Azure Environment leveraging services such as Feature Store, MLFlow, building post processing pipelines to infer the model outputs as well as orchestrating pipelines based on the model outputs.
• Handled Documentation and Code Management through both GitHub and Azure DevOps Services.
• Transforming multiple unorganized datasets received directly from clients to provide a comfortable picture of the datasets to extract maximum information using Python and SQL.