AKHILESH PILLAI

Data Engineer

Cognizant

10.2021 - 03.2024

Company Overview: Spearheaded the strategic initiative transition of the client Dun&Bradstreet's analytics production platform from AWS and Cloudera to Google Cloud
This resulted in optimized performance and scalability, resulting in decrease in infrastructure expenditure and improved data processing capabilities
Involved in migration of several workflows migrating the data pipeline end to end with the help of GCP Cloud composer and Databricks in case of Spark, triggering Spark jobs with Databricks operator in Airflow DAG and storing the final outputs in GCS buckets
Streamlined the conversion process of HQL to Pyspark notebooks during the migration from Hive to Databricks, ensuring seamless integration and accurate functionality
Conducted comprehensive unit testing to validate Pyspark code, and incorporated email notifications and file delivery tasks in the Airflow DAG to facilitate efficient downstream delivery
Utilized implementation of a robust CI/CD pipeline, resulting in a 50% reduction in deployment time and an 80% decrease in manual errors when deploying critical components, such as a workflow notebook, properties file, Apache Airflow DAGs, and shell scripts, into the Quality Assurance (QEA) & production environment
Responsible for getting QEA signoff and UAT approval which results in deployment to production on the release date
Have also backtracked and solved errors encountered in workflow failures during scoring phase
Was proficient in client interaction and also updating the daily progress with agile methodologies and JIRA
In earlier stages of my involvement in the project included data integrity testing, change detection validation, scoring and data ingestion
Have also automated scenario of count modernization as a value to eliminate repetitive manual effort using Pyspark in real time project
Spearheaded the strategic initiative transition of the client Dun&Bradstreet's analytics production platform from AWS and Cloudera to Google Cloud
This resulted in optimized performance and scalability, resulting in decrease in infrastructure expenditure and improved data processing capabilities