Senior Python and Airflow Developer with over 5+ years of experience in designing, developing, and optimizing scalable data pipelines and workflows. Extensive expertise in deploying Apache Airflow to orchestrate complex, mission-critical workflows. Skilled in leading teams, mentoring junior developers, and collaborating with cross-functional teams to deliver high-quality solutions. Proven ability to enhance performance and scalability in data-driven environments, leveraging knowledge of cloud services (AWS, Azure) and DevOps practices.
Overview
9
9
years of professional experience
Work History
Lead Software Engineer
Societe Generale Global Solution Centre
Bengaluru
07.2019 - Current
Lead the design and implementation of data pipelines using Apache Airflow to support complex processes, ensuring the scalability and reliability of workflows for real-time and batch data processing.
Designed Airflow Dags to create Airflow instances for clients.
Containerize Airflow using Docker and manage deployments with Kubernetes.
Built and deployed custom Python operators, sensors, and hooks in Airflow to automate various data integrations (APIs, databases, cloud storage).
Optimized DAG performance by reducing task execution times by 40%, implementing dynamic task generation, and parallel task execution.
Mentored junior developers and provided technical guidance on Python best practices, DAG design, and Airflow's advanced features, contributing to team knowledge sharing and code quality improvement.
Managed CI/CD pipelines for continuous Airflow DAG deployment using Jenkins, ensuring streamlined and automated deployments across development, staging, and production environments.
Provided leadership and mentorship to a team of 5, fostering a collaborative environment and ensuring adherence to best practices.
Senior Operations Professional
IBM India Pvt Ltd
Bengaluru
04.2015 - 07.2019
Architect and lead the implementation and administration of IBM BigFix for over 100,000 endpoints across Windows, Linux, and macOS platforms, providing comprehensive automation for patch management, software deployment, and security compliance.
Designed and developed advanced automation scripts (PowerShell, Python) and custom BigFix Fixlets, Tasks, and Baselines to automate critical operations, reducing manual intervention by 60%.
Spearheaded the integration of BigFix with ServiceNow for automated incident management, resulting in a 40% reduction in mean time to resolution (MTTR) for endpoint-related issues.
Led the migration of BigFix infrastructure to AWS, including cloud-based relays and disaster recovery solutions, improving scalability and reducing operational costs by 25%.
Established best practices for system architecture, compliance management, and monitoring to ensure security and performance of the BigFix environment.
Provided leadership to a team of 6 BigFix engineers, managing task assignments, conducting code reviews, and mentoring junior engineers on BigFix and automation strategies.
Optimized patch deployment workflows by 50%, automating deployment windows and leveraging BigFix’s dynamic grouping and relevance language to improve efficiency and ensure compliance across global data centers.