ENIYAN PARAMASIVAM
Chennai
Summary
Over 18 years of experience as a Lead Data Engineer, with 11 years focused on developing impactful data platforms in aviation, banking, and retail. Proficient in creating end-to-end data pipelines on cloud platforms such as AWS, Azure, and Databricks, while leading modernization initiatives. Expertise in translating business needs into scalable architectures, optimizing costs and performance, and ensuring data governance. Currently leading proof-of-concepts for generative AI on Databricks, investigating agentic AI patterns and responsible AI practices.
Overview
18
18
years of professional experience
1
1
Certification
Work History
Lead Data Engineer & GenAI POC Lead
Virgin Australia
Sydney
08.2023 - Current
- Drove modernization of airline’s data platform on AWS and Databricks, migrating legacy Global Data Warehouse to scalable lakehouse framework.
- Architected ingestion pipelines utilizing Databricks Autoloader, REST APIs, and custom frameworks for dynamic schema handling of terabytes of semi-structured data.
- Developed real-time pipelines with Delta Live Tables for sources including Salesforce and Kafka, establishing reusable event-driven ingestion frameworks.
- Implemented DBT-based transformation pipelines for silver-to-gold layer processing, ensuring data lineage, quality, and governance via Unity Catalog.
- Automated infrastructure and orchestrated CI/CD pipelines using Terraform and GitLab to standardize deployments and monitoring.
- Collaborated with business analysts to validate mappings, resolve historical data anomalies during GDW migration, and optimize pipeline performance.
- Led proof-of-concepts for generative AI on Databricks, iterating from requirements gathering to production implementation.
- Designed retrieval-augmented generation prototypes leveraging Mosaic AI capabilities, integrating chatbots that synthesize enterprise data for context-aware responses.
Lead Azure Data Engineer
Westpac
Sydney
05.2020 - 07.2023
- Executed end-to-end migration of on-premise platforms to Azure Data Lake, delivering a governed mortgage data product.
- Designed multi-tier data models using Azure Data Factory and Databricks pipelines to ensure auditable data promotion.
- Established reconciliation and control frameworks to guarantee data quality and regulatory compliance.
- Integrated Adobe Analytics data with mortgage datasets for comprehensive customer funnel insights, delivering dashboards via Synapse and Power BI.
- Directed cross-functional team of data engineers, optimizing PySpark workloads to reduce processing time by 40 percent.
- Oversaw seamless production releases, ensuring all processes adhered to quality standards.
- Developed strategies for effective data governance and compliance throughout migration process.
- Collaborated with stakeholders to align project objectives with organizational goals.
Senior Data Engineer / AWS SME
Qantas
Sydney
01.2014 - 04.2020
- Supported legacy reservation systems and became subject-matter expert in EDIFACT feeds and booking data.
- Migrated nine years of reservation history to Amazon Redshift using S3, Glue, and EMR.
- Built pipelines for Amadeus XML to S3, Glue/Spark, and Redshift, integrating with Teradata.
- Implemented SCD Type-2 and maintained raw data in S3 for analytics purposes.
- Led CRM migration from SQL Server to Salesforce utilizing AWS Glue and custom Python APIs.
- Designed mapping rules, reconciliation checks, and business-rule validations for data integrity.
- Migrated crew scheduling data to AWS, creating XML to Parquet pipelines with Glue/Spark.
- Orchestrated workflows via Lambda and S3 to enhance data processing efficiency.
Lead Developer
The Home Depot
Chennai
07.2009 - 12.2013
- Converted COBOL logic into BTEQ scripts to enhance processing efficiency.
- Scheduled daily batch jobs in Unix to ensure timely data updates.
- Maintained merchandising systems to optimize inventory management and performance.
- Supported demand chain management solutions to streamline operational workflows.
Developer
The Home Depot
Chennai
07.2007 - 06.2009
- Enhanced mainframe applications utilizing COBOL, DB2, and JCL for optimal performance.
- Scheduled batch jobs to ensure smooth operational flows through CA7.
- Maintained operational efficiency by managing job schedules and workflows effectively.
Education
Bachelor’s degree - Chemistry
Thiruvalluvar University
Tirupattur, India06-2007
Skills
- Databricks platform expertise
- Delta Lake architecture
- Cloud computing skills (AWS, Azure)
- Object storage systems (S3, ADLS)
- Analytical databases (Redshift, Synapse)
- ETL frameworks (Glue, ADF)
- Distributed computing tools (EMR, Apache Spark)
- Coding proficiency in Python and SQL
- Data pipeline management with DBT and Delta Live Tables
- Workflow management with Airflow
- DevOps practices using Terraform
- Source control with GitLab CI/CD
- Modeling methodologies for data structures
- Schema design strategies for Fact/Dimension/Star schema and Data Vault
- Incremental updates with SCD Type-2
- Compliance and governance with Purview
- AI model deployment strategies
- Advanced search capabilities with vector search
- Automated workflows leveraging AI technologies
Notable Projects Pocs
- Developed a proof-of-concept generative AI agent that generates SQL queries from natural language and summarises data quality metrics, using Databricks Model Serving and Unity Catalog for data governance.
- Implemented a POC for an airline operations assistant chatbot using RAG: integrated flight operations data and knowledge articles into a vector store, built retrieval pipelines and used Mosaic AI evaluation to tune prompts and ensure accurate responses.
- Explored data summarization and anomaly detection using large language models; compared open-source models fine-tuned on domain data with commercial models; documented evaluation metrics and recommended best practices for production.
Education Certifications
- Bachelor’s degree in Computer Science (or related field), University Name, Year
- Databricks Certified Data Engineer Associate
- Databricks Generative AI Practitioner, in progress
- AWS Certified Solutions Architect – Associate
- Azure Data Engineer Associate
Certification
· Databricks Certified Data Engineer Professional – 2025
· Databricks Certified Data Engineer Associate – 2025
· Databricks Generative AI Fundamentals – 2025
· Databricks Certified Generative AI Engineer Associate – In Progress
· Microsoft Certified: Azure Data Engineer Associate (DP-203) – 2021, 2022
· Microsoft Certified: Azure Fundamentals (DP-900) – 2020
Languages
Tamil
First Language
English
Proficient (C2)
C2
Timeline
Lead Data Engineer & GenAI POC Lead
Virgin Australia
08.2023 - Current
Lead Azure Data Engineer
Westpac
05.2020 - 07.2023
Senior Data Engineer / AWS SME
Qantas
01.2014 - 04.2020
Lead Developer
The Home Depot
07.2009 - 12.2013
Developer
The Home Depot
07.2007 - 06.2009
Bachelor’s degree - Chemistry
Thiruvalluvar University
Similar Profiles
Belinda VostiBelinda Vosti
Flight Attendant at Virgin AustraliaFlight Attendant at Virgin Australia
Sohail KhanSohail Khan
Baggage Handler at Virgin AustraliaBaggage Handler at Virgin Australia
Jeff MaricarJeff Maricar
Regional Cabin Crew at Virgin AustraliaRegional Cabin Crew at Virgin Australia
Caitlan AyoubCaitlan Ayoub
IFSC at Virgin AustraliaIFSC at Virgin Australia
Vedavalli SVedavalli S
Senior Technical Lead at Alten Calsoft LabsSenior Technical Lead at Alten Calsoft Labs