ENIYAN PARAMASIVAM
Senior Lead Data Engineer
Chennai
Summary
Experienced Senior Lead Data Engineer with over 18 years of expertise in designing and implementing large-scale data engineering, AI, and cloud solutions across AWS and Azure platforms. Skilled in building RAG-based AI assistants using Azure OpenAI (Mosaic AI), Databricks, and Microsoft Teams for enterprise automation. Proficient in Databricks Lakehouse architecture (Delta Lake, Photon, Unity Catalog) and ETL pipeline development using PySpark, Python, SQL, Kafka, DBT, and REST APIs. Strong hands-on experience with AWS (S3, Glue, EMR, Redshift, Lambda) and Azure (Data Lake, Synapse, ADF, Purview) for scalable, governed, and cost-efficient data platforms. Adept in MLflow-based tracking, Terraform, and GitLab CI/CD for automation and optimized cloud deployments.
Overview
18
18
years of professional experience
7
7
Certifications
2
2
Languages
Work History
Lead Databricks Data Engineer
Tata Consultancy Services (TCS)
08.2023 - Current
- Modernized the airline’s Global Data Warehouse to a Databricks Lakehouse on AWS, leveraging Delta Lake, Photon, and Unity Catalog for a scalable, governed, and cost-efficient architecture.
- Architected high-volume ingestion pipelines using Databricks Auto Loader, Kafka, and REST APIs, handling semi-structured data with dynamic schema evolution, and DLT pipelines for real-time processing.
- Implemented predictive optimization and liquid clustering on managed Delta tables, automating compaction and layout tuning; improved performance by 3× and reduced TCO by approximately 30%.
- Built transformation frameworks in DBT for silver-to-gold layers, with full lineage, data quality, and access governance via Unity Catalog.
- Automated infra provisioning and CI/CD deployments using Terraform and GitLab, integrating monitoring, alerting, and rollback strategies.
- Developed a Retrieval-Augmented Generation (RAG)–based AI Assistant chatbot, integrating Databricks, Azure OpenAI (Mosaic AI), and Microsoft Teams to automate crew and employee travel-policy inquiries using vectorized PDF document retrieval and natural-language responses.
- Implemented MLflow-based experiment tracking and cost analysis, monitoring token usage, embedding performance, and model accuracy, while optimizing prompt design and data chunking for scalable, low-cost enterprise deployment.
- Collaborated with business and analytics teams to validate data mappings, remediate historical issues during GDW migration, and optimize workloads using Photon and AQE.
Lead Azure Data Engineer
Tata Consultancy Services (TCS)
05.2020 - 07.2023
- Designed and delivered an end-to-end Azure Data Lakehouse using ADF, ADLS Gen2, and Databricks (Delta Lake), following Medallion architecture.
- Built ADF pipelines for batch and incremental ingestion; orchestrated Databricks notebooks for transformation and data quality checks.
- Developed Delta MERGE, Auto Loader, and Z-Ordering workflows to enable efficient CDC, schema evolution, and performance optimization.
- Implemented data quality and monitoring with ADF alerts, Databricks job notifications, and Webhook/Teams alerts; added SQL alerts for threshold KPIs.
- Configured Azure Monitor, Log Analytics, and Action Groups for centralized alerting and SLA dashboards.
- Integrated Bitbucket for source control, and Jenkins CI/CD for automated deployment across Dev, QA, and Prod environments.
- Secured access via AAD Managed Identities, Key Vault, and Private Endpoints; maintained lineage in Azure Purview.
- Supported Power BI and Synapse (serverless) reporting from curated gold tables.
Senior AWS Data Engineer
Tata Consultancy Services (TCS)
01.2014 - 04.2020
- Designed and built an AWS-based enterprise data lake for airline reservation and operational data sourced from Amadeus using S3, Glue, EMR, and Redshift.
- Developed ETL pipelines in PySpark and AWS Glue to ingest, cleanse, and transform large reservation datasets into curated and gold layers.
- Implemented data lakehouse design (Raw → Curated → Gold), ensuring schema consistency, lineage, and PII compliance.
- Optimized Redshift using DISTKEY/SORTKEY, compression encoding, and materialized views to improve query performance by 40 %.
- Managed orchestration and automation through Step Functions, Lambda, and CloudWatch Events, with SLA alerts via SNS.
- Ensured data security and governance with KMS encryption, IAM policies, Macie for PII detection, and CloudTrail auditing.
- Built a data reconciliation framework tracking counts and error thresholds between ingestion stages, and visualized KPIs via QuickSight dashboards.
- Implemented cost optimization strategies using Spot instances, S3 lifecycle policies, and RA3 nodes, reducing total cost of ownership by ≈45 %.
- Collaborated with business teams to deliver Power BI and QuickSight reports for flight revenue, booking trends, and operational insights.
Lead Developer
Tata Consultancy Services (TCS)
07.2009 - 12.2013
- Converted COBOL logic into BTEQ scripts to enhance processing efficiency.
- Scheduled daily batch jobs in Unix to ensure timely data updates.
- Maintained merchandising systems to optimize inventory management and performance.
- Supported demand chain management solutions to streamline operational workflows.
Developer
Tata Consultancy Services (TCS)
07.2007 - 06.2009
- Enhanced mainframe applications utilizing COBOL, DB2, and JCL for optimal performance.
- Scheduled batch jobs to ensure smooth operational flows through CA7.
- Maintained operational efficiency by managing job schedules and workflows effectively.
- Client: The Home Depot (Retail)
Education
Bachelor's degree - Chemistry
Thiruvalluvar University
Tirupattur, India06.2007
Skills
Databricks platform (Delta Lake, Photon, Unity Catalog)
Cloud platforms: AWS (S3, Glue, EMR, Redshift), Azure (Data Lake, Synapse, OpenAI)
AI/ML: RAG, NLP, Vector Search, MLflow tracking, prompt optimization
Real-time data pipelines (Kafka, REST APIs, Auto Loader, DLT)
ETL/ELT frameworks: PySpark, DBT, AWS Glue, ADF
DevOps automation: Terraform, GitLab CI/CD
Data modeling: Star Schema, Data Vault, SCD Type-2
Governance & compliance: Unity Catalog, IAM, KMS, Purview
Visualization: Power BI, QuickSight
Programming: PySpark, Python, SQL
Certification
Databricks Certified Data Engineer Professional, 2025
Timeline
Lead Databricks Data Engineer
Tata Consultancy Services (TCS)
08.2023 - Current
Lead Azure Data Engineer
Tata Consultancy Services (TCS)
05.2020 - 07.2023
Senior AWS Data Engineer
Tata Consultancy Services (TCS)
01.2014 - 04.2020
Lead Developer
Tata Consultancy Services (TCS)
07.2009 - 12.2013
Developer
Tata Consultancy Services (TCS)
07.2007 - 06.2009
Bachelor's degree - Chemistry
Thiruvalluvar University