DAYANAND PALLEMONI

Results-driven Data Engineer with over 6+ years of expertise in building and optimizing large-scale data processing solutions. Skilled in Spark Scala, SQL, Hadoop, Hive, and Google Cloud Platform (GCP), I specialize in designing high-performance data pipelines for seamless ingestion, transformation, and analysis. My experience includes developing and optimizing ETL workflows using Spark Scala and SQL for massive datasets, building scalable data architectures on GCP to ensure efficiency and reliability, and orchestrating workflows with Apache Airflow for automation and seamless execution. With a strong focus on performance tuning and resource optimization, I enhance data processing speed while ensuring data integrity, security, and compliance with industry best practices. Collaborating with cross-functional teams, I have successfully improved data pipeline efficiency, reduced processing time, and automated workflows, leading to better decision-making and operational excellence. Passionate about delivering scalable, cost-effective, and high-performance data solutions, I continuously strive to drive innovation and business success through data.

• Reduced data processing time by 35% through advanced performance optimization.
• Successfully migrated on-premises data pipelines to Google Cloud Platform, improving scalability and reducing costs by 20%.
• Automated over 90% of ETL processes, significantly reducing manual effort and enhancing operational efficiency.
• Enhanced data security by implementing strict IAM roles and policies.

Technologies: GCP (BigQuery, Dataflow, Cloud Storage, Cloud Pub/Sub), PySpark, Python, SQL

• Led the end-to-end implementation of a centralized GCP-based data lake, enabling enterprise-wide data management.
• Designed and developed real-time and batch data ingestion pipelines using Cloud Pub/Sub and Dataflow, enhancing data availability and reducing processing latency.
• Utilized BigQuery for scalable data warehousing and complex analytical queries, enabling faster business insights.
• Built PySpark-based data transformation workflows, ensuring data quality, consistency, and reliability across multiple sources.
• Automated data pipeline deployment and monitoring using Apache Airflow, reducing manual intervention by 90%.
• Integrated machine learning models into data pipelines, enhancing predictive analytics for improved business decisions.
• Implemented data security best practices, including IAM roles, encryption, and audit trails to ensure compliance.
• Conducted cost optimization and capacity planning, leading to a 20% reduction in cloud expenses.
• Provided comprehensive documentation and conducted training sessions for internal teams, ensuring a smooth transition and effective platform maintenance.

Technologies: PySpark, Python, SQL, Hive, Spark (Scala), GCP

• Developed and optimized data pipelines to streamline customer matching and profiling for a financial services client.
• Utilized Apache Spark (Scala and PySpark) for efficient large-scale data processing and transformation.
• Designed ETL workflows to ensure timely and accurate data processing for reporting and analytics.
• Collaborated with business stakeholders to align data solutions with operational and compliance requirements.

Technologies: Apache Spark, Spark SQL, Scala, Hive, Shell Scripting, SQL

• Engineered and maintained scalable data pipelines to support collection and recovery operations for financial services.
• Leveraged Apache Spark for large-scale data ingestion, transformation, and analytics, ensuring timely insights.
• Optimized ETL processes to enhance data accuracy, consistency, and performance.
• Partnered with cross-functional teams to develop custom data solutions tailored to business needs.
• Designed and managed data warehouses using Hive, improving query performance and accessibility for reporting teams.