Shuvamoy Mondal
Bangalore
Summary
With over 14.5 years of experience, including onsite work in the USA, I possess a proven ability in client/server and distributed multi-tier application development, database design, data processing, and data warehousing. Furthermore, I have accumulated nearly 8 years of experience working with big data platforms and have gained hands-on expertise in various cloud environments such as Amazon Web Services (AWS), Google Cloud (GCP), and Azure. As a graduate from WBUT, my passion lies in problem-solving and making a meaningful impact in my next endeavor.
Overview
15
15
years of professional experience
1
1
Certification
Work History
Senior Data Engineer
Telstra
08.2023 - Current
- Designed an architecture for complex ETL/ELT pipelines to ingest and transform data from diverse sources (Azure Storage,SFTP server, Database) into data lakes using Azure Data Factory and integrated with Azure DataBricks along with Delta lake Feature
- Design a serverless ETL Pipeline architecture in AWS to leverage AWS Glue, Lambda, Amazon S3, and AWS Step Functions, SQS/Kinesis/SNS
- Leveraged Delta tables in DataBricks for atomic operations and schema evolution without downtime, maintaining data consistency in streaming and batch scenarios
- Worked closely with DevOps, Development, and Product teams to ensure alignment on cloud strategy, architectural guidelines, and overall project goals
- Implemented CI/CD pipelines using Azure DevOps for automated deployment of application code and infrastructure changes
- Optimized Delta Lake performance by implementing file compaction (OPTIMIZE) and Z-Ordering, reducing query execution time by 30% and improved metadata management by implementing vacuum retention policies, reducing storage costs
Senior Data Engineer (Manager)
Deloitte
04.2019 - 08.2023
- Design a Data pipeline using Microservices like Docker and Kubernetes(GKE) in GCP to build the image which read the event from Kafka and write a raw data into GCS, which is further processed and loaded into Snowflake Database
- Designed and implemented serverless ETL pipelines using Google Cloud Dataflow (Apache Beam), Cloud Functions, and BigQuery
- Developed and optimized data ingestion, transformation, and orchestration using Cloud Composer (Airflow) and Pub/Sub for nearly event-driven processing
- Designed and implemented scalable ETL pipelines using AWS Glue, Lambda, and Step Functions for batch and real-time data ingestion
- Worked on building the framework in spark using scala language and implemented the ETL logic
- Experience to automate the code using pipelines in Jenkins and terraform deployment tool
- Worked on AWS Lambda to pull data from S3 and load into DynamoDb Table once the event arrived in S3
- Automated infrastructure deployment and resource management using Terraform and CloudFormation
- Ensured data governance, security, and compliance with IAM roles, encryption (KMS) and Optimize performance and cost by implementing partitioning, clustering, and query optimization techniques in BigQuery
- Built and deployed ML models using AWS SageMaker for predictive analytics
- Developed deep learning models for image classification
Big Data Hadoop Developer
Cognizant
08.2011 - 03.2019
- Experienced in handling large datasets using Partitions, Spark in Memory capabilities, Broadcasts in Spark, Effective & efficient Joins, Transformations and other during ingestion process itself
- Involved in creating Hive tables and loading and analysing data using hive queries
- Worked in Data migration from On Prem data lake to S3 storage using DMS
- Experience in data ingestion from Kafka using Spark Streaming in EMR and land to S3 storage to process further to transform from raw layer and load it into Redshift Database
- Involved in database development by creating PL/SQL functions, procedures, triggers, and packages and Materialized views
- Design and developed a project using HP Vertica columnar database
- Expert level experience on TeraData that includes Cursors, Procedures, Functions and Partitioned Tables, Triggers, Dynamic SQL
- Creating Script for Multi-load, Fast Load and BTEQ in TeraData
Pl/SQL Developer
Mitra Systems Inc
10.2009 - 07.2011
- Created Custom Packages, Stored Procedures, Function and SQL Scripts to load data into Pharmacy warehouse from different sources
- Written new PL/SQL packages, modified existing code perform Certain Specialized functions / enhancement on oracle Application
Education
Master of Science - Information Systems
Stratford University
Virginia,USA08.2009
Bachelor of Technology (B.Tech.) - Electronics and Communications
JIS College of Engineering
Kalyani, West Bengal01.2006
Skills
- Azure Cloud
- Azure Data Factory
- Azure DataBricks
- Synapse
- ADLS
- Azure Functions
- PySpark
- Spark
- GCP
- BigQuery
- GCP-DataProc
- Cloud SQL
- GKE
- GCP-DataFlow
- Cloud Function
- AWS
- AWS-Lambda
- AWS-StepFunction
- AWS-Glue
- AWS-Athena
- AWS-Redshift
- AWS-DynamoDB
- AWS-EMR
- AWS-EC2
- KAFKA
- Scala
- SnowFlake
- HDFS
- Docker
- Kubernetes
- Cassandra
- Terraform
- Jenkins
- Azure DevOps
- Oracle
- Teradata
- PL/SQL
- Unix ShellScript
- Machine Learning
- Deep Learning
- Git version control
- ETL development
- Big data processing
- Python programming
- NoSQL databases
- Hadoop ecosystem
- Data warehousing
Certification
AWS Solution Architect Associate, 08/01/18, 08/01/20
Languages
English
Advanced (C1)
Timeline
Senior Data Engineer
Telstra
08.2023 - Current
Senior Data Engineer (Manager)
Deloitte
04.2019 - 08.2023
Big Data Hadoop Developer
Cognizant
08.2011 - 03.2019
Pl/SQL Developer
Mitra Systems Inc
10.2009 - 07.2011
Bachelor of Technology (B.Tech.) - Electronics and Communications
JIS College of Engineering
Master of Science - Information Systems
Stratford University
Similar Profiles
Neha ParkarNeha Parkar
Assistant Store Manger at TelstraAssistant Store Manger at Telstra
Karanveer SinghKaranveer Singh
Retail Assistant Manager at TelstraRetail Assistant Manager at Telstra
Kirstie HardingKirstie Harding
Salesperson at TelstraSalesperson at Telstra
James NaumovskiJames Naumovski
Telecommunications Technician at TelstraTelecommunications Technician at Telstra
Arti NaikArti Naik
Cluster Manager at Medifit Consultants Private LimitedCluster Manager at Medifit Consultants Private Limited