Sagar Saini
Senior Big Data Engineer
Gurgaon
Summary
Experienced Senior Big Data Engineer with a proven track record of designing, developing, and implementing robust data solutions. Expertise in leveraging advanced technologies and frameworks including Apache Spark, Hadoop, Kafka, and cloud platforms such as AWS and Azure. Skilled in architecting scalable ETL pipelines, optimizing data workflows, and ensuring high data quality and integrity. Adept at integrating real-time streaming technologies for monitoring and analyzing critical business metrics. Strong problem-solving abilities with a focus on performance optimization and automation, reducing operational complexities and enhancing overall efficiency. Proven leadership in mentoring teams, driving technical excellence, and delivering innovative solutions to meet organizational goals and challenges.
Overview
6
6
years of professional experience
Work History
Senior Big Data Engineer
83Incs Softech Pvt Ltd
05.2022 - Current
- Java Programming: Worked on Java programming.
- ETL Development: Created Spark structured streaming ETLs using Java, MQTT, Kafka, Cloudera, HDFS, Spark, Cassandra, and MongoDB.
- Error Diagnosis and Correction: Diagnosed and corrected errors within code to enable application utilization and connections.
- Technology Utilization: Leveraged knowledge in Java, MQTT, Kafka, Cloudera, HDFS, Spark, Cassandra, and MongoDB to develop and optimize solutions.
- Data Quality Enhancement: Enhanced data quality by developing robust validation strategies to identify and correct inconsistencies.
- Real-Time Integration: Integrated real-time streaming technologies such as Azure ADX, Azure EventHub, and IoTHub for accurate monitoring of critical business metrics.
- Automation: Automated routine tasks through scripting languages, reducing manual effort and human error risks.
- Performance Optimization: Proactively addressed potential bottlenecks in the ETL process through regular monitoring, enabling seamless workflow operations.
- Batch and Streaming Jobs: Developed Spark batch jobs and utilized Spark Structured Streaming for efficient data processing.
- Security and Compliance: Ensured code security and compliance using Black Duck to identify and mitigate vulnerabilities.
Senior Data Engineer
Crisp Analytics Pvt Ltd
10.2021 - 04.2022
- Data Pipeline Development: Design, develop, and maintain data pipelines using AWS Glue to extract, transform, and load (ETL) data from various sources into data lakes and data warehouses.
- Data Storage Management: Utilize Amazon S3 for efficient data storage and management, ensuring data availability, durability, and security.
- Data Query and Analysis: Use AWS Athena for querying and analyzing large datasets stored in Amazon S3, optimizing query performance and cost.
- Data Migration: Implement and manage data migration processes using AWS DMS to migrate data between heterogeneous and homogeneous databases.
- Data Integration: Integrate data from various sources into the data ecosystem, ensuring data consistency and integrity across different systems.
- Performance Optimization: Optimize data processing workflows for performance and cost-efficiency, leveraging AWS services, PySpark, and best practices.
- Security and Compliance: Ensure data security and compliance with organizational and regulatory standards, implementing appropriate data access controls and encryption.
- Monitoring and Troubleshooting: Monitor data pipelines and workflows for performance and reliability, troubleshooting issues and implementing fixes as needed.
- Collaboration: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver high-quality data solutions.
Big Data Engineer
83incs softech pvt ltd
1 2019 - 10.2021
- Team Collaboration: Worked with a team of 50 people, including developers, QA, and scrum masters.
- Quality Metrics and Delivery: Collaborated to establish quality metrics and expectations for delivery, and worked with these throughout the sprint.
- Software Development: Engaged in coding, compiling, unit testing, integration, packaging, and deployment of developed software.
- Requirements Analysis: Understood the requirements and functional specifications to ensure alignment with project goals.
- Technical Investigation: Conducted technical investigation and analysis of incidents, problem replication, fault diagnosis, code fixes, and ensured customer satisfaction upon call closure.
- Testing Processes and Frameworks: Worked with other software leads to develop testing processes and frameworks.
- ETL Pipeline Design and Development: Designed, architected, and wrote ETL pipelines for our IoT application using technologies such as Java, MQTT, Kafka, Cloudera, HDFS, Spark, Cassandra, MongoDB, S3, AWS (EC2 instances and volumes), and Kubernetes.
- Continuous Deployment Pipeline: Developed a continuous deployment pipeline, integrating Git and Jenkins.
- System Testing and Validation: Participated in system testing and validation procedures, programming, and documentation.
- Big Data and Streaming: Deep knowledge of Kafka, Zookeeper, Cloudera, HDFS, Spark Structured Streaming, Spark Core, MQTT.
- Databases: Worked with MongoDB, Cassandra, Timescale, Hive.
- DevOps and Tools: Knowledge of Jira and Confluence, AWS services (EC2, S3), Kubernetes.
Software Engineer
ProfEdge Solutions Pvt Ltd
1 2018 - 12.2018
Responsibilities:
- Java Programming: Worked on Java programming tasks under the supervision of the Team Lead.
- ETL Development: Created Spark structured streaming ETLs using Java, MQTT, Kafka, Cloudera, HDFS, Spark, Cassandra, and MongoDB.
- Error Diagnosis and Correction: Diagnosed and corrected errors within code to enable application utilization and connections.
- Technology Utilization: Leveraged knowledge in Java, MQTT, Kafka, Cloudera, HDFS, Spark, Cassandra, and MongoDB to develop and optimize solutions.
Education
10th -
HBSE, SONEPAT
12th - undefined
HBSE, SONEPAT
B.TECH in ECE -
DEEN BANDHU CHOTU RAM UNIVERSITY OF TECH. & MANAGEMENT(DCRUST), SONEPAT
SonepatSkills
Swagger
Postman
Intellij Idea
Gradle
Jenkins
GitHub
JIRA
Robo3T
English
Hindi
MySQL
MongoDB
Cassandra
AWS - EC2
IAM
S3
Shell/ Bash Scripting
Kubernetes
Java
MQTT
Kafka
Cloudera
Hadoop
Spark
Jobtitle
BIG DATA ENGINEER
Timeline
Senior Big Data Engineer
83Incs Softech Pvt Ltd
05.2022 - Current
Senior Data Engineer
Crisp Analytics Pvt Ltd
10.2021 - 04.2022
Big Data Engineer
83incs softech pvt ltd
1 2019 - 10.2021
Software Engineer
ProfEdge Solutions Pvt Ltd
1 2018 - 12.2018
10th -
HBSE, SONEPAT
12th - undefined
HBSE, SONEPAT
B.TECH in ECE -
DEEN BANDHU CHOTU RAM UNIVERSITY OF TECH. & MANAGEMENT(DCRUST), SONEPAT