Suman Singh

• Led the migration of RDBMS data into a Hadoop data lake, enabling efficient processing and analysis of massive datasets (2-2.5 TB daily).
• Optimized data extraction and transformation pipelines using Spark (Python/Dataframes, RDDs) for faster consumption by data visualization tools (e.g., PowerBI).
• Reduced processing time through performance optimization techniques in HDFS, Spark, and Impala.
• Developed and implemented data processing pipelines to support ad-hoc business requests, fostering data-driven decision making.
• Mentored and managed a Hadoop developer team, ensuring successful project delivery and fostering team growth.

Key Achievements:

• Successfully designed and built a scalable data ingestion mechanism for the NSDL data lake, consolidating data from various sources into a unified Common Data Model.
• Optimized data processing pipelines, resulting in significant improvements in data availability and time-to-insights for business users.
• Established a DevOps culture within the team, leveraging tools like GitHub, Rundeck, and Ansible scripting to automate deployments and streamline workflows.

• Led the implementation of Site Reliability Engineering (SRE) practices to improve data platform uptime, reliability, and scalability.
• Designed and developed monitoring dashboards (PowerBI & Microsoft Data Studio) to visualize key SLI metrics against defined SLOs using GCP (Google Big Query). This enabled proactive identification and resolution of potential issues.
• Established and maintained coding best practices, including documented SOPs and knowledge bases in Confluence, ensuring code quality and maintainability.
• Managed data flow automation (DataLake to DataPond) using tools like Informatica, Big Data, SQL, Unix, Minifi, Nifi, and Rundeck.
• Championed knowledge sharing by introducing Confluence as a central knowledge repository.
• Implemented a data platform monitoring and communication process to ensure timely issue identification and resolution.
• Owned the definition and implementation of an RACI matrix to clearly define roles and responsibilities within the SRE team.

• Led a high-performing team of 9 data engineers responsible for building and maintaining data pipelines. This ensured timely and accurate data flow for the Prime Brokerage project.
• Developed automation scripts (Shell) to streamline critical tasks including data extraction, performance reporting, and table space management. These scripts significantly reduced manual effort and improved overall data processing efficiency.
• Built interactive dashboards (Power BI) to visualize daily rejection records, enabling stakeholders to quickly identify and address data quality issues.
• Implemented an hourly status monitoring system through automated emails, providing real-time insights into data pipeline health and proactively alerting teams of potential problems.
• Leveraged expertise in Hadoop, Shell scripting, and SQL to troubleshoot and resolve complex data loading and performance issues. This ensured data integrity and maintained optimal system performance.
• Proactively identified opportunities for automation and developed custom Shell scripts to streamline data processing tasks and minimize ongoing support requirements. This focus on continuous improvement resulted in a more efficient and scalable data management environment.

• Led a team of application support specialists in providing L2 support and technical guidance for the LCDB (Legal and Compliance Database).
• Ensured application uptime through 24/7 support, addressing performance, resource, software, and application problems.
• Implemented production changes, documented processes, and performed root cause analysis to improve application stability.

• Supported daily data loads for prepaid and post-paid systems, ensuring accurate and timely processing.
• Developed Shell Scripts and PLSQL queries in UNIX to meet specific reporting needs.
• Developing a scalable batch framework to efficiently process partial Usage Detail Records (UDRs).