MANOJKUMAR RAJENDRAN
Staff Data Engineer
Chennai,Tamilnadu
Summary
- 16+ years of professional experience spanning InfraOps, Healthcare, Banking, Energy and MLOps domains.
- A full stack Python data engineering manager with extensive hands-on experience in design/execution of the end-to-end architecture.
- Driven multiple analytic projects with high complexity, strategic value, and executive visibility.
- Built high performance teams and infused a culture of Innovation, Ownership, Excellence and Intrapreneurship among team members. Extensive hands-on experience in Data modelling, Data Ingestion, Data analytics, Statistical models, Data visualization/reporting.
- Designed and executed ML pipelines using Supervised & Unsupervised learning methods.
- Designed and implemented scalable big data analytics solutions
- Solutioning of architecture on both on-premises and AWS.
- Served as onsite tech lead and spearheaded project execution from conception till closure.
- Customer-focused, results-focused individual; passionate professional; pro- active in providing efficient & scalable solutions which equates to customer success.
- Implemented many enterprise-wide scalable automation initiatives for various labor- consuming processes. Proponent of best programming practices and evangelize across organisation.
- Technical lead, mentoring juniors & interns and bringing out their best.
Overview
16
16
years of professional experience
6
6
years of post-secondary education
Work History
Staff Data Engineer
Oportun
Chennai
02.2022 - Current
- Developed a proprietary FastAPI based MLOps platform providing services like ML model management, model versioning, model validation, model auditing, model orchestration, and model deployment.
- Designed and implemented the MLOps platform by using custom lightweight DAGs using native Python, resulting in 100x increase in performance and hassle-free orchestration, compared to using industry-familiar orchestration frameworks.
- Implemented validation mechanism which allowed customers to validate their model in the MLOps platform using huge datasets.
- Developed model metrics engine for reporting to business leaders.
TECHNICAL MANAGER
GAVS TECHNOLOGIES INDIA PVT LTD
04.2020 - 01.2022
- Designed and implemented a Pyspark based AIOps platform for infra monitoring of physical/digital assets of organization, with an aim of reducing the MTTR.
- Liaised between business and technology units to manage delivery schedules for the AIOps platform.
- Technically mentored the team in development of the AIOps platform
- Developed many features of platform such as Suppression of alerts, correlation of events, Pattern based clustering, Incident triaging, Dynamic threshold of alerts, Automated remediation, Root cause insights, etc.
- A lightweight distributed processing engine using the multiprocessing features of Python as an alternative to Spark for processing huge data.
LEAD ENGINEER
GAVS TECHNOLOGIES INDIA PVT LTD
07.2013 - 03.2020
- Developed a parallel processing OOPS-based engine using Airflow + Celery for executing Statistical algorithms to compute strategic insights on decades of healthcare data.
- Implemented a Flask based web application to display the prediction accuracy of various statistical models
over time. - Authored a CI/CD solution using Teamcity, Chef for automatic deployment of web apps.
- Designed a RPA bot using selenium to integrate web reports with proprietary governance portal.
- Developed an automated utility to generate LMS certificates and PMS letters.
- Mentored juniors in improving their technical knowledge to boost their productivity.
SENIOR APPLICATIONS DEVELOPER
INAUTIX TECHNOLOGIES INDIA PVT LTD
07.2010 - 07.2013
- Developed and enhanced a SWIFT based settlement system for Forex trades.
- Created periodic jobs for hassle free settlement of trades on their settlement day.
- Learned and shared knowledge of new technologies to support technically challenging projects and assume various responsibilities.
- Implemented consistent coding practices.
- Estimated work hours and tracked progress using Scrum methodology.
- Tested functional compliance of company products.
APPLICATIONS DEVELOPER
INAUTIX TECHNOLOGIES INDIA PVT LTD
05.2007 - 07.2010
- Designed and developed application scripts for test scenarios.
- Wrote code for SWIFT based proprietary software application for settlement of trades.
Education
PGWPM - IT, operations
GREAT LAKES INSTITUTE OF MANAGEMENT
09.2015 - 09.2017
BACHELOR OF ENGINEERING - MEPCO, Electronics and communications
SCHLENK ENGG. COLLEGE
04.2003 - 04.2007
Skills
Python 36Supervised and Unsupervised learningSpark 300 and Pyspark programming
Spark Structured Streaming and Kafka message bus
Docker containerizationAWS services such as IAM,EC2,EMR,EBS,ELB,ALB,S3
Deployment of codebase via Azure Devops pipelineevops
Hadoop: Sqoop, Flume, HDFS, Hive, Phoenix, YARNHBase
Data analytics toolkit - pandas, numpySeaborn, matplotlibRDBMS tools – Oracle SQL, Postgre SQLNoSQL databases - MongoDB, Scylla DB
Linux Centos, IBM AIX, Sun SolarisShell (Korn, C, Bash) scriptingPerl 5 scriptingPowerShell scriptingDevOps tools: Teamcity, Chef, AWSCloud formation, code Deploy, S3ETL suites -SAP Business Objects Suite XITalend 6GitFtp (Filezilla), sftp (winScp)Knowledge on R, TableauProject management: Azure Devops(VSTS), JiraFast API, Flask and Django
Airflow, Dagster
Accomplishments
- Designed and implemented a MLOps platform for the organization for seamless deployment and management of ML models thereby facilitating fast learning and continuous improvement.
- Conceptualized, designed and built/mentored a AIOps platform (https://zif.ai) for accelerated businessoutcomes with the following features:
- Prediction of system outages: Timeseries forecasting/Prediction of system down time fromperformance data of various system counters like CPU, Memory, Disk, Application latency using RandomForestRegressor for future time series
- Compute the ETI (Expected Time to Impact) of the physical assets based on Forecasted utilization and present them as Opportunities for L1/L2engineers for proactive maintenance of physical assets
- Established 95% accuracy in predicting system outages
- Anomaly detection: An anomaly detection model using statistical methods like z-score, IQR and
- Mean anomaly with vote-based scoring, Isolation Forest and percentiles and use the modelinsights to compute Dynamic Threshold of physical assets
- Pattern-based clustering of alerts: An unsupervised model to correlate unrelated events based onpattern co-occurrence using DBSCAN algorithm
- Noise reduction of alerts: A fault-tolerant and reliable Spark Structured streaming engine in multi-node clustered environment with capability of analyzing 10 lakh events in a minute
- Persisting theanalytics insights in a distributed mongo DB cluster
- Event correlation: A business rules-based algorithm to correlate events & alerts from multipleservers based on dynamically changing physical & logical topology of devices
- Root cause Insights: An algorithm to derive the RCA of an issue based on 360-degree analysis ofvarious data at hand
- Automated Remediation: Algorithm to auto close cases/tickets to reduce operational burden of L1/L2 engineers
- Automated Incident triaging: Algorithm to recommend technician to assign to a new ticket basedon patterns learned from historical incidents/tickets warehouse
- Dynamic threshold-based flagging: Algorithm to distinguish Actionable insights with non-actionable insights based on Dynamic threshold of various counters
- Prioritization of alerts/Insights: Algorithm to prioritize cases/tickets for prompt action of Operational team within SLA
- Acts on user-driven inputs configured in platform
- A lightweight distributed processing engine using the multiprocessing features of Python as analternative to Spark for processing huge data
- Migrated the Spark components from on-premises to AWS EMR
- An analytics engine for pre/post processing data of Statistical modelling using factory designpattern (OOPS paradigm in Python)
- A parallel processing engine using Airflow + Celery for executing Statistical algorithms tocomputing insights on decades of healthcare data
- An end-to-end ETL solution using SAP Business Objects Suite XI
- Tools used: Universe designer,
- Crystal reports, CMC, Infoview
- A Flask based web application to display the prediction accuracy of various statistical models overtime
- A CI/CD solution using Teamcity & Chef for automatic deployment of web apps
- An RPA bot using selenium to integrate web reports with proprietary governance portal
- An automated utility to generate LMS certificates and PMS letters
- A utility to monitor remote Linux servers and apply remediation measures in case of thresholdbreaches/anomalies
- A customized version of Project management framework using open-source enterpriseframework called Project Open to suit the needs of organization
- A utility to periodically refresh BIC loads using Perl and generate custom reports for integrating to Dashboards.
Timeline
Staff Data Engineer
Oportun
02.2022 - Current
TECHNICAL MANAGER
GAVS TECHNOLOGIES INDIA PVT LTD
04.2020 - 01.2022
PGWPM - IT, operations
GREAT LAKES INSTITUTE OF MANAGEMENT
09.2015 - 09.2017
LEAD ENGINEER
GAVS TECHNOLOGIES INDIA PVT LTD
07.2013 - 03.2020
SENIOR APPLICATIONS DEVELOPER
INAUTIX TECHNOLOGIES INDIA PVT LTD
07.2010 - 07.2013
APPLICATIONS DEVELOPER
INAUTIX TECHNOLOGIES INDIA PVT LTD
05.2007 - 07.2010
BACHELOR OF ENGINEERING - MEPCO, Electronics and communications
SCHLENK ENGG. COLLEGE
04.2003 - 04.2007