Atul Singhal
Senior Software Engineer
Noida
Summary
Highly skilled Data Engineer with 5+ years of experience in Finance domain. Having extensive experience in TSQL, Databricks, SSIS, PySpark, API integration, and Python. Proven ability to develop and optimize ETL processes, design and maintain data pipelines, and consume RESTful APIs (Endpoints). Adept at ensuring data integrity, governance, and quality control. Strong problem-solving, analytical, and communication skills with a solid understanding of data warehousing concepts and best practices.
Overview
6
6
years of professional experience
7
7
years of post-secondary education
7
7
Certifications
Work History
Senior Software Engineer
incedo
11.2018 - Current
- Developed and optimized T-SQL queries, stored procedures, and functions to support data ETL processes, improving query performance by 10 times better using indexing
- Designed, implemented, and maintained data pipelines using Databricks, PySpark ensuring efficient processing and analysis of large volumes of data, resulting in a 40% reduction in processing time
- Documented data models, processes, and procedures, ensuring knowledge transfer and compliance with best practices, reducing onboarding time for new team members by 30%
- Stayed up to date with emerging technologies and industry trends, leading to a 25% improvement in team efficiency
- Developed KPI models for business reporting using DBT/Databricks, enhancing reporting accuracy and speed by 50%
- Wrote Scala notebooks for Cobol files transformation and loading them to SQL-DB and AWS S3, processing over 100 data files daily
- Led a 3-member team in a Loan onboarding project using CloverDX, T-SQL, successfully onboarding 1000+ loans per month
- Designed ETL packages in SSIS for data loading and transformation, ensuring seamless data flow across systems
- Developed PowerShell scripts for GoAnywhere API integration
- Managed Active Batch jobs and deployments, ensuring timely and accurate data processing, with a 98% on-time job completion rate
- Built ETL pipelines with Azure Data Factory, Data Flows, enhancing the efficiency and scalability of data operations
- Optimized and tuned ETL processes and SQL queries for performance, reducing processing time by 35%
- Provided mentorship and training to team members, fostering a collaborative and productive work environment, improving team performance by 15%
- Collaborated with cross-functional teams to understand data requirements and deliver efficient solutions, resulting in a 20% increase in project delivery speed
- Conducted regular reviews and updates of data models and processes to ensure alignment with best practices and business needs, leading to a 30% improvement in data quality.
Education
MCA -
01.2008 - 04.2011
BCA - undefined
01.2005 - 04.2008
Skills
T-SQL (SQL Server)
Databricks
Spark structured API
PySpark
DBT
Python
ETL/ELT Process Development
Data Pipeline Development
Data Modeling and Database Design
SQL Server Integration Services (SSIS)
API Integration
Data Warehouse
CloverDX
Azure Data Factory
Azure Data Lake
Azure synapse Analytics
Azure Databricks
Azure CI/CD
Azure Data Flows
AWS S3
AWS Lambda
Active Batch
Azure Logic App (to set-up email only through ADF)
Certification
CloverDX
Projects
BSI - KPI using Databricks/DBT, 02/24, 06/24, Created KPI models for business reporting using DBT/Databricks. BSI - Databricks/scala, 11/22, 09/23, Developed Scala notebook to transform Cobol files and load to SQL-DB and S3. BSI - Loan Onboarding project - CloverDX, 03/23, 09/23, Lead a team of 3 engineers in designing pipeline, guided team in SSIS to CloverDX conversion, and developed TSQL scripts. BSI - Copy Book, 02/22, Present, Supported and modified existing SSIS packages for Cobol file transformation. AssetMark - OBS Data Migration Bank, 07/21, 12/21, Enhanced functionality for multiple banks and provided production support. AssetMark - OBS Retirement, 04/21, 06/21, Enhanced sales data process for additional providers. AssetMark - OBS Data Migration Bank Trust, 10/20, 03/21, Loaded, merged, and transformed flat files, utilizing GoAnywhere project via REST API. AssetMark - OBS Off platform, 04/20, 09/20, Loaded, validated, and transformed CSV files, resolving errors and sending notifications.
Timeline
Senior Software Engineer - incedo
11.2018 - Current
IMS Dehradun - MCA,
01.2008 - 04.2011
Kurukshetra University - BCA,
01.2005 - 04.2008