Rahul Raj
Kolkata
Summary
Adept Data Engineer with a proven track record at Tata Consultancy Services, where I spearheaded the implementation of innovative data solutions, including a 99% reduction in manual workload through automation. Skilled in Python, SQL, and AWS, I excel in problem-solving and ensuring data integrity, showcasing both technical prowess and strategic thinking.
Overview
4
4
years of professional experience
1
1
Certification
Work History
Data Engineer
Tata Consultancy Services
Kolkata
01.2021 - Current
- Implemented a robust data pipeline utilizing AWS Lambda, extracting data from AWS S3 and seamlessly loading it into Snowflake
- Engineered a AWS Lambda python code using boto3 and snowflake capable of identifying and managing missed files on specific days, ensuring data integrity by updating a dedicated log table in Snowflake
- This solution halted data staging processes until the missed files were successfully loaded
- Spearheaded the development of an end to end automation solution for daily monitoring process, leveraging AWS Lambda, Python, and boto3
- Orchestrated through Terraform, this system delivered scheduled alerts via email and Slack channels, resulting in a remarkable 99% reduction in manual workload
- Utilized industry-leading ETL tools such as Matillion to architect a cutting-edge solution for filtering and staging the latest records for specific identifiers within DELTA filetype data
- Leveraged GIT version control to streamline the deployment into prod
- Pioneered the creation of an automated script responsible for extracting metadata from Excel files and dynamically generating tables in Snowflake
- This innovation drastically reduced workload by 98%, enabling swift creation of thousands of tables within minutes when required
- Executed a seamless migration initiative between PostgreSQL and Snowflake, crafting a Python script to meticulously compare column counts and highlight any disparities
- Results were intelligently presented in detail within Excel, facilitating efficient resolution of mismatches
- Employed advanced SQL techniques to develop a highly scalable SQL script capable of generating metadata for thousands of tables concurrently, streamlining operational efficiencies
Education
B.Tech/B.E. - Computer Science Engineering
Heritage Institute of Technology
Kolkata, INDIA01.2021
12th - English
CBSE
01.2016
10th - English
CBSE
01.2014
Skills
- Languages: Python, SQL
- Data Analysis and Processing Libraries/Frameworks: Pandas, PySpark
- Data warehouses: Snowflake, Redshift
- ETL development
- Cloud: AWS services like S3, Lambda, SQS, SNS, DynamoDB, Athena, Glue, SDK, etc
- Tools: GIT (VCS), GitHub, Matillion, Jira
- Data Pipeline
- Gen-AI
- RDBMS
- System design
Languages
- English
- Hindi
Accomplishments
- Received On the Spot Award from TCS
- Received Best Team Award from TCS
- GATE : 3 x GATE CS Qualified, 1 x GATE DA Qualified
- TCS Codevita 2020 : Secured AIR-1676 in TCS-Codevita 2020 out of around 2 lakh students.
Certification
- AWS Certified AI Practitioner - AWS
- Snowflake - Build and Architect Data Pipelines using AWS - udemy business
- System Design - Coding ninjas
Languages
Hindi
First Language
English
Proficient (C2)
C2
Timeline
Data Engineer
Tata Consultancy Services
01.2021 - Current
B.Tech/B.E. - Computer Science Engineering
Heritage Institute of Technology
12th - English
CBSE
10th - English
CBSE
Similar Profiles
Vrunda ChindhadeVrunda Chindhade
HR Manager at Tata Consultancy Services, TCS, Tata Consultancy Services LimitedHR Manager at Tata Consultancy Services, TCS, Tata Consultancy Services Limited
Sreya BasuSreya Basu
Procare Analyst at Tata Steel(Tata Consultancy Services)Procare Analyst at Tata Steel(Tata Consultancy Services)
Arun VArun V
Program Manager at Tata consultancy services | Financial Ombudsman ServicesProgram Manager at Tata consultancy services | Financial Ombudsman Services
Naimish AgnihotriNaimish Agnihotri
Assistant Delivery Manager at Tata Consultancy Services, Global Shared ServicesAssistant Delivery Manager at Tata Consultancy Services, Global Shared Services
Ankan BeraAnkan Bera
Application Architect at WiproApplication Architect at Wipro