Senior Data Engineer
DTDC Express
12.2024 - Current
Krishnanand Anil
Data engineer
Bengaluru,KA
Timeline
Senior Data Engineer
Quantiphi Analytics Solution Private
02.2024 - 11.2024
M-Tech - Data Science And Engineering
Birla Institute of Technology
2022.07 - 2024.07
Data Engineer
Cognizant Technologies
08.2020 - 02.2024
B-Tech - Mechanical Engineering
Amrita School of Engineering
2016.07 - 2020.08
Work Preference
Work Type
Full TimeLocation Preference
On-SiteRemoteHybridImportant To Me
Career advancementCompany CulturePersonal development programsHealthcare benefitsWork from home optionWork History
Senior Data Engineer
DTDC Express
12.2024 - Current
- Created an event driven pipeline which aggregates data to its final form necessary for reporting, made the reporting layer near realtime on huge amounts of data .
- Designed and created multiple ETL pipelines for data aggregation and load.
- Migrated data from multiple cloud warehouses to on premis system such as GCP and AWS.
- Worked on cutting edge AI workflows for automated report generation .
- Collaborated with cross-functional teams to gather requirements for data processing projects.
- Developed scripts for automating data extraction and transformation processes.
- Participated in troubleshooting data inconsistencies and resolving technical issues efficiently.
- Documented workflows and processes to streamline onboarding of new team members.
- Engaged in continuous learning of emerging technologies relevant to data engineering practices.
- Championed the adoption of agile methodologies within the team, resulting in faster delivery times and increased collaboration among team members.
- Enhanced system performance by designing and implementing scalable data solutions for high-traffic applications.
Senior Data Engineer
Quantiphi Analytics Solution Private
02.2024 - 11.2024
- Acted as a trusted advisor for clients by providing thought leadership on best practices in data engineering, ensuring their systems were optimized for performance and scalability.
- Compiled, cleaned and manipulated data for proper handling.
- Mentored junior team members in best practices for software development, code optimization, and troubleshooting techniques.
- Ensured data quality through rigorous testing, validation, and monitoring of all data assets, minimizing inaccuracies and inconsistencies.
- Spearheaded efforts to migrate legacy systems onto cloud-based platforms, resulting in improved scalability and costefficiency.
- Ran statistical analyses within software to process large datasets.
- Created a near real time streaming pipleine using aws kinesis, lambda and redshift
- Created auto refresh MV concepts which required heavy data manipulation which the native auto mvs couldn't handle.
- Pioneered pyspark data transformations for consolidating reports and auto mailing.
- Created piplines from apis which needed handling multiple api concepts and throughput hadling
Data Engineer
Cognizant Technologies
08.2020 - 02.2024
- Developed data pipelines to support ETL processes for diverse datasets.
- Collaborated with cross-functional teams to design and implement data models.
- Optimized database performance through indexing and query tuning strategies.
- Assisted in maintaining data integrity by conducting regular audits and validation checks.
- Migrated from legacy hadoop system to aws cloud redshift environment, almost 300 tables and 20-30 reporting layer
- Worked on integrating scoop export scripts and pyspark etl in hadoop environment
- Pioneered api integrations to autosys jobs for realtime monitoring
Overview
5
5
years of professional experience
6
6
years of post-secondary education
Education
M-Tech - Data Science And Engineering
Birla Institute of Technology
Pilani 2022.07 - 2024.07
B-Tech - Mechanical Engineering
Amrita School of Engineering
Kollam Kerala 2016.07 - 2020.08
Skills
- Git version control
- ETL development
- Big data processing
- Python programming
- NoSQL databases
- Kafka streaming
- Data pipeline design
- Data modeling
- Performance tuning
- Hadoop ecosystem
- Data warehousing
- Spark development
- Advanced SQL
- Machine learning
- Data security
- Data quality assurance
- Metadata management
- Real-time analytics
- Data integration
- SQL and databases
- Database design
- SQL programming
- RDBMS
- Data migration
- Infrastructure as Code
- CloudFormation templates
- Lambda functions
- Data lake management
- DynamoDB experience
- Real-time data streaming
- AWS glue ETL management
- AWS redshift expertise
- Amazon S3 proficiency
- Cost optimization
Summary
Senior engineering professional with deep expertise in data architecture, pipeline development, and big data technologies. Proven track record in optimizing data workflows, enhancing system efficiency, and driving business intelligence initiatives. Strong collaborator, adaptable to evolving project demands, with focus on delivering impactful results through teamwork and innovation. Skilled in SQL, Python, Spark, and cloud platforms, with strategic approach to data management and problem-solving.