Karishma Kotian
Bangalore
Summary
Senior engineering professional with deep expertise in data architecture, pipeline development, and big data technologies. Proven track record in optimizing data workflows, enhancing system efficiency, and driving business intelligence initiatives. Strong collaborator, adaptable to evolving project demands, with focus on delivering impactful results through teamwork and innovation. Skilled in SQL, Python, Spark, and cloud platforms, with strategic approach to data management and problem-solving.
Overview
8
8
years of professional experience
1
1
Certification
Work History
Senior Data Engineer
Randstad Digital
02.2025 - Current
- Collaborating with cross-functional teams (data scientists, business analysts, product managers) to translate business requirements into scalable data solutions.
- Worked on data ingestion from legacy SQL Server to S3 storage using pyspark and writing Impala queries to transform the data as required.
- Involved in designing a scalable data model for a customer analytics platform to improve reporting efficiency and support real-time insights.
- Designing a data vault model in CDE using DBT and Apache Airflow.
Senior Data Engineer
Randstad Digital
10.2024 - 01.2025
- Involved in large-scale migration of booking records from legacy Hadoop system to AWS Cloud platform.
- Successfully migrated 30+ pyspark workflows to BDX with client satisfaction.
- Code development done using Visual Studio and implemented CI/CD pipelines using GitLab.
- Designed and implemented production-grade data pipelines using Apache Airflow DAGs.
- Also involved in data validation post migration and preparing evidence document for the same,achieved 98% data accuracy.
Data Engineer
IBM India Pvt Ltd
07.2023 - 10.2024
- Collaborating with cross-functional teams (data scientists, business analysts, product managers) to translate business requirements into scalable data solutions
- Worked on data ingestion from legacy SQL Server to S3 storage and writing Impala queries to transform the data as required
- Involved in designing a scalable data model for a customer analytics platform to improve reporting efficiency and support real-time insights
- Designing a data vault model in CDE using DBT and Apache Airflow
- Involved in large-scale migration of booking records from legacy Hadoop system to AWS Cloud platform
- Successfully migrated 20+ pyspark workflows to BDX with client satisfaction
- Code development done using Visual Studio and implemented CI/CD pipelines using GitLab
- Designed and implemented production-grade data pipelines using Apache Airflow DAGs
- Also involved in data validation post migration and preparing evidence document for the same, achieved 98% data accuracy
- Developed Spark code to flatten complex JSON structures within the parquet files stored in scality, improving data accessibility and analysis capabilities
- Designed and implemented spark transformations for generating dimensions and fact tables, enabling efficient querying and analysis
- Developed and optimized data processing workflows using Apache Spark on Kubernetes clusters, processing gigabytes of data daily, resulting in improved efficiency by 30%
Data Engineer
IBM India Pvt Ltd
10.2022 - 06.2023
- Maintained Data pipelines up time of 99.8% while ingesting transactional data across different primary sources using Azure ADF
- Implemented Azure Services to successfully develop, deploy and maintain data, resulting in a 30% increase in data processing efficiency
- Executed data modeling and ETL techniques to collect data from different source systems, resulting in the generation of actionable insights that contributed to a 15% improvement in business performance
- Successfully managed and processed complex user data sets, ensuring data integrity, confidentiality and compliance with privacy regulations
- Developed and optimized data transformation workflows using Azure Databricks and Apache Spark SQL, improving data quality and accuracy
- Design, develop, and execute test cases to validate ETL processes
- Verify data extraction from multiple data sources, data transformation logic, and accurate loading into target systems (data warehouses and data lakes)
- Ensure that the data in the target system matches the expected output in terms of quality, accuracy, and completeness
- Performed thorough data validation and quality checks at various stages of the ETL pipeline
- Developed SQL queries for data validation and testing purposes
- Identify and report data inconsistencies, errors or any data quality issues
- Worked closely with data engineers and developers to monitor ETL job performance and identify bottlenecks or failure points
- Maintained detailed documentation of test results, issues and resolutions
- Ensured data quality and reliability through rigorous testing and validation process, maintaining high standards of data integrity
- Monitoring Spark jobs via ESP tool and fixing the jobs within the timeframes
Data Engineer
Tata Consultancy Services
06.2021 - 09.2022
- Maintained Data pipelines up time of 99.8% while ingesting transactional data across different primary sources using Azure ADF
- Implemented Azure Services to successfully develop, deploy and maintain data, resulting in a 30% increase in data processing efficiency.
- Executed data modeling and ETL techniques to collect data from different source systems, resulting in the generation of actionable insights that contributed to a 15% improvement in business performance.
- Successfully managed and processed complex user data sets, ensuring data integrity, confidentiality and compliance with privacy regulations
- Developed and optimized data transformation workflows using Azure Databricks and Apache Spark SQL, improving data quality and accuracy.
Senior Test Engineer
LG SOFT India Pvt Ltd
11.2019 - 05.2021
- Design,develop, and execute test cases to validate ETL processes.
- Verify data extraction from multiple data sources,data transformation logic,and accurate loading into target systems(data warehouses and data lakes)
- Ensure that the data in the target system matches the expected output in terms of quality,accuracy,and completeness. Performed thorough data validation and quality checks at various stages of the ETL pipeline.
- Developed SQL queries for data validation and testing purposes. Identify and report data inconsistencies,errors or any data quality issues.
- worked closely with data engineers and developers to monitor ETL job performance and identify bottlenecks or failure points. Maintained detailed documentation of test results,issues and resolutions.
Senior Test Engineer
LG SOFT India Pvt Ltd
09.2017 - 10.2019
- Creating,executing, and managing test cases, and ensuring that the SQL databases are performing as expected.
- Design and execute test plans,test cases, and test scripts to validate database systems and applications.
- Developed SQL queries and scripts to validate data accuracy and system functionality.
- Identity,track,and document defects found during testing in collaboration with development teams.
- Work closely with the development and QA teams to detect and resolve issues before deployment.
Education
Bachelor of Science - Computer Science
Mangalore University
Mangalore, India05-2016
Skills
- Python programming
- SQL and databases
- Spark development
- Hadoop ecosystem
- ETL development
- Big data processing
- Azure ADF
- Azure databricks
- Amazon S3
- Snowflake, DBT
- Git version control
- Data pipeline design
- Data modeling
- Performance tuning
- Data warehousing
- Data quality assurance
- Data integration
- Data migration
- Data analysis
- Data governance
- Airflow
Certification
Microsoft Azure Fundamentals (AZ-900)
Languages
English
Hindi
Kannada
Timeline
Senior Data Engineer
Randstad Digital
02.2025 - Current
Senior Data Engineer
Randstad Digital
10.2024 - 01.2025
Data Engineer
IBM India Pvt Ltd
07.2023 - 10.2024
Data Engineer
IBM India Pvt Ltd
10.2022 - 06.2023
Data Engineer
Tata Consultancy Services
06.2021 - 09.2022
Senior Test Engineer
LG SOFT India Pvt Ltd
11.2019 - 05.2021
Senior Test Engineer
LG SOFT India Pvt Ltd
09.2017 - 10.2019
Bachelor of Science - Computer Science
Mangalore University
Similar Profiles
Rahul RamamoorthyRahul Ramamoorthy
Senior Operations Manager, Talent Solutions at Randstad DigitalSenior Operations Manager, Talent Solutions at Randstad Digital
Ross BarringtonRoss Barrington
Technical Helpdesk Specialist at Randstad DigitalTechnical Helpdesk Specialist at Randstad Digital
Amey VartakAmey Vartak
Technology Manager at Randstad DigitalTechnology Manager at Randstad Digital