Anil Reddaboina
Lead Data Engineer
Hyderabad
Summary
Forward-thinking Lead data engineer with over 13 years of experience, excelling in dynamic environments. Proven track record in designing and developing large-scale systems. A proud team player committed to achieving project objectives with speed and accuracy.
Overview
12
12
years of professional experience
1
1
Certification
Work History
Lead Data Engineer
ThoughtWorks
6 2022 - Current
- Design and Develop ETL Pipelines: Create and maintain ETL pipelines using Apache Airflow as an orchestration tool to ensure efficient data processing.
- Optimize Spark Jobs: Enhance the performance of Spark jobs, reducing runtime from 8 hours to 3 hours by implementing dynamic Spark configurations.
- File Format Optimization: Transition underlying file formats from CSV/Parquet to Delta Lake, improving data reliability and query performance.
- Workshops and Technical Feasibility: Lead workshops to solve business problems, run technical spikes to assess feasibility, and convert findings into actionable stories and sprint plans.
- Build Idempotent Pipelines: Develop idempotent pipelines to maintain data quality and integrity, ensuring consistent results even when jobs fail.
- Business Interaction and Requirement Gathering: Engage with business stakeholders to understand customer requirements and design corresponding data models and pipelines.
- Agile Practices and Team Leadership: Foster agile practices within the team, providing guidance and mentorship to ensure high productivity and adherence to best practices.
- Constructive Feedback and Growth: Offer constructive feedback to team members, supporting their professional growth and development.
- Deployment and Release Planning: Collaborate with the release management team to plan and execute deployments and releases efficiently.
Staff Software Engineer
General Electric
03.2019 - 06.2021
- Build and support scalable and durable data solutions that can enable self-service advanced analytics at GE Digital using both traditional (SQL server) and modern DW technologies (Hadoop, Spark, Cloud, NoSQL etc.) in agile manner
- Proven technical lead using technologies such as AWS Redshift, Postgres, Hadoop, Cassandra, Java 8, SpringBoot, Spark, AWS S3, AWS Lambda, AWS Glue, Airflow, Python, CloudFoundary, GE Predix platform and Event Hub messaging system
- Owned Cassandra to AWS Redshift database complete transformation.
- Experience with of all aspects of data systems(both Big data and traditional) including database design, ETL, aggregation strategy, performance optimization
- Designed and developed in house component to support per Tenant Database, that help each microservice works for multiple Tenants
Analyst -I Apps Programmer
BA Continuum
06.2017 - 03.2019
- Mentored junior developers, fostering professional growth and enhancing team productivity.
- Used Spark, Scala, SQL, and Java skills with object-oriented analysis and extensively involved end-to-end delivery.
- Collaborated with clients to develop customized architectural solutions, meeting their unique needs and preferences.
Associate Software Engineer
Cognizant Technologies
09.2014 - 05.2017
Software Engineer
HSBC Technology & Services
07.2012 - 09.2014
Education
Master of Computer Applications -
Kakatiya University
Telangana 06.2007 - 03.2010
Skills
Apache Spark, scala, delta-lake, and YARN
Apache Airflow, Livy, Databricks datalake,Python
Storage systems: SQL, AWS Redshift, Cassandra, Postgres, and Oracle
Cloud: AWS, Azure, S3, EC2, RDS
Other programming: Java, Spring boot , REST API, Golang, and Microservices
CI & CD: Docker, Jenkins, Gitlab, Gocd, Github, Maven, Gradle, Agile-Scrum, JIRA
Certification
AWS Certified Cloud Practitioner
Timeline
AWS Certified Cloud Practitioner
01-2022
Staff Software Engineer
General Electric
03.2019 - 06.2021
Analyst -I Apps Programmer
BA Continuum
06.2017 - 03.2019
Associate Software Engineer
Cognizant Technologies
09.2014 - 05.2017
Software Engineer
HSBC Technology & Services
07.2012 - 09.2014
Lead Data Engineer
ThoughtWorks
6 2022 - Current
Master of Computer Applications -
Kakatiya University
06.2007 - 03.2010