Accomplished Data Engineer with a proven track record at Koch Industries, adept in data migration, governance, and pipeline design. Leveraged Python and Snowflake to enhance operational efficiency and data accuracy. Recognized for innovative solutions in metadata management and ETL development, demonstrating exceptional problem-solving skills and a commitment to excellence.
✔ Automated Data Processing: Developed a workflow to extract data from Snowflake tables, convert it into XML format, and seamlessly transfer it to AWS S3 via SFTP, reducing manual effort, and increasing accuracy.
✔ Metadata Management Application: Designed and developed a Streamlit-based application for handling metadata, enabling smooth data governance, and operational efficiency.
✔ Version Control & Deployment: Hands-on experience with GitLab CI/CD pipelines for streamlined development and deployment.
✔ Kubernetes and Cloud Technologies: Familiar with containerization and orchestration to manage cloud-based applications efficiently.
✔ Migration: Led the migration of legacy Talend-based ETL workflows to Snowflake, ensuring seamless data movement from MSSQL, Restful APIs, and Salesforce. Migrated data pipelines from Talend to Snowflake, integrating data from MSSQL, APIs, and Salesforce.
Technology Used :
Technology Used:
Technology Used: