KAMALI KAMESH KRISHNA

As a Data Engineer at Comcast, I designed, built, and maintained end-to-end data systems and pipelines to capture, store, and transform data from diverse sources. This involved developing data structures, organizing data, and utilizing both distributed cloud and local databases to meet business and technical needs. My role also included providing primary on-call support for production jobs (one week/month), mentoring interns, maintaining the team's knowledge base WIKIs and supporting cross-functional teams dependent on our data pipelines.

Here are some of my key achievements:

• Migrated workloads to Databricks clusters, increasing subnet availability and improving performance by 20%.
• Migrated jobs from Gears to Airflow, designing and automating DAG placement to reduce workflow setup time.
• Built and dockerized a Python + S3FS CLI utility, improving file operations efficiency.
• Refactored PySpark ETL pipelines, reducing runtime by 35% and resource consumption by 25%.
• Upgraded Spark version in existing framework which reduced potential failures in 90+ PROD jobs.
• Developed a CI/CD pipeline with GitHub Actions to automate Docker image deployment.
• Designed a cost-effective ETL pipeline using Airflow, Kubernetes, PySpark, Docker, SQL, and AWS S3, lowering Databricks cluster costs.

During my time with Accenture as a Data Engineering Analyst, I worked for two projects. Here's a brief of my responsibilities within these projects:

Project for a Leading Canadian Bank

• Streamlined data collection and migration processes using Python and ETL tools like Datameer and Lotus Notes.
• Ensured daily delivery of critical data with a focus on accuracy and timeliness while automating manual tasks to enhance efficiency.

Worked on project for a Global Japanese Pharmaceutical Company

• Independently upgraded and deployed production Lambda functions, maintaining version consistency and ensuring seamless operations. Designed and implemented AWS pipelines for streaming data, enabling smooth transfers from source to production and improving data accessibility.
• Built PySpark pipelines to migrate production data into Databricks tables, managed real-time updates, and integrated AWS S3 for enhanced data workflows. Automated production table refreshes using AWS services (S3, Lambda, EventBridge, SNS), optimizing workflows and ensuring up-to-date data availability.
• Automated workflows with Python, AWS Lambda, and CloudWatch, reducing manual effort and improving monitoring. Built PySpark pipelines in Databricks to process real-time data from AWS S3, and automated table refreshes using AWS S3, Lambda, EventBridge, and SNS, keeping production pipelines reliable and up to date.