Suryateja Amara

• Designed and implemented real-time data ingestion pipelines using Kafka streams to process real time contract data.
• Developed and maintained Databricks batch and streaming workflows using Apache Spark and Delta Lake.
• Migrated legacy SSIS jobs to Databricks Workflows, enabling scalable and reliable cloud-native processing.
• Developed AWS data processing pipelines utilizing AWS CloudWatch events, Lambda, Step Functions, and EMR for efficient data handling.
• Ingesting data from multiple sources (s3, RDS, APIs) and apply business logic in Spark and write back to s3/RDS.
• Building AWS API pipelines for delivering data through APIs using lambda, API gateway and cognito and RDS.
• Building CI/CD pipelines using Concourse and GIT integration and Terraform.
• Migrating all pipelines to databricks from EMR.
• Creating unity catalog tables and enabling downstream consumers to access the data through unity tables.
• Implementation of medalian architecture to all the datasets.
• Optimized Databricks job execution costs by analyzing runtimes, cluster utilization, and tuning Spark configurations.
• Implemented monitoring, alerting, and failure recovery mechanisms for production-grade data pipelines.
• Collaborated with analytics, product, and platform teams to onboard new data sources, enhancing downstream reporting and insights.
• Led cross-functional team members by providing design and technical specifications, improving delivery speed and quality.

Client Name : Leading Healthcare company in USA

Role : Data Engineer / Spark developer

• Developed ETL spark code in Scala to read the source data in hive Datawarehouse and apply business transformations and loading the data to Exasol database.
• Implemented data quality checks to enhance data integrity across reporting frameworks.
• Identified bottlenecks and optimized Spark jobs, resulting in reduced cycle time.
• Implemented static code set approach which helped in code reusability and effective maintenance.
• This project is to develop a data mart for reporting application. All the required data for analytical reporting will be extracted from a Datawarehouse and transformed using spark and load the data to reporting database for dashboard reporting.
• Technologies : Spark, Hive, S3, Exasol, Scala, Shell Scripting.

Client Name : Nielsen

Role : ETL/ Spark developer

Technologies : Spark, Hive, S3, Redshift, EMR, Java, Shell Scripting.

• Ingested flat files from data lake and loaded them into hive tables.
• Finding the incremental records from the Raw table (used creating the flat files available in data lake) and maintain the latest and greatest records in another layer called merge.
• Applying the spark transformations on the input data available merge hive tables and load the transformed data into target hive tables.
• Developed ETL logic to extract data from Netezza warehouse, transformed it using business logic, and transferred final CSV files to reporting server.
• Scheduling the jobs using JLE (Java load engine) a custom ETL tool.
• Automated the ETL DQR checks to maintain data integrity.
• Worked huge volume of data comparisons using SQL.
• Automated the comparison scripts using Fitnesse tool and integrated it with jira.
• As part of cloud migration, worked as a spark developer to convert the existing relational database sql to spark sql.
• Converted existing ETL flow to Spark, extracting data from relational databases and transforming it using Spark SQL for storage as parquet files in S3 and HDFS.
• Hands on experience to run spark application on AWS EMR cluster.
• Led project focused on enterprise data warehousing.