Palaka HariPrasad

LLOYD'S BANKING GROUP IN GCP PLATFORM

• I have experience in both on-premise and Google Cloud Platform.
• For processing the data, we are using Python, PySpark DataFrame.
• As a part of Google Cloud, we are using Google Cloud Storage, BigQuery, Dataproc, Cloud Composer, Airflow DAGs. These are the services that we mainly used in our project.
• We are using GCS for storing all our code, files, and props in GCS buckets.
• On top of that bucket, we create an Airflow DAG to perform Hive job and Spark job.
• With the help of DATAPROC, perform all the 'Data processing work'.
• Once done with the transformations, a file is created in the target GCS bucket.
• Then we load all the processed data files into 'BIG QUERY'.
• Then we do all the analysis in 'BIG QUERY Tables'.
• We are using CLOUD COMPOSER for monitoring and checking the logs.
• To schedule the workflows, we are using 'AIRFLOW DAGS'.

LLOYD'S BANKING GROUP

• The project is based on a Big Data platform.
• My responsibilities is to develop as per the requirements, as well as test the code to be deployed, and curate the correct data as expected in the Big Data platform.
• As a data engineer, my responsibilities is to perform the ETL process.
• We have a huge amount of data. We extract the data from schema layer tables into staging layer database tables.
• After completion of the ETL process, create a target table in the curation layer with the correct DDL in the target database.
• Then set the application properties and jar in our path.
• After running the spark submit command. We will get a successful result.
• Finally, checking the target DB tables to verify if the records are coming as per the mapping requirement or not.
• After validating the data in the curation layer, we can transform the curated data into the consumption layer.
• Then, generate the IFS file in the proper HDFS path for the analysis requirements.