Murali Krishna

EDP DATA LAKE DATA INTEGRATION HUB

Data ingestion and integration frameworks provide a standardized, easy and reliable way for various Barclays departments to load their data to data-warehouse. Frameworks are built using enterprise et1 tools. Frameworks reduce coding complexity from users, and tool specific knowledge, and at the same time let applications use enterprise level features such as security, data lineage, high performance, high reliability. Standardized, reusable solution for ingesting and integrating data from various sources at the bank, to central data warehouses. The frameworks are used by Barclays UK, Cards and payments, Corporate, Investment bank, Group risk and technology, legal, Fin-crime, HR, compliance tech, wealth.

• Collaborating with Stakeholders to gather requirements and understand data integration needs.

• Developing data integration using aws services and defining integration patterns.

• Performance monitoring and optimization of data integration workflows.

• Orchestrating the ETL jobs using AWS Step Functions.

• Develop and maintain ETL jobs using AWS Glue for data extraction, transformation, and loading.

• Developing scripts to define data transformation logic.

Working with other roles to establish data partitioning, organization, and backup strategies

• Company Overview: Working for CIGNA in MOS (Monitoring Operating System) team.
• Developed Spark scripts based on PySpark as per requirements.
• Implemented Spark by using data frame API's, SparkSql for faster data processing.
• Involved in extracting data from various data sources into AWS S3.
• Maintain the day-to-day data in Datawarehouse without lagging from different source systems.
• Developing the ETL AWS GLUE Jobs to transfer data from source to AWS Athena and then after performing transformations on Athena tables moving data from Athena to Postgres SQL.
• Load the daily injecting data into ATHENA tables and to create Partitioning to improve the query performances.
• To Check the Postgres SQL Database for monitoring daily ingestion jobs are successful if not debugging the issue.
• Orchestrating the ETL jobs using AWS Step Functions.
• Running Analytics Pipeline for Aggregating and joining the multiple daily ingestion tables to meet the stakeholder's requirements.
• Scheduled the jobs and transferred final output data to Business team for reporting purpose.
• Creating the cluster group in Databricks for running Daily ingestion, Analytics Pipeline jobs.

• Worked on creating Hive managed and external tables based on the requirement.
• Implemented Partitioning on Hive tables for better performance.
• Installing, Upgrading and Managing Spark Cluster in Qubole Data Lake Collaborated with the infrastructure, network, database.
• Creating the tables in Athena and integrating with looker dashboard. In Looker Stakeholders will create there-own dashboards with processed final output data for business requirements
• Deploying the oozie workflows by using the automated Jenkins Tool.