Tarun T

• Built data pipelines using Google Cloud Dataflow, Kubernetes, Apache Beam, Azure Databricks, and PySpark to ensure smooth and scalable data processing.
• Performed data cleaning and transformation in BigQuery and managed storage using GCS, BigQuery, and Spanner.
• Used Terraform to manage and maintain cloud infrastructure as code.
• Created JUnit test cases and implemented CI/CD pipelines with Jenkins and Spinnaker for automated deployments.
• Monitored and optimized data pipelines for performance, cost, and security, and maintained data dictionaries in Collibra.

• Analyze the existing Hadoop/Big Data project and map source datasets, tables, and columns to GCP/BigQuery.
• Convert the Hadoop/Big Data code to GCP/BigQuery and deploy it into the development framework.
• Update configuration files, perform dry-run testing, and validate output tables using sample data.
• Migrate the project to QA, update configs, and run both Hadoop and GCP versions with the same data to compare results and prepare test cases and runbook.

Here are the points rewritten in simple and clear form:

• Gather requirements from the Product Owner and request database access as needed.
• Prepare the Technical Design Document and get approval.
• Write HQL statements to extract and transform data from source to staging and target.
• Review code with the Tech Anchor and upload HQL files into the GDIA framework.
• Create Oozie bundles, run the jobs, and prepare unit test cases after batch completion.
• Review results with the Product Owner, get approval for production launch, and prepare the runbook.
• Migrate the application from QA to Production, perform basic checks, and monitor initial job runs.

• Designed the Hive and Impala architecture for Mentis masking applications.
• Planned product features, got approvals, and guided the team to develop as per the plan.
• Coordinated daily with managers, clients, and internal teams to resolve issues and gather information.
• Developed Discovery, Static Masking, and Dynamic Masking products, including Hive/Impala UDFs and Spark UDFs.
• Created masking views, handled runtime and performance issues, and processed Hive tables using Spark.
• Connected Hive with Spark SQL to perform transformations and analysis.
• Monitored and debugged jobs from MENTIS UI and handled deployments with required client privileges.
• Supported testing, Kerberos and Sentry setup, and delivered end-to-end masking products, including documentation and client sign-off.

• Analyzed functional requirements and created Hive tables for data processing.
• Used partitions, bucketing, complex data types, and wrote HQL queries for efficient performance.
• Imported and exported large datasets between Hadoop and relational databases using Sqoop.
• Developed Hive DDLs and converted SQL queries into Hive queries.