Shivendra Thakur

Client - Tesco ( Retail)

• Designed and developed near-real-time data ingestion pipelines using Apache Kafka ,Java and Spark to consume, parse, and process streaming JSON data into structured Databricks Delta Lake tables.
• Implemented JSON parsing, data normalization, and transformation workflows, establishing relationships across Delta datasets to support downstream analytics.
• Collaborated with business stakeholders to understand reporting requirements and developed complex SQL transformations, publishing processed Reports to PostgreSQL for UI and backend applications.
• Designed and implemented automated Data Quality (DQ) frameworks using Great Expectations and Azure Data Factory to validate data accuracy, completeness, and consistency.
• Contributed to Spark-Scala framework development and code optimization, improving code reusability, maintainability, and data processing efficiency.

• Designed and developed end-to-end PySpark data pipelines to calculate field agent incentives based on dynamic business rules configured in the Rule Engine, enabling transparent incentive visibility and payout generation.
• Developed and deployed PySpark jobs to transform raw data into structured datasets, processing parameterized business incentive charters with complex rule-based calculations.
• Implemented transformation workflows for incentive-to-payout processes, generating JSON payout structures for multiple activity tags and lead-level incentives with daily/monthly agent payouts.
• Collaborated with business analysts and frontend teams for payout reconciliation and discrepancy resolution, maintaining rule engine input structures while optimizing pipeline performance.

Copied data ingested into Greenplum database from kafka topics to S3 by transforming CSV files, removing duplicates, and ingesting incremental data.

Client - PNC Bank

Ingested structured data from databases such as Oracle, SQL Server, and DB2 into HDFS to enable future analytics and dashboard generation using YARN and MapReduce techniques.

• Ingested CSV, mainframe, and text files from sources into HDFS while utilizing version control system Git.

• Created Python framework scripts based on project requirements to streamline data processing tasks.