Charan V

Project Title: Insurance Data Processing System and Market Analytics & Prediction

Client: Wipro
Technologies: AWS (S3, Glue, EMR, Lambda, Kinesis, Redshift, Athena, Databricks) Power BI, and Azure Data Factory, Azure Databricks, ADLS Gen2, Synapse

RESPONSIBILITIES:

• Designed end-to-end ETL pipelines using Azure Data Factory and Databricks.
• Built real-time and batch processing pipelines for high-volume data.
• Integrated multiple data sources into Azure Data Lake and Azure SQL DB.
• Optimized pipelines using parallel processing and caching techniques.
• Implemented data quality frameworks and validation checks
  Developed CI/CD pipelines for automated deployment.
• Monitored pipelines using Azure Monitor and Log Analytics.
• Developed Spark/PySpark-based ETL pipelines for large-scale data migration.
• Processed structured and semi-structured data into enterprise data lake architecture.
• Optimized Spark jobs using partitioning, caching, and query tuning techniques.
• Built data models and ETL workflows based on business requirements.
• Integrated data pipelines with Power BI dashboards for reporting.
• Developed Python-based data processing scripts and automation frameworks.
• Performed data cleansing, transformation, and validation.
• Designed and built real-time data pipelines using AWS Glue, Lambda, and Kinesis for streaming data ingestion.
• Developed PySpark ETL jobs on EMR and Databricks for large-scale data processing.
• Built S3-based data lake with partitioned Parquet datasets, improving query performance by 60%.
• Designed and optimized Redshift schemas, improving query performance by 50%.
• Implemented event-driven architecture using Lambda for real-time processing.
• Created Athena queries for ad-hoc analysis and reportin.
• Built streaming solutions using Kinesis + Lambda + Redshift for near real-time analytics.

Project Title: Data Integration and Migration Platform

Client: Unisys
Technologies: Azure Data Factory (ADF), Azure Databricks, Azure Key Vault, Azure Logic Apps, Azure Data Lake Storage, Azure SQL Database

RESPONSIBILITIES:

• Migrated on-prem data to Azure Data Lake and Azure SQL Database.
• Integrated Databricks notebooks with ADF pipelines.
• Implemented secure data handling using Azure Key Vault.
• Automated workflows using Logic Apps and event-driven architecture.
• Built Power BI dashboards for business insights.
• Designed and developed Azure Data Factory (ADF) pipelines, Linked Services, and Datasets for seamless data migration and transformation.
• Tailored ADF pipelines to meet specific project requirements, ensuring efficient data
  movement and transformation.
• Implemented Azure Logic Apps for real-time email notifications to monitor data migration progress.
• Collaborated with senior team members to understand complex data migration requirements and develop effective solutions.

Project Title: Inventory Management - E commerce

Client: US
Role: Internship
Technologies: S3, Glue, Redshift, EMR, Lambda, Kinesis, Athena, RDS, DynamoDB, Step Functions and Azure Data Factory, Azure Databricks, ADLS Gen2, Azure Synapse, Azure SQL DB, Azure Key Vault, Logic Apps,Data Lake.

RESPONSIBILITIES:

• Implemented data quality checks, improving accuracy of reporting outputs for business intelligence.
• Utilized SQL for querying databases, enhancing data retrieval efficiency across projects.
• Assisted in the design and implementation of data models to support analytical needs.
• Monitored system performance, troubleshooting issues to minimize downtime and enhance reliability.
• Streamlined complex workflows by breaking them down into manageable components for easier implementation and maintenance.
• Optimized data processing by implementing efficient ETL pipelines and streamlining database design.
• Migrated legacy systems to modern big-data technologies, improving performance and scalability while minimizing business disruption.
• Increased efficiency of data-driven decision making by creating user-friendly dashboards that enable quick access to key metrics.
• Documented technical processes and workflows, facilitating knowledge sharing among team members.
• Collaborated on ETL (Extract, Transform, Load) tasks, maintaining data integrity and verifying pipeline stability.
• .Fine-tuned query performance and optimized database structures for faster, more accurate data retrieval and reporting.
• Enhanced data quality by performing thorough cleaning, validation, and transformation tasks.
• Designed and developed scalable data pipelines for ingesting, processing, and transforming large datasets using Python and SQL.
• Utilized Microsoft Azure services including Azure Data Factory, Azure Data Lake, Synapse Analytics, and Databricks for data integration and transformation.
• Developed and optimized data models and schemas to support business intelligence and reporting needs.
• Ensured data quality, validation, and governance using monitoring and logging frameworks.
• Tuned and optimized query performance and data processing jobs for cost and efficiency.