Karthik K

Tech stack: Azure Databricks, Azure blob storage, Pyspark, Databricks, Azure Data factory, Power BI, Team Management, ETL, Data pipeline, snowflake, Agile Methodology

Roles and Responsibilities:

• Creating Pyspark scripts to load the data and perform transformation as per the business unit document.
• Optimising the existing data pipelines with the front-end and back-end techniques.
• Understand the pre-scripted Scala notebook and optimising the scripts.
• Developing the data pipeline using Azure Data Factory. Which will load the data from Azure SQL database.
• Refine the data to the most granular level which will be used to generate reports.
• Understand the limitations of PowerBI and help the PowerBI architect's with Databricks cluster configuration.
• Aggregating the data based on the business requirements using SQL and Pyspark.

Tech stack: Apache spark, Spark Databricks, Pyspark, SQL, AWS, Snowflake, Snowflake SQL, Data pipeline, SIP, Git hub, Agile methodology, Team Management

• Ensured data quality through rigorous testing, validation, and monitoring of all data assets, minimizing inaccuracies and inconsistencies.
• Developed the AWS step function to trigger the Databricks job and monitor the run logs
• Developed the AWS Lambda Function based on the business requirement and attached the Amazon resource machine to step function
• Experience in working with the complex SQL queries
• Experience in fetching the data from multiple data tables and to build the logic to match the data that is requested by the business
• Experience in creating and selecting the best suitable job cluster required for the data flow
• Having good exposure into Snowflake
• Experience in handling team, and also involved in the capacity planning for the team
• Involved in developing the data pipeline using the databricks workflows
• Proficiency in PySpark and extensive hands-on experience in building and optimizing data pipelines using DataBricks

Tech-stack:

Talend ETL, Pyspark, Databricks, AWS S3, ECMS, Oracle database, Python, SQL.

• Connecting with Business managers for the different sectors of freight (price) changes.
• Based on the contract document, created Pyspark scripts for few sectors like FMCG, Beverage, etc.
• Using SQL, created the Adhoc orders.
• Creating the separate schema for different sectors and tables for different business unit.
• Working closely with the application back-end to configure the framework for the new business units.
• Using Talend ETL created automated scripts for the periodic freight sheets.
• Documenting the work and posting to the superiors.

Tech stack: SQL, Microsoft tools

• Developed and optimized complex SQL queries to retrieve, manipulate, and analyse large datasets.
• Designed and maintained database schemas, ensuring efficient data storage and retrieval.
• Implemented stored procedures and triggers to automate routine tasks, improving system performance.
• Collaborated with cross-functional teams to understand and address database-related requirements.
• Conducted performance tuning and optimization to enhance overall database efficiency.
• Created and maintained documentation for database structures, procedures, and best practices.