I have more than 2 years experience with strong emphasis on Design, Development,Implementation,Testing and Deployment of Applications in Data warehouse. Designed and developed data loading strategies,transformation for business to analyze the datasets. 2+ year of experience in big data technologies and tools such as Spark/PySpark, Hadoop,sqoop, Hive. And also having knowledge on snowflake. Experienced in working with DBT tool,spark ecosystems using SparkCore,Spark SQL and Pyspark on different formats from different sources. Ability to work independently and across multiple teams Developing thought leadership materials and being recognized in the Information Technology industry as a credible participant in the digital and cloud dialogue Excellent communication skills with the ability to influence client business and IT teams Experienced professional with a strong background in technology-related roles. Proficient in software development, system administration, and technical support. Skilled in problem-solving and optimizing performance. Capable of managing projects and collaborating effectively with teams. Committed to continuous learning and staying current with industry trends to contribute to organizational success.
· Developed queries using DTF(aframeworkforpyspark)to perform data transformation and implement SCD type 2 and store in Big Query.
· Developed queries to perform UnitTesting,FunctionalUnitTesting.
· Experience in using Dataproc Service of GCP.
· Used jenkins for CI/CD pipeline.
Having knowledge of Dag Creation and BigQuery table creation using terraform and having knowledge on spinnaker and scheduling jobs in TWS scheduling tool.
· Developed models in DBT to perform data transformations,data quality checks, data control and store data in big query.
· Developed queries to perform unit testing in DBT.
· Worked on code optimizations and creating and utilizing of macros in DBT.
· Used Jenkins for CI/CD,GitHub for Version Control.