Javed Aalam

Software Engineer

Hexaware Technologies Ltd.

Chennai

06.2022 - Current

In the project, I worked on the role of Data Governance.
We used tools such as Azure Synapse, Microsoft Purview, and SSMS.
In this first, we moved data from an on-prem server of our client using an ETL Pipeline in Synapse to our ADLS Gen2 Storage into different folder structures such as Raw, Standardized, and Curated.
This is further classified as different layers. The data is first ingested into the Raw layer as a CSV file, and then further transformed into other layers such as the Standardized and Curated layers.
Then, using Purview, we connected to the Storage account, created a custom scan rule set to ingest the data into Purview for better data governance, and used the same ETL pipeline to push lineage across the layer.
Also, in the Purview, we create a Data Glossary template to upload Glossary terms and form mappings between the same Glossary terms and all the assets present in the collection, so that we can see a comprehensive image of how data is flowing.
Here, I also developed a script to perform the manual task of creating custom lineage from assets present in the collection to the DataMart objects.
Also developed script to remove manual task of mapping Glossary terms to assets.
I also got the knowledge working on Fabric.
Create lakehouses, warehouse, pipelines, and also the Power BI Reports.
Using the Pipeline in Fabric, data was moved from Azure Blob Storage with transformation into a lakehouse. And, using the semantic model, create the Power BI reports.
Also, in the role of Data Governance, I also got the opportunity to work on Profisee.
In Profisee, implemented a Master Data Management system to ensure consistency and accuracy of core business data.
Goals include eliminating duplicates, enhancing data quality, and facilitating cross-functional collaboration.
The project involves designing workflows, integrating with existing systems, and enforcing security measures.
Key phases include assessment, development, testing, deployment, and ongoing optimization.