Arindam Das

Creating Data pipelines to fetch data, parse and store into hive.

Perform transformation based on business requirements.

Tools Used: Apache spark, hive,MySQL, Python/Shell Scripting, Scala ,Autosys, Kafka etc.

• Creating data pipeline that collects data from different sources and runs analytical jobs, generating reports etc.
• Migrated the existing data pipeline to AWS and modernized it using Amazon EMR, Amazon S3 and underlying AWS Cloud infrastructure along with Apache Spark.
• Streamlined data processing time for majority workloads by 98 percent.
• Improved data availability by 30 percent and data infrastructure cost by 30 percent.
• Conceptualized & Created a complete SPARK, BigQuery & Python based framework to partially process data on Google BigQuery and partially in our Spark clusters to produce business useful insights.
• Successfully completed the POC of Dremio Query Engine for optimizing querying platform.

• Collaborated on ETL (Extract, Transform, Load) tasks, maintaining data integrity and verifying pipeline stability.
• Analyzed complex data and identified anomalies, trends and risks to provide useful insights to improve internal controls.
• Employed data cleansing methods, significantly Enhanced data quality.
• Designed and implemented effective database solutions and models to store and retrieve data.
• Designed compliance frameworks for multi-site data warehousing efforts to verify conformity with state and federal data security guidelines.
• Developed database architectural strategies at modeling, design and implementation stages to address business or industry requirements.