Sumeet Agrawal

Max Devices Supply Chain Data Platform

• Working on large Data platform of Microsoft multi billion Devices supply chain analytics
• Manage data storage of over 1.3 PB, 1000+ data pipelines and spark transformations, triggers and over 50+ source systems
• Works as subject matter expert to manage strategic data platform in various areas like telemetry and instrumentation, source code management , data growth planning, spark optimization techniques, scheduling and SLA tracking
• Developed several frameworks including Data Quality , Data Test framework, Metadata driven ETL framework to used by across the platform
• Developed generic libraries to read/write/log etc in Scala functional language to bring code and data governance across Data Platform
• Explore next gen tech stack and opportunities to adopt in Data platform
• Designed and implemented incremental process of data transformation to reduce latency for Return Analytics

Liquidity Risk Analytics

• Worked for Corporate Technology division in liquidity Risk Analytics
• Extensively worked on pyspark (Python+Spark) for data transformations, Unit Testing and production scalable code
• Define real-time and batch ingestion architecture using lambda approach including kafka, spark streaming, HBase real time as well as sqoop and hive for batch layer
• Spark up-gradation of production code from 1.6 to major version 2.1
• Worked with Data Scientists on evaluating and integrating data from complex and high velocity transaction systems

Johnson & Johnson

• Replace existing SAS ETL job with big data spark solution
• Understanding the business requirements and make plans and design in Apache spark
• Data migration from Amazon S3 to Redshift database using Spark framework
• Wrote robust scala codes, performance oriented for ETL workflow from 20+ sources
• Performed complex operation using scala on data frames to get the master tables used for tableau reporting

Cox Comm

• Lead Management System Worked on real time data ingestion from Amazon S3 to Hadoop
• Parsing and designing of unstructured/JSON data into hive
• Deriving complex variables records from the unstructured data to be consume by analytical tools
• Automation of the whole process using shell scripting

• Worked as a java programmer for Big data Map reduce batch jobs
• Work as a developer to operationalize data management and governance tools on open source frameworks
• Cloudera cluster building , deployment and up gradation of services using cloudera manager
• Actively particpated in requirement gathering, POC's on big data migration from SQL Databases like Teradata, Oracle in Hadoop distributed file systems
• Written Hive/Pig/UDF scripts to generate reports and scheduled jobs using cloudera tool oozie
• Providing Technical Architecture (Selections of tools and technologies) Measure performance bench marking of data formats, compression techniques, Optimization techniques in Hive