Big Data Architect

Work Location: Bangalore & Pune

Experience: 6-10 Years

  • Data Profiling/Analysis, design of ETL jobs
  • Development & enhancements of several applications that run on Hadoop – MDE, IMV, LA , BAT.
  • Develop Pig, mapreduce, spark jobs for ETL
  • Performance tune the SOLR load process
  • Develop Java based Utilities for cleansing of source data, conversion of data formats to support AVRO, parquet storage.
  • Develop real time integration (Java) of real time SYW.com interactions and build process to process the json objects and create batch update process to store the data on HDFS for profile and interactions application for SYW.com profile development
  • Performance tune the SOLR load process
  • Develop Morphine code to load SOLR indexes and create collections
  • Develop right strategy for leveraging hive for creating partitions for high volume data
  • Develop Morphine code to load SOLR indexes and create collections
  • Develop python code to load data to Mongo collections for IMV
  • Develop process for data prep for GCP data load Schema

Develop code to pull data from API and capability of publishing the data thru API for downstream consumption