Engineering On The Edge

I have a full plate of meetings at Supercomputing 2011 this year. Below are some of the points of interest I am learning about along the way that I wanted to share.

Platform Computing
Yesterday I attended the 7:30 breakfast meeting for Platform Computing’s MapReduce. This is a policy-driven workload manager and scheduler that handles mixed types of workloads running on the same cluster. It uses an open architecture to support multiple applications for jobs built with Hadoop MapReduce technology. Applications include Pig, Hive, Java, Oozie, Cumbo, and natively written Java MapReduce programs.

The scheduler can give the high-performance computing (HPC) manager options to schedule job submissions with a Fairshare Scheduler, Preemptive Scheduler, Threshold-based Scheduler or Task Scheduler. It also helps to work with resource draining.

MapReduce sends the application to a Job Controller, which decides what data should be mapped to an input folder and what mapped tasks should go to the local storage. It can split data so only that data that needs to be in the compute schedule goes to the CPU. It also uses Resource Groups to move data between local workstations.

Platform Computing’s MapReduce can allow clusters to be grouped into one large resource. Platform is used as a massive scheduler. The user has control of the job not just when it is in the queue, but while the job is running. Continue reading →