InputFormatInfo
Related Docs:
class InputFormatInfo
| package scheduler
Computes the preferred locations based on input(s) and returned a location to block map.
Computes the preferred locations based on input(s) and returned a location to block map. Typical use of this method for allocation would follow some algo like this:
a) For each host, count number of splits hosted on that host. b) Decrement the currently allocated containers on that host. c) Compute rack info for each host and update rack -> count map based on (b). d) Allocate nodes based on (c) e) On the allocation result, ensure that we don't allocate "too many" jobs on a single node (even if data locality on that is very high) : this is to prevent fragility of job if a single (or small set of) hosts go down.
go to (a) until required nodes are allocated.
If a node 'dies', follow same procedure.
PS: I know the wording here is weird, hopefully it makes some sense !