Home > other >  Just grab from a remote node over spark shuffleread data is placed outside the heap memory? If is ve
Just grab from a remote node over spark shuffleread data is placed outside the heap memory? If is ve

Time:09-25

As title, shuffleRead use netty fetching data from a remote node returns a managerBuf, then the buf is used outside the heap memory or a pile of internal memory? You need to be out of memory, the memory is not affected by spark memoryManager management? Code doesn't seem to be memoryManager management, the case, if a map output is large, after the netty caught on outside the heap memory, wouldn't it be easier to oom? I want to know, netty crawl over data will be written to a file? Of netty are not ripe, they forget the great god, for a long time for a long time,
  • Related