Hadoop log aggregation error
What's inside this article
⌄
- Hadoop log aggregation not initialized
- Yarn log aggregation error
- Hadoop logging initialization error
- Fix Yarn log aggregation
You may encounter with Hadoop Yarn exception in NodeManager logs that states:
Log aggregation is not initialized for application_1657793849922_0033, did it fail to start?
It may happen because NM reboot. The newly launched NM inherited the running application, and it does not know how to collect logs from it.
According to the Hadoop Yarn NodeManager source code, instances of log collector classes for each running application are stored in ConcurrentHashMap.
When the application starts, NM puts a log collector class object to HashMap. It is made once, and during the NM reboot, this cache is simply erased.
As a result, such an error happens (according to the code it is issued if the required log collector instance does not exist in the map).