High-impact and low-probability events have occurred more frequently than before, which can seriously damage energy supply infrastructures. As localized small energy systems, multi-energy microgrids (MEMGs) can provide a viable solution for the system- wise load restoration of integrated energy systems (IESs), due to their enhanced flexibility and controllability. However, existing literature tends to realize MEMGs as corrective response rather than load restoration resource after extreme events, which cannot fully exploit the benefits of multi-MEMGs on IES resilience. This paper introduces a decentralized operating paradigm for the real-time coordination of local multi-MEMGs towards system- wise IES load restoration, while a novel topology-aware multi-task reinforcement learning method with soft modularization is proposed to solve it. The multi-task learning framework enables MEMGs to simultaneously learn scheduling decisions across different network topologies and better adapt to unanticipated contingencies. Additionally, to avoid insecure MEMG operations, a physics-informed safety layer is embedded on top of the multi-task learning framework for action corrections. Case studies have been conducted on two IESs (33- bus power, 20-node gas, and 20-node heat network as well as 69- bus power, 40-node gas, and 62-node heat network) to evaluate the effectiveness of the proposed method in enabling effective coordination among multi-MEMGs towards system- wise IES load restoration.