During Elastic EP rank recovery, the recovered rank must re-run expensive initialization (especially CUDA graph capture).
Previously, this required the recovered and healthy ranks to enter the same communication phase, which paused healthy-rank inference and increased recovery disruption.
This change enables **asynchronous recovered-rank initialization** in Mooncake PG: recovered ranks can initialize in isolation first, then join the live process group after local recovery work is finished.
* feat(tone_tests): add E2E test cases and bilingual documentation
Signed-off-by: lukotong-7 <shicanwei.scw@alibaba-inc.com>
* update T-One job link to show real-time logs
Signed-off-by: lukotong-7 <shicanwei.scw@alibaba-inc.com>
* Add description for test_hicache_storage_mooncake_backend.py
Signed-off-by: lukotong-7 <shicanwei.scw@alibaba-inc.com>
* Add parse and cleanup command options to test_1p1d_erdma.sh
Signed-off-by: lukotong-7 <shicanwei.scw@alibaba-inc.com>
* Ensure that cleanup and parse are always executed, and replace exit with return.
Signed-off-by: lukotong-7 <shicanwei.scw@alibaba-inc.com>
---------
Signed-off-by: lukotong-7 <shicanwei.scw@alibaba-inc.com>
Xun Sun