Commit Graph

53 Commits

Author SHA1 Message Date
TinaMengtingZhang 67c98923d5 Add st for dump 2022-04-07 16:00:02 -04:00
TinaMengtingZhang 438e261bbc wait until dumping finished 2022-03-18 11:44:09 -04:00
i-robot 1863766fad
!31063 Fix a+m dump probilistic failure in ci
Merge pull request !31063 from TinaMengtingZhang/fix_st_error2
2022-03-16 07:19:48 +00:00
huanghui 4482acf586 Support API set_dump for more type of ops 2022-03-15 10:24:15 +08:00
TinaMengtingZhang 646909d3f4 fix a+m dump probilistic failure in ci: 1. do not parallel to dump statistics and remove file lock. 2. when tensor size is small, do it in single thread 2022-03-14 17:11:55 -04:00
i-robot dcb5cd670c
!30953 Dynamic Weight Decay
Merge pull request !30953 from wanyiming/dynamic_wd
2022-03-14 06:03:24 +00:00
wanyiming a124ec4de7 add dynamic_decay 2022-03-10 11:02:27 +08:00
TinaMengtingZhang 78449afc49 modify st level for stat dump 2022-03-07 21:56:29 -05:00
TinaMengtingZhang 4a8d3defe7 Enhance the perfomance of A+M dump: Parallelize ConverFormatForTensorAndDump and refactoring 2022-03-01 23:08:38 -05:00
TinaMengtingZhang e576292862 dump statistic file in kernel by kernel mode 2022-02-15 16:58:00 -05:00
Parastoo Ashtari f6bebc7d97 use origin_parameter_order to load and dump params mindRT
Refactor mindRT code
Fix DumpConstantData issue
2022-02-10 14:18:08 -05:00
i-robot 8d648e0f1e !19673 Seperate constant from pb file
Merge pull request !19673 from sabrinasun_59ee/constant
2021-12-15 23:52:33 +00:00
sabrinasun 542d715217 seperate constant from pb 2021-12-14 13:19:14 -05:00
Jimmy Qi 6f2b70c21f Print null for min, max, avg values for NaN tensor 2021-12-12 19:31:21 +00:00
Jimmy Qi b9d1a4920c Add statistic dump for ascend 2021-12-06 20:11:44 +00:00
TinaMengtingZhang 16a19be56f Enable cann api allback register function
Add testcases for cann api feature
2021-12-03 17:36:15 -05:00
sabrinasun c095908772 fix enable=false still dump issue for cell dump 2021-11-30 22:10:20 -05:00
TinaMengtingZhang 07b653103e Support Cann callback api for ascend async dump 2021-11-24 10:35:29 -05:00
sabrinasun 99a434672f support cell dump when dump_mode=2 2021-11-16 16:01:13 -05:00
Jimmy Qi b21c099767 Add option to dump tensor statistics in csv format 2021-11-15 19:00:59 +00:00
i-robot 257bb3500d !25785 change dump path to avoid invalid symbol in ci path
Merge pull request !25785 from yelihua/dev
2021-11-04 13:08:01 +00:00
parastooashtari 1a59dc37bf add graph execution order history to dump 2021-11-02 11:00:38 -04:00
yelihua f9f2058055 change dump path to avoid invalid symbol in path 2021-11-02 17:29:35 +08:00
sabrinasun 332e0dbb0f add overflow st test and checkdumpstructure test 2021-10-22 06:28:58 +08:00
sabrinasun 220245f592 add security isolation to online and offline debugger 2021-09-12 23:01:15 -04:00
yelihua 7c3994e48e use Common::CreatePrefixPath instead of Common::GetRealPath 2021-09-09 20:34:25 +08:00
TinaMengtingZhang 424e94a7d4 Adapt async dump conversion tool with new run package (Sep03 version) 2021-09-07 13:41:26 -04:00
yanghaoran 1fd655787a upgrade Ascend packages 04 Sep 21 2021-09-06 21:24:07 +08:00
TinaMengtingZhang b17b2bc687 Fix convert async dump files failed issue and refactor convert_async.py 2021-09-01 15:10:55 -04:00
Xiaoda Zhang b2703879c6 fix the scope setting error when cloning nodes 2021-08-26 10:25:38 +08:00
ms_yan 36a8886ca2 Revert "[feat] [assistant] [I3T96T] add new Dataset operator CMUARCTICDataset"
This reverts commit b077aa1cab.

Revert "[feat] [assistant] [I3T96X] add new Dataset operator LibriSpeechDataset"

This reverts commit 4e6f7dc97d.

delete pass_registry_test.cc

comment  hiai_nlu_model_multi.pb related  line
2021-08-23 01:46:38 +08:00
djc b077aa1cab [feat] [assistant] [I3T96T] add new Dataset operator CMUARCTICDataset 2021-08-22 16:26:45 +08:00
djc 4e6f7dc97d [feat] [assistant] [I3T96X] add new Dataset operator LibriSpeechDataset 2021-08-22 13:39:37 +08:00
yelihua a3cba3857e use temporary dir as dump dir 2021-08-17 20:11:33 +08:00
yelihua a6dc9a0a07 get rank id when set hccl env for single card train 2021-08-16 13:44:02 +08:00
yelihua ca835a7d2d change the async dump testcase level to level1 2021-08-05 19:52:35 +08:00
TinaMengtingZhang 5b8e846fe9 Support MS_DIAGNOSTIC_DATA_PATH in configuring dump path 2021-07-22 13:34:59 -04:00
limingqi107 06a6e8d186 fix the test case of CPU dump 2021-07-12 22:58:34 +08:00
limingqi107 36b1ff25b4 Enable mindRT on CPU device 2021-07-07 13:19:37 +08:00
limingqi107 e761655a42 actor runtime support CPU dump 2021-07-04 11:19:09 +08:00
TinaMengtingZhang 2fa05b66a1 change device_id to rank_id in dump path 2021-06-06 21:27:20 -04:00
TinaMengtingZhang f11d5fa8ad Unify file name for parameter and CPU dump 2021-06-02 16:31:03 -04:00
John Tzanakakis 1d62e1653e Unified dump entry, async dump multiple steps in different dirs, and async dump multiple specific steps in one training session - finer grain control 2021-05-31 09:47:20 -04:00
TinaMengtingZhang 4926d74570 unify dir path
dump hccl and config json files to dir

update filename for sync dump except cpu dump

update testcases
2021-05-26 16:22:18 -04:00
TinaMengtingZhang 17bc01382a Revert the change in GenerateDumpPath 2021-05-12 22:11:28 -04:00
buxue 0eab785cfa unify dump path and support npy format 2021-05-08 09:04:33 +08:00
rainyhorse d2432a962a update the convert tool path 2021-03-09 15:00:09 +08:00
rainyhorse 1186d34622 dump tool update 2021-02-08 10:08:33 +08:00
jjfeing 502be04491 upgrade 0204 2021-02-06 10:51:43 +08:00
l00591931 9ec100d069 Change TensorAdd to Add, from r1.1 to master 2021-02-01 17:53:52 +08:00