TinaMengtingZhang
67c98923d5
Add st for dump
2022-04-07 16:00:02 -04:00
TinaMengtingZhang
438e261bbc
wait until dumping finished
2022-03-18 11:44:09 -04:00
i-robot
1863766fad
!31063 Fix a+m dump probilistic failure in ci
...
Merge pull request !31063 from TinaMengtingZhang/fix_st_error2
2022-03-16 07:19:48 +00:00
huanghui
4482acf586
Support API set_dump for more type of ops
2022-03-15 10:24:15 +08:00
TinaMengtingZhang
646909d3f4
fix a+m dump probilistic failure in ci: 1. do not parallel to dump statistics and remove file lock. 2. when tensor size is small, do it in single thread
2022-03-14 17:11:55 -04:00
i-robot
dcb5cd670c
!30953 Dynamic Weight Decay
...
Merge pull request !30953 from wanyiming/dynamic_wd
2022-03-14 06:03:24 +00:00
wanyiming
a124ec4de7
add dynamic_decay
2022-03-10 11:02:27 +08:00
TinaMengtingZhang
78449afc49
modify st level for stat dump
2022-03-07 21:56:29 -05:00
TinaMengtingZhang
4a8d3defe7
Enhance the perfomance of A+M dump: Parallelize ConverFormatForTensorAndDump and refactoring
2022-03-01 23:08:38 -05:00
TinaMengtingZhang
e576292862
dump statistic file in kernel by kernel mode
2022-02-15 16:58:00 -05:00
Parastoo Ashtari
f6bebc7d97
use origin_parameter_order to load and dump params mindRT
...
Refactor mindRT code
Fix DumpConstantData issue
2022-02-10 14:18:08 -05:00
i-robot
8d648e0f1e
!19673 Seperate constant from pb file
...
Merge pull request !19673 from sabrinasun_59ee/constant
2021-12-15 23:52:33 +00:00
sabrinasun
542d715217
seperate constant from pb
2021-12-14 13:19:14 -05:00
Jimmy Qi
6f2b70c21f
Print null for min, max, avg values for NaN tensor
2021-12-12 19:31:21 +00:00
Jimmy Qi
b9d1a4920c
Add statistic dump for ascend
2021-12-06 20:11:44 +00:00
TinaMengtingZhang
16a19be56f
Enable cann api allback register function
...
Add testcases for cann api feature
2021-12-03 17:36:15 -05:00
sabrinasun
c095908772
fix enable=false still dump issue for cell dump
2021-11-30 22:10:20 -05:00
TinaMengtingZhang
07b653103e
Support Cann callback api for ascend async dump
2021-11-24 10:35:29 -05:00
sabrinasun
99a434672f
support cell dump when dump_mode=2
2021-11-16 16:01:13 -05:00
Jimmy Qi
b21c099767
Add option to dump tensor statistics in csv format
2021-11-15 19:00:59 +00:00
i-robot
257bb3500d
!25785 change dump path to avoid invalid symbol in ci path
...
Merge pull request !25785 from yelihua/dev
2021-11-04 13:08:01 +00:00
parastooashtari
1a59dc37bf
add graph execution order history to dump
2021-11-02 11:00:38 -04:00
yelihua
f9f2058055
change dump path to avoid invalid symbol in path
2021-11-02 17:29:35 +08:00
sabrinasun
332e0dbb0f
add overflow st test and checkdumpstructure test
2021-10-22 06:28:58 +08:00
sabrinasun
220245f592
add security isolation to online and offline debugger
2021-09-12 23:01:15 -04:00
yelihua
7c3994e48e
use Common::CreatePrefixPath instead of Common::GetRealPath
2021-09-09 20:34:25 +08:00
TinaMengtingZhang
424e94a7d4
Adapt async dump conversion tool with new run package (Sep03 version)
2021-09-07 13:41:26 -04:00
yanghaoran
1fd655787a
upgrade Ascend packages 04 Sep 21
2021-09-06 21:24:07 +08:00
TinaMengtingZhang
b17b2bc687
Fix convert async dump files failed issue and refactor convert_async.py
2021-09-01 15:10:55 -04:00
Xiaoda Zhang
b2703879c6
fix the scope setting error when cloning nodes
2021-08-26 10:25:38 +08:00
ms_yan
36a8886ca2
Revert "[feat] [assistant] [I3T96T] add new Dataset operator CMUARCTICDataset"
...
This reverts commit b077aa1cab .
Revert "[feat] [assistant] [I3T96X] add new Dataset operator LibriSpeechDataset"
This reverts commit 4e6f7dc97d .
delete pass_registry_test.cc
comment hiai_nlu_model_multi.pb related line
2021-08-23 01:46:38 +08:00
djc
b077aa1cab
[feat] [assistant] [I3T96T] add new Dataset operator CMUARCTICDataset
2021-08-22 16:26:45 +08:00
djc
4e6f7dc97d
[feat] [assistant] [I3T96X] add new Dataset operator LibriSpeechDataset
2021-08-22 13:39:37 +08:00
yelihua
a3cba3857e
use temporary dir as dump dir
2021-08-17 20:11:33 +08:00
yelihua
a6dc9a0a07
get rank id when set hccl env for single card train
2021-08-16 13:44:02 +08:00
yelihua
ca835a7d2d
change the async dump testcase level to level1
2021-08-05 19:52:35 +08:00
TinaMengtingZhang
5b8e846fe9
Support MS_DIAGNOSTIC_DATA_PATH in configuring dump path
2021-07-22 13:34:59 -04:00
limingqi107
06a6e8d186
fix the test case of CPU dump
2021-07-12 22:58:34 +08:00
limingqi107
36b1ff25b4
Enable mindRT on CPU device
2021-07-07 13:19:37 +08:00
limingqi107
e761655a42
actor runtime support CPU dump
2021-07-04 11:19:09 +08:00
TinaMengtingZhang
2fa05b66a1
change device_id to rank_id in dump path
2021-06-06 21:27:20 -04:00
TinaMengtingZhang
f11d5fa8ad
Unify file name for parameter and CPU dump
2021-06-02 16:31:03 -04:00
John Tzanakakis
1d62e1653e
Unified dump entry, async dump multiple steps in different dirs, and async dump multiple specific steps in one training session - finer grain control
2021-05-31 09:47:20 -04:00
TinaMengtingZhang
4926d74570
unify dir path
...
dump hccl and config json files to dir
update filename for sync dump except cpu dump
update testcases
2021-05-26 16:22:18 -04:00
TinaMengtingZhang
17bc01382a
Revert the change in GenerateDumpPath
2021-05-12 22:11:28 -04:00
buxue
0eab785cfa
unify dump path and support npy format
2021-05-08 09:04:33 +08:00
rainyhorse
d2432a962a
update the convert tool path
2021-03-09 15:00:09 +08:00
rainyhorse
1186d34622
dump tool update
2021-02-08 10:08:33 +08:00
jjfeing
502be04491
upgrade 0204
2021-02-06 10:51:43 +08:00
l00591931
9ec100d069
Change TensorAdd to Add, from r1.1 to master
2021-02-01 17:53:52 +08:00