yankf21
/
mindspore2022
forked from huawei/mindspore2022
0
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
42,203 Commits 18 Branches 22 Tags 3.6 GiB
Commit Graph

557 Commits

Author SHA1 Message Date
yangzhenzhang 2a0b528084 support opt parallel for adafactor 2021-12-13 11:21:44 +08:00
i-robot 938dc8abd0 !27439 [Auto parallel] Add new operatorInfo for Parallel: CumSum 
Merge pull request !27439 from Xiaoda/117-add-cumsum-op
 2021-12-10 07:00:06 +00:00
i-robot 0e358f4cb3 !27428 revert insert VirtualDataset node for master 
Merge pull request !27428 from lilei/modify_virtualdataset_for_master
 2021-12-09 02:47:13 +00:00
Xiaoda Zhang 8042c88223 add the new operatorInfo for parallel: CumSum 2021-12-09 09:21:52 +08:00
lilei 2edf6ab33b revert insert VirtualDataset node for master 2021-12-08 17:39:42 +08:00
i-robot faaec746f7 !27401 add more ut tests for allreduce fusion 
Merge pull request !27401 from jiahongQian/master
 2021-12-08 09:38:37 +00:00
jiahongQian b03c8d18d3 add more ut tests 2021-12-08 14:17:02 +08:00
i-robot ffca7b08a5 !27237 auto insert VirtualDataset node for master 
Merge pull request !27237 from lilei/modify_virtualdataset_for_master
 2021-12-07 09:02:50 +00:00
i-robot f40668ef73 !27251 test_micro_batch_Interleaved 
Merge pull request !27251 from lilei/add_parallel_ut
 2021-12-07 02:18:09 +00:00
lilei 05189459ab auto insert VirtualDataset node for master 2021-12-06 21:39:37 +08:00
lilei e933aa268b test_micro_batch_Interleaved 2021-12-06 18:37:21 +08:00
i-robot 2d23b698a6 !27024 add allreduce fusion by size 
Merge pull request !27024 from jiahongQian/master
 2021-12-06 05:32:12 +00:00
q00596439 de36fdc169 add allreduce fusion size and unify the interface 2021-12-05 21:35:11 +08:00
huangxinjing 8c9b2b93a8 Add transformer 2021-12-03 09:08:57 +08:00
yangzhenzhang 7454b8f8f2 check args for shard 2021-12-02 11:51:29 +08:00
Xiaoda Zhang 364858cbc9 In sharding propagation, to keep strategy consistent of parameter being used by multiple operators, we check the edge with one node of TmpIdentityInfo 2021-11-30 18:58:33 +08:00
Xiaoda Zhang 04db51a528 In a previous PR (https://gitee.com/mindspore/mindspore/pulls/26807/), we replaced 'auto_parallel_search_mode' by 'search_mode' directly. 
However, to be forward compatitable, it is suitable to keep 'auto_parallel_search_mode' available. This PR recovers the 'auto_parallel_search_mode' interface and adds a warning when using this old interface.

This PR also deals with other codestyle things.
 2021-11-29 15:40:23 +08:00
i-robot 9f8ec2c5ab !26807 [Auto parallel] [Sharding propagation] Interface change of sharding propagation 
Merge pull request !26807 from Xiaoda/113-auto-parallel-search-mode-changes-to-search-mode
 2021-11-26 01:48:58 +00:00
i-robot 6ecbc97fd6 !26804 virtual_dataset_avoid_auto_parallel 
Merge pull request !26804 from yao_yf/virtual_dataset_avoid_auto_parallel
 2021-11-26 01:48:53 +00:00
i-robot b282414de7 !26619 arallel_ut_refactoring 
Merge pull request !26619 from 王志伟/parallel_ut_refactoring1
 2021-11-26 01:34:49 +00:00
Xiaoda Zhang ad5ac77ae8 1) 'auto_parallel_search_mode' changes to 'search_mode'; 
2) 'sharding_propagation' moves to 'search_mode';
 2021-11-25 19:38:21 +08:00
yao_yf f29ce1fb60 virtual dataset avoid auto parallel 2021-11-25 18:47:35 +08:00
i-robot 519f14a909 !26006 slice recompute activation 
Merge pull request !26006 from yao_yf/add_transformer_slice_activation_config
 2021-11-24 06:14:28 +00:00
wzw 86c5ad20c8 parallel_ut_refactoring1 2021-11-24 09:41:52 +08:00
i-robot 1b8c2ff0e9 !26414 fault_recover_by_mirror_group_fix_opt_shard 
Merge pull request !26414 from yao_yf/fault_recover_by_mirror_group_fix_opt_shard
 2021-11-20 03:06:45 +00:00
yao_yf 188d39da83 slice_activation_in_recompute 
slice recompute activation
 2021-11-19 10:51:54 +08:00
yao_yf 01dc4bbdf9 fix fault recover in optimizer shard 2021-11-19 09:46:27 +08:00
Xiaoda Zhang df67e74eaf making sharding_propagation smooth, add a reshape justification: 
1) when propagate sharding strategy from one op to another, try to find the strategy with zero communication cost;
2) if there is no such strategy, find the strategy with minimum communication cost, and raise a warning;
 2021-11-18 15:55:50 +08:00
i-robot 9f52343a6a !26350 add check for resizenearestneighbor parallel op 
Merge pull request !26350 from yangzhenzhang/add-check-for-resize-op
 2021-11-18 05:45:57 +00:00
yangzhenzhang ba99e4c505 add check for resize op 2021-11-16 15:39:52 +08:00
ttudu 33ac1de062 fix bug 2021-11-16 11:14:58 +08:00
i-robot 7a73bae5c3 !26036 add output strategy for matmul operator 
Merge pull request !26036 from yangzhenzhang/add-output-strategy-for-op-init
 2021-11-12 01:22:22 +00:00
Xiaoda Zhang a772767265 support reshape in sharding propagation: 
1) using 'swc index of strategy_cost_' as reshape's selected strategy;
2) when encountering reshape in BFS, select the 'swc index' with zero communication cost;
3) when encountering a reshape that is already visited, check whether there exists communication between reshape and current operator. It is OK if communication happens between two configured operators;
4) currently, two consecutive reshapes are not supported;
5) adjusting BFS structure in graph_costmodel.cc;
6) adjusting some code in step_auto_parallel.cc to avoid cyclomatic complexity.
 2021-11-11 11:35:43 +08:00
yangzhenzhang 8431ba616c add output strategy for op init 2021-11-10 15:35:45 +08:00
huangxinjing f354ab22a3 add pipeline shard interface 
Add support for no pipeline accugradient

Add delay tag for fusion op

Optimizer the visite order

add mirror for mini step control

Move the group to attributes

Add gradient_shard control for the mini step

Fix code stype

Fix ut description

Add interface
 2021-11-09 09:11:13 +08:00
i-robot ded1c77bbf !25765 neighborExchangeV2 & grad 
Merge pull request !25765 from TuDouNi/neighborExchangeV2
 2021-11-05 09:31:03 +00:00
ttudu e953c15cd2 NeighborExchangeV2 & Grad 2021-11-05 11:00:46 +08:00
yangzhenzhang 6ad6304b77 add output strategy 2021-11-04 16:28:54 +08:00
yangzhenzhang c42081619e add parallel op for resizenearestneighbor 2021-10-28 11:31:00 +08:00
wanyiming 4fbc59a98a utfixs 2021-10-18 18:06:37 +08:00
i-robot f83070728d !24790 support user define strategy gen method under auto parallel context 
Merge pull request !24790 from zhuyuxiao/master
 2021-10-14 09:03:36 +00:00
zhuyuxiao 1907246931 change api 2021-10-13 10:28:30 +08:00
i-robot 3fd94000c5 !24568 Apply batch parallel in auto_parallel mode when strategies are not specified 
Merge pull request !24568 from zhuyuxiao/master
 2021-10-13 01:20:15 +00:00
zhuyuxiao cf76c76745 apply batch parallel in auto_parallel mode when strategies are not specified 2021-10-12 15:44:10 +08:00
yao_yf b303d6001c parallel ut refactor 2021-10-12 15:18:51 +08:00
i-robot 7cde7731b0 !23537 Update pangu reshape and softmax. 
Merge pull request !23537 from linqingke/pangu
 2021-09-24 08:12:13 +00:00
i-robot e7cb505e68 !23569 Produce parallel operators for ResizeBilinear and ResizeNearestNeighbor 
Merge pull request !23569 from Bert0108/resizebilinear_parallel_ops
 2021-09-24 03:05:23 +00:00
i-robot d37fccc56f !23544 remove deprecated gather op 
Merge pull request !23544 from zhuyuxiao/master
 2021-09-24 01:20:53 +00:00
Bert0108 2d3d0b673e parallel operators for ResizeBilinear and ResizeNearestNeighbor 2021-09-24 09:07:49 +08:00
linqingke acde7febef update pangu reshape and softmax performance. 
Add layer norm judge

Fix layer norm name error

Fix input tyoe check

Fix ut test

Add 3d supports
 2021-09-24 09:06:16 +08:00
zhoufeng 1f934bd782 check neighbor attr type 
Signed-off-by: zhoufeng <zhoufeng54@huawei.com>
 2021-09-17 21:15:37 +08:00
zhuyuxiao 79d99323a2 rename gather 2021-09-16 16:07:05 +08:00
huangxinjing 0b89d5c9c4 fix batch size error 2021-09-16 10:56:40 +08:00
i-robot bbdacd41f4 !20585 add security isolate for save_graphs 
Merge pull request !20585 from huanghui/add-security-isolate-for-DumpIR
 2021-09-15 01:25:46 +00:00
i-robot db19a40280 !23365 add print op security 
Merge pull request !23365 from fangzehua/print_sec
 2021-09-14 09:01:31 +00:00
huanghui ba66c0d491 add security isolate for save_graphs 2021-09-14 10:24:12 +08:00
fangzehua 4ccc635a07 add print security 2021-09-13 17:45:36 +08:00
yangzhenzhang 1b8eb283e4 modify batch parallel info 2021-09-13 16:28:56 +08:00
huangxinjing e02f553010 Fix spell error and add mode check 2021-09-09 18:50:53 +08:00
Xiaoda Zhang 5613c0b974 add a moe implementation: 
1) extend the Liner cell for including BatchMatMul implementation, in
which the first dimension indicates the expert number;
2) implement a Switch (top1) router;
3) implement a MoE cell, which extends the FeedForward cell.
 2021-09-09 11:53:42 +08:00
i-robot 77424eaad5 !23004 Add args Check for Transformer 
Merge pull request !23004 from huangxinjing/args_check
 2021-09-08 08:24:06 +00:00
huangxinjing 6cea07f749 Add args check 2021-09-08 09:12:35 +08:00
yao_yf 3ef26288a2 parallel_sparse_attention_ops_fix_repeated_cal 2021-09-06 19:57:22 +08:00
i-robot fa12d62d4d !21776 set device_id master 0813 
Merge pull request !21776 from mindspore_ding/set_device_id_master_0813
 2021-09-06 03:02:58 +00:00
dingpeifei b4bc6000dc set device id master 0813 2021-09-02 20:51:50 +08:00
i-robot d87d0e07c2 !22255 recompute_interface_modify 
Merge pull request !22255 from yao_yf/recompute_interface_modify
 2021-09-01 08:09:36 +00:00
yao_yf 39055af6e4 recompute interface modify 2021-09-01 10:59:59 +08:00
i-robot cc8d614b25 !22650 fixed sparse attention modify 
Merge pull request !22650 from yao_yf/fixed_sparse_attention_modify
 2021-09-01 01:29:45 +00:00
i-robot 389f3a6b6c !21835 make alltoall and neighborexchange to be interface && revert pr 21395 
Merge pull request !21835 from zhoufeng/revert-same-input-to-comm-op
 2021-08-31 08:53:46 +00:00
yao_yf 82889ec56b fixed sparse attention 2021-08-31 16:11:56 +08:00
yao_yf 68dd138462 add parallel sparse attention ops: dsd_matmul 2021-08-30 18:46:52 +08:00
zhoufeng ecae690a19 Revert "fix same node is used by two comm op" 
This reverts commit b09d411dc4.

add AlltoAll and NeighborExchange as interface
 2021-08-30 16:54:38 +08:00
yao_yf b8a9cbe2a3 add cus_matmul_dds parallel ops 2021-08-30 09:44:48 +08:00
Zhang Qinghua a137fa1d0b Optimize the Executors routines. 
- Fix the key generating.
- Distinguish the executors.
 2021-08-28 17:36:12 +08:00
zhihenghu ce12c02343 Add Sparse Attention 
adjut the file structure and name

Deleted extra information

Do some formatting work

Add test case and fix some document

fix imports
 2021-08-28 11:50:58 +08:00
i-robot e6e1f37ae4 !22346 [Core] Fix the bug of scope setting when cloning nodes 
Merge pull request !22346 from Xiaoda/86-fix-the-fullname-scope-bug
 2021-08-26 13:04:15 +00:00
i-robot 8d00a8d803 !22360 Fix Transformer Mirror Error 
Merge pull request !22360 from huangxinjing/fix_transformer_mirror_error
 2021-08-26 08:16:33 +00:00
Xiaoda Zhang b2703879c6 fix the scope setting error when cloning nodes 2021-08-26 10:25:38 +08:00
i-robot edcbb68d71 !22386 fix neighborexchange empty input case 
Merge pull request !22386 from zhoufeng/fix-neighbor-empty-input-bak
 2021-08-26 01:52:10 +00:00
zhoufeng e5a1582e4b fix neighborexchange empty input case 
Signed-off-by: zhoufeng <zhoufeng54@huawei.com>
 2021-08-25 21:41:22 +08:00
huangxinjing 62496d75f3 less the interface exposed 2021-08-25 21:22:40 +08:00
yangzhenzhang 0b9b2a9458 add test cases 2021-08-25 17:22:20 +08:00
lichenever 5812076512 Refactor_part_of_pipeline 2021-08-24 21:37:41 +08:00
yangzhenzhang f1afaeac5a modify check strategy for conv2d 2021-08-24 16:06:16 +08:00
ms_yan 36a8886ca2 Revert "[feat] [assistant] [I3T96T] add new Dataset operator CMUARCTICDataset" 
This reverts commit b077aa1cab.

Revert "[feat] [assistant] [I3T96X] add new Dataset operator LibriSpeechDataset"

This reverts commit 4e6f7dc97d.

delete pass_registry_test.cc

comment  hiai_nlu_model_multi.pb related  line
 2021-08-23 01:46:38 +08:00
djc b077aa1cab [feat] [assistant] [I3T96T] add new Dataset operator CMUARCTICDataset 2021-08-22 16:26:45 +08:00
djc 4e6f7dc97d [feat] [assistant] [I3T96X] add new Dataset operator LibriSpeechDataset 2021-08-22 13:39:37 +08:00
huangxinjing d777742904 1. Move the class to mindspore.parallel, support activation sharding 2021-08-20 14:34:10 +08:00
i-robot dde05c36b8 !21551 auto_parallel_context interface dataset_strategy union 
Merge pull request !21551 from yao_yf/auto_parallel_context_dataset_strategy_union
 2021-08-18 09:03:18 +00:00
i-robot 415275ae17 !21805 support adafactor model parallel 
Merge pull request !21805 from yangzhenzhang/auto-parallel-support-adafactor-opt
 2021-08-18 07:20:39 +00:00
i-robot 0d839fa7c6 !21809 Improved Transformer Struture and Add Args Check 
Merge pull request !21809 from huangxinjing/transformer_improved
 2021-08-17 12:05:25 +00:00
i-robot a77a0b968d !21761 comm_recompute_interface. 
Merge pull request !21761 from yao_yf/comm_recompute_interface
 2021-08-17 09:14:29 +00:00
yangzhenzhang 7ca64d2235 auto parallel support adafactor opt 2021-08-17 10:36:42 +08:00
yao_yf 5277b229be add cell comm recompute interface 2021-08-17 09:23:21 +08:00
huangxinjing 18044aff0f 1. Add docstring, elimitate attention mask, tuple append the deocoder return layer past 
split attention inputs from two into three

Add input check and modify test case

Fix check error and add check to transformer inputs

add input check to the transformers
 2021-08-16 21:08:48 +08:00
yao_yf a83bf73298 union auto_parallel_context interface dataset_strategy 2021-08-16 10:54:46 +08:00
yao_yf e233880e41 fix reshape depend reshape in auto parallel 2021-08-12 11:45:28 +08:00
i-robot 63445ff6fd !21627 alltoall exception handle 
Merge pull request !21627 from zhoufeng/alltoall_exception
 2021-08-12 01:40:22 +00:00
yangzhenzhang d18c813ee4 check strategy for conv2d 2021-08-11 19:12:48 +08:00
zhoufeng 03a56f2bb0 alltoall exception handle 
Signed-off-by: zhoufeng <zhoufeng54@huawei.com>
 2021-08-11 17:30:49 +08:00
i-robot 4aaa8126a0 !21528 Add Parallel Print Support 
Merge pull request !21528 from huangxinjing/add_print
 2021-08-09 10:55:04 +00:00
huangxinjing 92bad162bd Add print 2021-08-09 11:14:25 +08:00
huangbingjian 53b31abf12 remove useless depend 2021-08-05 20:21:24 +08:00
yangzhenzhang ef0361a449 fix bugs for conv2d 2021-08-03 10:48:05 +08:00
Xiaoda Zhang 4b4b3cdaf4 add reduceany operator and extend onehot to multi-dimensions 2021-08-02 08:53:43 +08:00
huangxinjing 615d1a179d Add transformer layer 2021-07-29 11:10:43 +08:00
i-robot 9f296c58d6 !20960 [AutoParallel]Add replace graph for conv2d 
Merge pull request !20960 from lichen/add_replace_graph_for_conv2d
 2021-07-28 07:06:41 +00:00
lichenever a7f8024c29 add_replace_graph_for_conv2d 2021-07-28 11:02:21 +08:00
yao_yf dc7dc7d3fa dataset strategy set 2021-07-27 18:44:10 +08:00
yangzhenzhang 80e5cc0e52 add parallel op for gatherd 2021-07-26 11:14:25 +08:00
Xiaoda Zhang bb5d4212f7 enable All2All in infering redistribution ops 2021-07-24 09:09:01 +08:00
lichenever 3c7cfb7c08 auto_parallel_support_control_flow 2021-07-22 17:31:31 +08:00
i-robot a7d40fc220 !20520 [AutoParallel]Add op AllToAllv 
Merge pull request !20520 from lichen/add_op_AllToAllv
 2021-07-20 07:57:03 +00:00
lichenever 8c1998fd6b add_op_AllToAllv 2021-07-20 10:49:28 +08:00
i-robot c9d3c1d346 !20411 enable optimizer parallel for inference 
Merge pull request !20411 from gziyan/enable_opt_shard_predict
 2021-07-20 01:37:06 +00:00
yangzhenzhang b31cd27a08 update check strategy for conv2d 2021-07-19 15:03:26 +08:00
Ziyan 1c9166e0a6 remove restriction for opt shard in inference 2021-07-16 17:16:12 +08:00
Xiaoda Zhang 04381273b3 Add the sharding propagation function: 
1) users configure sharding strategies for operators;
2) framework will propagate the strategies from configured-ops to
non-configured ops using BFS;
3) the propagation goal is to minimize redistribution communication
cost;
 2021-07-14 10:54:18 +08:00
chenhaozhe 086a871975 Change Loss to LossBase 2021-07-02 16:41:02 +08:00
lichenever db8850a4a3 pipeline_support_predict_master 2021-06-26 09:25:59 +08:00
Ziyan be1f5a43d7 opt shard fit micro batch 2021-06-24 14:52:19 +08:00
yangzhenzhang 69acf757d0 add parallel op for conv2d backprop input 2021-06-24 10:01:49 +08:00
yangzhenzhang 24370b5613 add parallel op for maxpool 2021-06-21 09:00:56 +08:00
yangzhenzhang af0d28de48 add parallel op for batchnorm 2021-06-17 15:43:24 +08:00
i-robot 85d860e6a2 !16457 [AutoParallel]pipeline_split_adapt_master 
Merge pull request !16457 from lichen/pipeline_split_adapt_master
 2021-06-11 11:37:40 +08:00
lichenever db5d508356 pipeline_split_adapt_master 2021-06-10 20:17:33 +08:00
yangzhenzhang 7a40741048 add parallel operator for conv2d 2021-06-10 14:58:51 +08:00
Ziyan 95ac0f6d58 fix optimizer weight shard config 2021-06-08 10:44:17 +08:00
chenhaozhe 9da8534396 change _Loss to Loss 2021-06-03 15:26:59 +08:00
mindspore-ci-bot 1c8fda25ef !16478 handle load op in step parallel 
From: @gong_zi_yan
Reviewed-by: @yangzhenzhang,@stsuteng
Signed-off-by: @stsuteng
 2021-05-29 09:32:35 +08:00
mindspore-ci-bot b45b63fc58 !17239 add parallel gathernd test case 
From: @hanyang001
Reviewed-by: @stsuteng,@yangzhenzhang
Signed-off-by: @stsuteng
 2021-05-29 09:11:25 +08:00
Wan Hanyang c51dff2634 add parallel gathernd test case 2021-05-28 17:21:11 +08:00
Wan Hanyang 3ce521d78f add parallel layernorm test case 2021-05-28 17:20:04 +08:00
Ziyan 4b17493e52 handle load in step parallel 2021-05-28 09:16:40 +08:00
yangzhenzhang d711d98f07 clean duplicate code 2021-05-25 17:25:34 +08:00
yao_yf 732d13ccff parallel dropout support repeated compute 2021-05-20 19:42:13 +08:00
yangzhenzhang 6aa3859131 modify check strategy for scatter update 2021-05-10 11:22:00 +08:00
Ziyan 2a752f24bf enable not fully use opt shard 2021-05-07 15:32:48 +08:00
yao_yf e967f1939b parallel envs variable check 2021-05-07 09:12:53 +08:00
mindspore-ci-bot 78fcdbc7c9 !15790 modify scatter update op 
From: @yangzhenzhang
Reviewed-by: @kisnwang,@stsuteng
Signed-off-by: @stsuteng
 2021-04-28 10:48:19 +08:00
yangzhenzhang 075f680a42 modify scatter update op 2021-04-27 20:23:14 +08:00
Xiaoda Zhang aa52399200 Making the Tile operator to have more parallel strategies 2021-04-27 11:36:40 +08:00
yao_yf 093ef784de dont insert virtualoutput for scalar 2021-04-26 19:55:16 +08:00
mindspore-ci-bot 3cfd58e8e0 !15643 insert virtual div only for first input of dropout do mask 
From: @yangzhenzhang
Reviewed-by: @stsuteng,@kisnwang
Signed-off-by: @stsuteng
 2021-04-26 09:21:00 +08:00
mindspore-ci-bot 49d6c029a6 !15542 split axis and batch for gather 
From: @yangzhenzhang
Reviewed-by: @kisnwang,@stsuteng,@stsuteng
Signed-off-by: @stsuteng,@stsuteng
 2021-04-25 19:33:09 +08:00
yangzhenzhang 5828973978 fix bug for dropout do mask 2021-04-25 16:47:44 +08:00
yao_yf 21276408b8 parallel virtual_out_ops 2021-04-25 11:18:54 +08:00
yangzhenzhang 213922574e split axis and batch for gatherv2 2021-04-23 16:59:35 +08:00
yangzhenzhang c2ca2232c5 add select op 2021-04-20 09:23:22 +08:00
mindspore-ci-bot 1c9d3c0aa0 !15353 add parallel operator for scatter update 
From: @yangzhenzhang
Reviewed-by: @kisnwang,@stsuteng
Signed-off-by: @stsuteng
 2021-04-20 09:03:08 +08:00