!20821 提交GRU模型GPU版本PR，优化Ascend性能与精度，合并GPU/Ascend为同一套代码
Merge pull request !20821 from 吕昱峰（Nate.River）/gru
This commit is contained in:
Gitee
commit
59d8e30203
|
|
@ -46,7 +46,7 @@ In this model, we use the Multi30K dataset as our train and test dataset.As trai
|
|||
|
||||
# [Environment Requirements](#content)
|
||||
|
||||
- Hardware（Ascend）
|
||||
- Hardware（Ascend or GPU）
|
||||
- Prepare hardware environment with Ascend processor.
|
||||
- Framework
|
||||
- [MindSpore](https://gitee.com/mindspore/mindspore)
|
||||
|
|
@ -81,15 +81,27 @@ nltk.download()
|
|||
After dataset preparation, you can start training and evaluation as follows:
|
||||
|
||||
```bash
|
||||
# run training example
|
||||
cd ./scripts
|
||||
bash run_standalone_train.sh [TRAIN_DATASET_PATH]
|
||||
# download dataset
|
||||
bash download_dataset.sh
|
||||
|
||||
# preprocess dataset
|
||||
bash preprocess.sh [DATASET_PATH]
|
||||
|
||||
# create mindrecord
|
||||
bash create_dataset.sh [DATASET_PATH] [DATASET_PATH]
|
||||
|
||||
# run training example
|
||||
bash run_standalone_train_{platform}.sh [TRAIN_DATASET_PATH]
|
||||
|
||||
# run distributed training example
|
||||
bash run_distribute_train_ascend.sh [RANK_TABLE_FILE] [TRAIN_DATASET_PATH]
|
||||
bash run_distribute_train_{platform}.sh [RANK_TABLE_FILE] [TRAIN_DATASET_PATH]
|
||||
# platform: ascend or gpu
|
||||
# do not need [RANK_TABLE_FILE] if you use GPU
|
||||
|
||||
# run evaluation example
|
||||
bash run_eval.sh [CKPT_FILE] [DATASET_PATH]
|
||||
bash run_eval_{platform}.sh [CKPT_FILE] [DATASET_PATH]
|
||||
# platform: ascend or gpu
|
||||
```
|
||||
|
||||
- Running on ModelArts (If you want to run in modelarts, please check the official documentation of [modelarts](https://support.huaweicloud.com/modelarts/), and you can start training as follows)
|
||||
|
|
@ -158,7 +170,6 @@ The GRU network script and code result are as follows:
|
|||
│ ├──local_adapter.py // Local adapter
|
||||
│ ├──moxing_adapter.py // Moxing adapter for ModelArts
|
||||
├── src
|
||||
| ├──gru.py // gru cell architecture.
|
||||
│ ├──create_data.py // Dataset preparation.
|
||||
│ ├──dataset.py // Dataset loader to feed into model.
|
||||
│ ├──gru_for_infer.py // GRU eval model architecture.
|
||||
|
|
@ -167,16 +178,24 @@ The GRU network script and code result are as follows:
|
|||
│ ├──lr_schedule.py // Learning rate scheduler.
|
||||
│ ├──parse_output.py // Parse output file.
|
||||
│ ├──preprocess.py // Dataset preprocess.
|
||||
| ├──rnn_cells.py // rnn cell architecture.
|
||||
| ├──rnns.py // rnn layer architecture.
|
||||
│ ├──seq2seq.py // Seq2seq architecture.
|
||||
| ├──utils.py // utils for rnn.
|
||||
│ ├──tokenization.py // tokenization for the dataset.
|
||||
│ ├──weight_init.py // Initialize weights in the net.
|
||||
├── scripts
|
||||
│ ├──create_dataset.sh // shell script for create dataset.
|
||||
│ ├──download_dataset.sh // shell script for download dataset.
|
||||
│ ├──parse_output.sh // shell script for parse eval output file to calculate BLEU.
|
||||
│ ├──preprocess.sh // shell script for preprocess dataset.
|
||||
│ ├──run_distributed_train.sh // shell script for distributed train on ascend.
|
||||
│ ├──run_eval.sh // shell script for standalone eval on ascend.
|
||||
│ ├──run_standalone_train.sh // shell script for standalone eval on ascend.
|
||||
│ ├──run_distributed_train_ascend.sh // shell script for distributed train on ascend.
|
||||
│ ├──run_distributed_train_gpu.sh // shell script for distributed train on gpu.
|
||||
│ ├──run_eval_ascend.sh // shell script for standalone eval on ascend.
|
||||
│ ├──run_eval_gpu.sh // shell script for standalone eval on gpu.
|
||||
│ ├──run_infer_310.sh // shell script for 310 inference.
|
||||
│ ├──run_standalone_train_ascend.sh // shell script for standalone eval on ascend.
|
||||
│ ├──run_standalone_train_gpu.sh // shell script for standalone eval on gpu.
|
||||
├── default_config.yaml // Configurations
|
||||
├── postprocess.py // GRU postprocess script.
|
||||
├── preprocess.py // GRU preprocess script.
|
||||
|
|
@ -188,7 +207,14 @@ The GRU network script and code result are as follows:
|
|||
|
||||
## [Dataset Preparation](#content)
|
||||
|
||||
Firstly, we should download the dataset from the WMT16 official net.After downloading the Multi30k dataset file, we get six dataset file, which is show as below.And we should in put the in same directory.
|
||||
Firstly, we should download the dataset from the WMT16 official net.
|
||||
|
||||
```bash
|
||||
cd scripts
|
||||
bash download_dataset.sh
|
||||
```
|
||||
|
||||
After downloading the Multi30k dataset file, we get six dataset file, which is show as below.And we should in put the in same directory.
|
||||
|
||||
```text
|
||||
train.de
|
||||
|
|
@ -250,14 +276,17 @@ Parameters for both training and evaluation can be set in config.py. All the dat
|
|||
|
||||
```bash
|
||||
cd ./scripts
|
||||
bash run_standalone_train.sh [DATASET_PATH]
|
||||
bash run_standalone_train_{platform}.sh [DATASET_PATH]
|
||||
# platform: ascend or gpu
|
||||
```
|
||||
|
||||
- Running scripts for distributed training of GRU. Task training on multiple device and run the following command in bash to be executed in `scripts/`:
|
||||
|
||||
``` bash
|
||||
cd ./scripts
|
||||
bash run_distributed_train.sh [RANK_TABLE_PATH] [DATASET_PATH]
|
||||
bash run_distributed_train_{platform}.sh [RANK_TABLE_PATH] [DATASET_PATH]
|
||||
# platform: ascend or gpu
|
||||
# do not need [RANK_TABLE_FILE] if you use GPU
|
||||
```
|
||||
|
||||
## [Inference Process](#content)
|
||||
|
|
@ -266,7 +295,8 @@ Parameters for both training and evaluation can be set in config.py. All the dat
|
|||
|
||||
``` bash
|
||||
cd ./scripts
|
||||
bash run_eval.sh [CKPT_FILE] [DATASET_PATH]
|
||||
bash run_eval_{platform}.sh [CKPT_FILE] [DATASET_PATH]
|
||||
# platform: ascend or gpu
|
||||
```
|
||||
|
||||
- After evalulation, we will get eval/target.txt and eval/output.txt.Then we can use scripts/parse_output.sh to get the translation.
|
||||
|
|
@ -354,35 +384,35 @@ perl multi-bleu.perl target.txt.forbleu < output.txt.forbleu
|
|||
|
||||
### Training Performance
|
||||
|
||||
| Parameters | Ascend |
|
||||
| -------------------------- | -------------------------------------------------------------- |
|
||||
| Resource | Ascend 910; OS Euler2.8 |
|
||||
| uploaded Date | 01/18/2021 (month/day/year) |
|
||||
| MindSpore Version | 1.1.0 |
|
||||
| Dataset | Multi30k Dataset |
|
||||
| Training Parameters | epoch=30, batch_size=16 |
|
||||
| Optimizer | Adam |
|
||||
| Loss Function | NLLLoss |
|
||||
| outputs | probability |
|
||||
| Speed | 50ms/step (1pcs) |
|
||||
| Epoch Time | 13.4s (1pcs) |
|
||||
| Loss | 2.5984 |
|
||||
| Params (M) | 21 |
|
||||
| Checkpoint for inference | 272M (.ckpt file) |
|
||||
| Scripts | [gru](https://gitee.com/mindspore/mindspore/tree/master/model_zoo/official/nlp/gru) |
|
||||
| Parameters | Ascend | GPU |
|
||||
| -------------------------- | ----------------------------- |---------------------------|
|
||||
| Resource | Ascend 910; OS Euler2.8 | GTX1080Ti, Ubuntu 18.04 |
|
||||
| uploaded Date | 06/05/2021 (month/day/year) | 06/05/2021 (month/day/year) |
|
||||
| MindSpore Version | 1.2.0 |1.2.0 |
|
||||
| Dataset | Multi30k Dataset | Multi30k Dataset |
|
||||
| Training Parameters | epoch=30, batch_size=16 | epoch=30, batch_size=16 |
|
||||
| Optimizer | Adam | Adam |
|
||||
| Loss Function | NLLLoss | NLLLoss |
|
||||
| outputs | probability | probability |
|
||||
| Speed | 35ms/step (1pcs) | 200ms/step (1pcs) |
|
||||
| Epoch Time | 64.4s (1pcs) | 361.5s (1pcs) |
|
||||
| Loss | 3.86888 |2.533958 |
|
||||
| Params (M) | 21 | 21 |
|
||||
| Checkpoint for inference | 272M (.ckpt file) | 272M (.ckpt file) |
|
||||
| Scripts | [gru](https://gitee.com/mindspore/mindspore/tree/master/model_zoo/official/nlp/gru) |[gru](https://gitee.com/mindspore/mindspore/tree/master/model_zoo/official/nlp/gru) |
|
||||
|
||||
### Inference Performance
|
||||
|
||||
| Parameters | Ascend |
|
||||
| ------------------- | --------------------------- |
|
||||
| Resource | Ascend 910; OS Euler2.8 |
|
||||
| Uploaded Date | 01/18/2020 (month/day/year) |
|
||||
| MindSpore Version | 1.1.0 |
|
||||
| Dataset | Multi30K |
|
||||
| batch_size | 1 |
|
||||
| outputs | label index |
|
||||
| Accuracy | BLEU: 30.30 |
|
||||
| Model for inference | 272M (.ckpt file) |
|
||||
| Parameters | Ascend | GPU |
|
||||
| ------------------- | --------------------------- |---------------------------|
|
||||
| Resource | Ascend 910; OS Euler2.8 | GTX1080Ti, Ubuntu 18.04 |
|
||||
| Uploaded Date | 06/05/2021 (month/day/year) | 06/05/2021 (month/day/year)|
|
||||
| MindSpore Version | 1.2.0 | 1.2.0 |
|
||||
| Dataset | Multi30K | Multi30K |
|
||||
| batch_size | 1 | 1 |
|
||||
| outputs | label index | label index |
|
||||
| Accuracy | BLEU: 31.26 | BLEU: 29.30 |
|
||||
| Model for inference | 272M (.ckpt file) | 272M (.ckpt file) |
|
||||
|
||||
# [Random Situation Description](#content)
|
||||
|
||||
|
|
|
|||
|
|
@ -36,6 +36,8 @@ scale_factor: 2
|
|||
scale_window: 2000
|
||||
warmup_ratio: 0.333333
|
||||
teacher_force_ratio: 0.5
|
||||
compute_type: mstype.float16
|
||||
dtype: mstype.float32
|
||||
|
||||
run_distribute: False
|
||||
dataset_path: ""
|
||||
|
|
|
|||
|
|
@ -20,6 +20,8 @@ import ast
|
|||
import argparse
|
||||
from pprint import pformat
|
||||
import yaml
|
||||
import mindspore.common.dtype as mstype
|
||||
|
||||
|
||||
class Config:
|
||||
"""
|
||||
|
|
@ -108,6 +110,24 @@ def merge(args, cfg):
|
|||
cfg[item] = args_var[item]
|
||||
return cfg
|
||||
|
||||
def parse_dtype(dtype):
|
||||
if dtype not in ["mstype.float32", "mstype.float16"]:
|
||||
raise ValueError("Not supported dtype")
|
||||
|
||||
if dtype == "mstype.float32":
|
||||
return mstype.float32
|
||||
if dtype == "mstype.float16":
|
||||
return mstype.float16
|
||||
return None
|
||||
|
||||
def extra_operations(cfg):
|
||||
"""
|
||||
Do extra work on config
|
||||
Args:
|
||||
config: Object after instantiation of class 'Config'.
|
||||
"""
|
||||
cfg.dtype = parse_dtype(cfg.dtype)
|
||||
cfg.compute_type = parse_dtype(cfg.compute_type)
|
||||
|
||||
def get_config():
|
||||
"""
|
||||
|
|
@ -121,6 +141,8 @@ def get_config():
|
|||
default, helper, choices = parse_yaml(path_args.config_path)
|
||||
args = parse_cli_to_yaml(parser=parser, cfg=default, helper=helper, choices=choices, cfg_path=path_args.config_path)
|
||||
final_config = merge(args, default)
|
||||
return Config(final_config)
|
||||
final_config = Config(final_config)
|
||||
extra_operations(final_config)
|
||||
return final_config
|
||||
|
||||
config = get_config()
|
||||
|
|
|
|||
|
|
@ -1,3 +0,0 @@
|
|||
nltk
|
||||
numpy
|
||||
pyyaml
|
||||
|
|
@ -17,7 +17,6 @@ echo "==========================================================================
|
|||
echo "Please run the script as: "
|
||||
echo "sh create_dataset.sh DATASET_PATH OUTPUT_PATH"
|
||||
echo "for example: sh create_dataset.sh /path/multi30k/ /path/multi30k/mindrecord/"
|
||||
echo "DATASET_NAME including ag, dbpedia, and yelp_p"
|
||||
echo "It is better to use absolute path."
|
||||
echo "=============================================================================================================="
|
||||
ulimit -u unlimited
|
||||
|
|
|
|||
|
|
@ -0,0 +1,33 @@
|
|||
#!/bin/bash
|
||||
# Copyright 2021 Huawei Technologies Co., Ltd
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
# ============================================================================
|
||||
CUR_PATH=`pwd`
|
||||
DATA_PATH=${CUR_PATH}/../data
|
||||
TRAIN_URL=http://www.quest.dcs.shef.ac.uk/wmt16_files_mmt/training.tar.gz
|
||||
VALID_URL=http://www.quest.dcs.shef.ac.uk/wmt16_files_mmt/validation.tar.gz
|
||||
TEST_URL=http://www.quest.dcs.shef.ac.uk/wmt16_files_mmt/mmt16_task1_test.tar.gz
|
||||
|
||||
mkdir ${DATA_PATH}
|
||||
cd ${DATA_PATH}
|
||||
wget --no-check-certificate ${TRAIN_URL}
|
||||
wget --no-check-certificate ${VALID_URL}
|
||||
wget --no-check-certificate ${TEST_URL}
|
||||
tar xvf training.tar.gz
|
||||
tar xvf validation.tar.gz
|
||||
tar xvf mmt16_task1_test.tar.gz
|
||||
/bin/rm training.tar.gz
|
||||
/bin/rm validation.tar.gz
|
||||
/bin/rm mmt16_task1_test.tar.gz
|
||||
|
||||
|
|
@ -47,6 +47,7 @@ exit 1
|
|||
fi
|
||||
|
||||
ulimit -u unlimited
|
||||
export DEVICE_TARGET="Ascend"
|
||||
export DEVICE_NUM=8
|
||||
export RANK_SIZE=8
|
||||
export RANK_TABLE_FILE=$PATH1
|
||||
|
|
@ -65,6 +66,6 @@ do
|
|||
cd ./train_parallel$i || exit
|
||||
echo "start training for rank $RANK_ID, device $DEVICE_ID"
|
||||
env > env.log
|
||||
python train.py --run_distribute=True --dataset_path=$DATASET_PATH &> log &
|
||||
python train.py --device_target=$DEVICE_TARGET --run_distribute=True --dataset_path=$DATASET_PATH &> log &
|
||||
cd ..
|
||||
done
|
||||
done
|
||||
|
|
|
|||
|
|
@ -0,0 +1,54 @@
|
|||
#!/bin/bash
|
||||
# Copyright 2021 Huawei Technologies Co., Ltd
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
# ============================================================================
|
||||
|
||||
if [ $# -ne 1 ]
|
||||
then
|
||||
echo "Usage: sh run_distribute_train_gpu.sh [DATASET_PATH]"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
get_real_path(){
|
||||
if [ "${1:0:1}" == "/" ]; then
|
||||
echo "$1"
|
||||
else
|
||||
echo "$(realpath -m $PWD/$1)"
|
||||
fi
|
||||
}
|
||||
|
||||
DATASET_PATH=$(get_real_path $1)
|
||||
echo $DATASET_PATH
|
||||
if [ ! -f $DATASET_PATH ]
|
||||
then
|
||||
echo "error: DATASET_PATH=$DATASET_PATH is not a file"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
ulimit -u unlimited
|
||||
export DEVICE_TARGET="GPU"
|
||||
export DEVICE_NUM=8
|
||||
|
||||
rm -rf ./train
|
||||
mkdir ./train
|
||||
cp ../*.py ./train
|
||||
cp ../*.yaml ./train
|
||||
cp *.sh ./train
|
||||
cp -r ../src ./train
|
||||
cp -r ../model_utils ./train
|
||||
cd ./train || exit
|
||||
echo "start training for $DEVICE_NUM GPUs"
|
||||
env > env.log
|
||||
mpirun --allow-run-as-root -n $DEVICE_NUM python train.py --run_distribute=True --device_target=$DEVICE_TARGET --dataset_path=$DATASET_PATH &> log &
|
||||
cd ..
|
||||
|
|
@ -0,0 +1,62 @@
|
|||
#!/bin/bash
|
||||
# Copyright 2021 Huawei Technologies Co., Ltd
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
# ============================================================================
|
||||
if [ $# -ne 2 ]
|
||||
then
|
||||
echo "Usage: sh run_eval_ascend.sh [CKPT_FILE] [DATASET_PATH]"
|
||||
exit 1
|
||||
fi
|
||||
ulimit -u unlimited
|
||||
export DEVICE_NUM=1
|
||||
export DEVICE_ID=0
|
||||
export RANK_ID=0
|
||||
export RANK_SIZE=1
|
||||
export DEVICE_TARGET="Ascend"
|
||||
|
||||
get_real_path(){
|
||||
if [ "${1:0:1}" == "/" ]; then
|
||||
echo "$1"
|
||||
else
|
||||
echo "$(realpath -m $PWD/$1)"
|
||||
fi
|
||||
}
|
||||
|
||||
CKPT_FILE=$(get_real_path $1)
|
||||
echo $CKPT_FILE
|
||||
if [ ! -f $CKPT_FILE ]
|
||||
then
|
||||
echo "error: CKPT_FILE=$CKPT_FILE is not a file"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
DATASET_PATH=$(get_real_path $2)
|
||||
echo $DATASET_PATH
|
||||
if [ ! -f $DATASET_PATH ]
|
||||
then
|
||||
echo "error: DATASET_PATH=$DATASET_PATH is not a file"
|
||||
exit 1
|
||||
fi
|
||||
rm -rf ./eval
|
||||
mkdir ./eval
|
||||
cp ../*.py ./eval
|
||||
cp ../*.yaml ./eval
|
||||
cp *.sh ./eval
|
||||
cp -r ../src ./eval
|
||||
cp -r ../model_utils ./eval
|
||||
cd ./eval || exit
|
||||
echo "start eval for device $DEVICE_ID"
|
||||
env > env.log
|
||||
python eval.py --device_target=$DEVICE_TARGET --ckpt_file=$CKPT_FILE --dataset_path=$DATASET_PATH &> log &
|
||||
cd ..
|
||||
|
|
@ -15,7 +15,7 @@
|
|||
# ============================================================================
|
||||
if [ $# -ne 2 ]
|
||||
then
|
||||
echo "Usage: sh run_eval.sh [CKPT_FILE] [DATASET_PATH]"
|
||||
echo "Usage: sh run_eval_gpu.sh [CKPT_FILE] [DATASET_PATH]"
|
||||
exit 1
|
||||
fi
|
||||
ulimit -u unlimited
|
||||
|
|
@ -23,6 +23,8 @@ export DEVICE_NUM=1
|
|||
export DEVICE_ID=0
|
||||
export RANK_ID=0
|
||||
export RANK_SIZE=1
|
||||
export DEVICE_TARGET="GPU"
|
||||
|
||||
get_real_path(){
|
||||
if [ "${1:0:1}" == "/" ]; then
|
||||
echo "$1"
|
||||
|
|
@ -56,5 +58,5 @@ cp -r ../model_utils ./eval
|
|||
cd ./eval || exit
|
||||
echo "start eval for device $DEVICE_ID"
|
||||
env > env.log
|
||||
python eval.py --ckpt_file=$CKPT_FILE --dataset_path=$DATASET_PATH &> log &
|
||||
python eval.py --device_target=$DEVICE_TARGET --ckpt_file=$CKPT_FILE --dataset_path=$DATASET_PATH &> log &
|
||||
cd ..
|
||||
|
|
@ -0,0 +1,54 @@
|
|||
#!/bin/bash
|
||||
# Copyright 2021 Huawei Technologies Co., Ltd
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
# ============================================================================
|
||||
if [ $# -ne 1 ]
|
||||
then
|
||||
echo "Usage: sh run_standalone_train_ascend.sh [DATASET_PATH]"
|
||||
exit 1
|
||||
fi
|
||||
ulimit -u unlimited
|
||||
export DEVICE_NUM=1
|
||||
export DEVICE_ID=0
|
||||
export RANK_ID=0
|
||||
export RANK_SIZE=1
|
||||
export DEVICE_TARGET="Ascend"
|
||||
get_real_path(){
|
||||
if [ "${1:0:1}" == "/" ]; then
|
||||
echo "$1"
|
||||
else
|
||||
echo "$(realpath -m $PWD/$1)"
|
||||
fi
|
||||
}
|
||||
|
||||
DATASET_PATH=$(get_real_path $1)
|
||||
echo $DATASET_PATH
|
||||
if [ ! -f $DATASET_PATH ]
|
||||
then
|
||||
echo "error: DATASET_PATH=$DATASET_PATH is not a file"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
rm -rf ./train
|
||||
mkdir ./train
|
||||
cp ../*.py ./train
|
||||
cp ../*.yaml ./train
|
||||
cp *.sh ./train
|
||||
cp -r ../src ./train
|
||||
cp -r ../model_utils ./train
|
||||
cd ./train || exit
|
||||
echo "start training for device $DEVICE_ID"
|
||||
env > env.log
|
||||
python train.py --device_target=$DEVICE_TARGET --dataset_path=$DATASET_PATH &> log &
|
||||
cd ..
|
||||
|
|
@ -15,14 +15,15 @@
|
|||
# ============================================================================
|
||||
if [ $# -ne 1 ]
|
||||
then
|
||||
echo "Usage: sh run_distribute_train_ascend.sh [DATASET_PATH]"
|
||||
echo "Usage: sh run_standalone_train_gpu.sh [DATASET_PATH]"
|
||||
exit 1
|
||||
fi
|
||||
ulimit -u unlimited
|
||||
export DEVICE_NUM=1
|
||||
export DEVICE_ID=4
|
||||
export DEVICE_ID=0
|
||||
export RANK_ID=0
|
||||
export RANK_SIZE=1
|
||||
export DEVICE_TARGET="GPU"
|
||||
get_real_path(){
|
||||
if [ "${1:0:1}" == "/" ]; then
|
||||
echo "$1"
|
||||
|
|
@ -38,7 +39,6 @@ then
|
|||
echo "error: DATASET_PATH=$DATASET_PATH is not a file"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
rm -rf ./train
|
||||
mkdir ./train
|
||||
cp ../*.py ./train
|
||||
|
|
@ -49,5 +49,5 @@ cp -r ../model_utils ./train
|
|||
cd ./train || exit
|
||||
echo "start training for device $DEVICE_ID"
|
||||
env > env.log
|
||||
python train.py --dataset_path=$DATASET_PATH &> log &
|
||||
python train.py --device_target=$DEVICE_TARGET --dataset_path=$DATASET_PATH &> log &
|
||||
cd ..
|
||||
|
|
@ -1,104 +0,0 @@
|
|||
# Copyright 2021 Huawei Technologies Co., Ltd
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
# ============================================================================
|
||||
"""GRU cell"""
|
||||
import mindspore.nn as nn
|
||||
import mindspore.ops.operations as P
|
||||
import mindspore.common.dtype as mstype
|
||||
from src.weight_init import gru_default_state
|
||||
|
||||
class BidirectionGRU(nn.Cell):
|
||||
'''
|
||||
BidirectionGRU model
|
||||
|
||||
Args:
|
||||
config: config of network
|
||||
'''
|
||||
def __init__(self, config, is_training=True):
|
||||
super(BidirectionGRU, self).__init__()
|
||||
if is_training:
|
||||
self.batch_size = config.batch_size
|
||||
else:
|
||||
self.batch_size = config.eval_batch_size
|
||||
self.embedding_size = config.encoder_embedding_size
|
||||
self.hidden_size = config.hidden_size
|
||||
self.weight_i, self.weight_h, self.bias_i, self.bias_h, self.init_h = gru_default_state(self.batch_size,
|
||||
self.embedding_size,
|
||||
self.hidden_size)
|
||||
self.weight_bw_i, self.weight_bw_h, self.bias_bw_i, self.bias_bw_h, self.init_bw_h = \
|
||||
gru_default_state(self.batch_size, self.embedding_size, self.hidden_size)
|
||||
self.reverse = P.ReverseV2(axis=[1])
|
||||
self.concat = P.Concat(axis=2)
|
||||
self.squeeze = P.Squeeze(axis=0)
|
||||
self.rnn = P.DynamicGRUV2()
|
||||
self.text_len = config.max_length
|
||||
self.cast = P.Cast()
|
||||
|
||||
def construct(self, x):
|
||||
'''
|
||||
BidirectionGRU construction
|
||||
|
||||
Args:
|
||||
x(Tensor): BidirectionGRU input
|
||||
|
||||
Returns:
|
||||
output(Tensor): rnn output
|
||||
hidden(Tensor): hidden state
|
||||
'''
|
||||
x = self.cast(x, mstype.float16)
|
||||
y1, _, _, _, _, _ = self.rnn(x, self.weight_i, self.weight_h, self.bias_i, self.bias_h, None, self.init_h)
|
||||
bw_x = self.reverse(x)
|
||||
y1_bw, _, _, _, _, _ = self.rnn(bw_x, self.weight_bw_i,
|
||||
self.weight_bw_h, self.bias_bw_i, self.bias_bw_h, None, self.init_bw_h)
|
||||
y1_bw = self.reverse(y1_bw)
|
||||
output = self.concat((y1, y1_bw))
|
||||
hidden = self.concat((y1[self.text_len-1:self.text_len:1, ::, ::],
|
||||
y1_bw[self.text_len-1:self.text_len:1, ::, ::]))
|
||||
hidden = self.squeeze(hidden)
|
||||
return output, hidden
|
||||
|
||||
class GRU(nn.Cell):
|
||||
'''
|
||||
GRU model
|
||||
|
||||
Args:
|
||||
config: config of network
|
||||
'''
|
||||
def __init__(self, config, is_training=True):
|
||||
super(GRU, self).__init__()
|
||||
if is_training:
|
||||
self.batch_size = config.batch_size
|
||||
else:
|
||||
self.batch_size = config.eval_batch_size
|
||||
self.embedding_size = config.encoder_embedding_size
|
||||
self.hidden_size = config.hidden_size
|
||||
self.weight_i, self.weight_h, self.bias_i, self.bias_h, self.init_h = \
|
||||
gru_default_state(self.batch_size, self.embedding_size + self.hidden_size*2, self.hidden_size)
|
||||
self.rnn = P.DynamicGRUV2()
|
||||
self.cast = P.Cast()
|
||||
|
||||
def construct(self, x):
|
||||
'''
|
||||
GRU construction
|
||||
|
||||
Args:
|
||||
x(Tensor): GRU input
|
||||
|
||||
Returns:
|
||||
output(Tensor): rnn output
|
||||
hidden(Tensor): hidden state
|
||||
'''
|
||||
x = self.cast(x, mstype.float16)
|
||||
y1, h1, _, _, _, _ = self.rnn(x, self.weight_i, self.weight_h, self.bias_i, self.bias_h, None, self.init_h)
|
||||
return y1, h1
|
||||
|
|
@ -240,3 +240,48 @@ class GRUTrainOneStepWithLossScaleCell(nn.Cell):
|
|||
succ = self.optimizer(grads)
|
||||
ret = (loss, cond, scaling_sens)
|
||||
return F.depend(ret, succ)
|
||||
|
||||
class GRUTrainOneStepCell(nn.TrainOneStepCell):
|
||||
"""
|
||||
Encapsulation class of GRU network training.
|
||||
Append an optimizer to the training network after that the construct
|
||||
function can be called to create the backward graph.
|
||||
Args:
|
||||
network (Cell): The training network. Note that loss function should have been added.
|
||||
optimizer (Optimizer): Optimizer for updating the weights.
|
||||
sens (Number): The adjust parameter. Default: 1.0.
|
||||
enable_clip_grad (boolean): If True, clip gradients in GRUTrainOneStepCell. Default: True.
|
||||
"""
|
||||
|
||||
def __init__(self, network, optimizer, sens=1.0, enable_clip_grad=True):
|
||||
super(GRUTrainOneStepCell, self).__init__(network, optimizer, sens)
|
||||
self.cast = P.Cast()
|
||||
self.hyper_map = C.HyperMap()
|
||||
self.clip_gradients = ClipGradients()
|
||||
self.enable_clip_grad = enable_clip_grad
|
||||
|
||||
def set_sens(self, value):
|
||||
self.sens = value
|
||||
|
||||
def construct(self,
|
||||
encoder_inputs,
|
||||
decoder_inputs,
|
||||
teacher_force,
|
||||
sens=None):
|
||||
"""Defines the computation performed."""
|
||||
|
||||
weights = self.weights
|
||||
loss = self.network(encoder_inputs,
|
||||
decoder_inputs,
|
||||
teacher_force)
|
||||
|
||||
grads = self.grad(self.network, weights)(encoder_inputs,
|
||||
decoder_inputs,
|
||||
teacher_force,
|
||||
self.cast(F.tuple_to_array((self.sens,)),
|
||||
mstype.float32))
|
||||
if self.enable_clip_grad:
|
||||
grads = self.clip_gradients(grads, GRADIENT_CLIP_TYPE, GRADIENT_CLIP_VALUE)
|
||||
grads = self.grad_reducer(grads)
|
||||
succ = self.optimizer(grads)
|
||||
return F.depend(loss, succ)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,130 @@
|
|||
# Copyright 2021 Huawei Technologies Co., Ltd
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
# ============================================================================
|
||||
'''RNN Cells module, include RNNCell, GRUCell, LSTMCell'''
|
||||
import math
|
||||
import numpy as np
|
||||
import mindspore.nn as nn
|
||||
import mindspore.ops as P
|
||||
from mindspore import Tensor, Parameter
|
||||
from mindspore.common.initializer import initializer, Uniform
|
||||
|
||||
def rnn_tanh_cell(inputs, hidden, w_ih, w_hh, b_ih, b_hh):
|
||||
'''RNN cell function with tanh activation'''
|
||||
if b_ih is None:
|
||||
igates = P.MatMul(False, True)(inputs, w_ih)
|
||||
hgates = P.MatMul(False, True)(hidden, w_hh)
|
||||
else:
|
||||
igates = P.MatMul(False, True)(inputs, w_ih) + b_ih
|
||||
hgates = P.MatMul(False, True)(hidden, w_hh) + b_hh
|
||||
return P.Tanh()(igates + hgates)
|
||||
|
||||
def rnn_relu_cell(inputs, hidden, w_ih, w_hh, b_ih, b_hh):
|
||||
'''RNN cell function with relu activation'''
|
||||
if b_ih is None:
|
||||
igates = P.MatMul(False, True)(inputs, w_ih)
|
||||
hgates = P.MatMul(False, True)(hidden, w_hh)
|
||||
else:
|
||||
igates = P.MatMul(False, True)(inputs, w_ih) + b_ih
|
||||
hgates = P.MatMul(False, True)(hidden, w_hh) + b_hh
|
||||
return P.ReLU()(igates + hgates)
|
||||
|
||||
def lstm_cell(inputs, hidden, w_ih, w_hh, b_ih, b_hh):
|
||||
'''LSTM cell function'''
|
||||
hx, cx = hidden
|
||||
if b_ih is None:
|
||||
gates = P.MatMul(False, True)(inputs, w_ih) + P.MatMul(False, True)(hx, w_hh)
|
||||
else:
|
||||
gates = P.MatMul(False, True)(inputs, w_ih) + P.MatMul(False, True)(hx, w_hh) + b_ih + b_hh
|
||||
ingate, forgetgate, cellgate, outgate = P.Split(1, 4)(gates)
|
||||
|
||||
ingate = P.Sigmoid()(ingate)
|
||||
forgetgate = P.Sigmoid()(forgetgate)
|
||||
cellgate = P.Tanh()(cellgate)
|
||||
outgate = P.Sigmoid()(outgate)
|
||||
|
||||
cy = (forgetgate * cx) + (ingate * cellgate)
|
||||
hy = outgate * P.Tanh()(cy)
|
||||
|
||||
return hy, cy
|
||||
|
||||
def gru_cell(inputs, hidden, w_ih, w_hh, b_ih, b_hh):
|
||||
'''GRU cell function'''
|
||||
if b_ih is None:
|
||||
gi = P.MatMul(False, True)(inputs, w_ih)
|
||||
gh = P.MatMul(False, True)(hidden, w_hh)
|
||||
else:
|
||||
gi = P.MatMul(False, True)(inputs, w_ih) + b_ih
|
||||
gh = P.MatMul(False, True)(hidden, w_hh) + b_hh
|
||||
i_r, i_i, i_n = P.Split(1, 3)(gi)
|
||||
h_r, h_i, h_n = P.Split(1, 3)(gh)
|
||||
|
||||
resetgate = P.Sigmoid()(i_r + h_r)
|
||||
inputgate = P.Sigmoid()(i_i + h_i)
|
||||
newgate = P.Tanh()(i_n + resetgate * h_n)
|
||||
hy = newgate + inputgate * (hidden - newgate)
|
||||
|
||||
return hy
|
||||
|
||||
class RNNCellBase(nn.Cell):
|
||||
'''Basic class for RNN Cells'''
|
||||
def __init__(self, input_size: int, hidden_size: int, bias: bool, num_chunks: int):
|
||||
super().__init__()
|
||||
self.input_size = input_size
|
||||
self.hidden_size = hidden_size
|
||||
self.bias = bias
|
||||
self.weight_ih = Parameter(Tensor(np.random.randn(num_chunks * hidden_size, input_size).astype(np.float32)))
|
||||
self.weight_hh = Parameter(Tensor(np.random.randn(num_chunks * hidden_size, hidden_size).astype(np.float32)))
|
||||
if bias:
|
||||
self.bias_ih = Parameter(Tensor(np.random.randn(num_chunks * hidden_size).astype(np.float32)))
|
||||
self.bias_hh = Parameter(Tensor(np.random.randn(num_chunks * hidden_size).astype(np.float32)))
|
||||
self.reset_parameters()
|
||||
|
||||
def reset_parameters(self):
|
||||
stdv = 1 / math.sqrt(self.hidden_size)
|
||||
for weight in self.get_parameters():
|
||||
weight.set_data(initializer(Uniform(stdv), weight.shape))
|
||||
|
||||
class RNNCell(RNNCellBase):
|
||||
'''RNNCell operator class'''
|
||||
_non_linearity = ['tanh', 'relu']
|
||||
def __init__(self, input_size: int, hidden_size: int, bias: bool = True, nonlinearity: str = "tanh"):
|
||||
super().__init__(input_size, hidden_size, bias, num_chunks=1)
|
||||
if nonlinearity not in self._non_linearity:
|
||||
raise ValueError("Unknown nonlinearity: {}".format(nonlinearity))
|
||||
self.nonlinearity = nonlinearity
|
||||
|
||||
def construct(self, inputs, hx):
|
||||
if self.nonlinearity == "tanh":
|
||||
ret = rnn_tanh_cell(inputs, hx, self.weight_ih, self.weight_hh, self.bias_ih, self.bias_hh)
|
||||
else:
|
||||
ret = rnn_relu_cell(inputs, hx, self.weight_ih, self.weight_hh, self.bias_ih, self.bias_hh)
|
||||
return ret
|
||||
|
||||
class LSTMCell(RNNCellBase):
|
||||
'''LSTMCell operator class'''
|
||||
def __init__(self, input_size: int, hidden_size: int, bias: bool = True):
|
||||
super().__init__(input_size, hidden_size, bias, num_chunks=4)
|
||||
self.support_non_tensor_inputs = True
|
||||
|
||||
def construct(self, inputs, hx):
|
||||
return lstm_cell(inputs, hx, self.weight_ih, self.weight_hh, self.bias_ih, self.bias_hh)
|
||||
|
||||
class GRUCell(RNNCellBase):
|
||||
'''GRUCell operator class'''
|
||||
def __init__(self, input_size: int, hidden_size: int, bias: bool = True):
|
||||
super().__init__(input_size, hidden_size, bias, num_chunks=3)
|
||||
|
||||
def construct(self, inputs, hx):
|
||||
return gru_cell(inputs, hx, self.weight_ih, self.weight_hh, self.bias_ih, self.bias_hh)
|
||||
|
|
@ -0,0 +1,311 @@
|
|||
# Copyright 2021 Huawei Technologies Co., Ltd
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
# ============================================================================
|
||||
'''RNN operators module, include RNN, GRU, LSTM'''
|
||||
import math
|
||||
import numpy as np
|
||||
import mindspore
|
||||
import mindspore.nn as nn
|
||||
import mindspore.ops as P
|
||||
from mindspore.ops.primitive import constexpr
|
||||
from mindspore import Tensor, Parameter, ParameterTuple
|
||||
from mindspore import log as logger
|
||||
from mindspore import context
|
||||
from src.rnn_cells import rnn_relu_cell, rnn_tanh_cell, lstm_cell, gru_cell
|
||||
from src.utils import Reverse, ReverseSequence
|
||||
|
||||
@constexpr
|
||||
def _init_state(shape, dtype, is_lstm):
|
||||
hx = Tensor(np.zeros(shape), dtype)
|
||||
cx = Tensor(np.zeros(shape), dtype)
|
||||
if is_lstm:
|
||||
return (hx, cx)
|
||||
return hx
|
||||
|
||||
class DynamicRNN(nn.Cell):
|
||||
'''Dynamic RNN module to compute RNN cell by timesteps'''
|
||||
def __init__(self, mode):
|
||||
super().__init__()
|
||||
if mode == "RNN_RELU":
|
||||
cell = rnn_relu_cell
|
||||
elif mode == "RNN_TANH":
|
||||
cell = rnn_tanh_cell
|
||||
elif mode == "LSTM":
|
||||
cell = lstm_cell
|
||||
elif mode == "GRU":
|
||||
cell = gru_cell
|
||||
else:
|
||||
raise ValueError("Unrecognized RNN mode: " + mode)
|
||||
self.cell = cell
|
||||
self.is_lstm = mode == "LSTM"
|
||||
|
||||
def recurrent(self, x, h_0, w_ih, w_hh, b_ih, b_hh):
|
||||
'''recurrent steps without sequence length'''
|
||||
time_step = x.shape[0]
|
||||
outputs = []
|
||||
t = 0
|
||||
h = h_0
|
||||
while t < time_step:
|
||||
x_t = x[t:t+1:1]
|
||||
x_t = P.Squeeze(0)(x_t)
|
||||
h = self.cell(x_t, h, w_ih, w_hh, b_ih, b_hh)
|
||||
if self.is_lstm:
|
||||
outputs.append(h[0])
|
||||
else:
|
||||
outputs.append(h)
|
||||
t += 1
|
||||
outputs = P.Stack()(outputs)
|
||||
return outputs, h
|
||||
|
||||
def variable_recurrent(self, x, h, seq_length, w_ih, w_hh, b_ih, b_hh):
|
||||
'''recurrent steps with sequence length'''
|
||||
time_step = x.shape[0]
|
||||
h_t = h
|
||||
if self.is_lstm:
|
||||
hidden_size = h[0].shape[-1]
|
||||
zero_output = P.ZerosLike()(h_t[0])
|
||||
else:
|
||||
hidden_size = h.shape[-1]
|
||||
zero_output = P.ZerosLike()(h_t)
|
||||
seq_length = P.Cast()(seq_length, mindspore.float32)
|
||||
seq_length = P.BroadcastTo((hidden_size, -1))(seq_length)
|
||||
seq_length = P.Cast()(seq_length, mindspore.int32)
|
||||
seq_length = P.Transpose()(seq_length, (1, 0))
|
||||
|
||||
outputs = []
|
||||
state_t = h_t
|
||||
t = 0
|
||||
while t < time_step:
|
||||
x_t = x[t:t+1:1]
|
||||
x_t = P.Squeeze(0)(x_t)
|
||||
h_t = self.cell(x_t, state_t, w_ih, w_hh, b_ih, b_hh)
|
||||
seq_cond = seq_length > t
|
||||
if self.is_lstm:
|
||||
state_t_0 = P.Select()(seq_cond, h_t[0], state_t[0])
|
||||
state_t_1 = P.Select()(seq_cond, h_t[1], state_t[1])
|
||||
output = P.Select()(seq_cond, h_t[0], zero_output)
|
||||
state_t = (state_t_0, state_t_1)
|
||||
else:
|
||||
state_t = P.Select()(seq_cond, h_t, state_t)
|
||||
output = P.Select()(seq_cond, h_t, zero_output)
|
||||
outputs.append(output)
|
||||
t += 1
|
||||
outputs = P.Stack()(outputs)
|
||||
return outputs, state_t
|
||||
|
||||
def construct(self, x, h, seq_length, w_ih, w_hh, b_ih, b_hh):
|
||||
if seq_length is None:
|
||||
return self.recurrent(x, h, w_ih, w_hh, b_ih, b_hh)
|
||||
return self.variable_recurrent(x, h, seq_length, w_ih, w_hh, b_ih, b_hh)
|
||||
|
||||
class RNNBase(nn.Cell):
|
||||
'''Basic class for RNN operators'''
|
||||
def __init__(self, mode, input_size, hidden_size, num_layers=1, has_bias=True,
|
||||
batch_first=False, dropout=0, bidirectional=False):
|
||||
super().__init__()
|
||||
if not 0 <= dropout <= 1:
|
||||
raise ValueError("dropout should be a number in range [0, 1] "
|
||||
"representing the probability of an element being "
|
||||
"zeroed")
|
||||
|
||||
if dropout > 0 and num_layers == 1:
|
||||
logger.warning("dropout option adds dropout after all but last "
|
||||
"recurrent layer, so non-zero dropout expects "
|
||||
"num_layers greater than 1, but got dropout={} and "
|
||||
"num_layers={}".format(dropout, num_layers))
|
||||
if mode == "LSTM":
|
||||
gate_size = 4 * hidden_size
|
||||
elif mode == "GRU":
|
||||
gate_size = 3 * hidden_size
|
||||
elif mode == "RNN_TANH":
|
||||
gate_size = hidden_size
|
||||
elif mode == "RNN_RELU":
|
||||
gate_size = hidden_size
|
||||
else:
|
||||
raise ValueError("Unrecognized RNN mode: " + mode)
|
||||
|
||||
self.is_ascend = context.get_context("device_target") == "Ascend"
|
||||
if self.is_ascend:
|
||||
self.reverse = P.ReverseV2([0])
|
||||
self.reverse_sequence = P.ReverseSequence(0, 1)
|
||||
else:
|
||||
self.reverse = Reverse(0)
|
||||
self.reverse_sequence = ReverseSequence(0, 1)
|
||||
self.hidden_size = hidden_size
|
||||
self.batch_first = batch_first
|
||||
self.num_layers = num_layers
|
||||
self.dropout = dropout
|
||||
self.dropout_op = nn.Dropout(float(1 - dropout))
|
||||
self.bidirectional = bidirectional
|
||||
self.has_bias = has_bias
|
||||
self.rnn = DynamicRNN(mode)
|
||||
num_directions = 2 if bidirectional else 1
|
||||
self.is_lstm = mode == "LSTM"
|
||||
|
||||
self.w_ih_list = []
|
||||
self.w_hh_list = []
|
||||
self.b_ih_list = []
|
||||
self.b_hh_list = []
|
||||
stdv = 1 / math.sqrt(self.hidden_size)
|
||||
for layer in range(num_layers):
|
||||
for direction in range(num_directions):
|
||||
layer_input_size = input_size if layer == 0 else hidden_size * num_directions
|
||||
suffix = '_reverse' if direction == 1 else ''
|
||||
|
||||
self.w_ih_list.append(Parameter(
|
||||
Tensor(np.random.uniform(-stdv, stdv, (gate_size, layer_input_size)).astype(np.float32)),
|
||||
name='weight_ih_l{}{}'.format(layer, suffix)))
|
||||
self.w_hh_list.append(Parameter(
|
||||
Tensor(np.random.uniform(-stdv, stdv, (gate_size, hidden_size)).astype(np.float32)),
|
||||
name='weight_hh_l{}{}'.format(layer, suffix)))
|
||||
if has_bias:
|
||||
self.b_ih_list.append(Parameter(
|
||||
Tensor(np.random.uniform(-stdv, stdv, (gate_size)).astype(np.float32)),
|
||||
name='bias_ih_l{}{}'.format(layer, suffix)))
|
||||
self.b_hh_list.append(Parameter(
|
||||
Tensor(np.random.uniform(-stdv, stdv, (gate_size)).astype(np.float32)),
|
||||
name='bias_hh_l{}{}'.format(layer, suffix)))
|
||||
self.w_ih_list = ParameterTuple(self.w_ih_list)
|
||||
self.w_hh_list = ParameterTuple(self.w_hh_list)
|
||||
self.b_ih_list = ParameterTuple(self.b_ih_list)
|
||||
self.b_hh_list = ParameterTuple(self.b_hh_list)
|
||||
|
||||
def _stacked_bi_dynamic_rnn(self, x, h, seq_length):
|
||||
"""stacked bidirectional dynamic_rnn"""
|
||||
pre_layer = x
|
||||
h_n = ()
|
||||
c_n = ()
|
||||
output = 0
|
||||
for i in range(self.num_layers):
|
||||
offset = i * 2
|
||||
if self.has_bias:
|
||||
w_f_ih, w_f_hh, b_f_ih, b_f_hh = \
|
||||
self.w_ih_list[offset], self.w_hh_list[offset], \
|
||||
self.b_ih_list[offset], self.b_hh_list[offset]
|
||||
w_b_ih, w_b_hh, b_b_ih, b_b_hh = \
|
||||
self.w_ih_list[offset + 1], self.w_hh_list[offset + 1], \
|
||||
self.b_ih_list[offset + 1], self.b_hh_list[offset + 1]
|
||||
else:
|
||||
w_f_ih, w_f_hh = self.w_ih_list[offset], self.w_hh_list[offset]
|
||||
w_b_ih, w_b_hh = self.w_ih_list[offset + 1], self.w_hh_list[offset + 1]
|
||||
b_f_ih, b_f_hh, b_b_ih, b_b_hh = None, None, None, None
|
||||
if self.is_lstm:
|
||||
h_f_i = (h[0][offset], h[1][offset])
|
||||
h_b_i = (h[0][offset + 1], h[1][offset + 1])
|
||||
else:
|
||||
h_f_i = h[offset]
|
||||
h_b_i = h[offset + 1]
|
||||
if seq_length is None:
|
||||
x_b = self.reverse(pre_layer)
|
||||
else:
|
||||
x_b = self.reverse_sequence(pre_layer, seq_length)
|
||||
output_f, h_t_f = self.rnn(pre_layer, h_f_i, seq_length, w_f_ih, w_f_hh, b_f_ih, b_f_hh)
|
||||
output_b, h_t_b = self.rnn(x_b, h_b_i, seq_length, w_b_ih, w_b_hh, b_b_ih, b_b_hh)
|
||||
if seq_length is None:
|
||||
output_b = self.reverse(output_b)
|
||||
else:
|
||||
output_b = self.reverse_sequence(output_b, seq_length)
|
||||
output = P.Concat(2)((output_f, output_b))
|
||||
pre_layer = self.dropout_op(output) if (self.dropout != 0 and i < self.num_layers - 1) else output
|
||||
if self.is_lstm:
|
||||
h_n += (h_t_f[0], h_t_b[0],)
|
||||
c_n += (h_t_f[1], h_t_b[1],)
|
||||
else:
|
||||
h_n += (h_t_f, h_t_b,)
|
||||
if self.is_lstm:
|
||||
h_n = P.Concat(0)(h_n)
|
||||
c_n = P.Concat(0)(c_n)
|
||||
h_n = h_n.view(h[0].shape)
|
||||
c_n = c_n.view(h[1].shape)
|
||||
return output, (h_n.view(h[0].shape), c_n.view(h[1].shape))
|
||||
h_n = P.Concat(0)(h_n)
|
||||
return output, h_n.view(h.shape)
|
||||
|
||||
def _stacked_dynamic_rnn(self, x, h, seq_length):
|
||||
"""stacked mutil_layer dynamic_rnn"""
|
||||
pre_layer = x
|
||||
h_n = ()
|
||||
c_n = ()
|
||||
output = 0
|
||||
for i in range(self.num_layers):
|
||||
if self.has_bias:
|
||||
w_ih, w_hh, b_ih, b_hh = self.w_ih_list[i], self.w_hh_list[i], self.b_ih_list[i], self.b_hh_list[i]
|
||||
else:
|
||||
w_ih, w_hh = self.w_ih_list[i], self.w_hh_list[i]
|
||||
b_ih, b_hh = None, None
|
||||
if self.is_lstm:
|
||||
h_i = (h[0][i], h[1][i])
|
||||
else:
|
||||
h_i = h[i]
|
||||
output, h_t = self.rnn(pre_layer, h_i, seq_length, w_ih, w_hh, b_ih, b_hh)
|
||||
pre_layer = self.dropout_op(output) if (self.dropout != 0 and i < self.num_layers - 1) else output
|
||||
if self.is_lstm:
|
||||
h_n += (h_t[0],)
|
||||
c_n += (h_t[1],)
|
||||
else:
|
||||
h_n += (h_t,)
|
||||
if self.is_lstm:
|
||||
h_n = P.Concat(0)(h_n)
|
||||
c_n = P.Concat(0)(c_n)
|
||||
h_n = h_n.view(h[0].shape)
|
||||
c_n = c_n.view(h[1].shape)
|
||||
return output, (h_n.view(h[0].shape), c_n.view(h[1].shape))
|
||||
h_n = P.Concat(0)(h_n)
|
||||
return output, h_n.view(h.shape)
|
||||
|
||||
def construct(self, x, h=None, seq_length=None):
|
||||
'''Defines the RNN like operators performed'''
|
||||
max_batch_size = x.shape[0] if self.batch_first else x.shape[1]
|
||||
num_directions = 2 if self.bidirectional else 1
|
||||
if h is None:
|
||||
h = _init_state((self.num_layers * num_directions, max_batch_size, self.hidden_size), x.dtype, self.is_lstm)
|
||||
if self.batch_first:
|
||||
x = P.Transpose()(x, (1, 0, 2))
|
||||
if self.bidirectional:
|
||||
x, h = self._stacked_bi_dynamic_rnn(x, h, seq_length)
|
||||
else:
|
||||
x, h = self._stacked_dynamic_rnn(x, h, seq_length)
|
||||
if self.batch_first:
|
||||
x = P.Transpose()(x, (1, 0, 2))
|
||||
return x, h
|
||||
|
||||
class RNN(RNNBase):
|
||||
'''RNN operator class'''
|
||||
def __init__(self, *args, **kwargs):
|
||||
if 'nonlinearity' in kwargs:
|
||||
if kwargs['nonlinearity'] == 'tanh':
|
||||
mode = 'RNN_TANH'
|
||||
elif kwargs['nonlinearity'] == 'relu':
|
||||
mode = 'RNN_RELU'
|
||||
else:
|
||||
raise ValueError("Unknown nonlinearity '{}'".format(
|
||||
kwargs['nonlinearity']))
|
||||
del kwargs['nonlinearity']
|
||||
else:
|
||||
mode = 'RNN_TANH'
|
||||
|
||||
super(RNN, self).__init__(mode, *args, **kwargs)
|
||||
|
||||
class GRU(RNNBase):
|
||||
'''GRU operator class'''
|
||||
def __init__(self, *args, **kwargs):
|
||||
mode = 'GRU'
|
||||
super(GRU, self).__init__(mode, *args, **kwargs)
|
||||
|
||||
class LSTM(RNNBase):
|
||||
'''LSTM operator class'''
|
||||
def __init__(self, *args, **kwargs):
|
||||
mode = 'LSTM'
|
||||
super(LSTM, self).__init__(mode, *args, **kwargs)
|
||||
self.support_non_tensor_inputs = True
|
||||
|
|
@ -18,8 +18,8 @@ from mindspore import Tensor
|
|||
import mindspore.nn as nn
|
||||
import mindspore.ops.operations as P
|
||||
import mindspore.common.dtype as mstype
|
||||
from src.gru import BidirectionGRU, GRU
|
||||
from src.weight_init import dense_default_state
|
||||
from src.rnns import GRU
|
||||
|
||||
class Attention(nn.Cell):
|
||||
'''
|
||||
|
|
@ -29,8 +29,8 @@ class Attention(nn.Cell):
|
|||
super(Attention, self).__init__()
|
||||
self.text_len = config.max_length
|
||||
self.attn = nn.Dense(in_channels=config.hidden_size * 3,
|
||||
out_channels=config.hidden_size).to_float(mstype.float16)
|
||||
self.fc = nn.Dense(config.hidden_size, 1, has_bias=False).to_float(mstype.float16)
|
||||
out_channels=config.hidden_size).to_float(config.compute_type)
|
||||
self.fc = nn.Dense(config.hidden_size, 1, has_bias=False).to_float(config.compute_type)
|
||||
self.expandims = P.ExpandDims()
|
||||
self.tanh = P.Tanh()
|
||||
self.softmax = P.Softmax()
|
||||
|
|
@ -39,6 +39,9 @@ class Attention(nn.Cell):
|
|||
self.concat = P.Concat(axis=2)
|
||||
self.squeeze = P.Squeeze(axis=2)
|
||||
self.cast = P.Cast()
|
||||
self.dtype = config.dtype
|
||||
self.compute_type = config.compute_type
|
||||
|
||||
def construct(self, hidden, encoder_outputs):
|
||||
'''
|
||||
Attention construction
|
||||
|
|
@ -58,9 +61,9 @@ class Attention(nn.Cell):
|
|||
energy = self.tanh(out)
|
||||
attention = self.fc(energy)
|
||||
attention = self.squeeze(attention)
|
||||
attention = self.cast(attention, mstype.float32)
|
||||
attention = self.cast(attention, self.dtype)
|
||||
attention = self.softmax(attention)
|
||||
attention = self.cast(attention, mstype.float16)
|
||||
attention = self.cast(attention, self.compute_type)
|
||||
return attention
|
||||
|
||||
class Encoder(nn.Cell):
|
||||
|
|
@ -76,8 +79,9 @@ class Encoder(nn.Cell):
|
|||
self.vocab_size = config.src_vocab_size
|
||||
self.embedding_size = config.encoder_embedding_size
|
||||
self.embedding = nn.Embedding(self.vocab_size, self.embedding_size)
|
||||
self.rnn = BidirectionGRU(config, is_training=is_training).to_float(mstype.float16)
|
||||
self.fc = nn.Dense(2*self.hidden_size, self.hidden_size).to_float(mstype.float16)
|
||||
self.rnn = GRU(input_size=self.embedding_size, \
|
||||
hidden_size=self.hidden_size, bidirectional=True).to_float(config.compute_type)
|
||||
self.fc = nn.Dense(2*self.hidden_size, self.hidden_size).to_float(config.compute_type)
|
||||
self.shape = P.Shape()
|
||||
self.transpose = P.Transpose()
|
||||
self.p = P.Print()
|
||||
|
|
@ -85,6 +89,8 @@ class Encoder(nn.Cell):
|
|||
self.text_len = config.max_length
|
||||
self.squeeze = P.Squeeze(axis=0)
|
||||
self.tanh = P.Tanh()
|
||||
self.concat = P.Concat(2)
|
||||
self.dtype = config.dtype
|
||||
|
||||
def construct(self, src):
|
||||
'''
|
||||
|
|
@ -99,8 +105,10 @@ class Encoder(nn.Cell):
|
|||
'''
|
||||
embedded = self.embedding(src)
|
||||
embedded = self.transpose(embedded, (1, 0, 2))
|
||||
embedded = self.cast(embedded, mstype.float16)
|
||||
embedded = self.cast(embedded, self.dtype)
|
||||
output, hidden = self.rnn(embedded)
|
||||
hidden = self.transpose(hidden, (1, 0, 2))
|
||||
hidden = hidden.view(hidden.shape[0], -1)
|
||||
hidden = self.fc(hidden)
|
||||
hidden = self.tanh(hidden)
|
||||
return output, hidden
|
||||
|
|
@ -118,7 +126,8 @@ class Decoder(nn.Cell):
|
|||
self.vocab_size = config.trg_vocab_size
|
||||
self.embedding_size = config.decoder_embedding_size
|
||||
self.embedding = nn.Embedding(self.vocab_size, self.embedding_size)
|
||||
self.rnn = GRU(config, is_training=is_training).to_float(mstype.float16)
|
||||
self.rnn = GRU(input_size=self.embedding_size + self.hidden_size*2, \
|
||||
hidden_size=self.hidden_size).to_float(config.compute_type)
|
||||
self.text_len = config.max_length
|
||||
self.shape = P.Shape()
|
||||
self.transpose = P.Transpose()
|
||||
|
|
@ -130,11 +139,13 @@ class Decoder(nn.Cell):
|
|||
self.log_softmax = P.LogSoftmax(axis=1)
|
||||
weight, bias = dense_default_state(self.embedding_size+self.hidden_size*3, self.vocab_size)
|
||||
self.fc = nn.Dense(self.embedding_size+self.hidden_size*3, self.vocab_size,
|
||||
weight_init=weight, bias_init=bias).to_float(mstype.float16)
|
||||
weight_init=weight, bias_init=bias).to_float(config.compute_type)
|
||||
self.attention = Attention(config)
|
||||
self.bmm = P.BatchMatMul()
|
||||
self.dropout = nn.Dropout(0.7)
|
||||
self.expandims = P.ExpandDims()
|
||||
self.dtype = config.dtype
|
||||
|
||||
def construct(self, inputs, hidden, encoder_outputs):
|
||||
'''
|
||||
Decoder construction
|
||||
|
|
@ -150,21 +161,22 @@ class Decoder(nn.Cell):
|
|||
'''
|
||||
embedded = self.embedding(inputs)
|
||||
embedded = self.transpose(embedded, (1, 0, 2))
|
||||
embedded = self.cast(embedded, mstype.float16)
|
||||
embedded = self.cast(embedded, self.dtype)
|
||||
attn = self.attention(hidden, encoder_outputs)
|
||||
attn = self.expandims(attn, 1)
|
||||
encoder_outputs = self.transpose(encoder_outputs, (1, 0, 2))
|
||||
weight = self.bmm(attn, encoder_outputs)
|
||||
weight = self.transpose(weight, (1, 0, 2))
|
||||
weight = self.cast(weight, self.dtype)
|
||||
emd_con = self.concat((embedded, weight))
|
||||
output, hidden = self.rnn(emd_con)
|
||||
output = self.cast(output, self.dtype)
|
||||
out = self.concat((embedded, output, weight))
|
||||
out = self.squeeze(out)
|
||||
hidden = self.squeeze(hidden)
|
||||
prediction = self.fc(out)
|
||||
prediction = self.dropout(prediction)
|
||||
prediction = self.cast(prediction, mstype.float32)
|
||||
prediction = self.cast(prediction, mstype.float32)
|
||||
prediction = self.cast(prediction, self.dtype)
|
||||
pred_prob = self.log_softmax(prediction)
|
||||
pred_prob = self.expandims(pred_prob, 0)
|
||||
return pred_prob, hidden
|
||||
|
|
|
|||
|
|
@ -0,0 +1,76 @@
|
|||
# Copyright 2021 Huawei Technologies Co., Ltd
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
# ============================================================================
|
||||
'''Utils for GPU version GRU, like Reverse operators'''
|
||||
import mindspore
|
||||
import mindspore.nn as nn
|
||||
import mindspore.ops as ops
|
||||
import mindspore.numpy as np
|
||||
|
||||
class Reverse(nn.Cell):
|
||||
"""Reverse operator, like Reverse in mindspore"""
|
||||
def __init__(self, dim):
|
||||
super().__init__()
|
||||
self.dim = dim
|
||||
|
||||
def construct(self, input_x):
|
||||
shape = input_x.shape
|
||||
dim_size = shape[self.dim]
|
||||
reversed_indexes = np.arange(dim_size-1, -1, -1)
|
||||
output = ops.Gather()(input_x, reversed_indexes, self.dim)
|
||||
return output
|
||||
|
||||
class ReverseSequence(nn.Cell):
|
||||
"""Reverse sequence operator, like ReverseSequenceV2 in mindspore"""
|
||||
def __init__(self, seq_dim, batch_dim=0):
|
||||
super().__init__()
|
||||
self.seq_dim = seq_dim
|
||||
self.batch_dim = batch_dim
|
||||
|
||||
def construct(self, x, seq_lengths):
|
||||
"""Defines the ReverseSequence operator computation performed."""
|
||||
batch_size = x.shape[self.batch_dim]
|
||||
max_seq_len = x.shape[self.seq_dim]
|
||||
seq_lens_type = seq_lengths.dtype
|
||||
|
||||
back = ops.Sub()(seq_lengths, ops.OnesLike()(seq_lengths))
|
||||
|
||||
batch_idx = self.make_shape((batch_size, max_seq_len), seq_lens_type, 0)
|
||||
forward_idx = self.make_shape((batch_size, max_seq_len), seq_lens_type, 1)
|
||||
|
||||
back = back.view(-1, 1)
|
||||
reverse_idx = ops.Sub()(back, forward_idx)
|
||||
|
||||
condition = ops.Less()(reverse_idx, ops.ZerosLike()(reverse_idx))
|
||||
reverse_idx = ops.Select()(condition, forward_idx, reverse_idx)
|
||||
|
||||
reverse_idx = ops.ExpandDims()(reverse_idx, 2)
|
||||
batch_idx = ops.ExpandDims()(batch_idx, 2)
|
||||
|
||||
if self.batch_dim > self.seq_dim:
|
||||
batch_idx = ops.Transpose()(batch_idx, (1, 0, 2))
|
||||
reverse_idx = ops.Transpose()(reverse_idx, (1, 0, 2))
|
||||
x = ops.Transpose()(x, (1, 0, 2))
|
||||
start_indices = ops.Concat(2)((batch_idx, reverse_idx))
|
||||
|
||||
output = ops.GatherNd()(x, start_indices)
|
||||
|
||||
return output
|
||||
|
||||
def make_shape(self, shape, dtype, range_dim):
|
||||
output = ops.Ones()(shape, mindspore.float32)
|
||||
output = ops.CumSum()(output, range_dim)
|
||||
output = ops.Cast()(output, dtype)
|
||||
output = output - 1
|
||||
return output
|
||||
|
|
@ -15,21 +15,7 @@
|
|||
"""weight init"""
|
||||
import math
|
||||
import numpy as np
|
||||
from mindspore import Tensor, Parameter
|
||||
|
||||
def gru_default_state(batch_size, input_size, hidden_size, num_layers=1, bidirectional=False):
|
||||
'''Weight init for gru cell'''
|
||||
stdv = 1 / math.sqrt(hidden_size)
|
||||
weight_i = Parameter(Tensor(
|
||||
np.random.uniform(-stdv, stdv, (input_size, 3*hidden_size)).astype(np.float32)), name='weight_i')
|
||||
weight_h = Parameter(Tensor(
|
||||
np.random.uniform(-stdv, stdv, (hidden_size, 3*hidden_size)).astype(np.float32)), name='weight_h')
|
||||
bias_i = Parameter(Tensor(
|
||||
np.random.uniform(-stdv, stdv, (3*hidden_size)).astype(np.float32)), name='bias_i')
|
||||
bias_h = Parameter(Tensor(
|
||||
np.random.uniform(-stdv, stdv, (3*hidden_size)).astype(np.float32)), name='bias_h')
|
||||
init_h = Tensor(np.zeros((batch_size, hidden_size)).astype(np.float16))
|
||||
return weight_i, weight_h, bias_i, bias_h, init_h
|
||||
from mindspore import Tensor
|
||||
|
||||
def dense_default_state(in_channel, out_channel):
|
||||
'''Weight init for dense cell'''
|
||||
|
|
|
|||
|
|
@ -15,17 +15,19 @@
|
|||
"""train script"""
|
||||
import os
|
||||
import time
|
||||
import mindspore.common.dtype as mstype
|
||||
from mindspore.context import ParallelMode
|
||||
from mindspore import context
|
||||
from mindspore.communication.management import init
|
||||
from mindspore.communication.management import init, get_rank
|
||||
from mindspore.train.callback import Callback, CheckpointConfig, ModelCheckpoint, TimeMonitor
|
||||
from mindspore.train import Model
|
||||
from mindspore.common import set_seed
|
||||
from mindspore.train.loss_scale_manager import DynamicLossScaleManager
|
||||
from mindspore.nn.optim import Adam
|
||||
from mindspore import log as logger
|
||||
|
||||
from src.seq2seq import Seq2Seq
|
||||
from src.gru_for_train import GRUWithLossCell, GRUTrainOneStepWithLossScaleCell
|
||||
from src.gru_for_train import GRUWithLossCell, GRUTrainOneStepWithLossScaleCell, GRUTrainOneStepCell
|
||||
from src.dataset import create_gru_dataset
|
||||
from src.lr_schedule import dynamic_lr
|
||||
|
||||
|
|
@ -72,13 +74,20 @@ class LossCallBack(Callback):
|
|||
cb_params.cur_step_num,
|
||||
str(cb_params.net_outputs)))
|
||||
with open("./loss_{}.log".format(self.rank_id), "a+") as f:
|
||||
f.write("time: {}, epoch: {}, step: {}, loss: {}, overflow: {}, loss_scale: {}".format(
|
||||
time_stamp_current - time_stamp_first,
|
||||
cb_params.cur_epoch_num,
|
||||
cb_params.cur_step_num,
|
||||
str(cb_params.net_outputs[0].asnumpy()),
|
||||
str(cb_params.net_outputs[1].asnumpy()),
|
||||
str(cb_params.net_outputs[2].asnumpy())))
|
||||
if context.get_context("device_target") == "Ascend":
|
||||
f.write("time: {}, epoch: {}, step: {}, loss: {}, overflow: {}, loss_scale: {}".format(
|
||||
time_stamp_current - time_stamp_first,
|
||||
cb_params.cur_epoch_num,
|
||||
cb_params.cur_step_num,
|
||||
str(cb_params.net_outputs[0].asnumpy()),
|
||||
str(cb_params.net_outputs[1].asnumpy()),
|
||||
str(cb_params.net_outputs[2].asnumpy())))
|
||||
else:
|
||||
f.write("time: {}, epoch: {}, step: {}, loss: {}".format(
|
||||
time_stamp_current - time_stamp_first,
|
||||
cb_params.cur_epoch_num,
|
||||
cb_params.cur_step_num,
|
||||
str(cb_params.net_outputs.asnumpy())))
|
||||
f.write('\n')
|
||||
|
||||
|
||||
|
|
@ -139,13 +148,32 @@ def modelarts_pre_process():
|
|||
@moxing_wrapper(pre_process=modelarts_pre_process)
|
||||
def run_train():
|
||||
"""run train."""
|
||||
context.set_context(mode=context.GRAPH_MODE, device_target="Ascend", device_id=get_device_id(), save_graphs=False)
|
||||
rank = get_rank_id()
|
||||
context.set_context(mode=context.GRAPH_MODE, device_target=config.device_target,
|
||||
device_id=get_device_id(), save_graphs=False)
|
||||
if config.device_target == "GPU":
|
||||
if config.compute_type != mstype.float32:
|
||||
logger.warning('GPU only support fp32 temporarily, run with fp32.')
|
||||
config.compute_type = mstype.float32
|
||||
|
||||
device_num = get_device_num()
|
||||
if config.run_distribute:
|
||||
context.set_auto_parallel_context(device_num=device_num, parallel_mode=ParallelMode.DATA_PARALLEL,
|
||||
gradients_mean=True)
|
||||
init()
|
||||
if config.device_target == "Ascend":
|
||||
rank = get_rank_id()
|
||||
context.set_auto_parallel_context(device_num=device_num,
|
||||
parallel_mode=ParallelMode.DATA_PARALLEL,
|
||||
gradients_mean=True)
|
||||
init()
|
||||
elif config.device_target == "GPU":
|
||||
rank = get_rank()
|
||||
init("nccl")
|
||||
context.set_auto_parallel_context(parallel_mode=ParallelMode.DATA_PARALLEL,
|
||||
gradients_mean=True)
|
||||
else:
|
||||
raise ValueError(config.device_target)
|
||||
else:
|
||||
rank = 0
|
||||
device_num = 1
|
||||
|
||||
mindrecord_file = config.dataset_path
|
||||
if not os.path.exists(mindrecord_file):
|
||||
print("dataset file {} not exists, please check!".format(mindrecord_file))
|
||||
|
|
@ -162,8 +190,10 @@ def run_train():
|
|||
scale_factor=config.scale_factor,
|
||||
scale_window=config.scale_window)
|
||||
update_cell = scale_manager.get_update_cell()
|
||||
netwithgrads = GRUTrainOneStepWithLossScaleCell(network, opt, update_cell)
|
||||
|
||||
if config.device_target == "Ascend":
|
||||
netwithgrads = GRUTrainOneStepWithLossScaleCell(network, opt, update_cell)
|
||||
else:
|
||||
netwithgrads = GRUTrainOneStepCell(network, opt)
|
||||
time_cb = TimeMonitor(data_size=dataset_size)
|
||||
loss_cb = LossCallBack(rank_id=rank)
|
||||
cb = [time_cb, loss_cb]
|
||||
|
|
@ -171,10 +201,10 @@ def run_train():
|
|||
if config.save_checkpoint:
|
||||
ckpt_config = CheckpointConfig(save_checkpoint_steps=config.ckpt_epoch * dataset_size,
|
||||
keep_checkpoint_max=config.keep_checkpoint_max)
|
||||
save_ckpt_path = os.path.join(config.outputs_dir, 'ckpt_' + str(get_rank_id()) + '/')
|
||||
save_ckpt_path = os.path.join(config.outputs_dir, 'ckpt_' + str(rank) + '/')
|
||||
ckpt_cb = ModelCheckpoint(config=ckpt_config,
|
||||
directory=save_ckpt_path,
|
||||
prefix='{}'.format(get_rank_id()))
|
||||
prefix='{}'.format(rank))
|
||||
cb += [ckpt_cb]
|
||||
netwithgrads.set_train(True)
|
||||
model = Model(netwithgrads)
|
||||
|
|
|
|||
Loading…
Reference in New Issue