diff --git a/model_zoo/official/nlp/gru/README.md b/model_zoo/official/nlp/gru/README.md
index 652b72de0f2..421ace5a515 100644
--- a/model_zoo/official/nlp/gru/README.md
+++ b/model_zoo/official/nlp/gru/README.md
@@ -46,7 +46,7 @@ In this model, we use the Multi30K dataset as our train and test dataset.As trai
 
 # [Environment Requirements](#content)
 
-- Hardware（Ascend）
+- Hardware（Ascend or GPU）
     - Prepare hardware environment with Ascend processor.
 - Framework
     - [MindSpore](https://gitee.com/mindspore/mindspore)
@@ -81,15 +81,27 @@ nltk.download()
     After dataset preparation, you can start training and evaluation as follows:
 
     ```bash
-    # run training example
     cd ./scripts
-    bash run_standalone_train.sh [TRAIN_DATASET_PATH]
+    # download dataset
+    bash download_dataset.sh
+
+    # preprocess dataset
+    bash preprocess.sh [DATASET_PATH]
+
+    # create mindrecord
+    bash create_dataset.sh [DATASET_PATH] [DATASET_PATH]
+
+    # run training example
+    bash run_standalone_train_{platform}.sh [TRAIN_DATASET_PATH]
 
     # run distributed training example
-    bash run_distribute_train_ascend.sh [RANK_TABLE_FILE] [TRAIN_DATASET_PATH]
+    bash run_distribute_train_{platform}.sh [RANK_TABLE_FILE] [TRAIN_DATASET_PATH]
+    # platform: ascend or gpu
+    # do not need [RANK_TABLE_FILE] if you use GPU
 
     # run evaluation example
-    bash run_eval.sh [CKPT_FILE] [DATASET_PATH]
+    bash run_eval_{platform}.sh [CKPT_FILE] [DATASET_PATH]
+    # platform: ascend or gpu
     ```
 
 - Running on ModelArts (If you want to run in modelarts, please check the official documentation of [modelarts](https://support.huaweicloud.com/modelarts/), and you can start training as follows)
@@ -158,7 +170,6 @@ The GRU network script and code result are as follows:
   │   ├──local_adapter.py                    // Local adapter
   │   ├──moxing_adapter.py                   // Moxing adapter for ModelArts
   ├── src
-  |   ├──gru.py                              // gru cell architecture.
   │   ├──create_data.py                      // Dataset preparation.
   │   ├──dataset.py                          // Dataset loader to feed into model.
   │   ├──gru_for_infer.py                    // GRU eval model architecture.
@@ -167,16 +178,24 @@ The GRU network script and code result are as follows:
   │   ├──lr_schedule.py                      // Learning rate scheduler.
   │   ├──parse_output.py                     // Parse output file.
   │   ├──preprocess.py                       // Dataset preprocess.
+  |   ├──rnn_cells.py                        // rnn cell architecture.
+  |   ├──rnns.py                             // rnn layer architecture.
   │   ├──seq2seq.py                          // Seq2seq architecture.
+  |   ├──utils.py                            // utils for rnn.
   │   ├──tokenization.py                     // tokenization for the dataset.
   │   ├──weight_init.py                      // Initialize weights in the net.
   ├── scripts
   │   ├──create_dataset.sh                   // shell script for create dataset.
+  │   ├──download_dataset.sh                 // shell script for download dataset.
   │   ├──parse_output.sh                     // shell script for parse eval output file to calculate BLEU.
   │   ├──preprocess.sh                       // shell script for preprocess dataset.
-  │   ├──run_distributed_train.sh            // shell script for distributed train on ascend.
-  │   ├──run_eval.sh                         // shell script for standalone eval on ascend.
-  │   ├──run_standalone_train.sh             // shell script for standalone eval on ascend.
+  │   ├──run_distributed_train_ascend.sh     // shell script for distributed train on ascend.
+  │   ├──run_distributed_train_gpu.sh        // shell script for distributed train on gpu.
+  │   ├──run_eval_ascend.sh                  // shell script for standalone eval on ascend.
+  │   ├──run_eval_gpu.sh                     // shell script for standalone eval on gpu.
+  │   ├──run_infer_310.sh                    // shell script for 310 inference.
+  │   ├──run_standalone_train_ascend.sh      // shell script for standalone eval on ascend.
+  │   ├──run_standalone_train_gpu.sh         // shell script for standalone eval on gpu.
   ├── default_config.yaml                    // Configurations
   ├── postprocess.py                         // GRU postprocess script.
   ├── preprocess.py                          // GRU preprocess script.
@@ -188,7 +207,14 @@ The GRU network script and code result are as follows:
 
 ## [Dataset Preparation](#content)
 
-Firstly, we should download the dataset from the WMT16 official net.After downloading the Multi30k dataset file, we get six dataset file, which is show as below.And we should in put the in same directory.
+Firstly, we should download the dataset from the WMT16 official net.
+
+```bash
+cd scripts
+bash download_dataset.sh
+```
+
+After downloading the Multi30k dataset file, we get six dataset file, which is show as below.And we should in put the in same directory.
 
 ```text
 train.de
@@ -250,14 +276,17 @@ Parameters for both training and evaluation can be set in config.py. All the dat
 
     ```bash
     cd ./scripts
-    bash run_standalone_train.sh [DATASET_PATH]
+    bash run_standalone_train_{platform}.sh [DATASET_PATH]
+    # platform: ascend or gpu
     ```
 
 - Running scripts for distributed training of GRU. Task training on multiple device and run the following command in bash to be executed in `scripts/`:
 
     ``` bash
     cd ./scripts
-    bash run_distributed_train.sh [RANK_TABLE_PATH] [DATASET_PATH]
+    bash run_distributed_train_{platform}.sh [RANK_TABLE_PATH] [DATASET_PATH]
+    # platform: ascend or gpu
+    # do not need [RANK_TABLE_FILE] if you use GPU
     ```
 
 ## [Inference Process](#content)
@@ -266,7 +295,8 @@ Parameters for both training and evaluation can be set in config.py. All the dat
 
     ``` bash
     cd ./scripts
-    bash run_eval.sh [CKPT_FILE] [DATASET_PATH]
+    bash run_eval_{platform}.sh [CKPT_FILE] [DATASET_PATH]
+    # platform: ascend or gpu
     ```
 
 - After evalulation, we will get eval/target.txt and eval/output.txt.Then we can use scripts/parse_output.sh to get the translation.
@@ -354,35 +384,35 @@ perl multi-bleu.perl target.txt.forbleu < output.txt.forbleu
 
 ### Training Performance
 
-| Parameters                 | Ascend                                                         |
-| -------------------------- | -------------------------------------------------------------- |
-| Resource                   | Ascend 910; OS Euler2.8                                                     |
-| uploaded Date              | 01/18/2021 (month/day/year)                                    |
-| MindSpore Version          | 1.1.0                                                          |
-| Dataset                    | Multi30k Dataset                                |
-| Training Parameters        | epoch=30, batch_size=16                                        |
-| Optimizer                  | Adam                                                           |
-| Loss Function              | NLLLoss                                                        |
-| outputs                    | probability                                                    |
-| Speed                      | 50ms/step (1pcs)                                              |
-| Epoch Time                 | 13.4s (1pcs)                                                   |
-| Loss                       | 2.5984                                                          |
-| Params (M)                 | 21                                                            |
-| Checkpoint for inference   | 272M (.ckpt file)                                              |
-| Scripts                    | [gru](https://gitee.com/mindspore/mindspore/tree/master/model_zoo/official/nlp/gru) |
+| Parameters                 | Ascend                        | GPU                       |
+| -------------------------- | ----------------------------- |---------------------------|
+| Resource                   | Ascend 910; OS Euler2.8       | GTX1080Ti, Ubuntu 18.04   |
+| uploaded Date              | 06/05/2021 (month/day/year)   | 06/05/2021 (month/day/year) |
+| MindSpore Version          | 1.2.0                         |1.2.0                      |
+| Dataset                    | Multi30k Dataset              | Multi30k Dataset          |
+| Training Parameters        | epoch=30, batch_size=16       | epoch=30, batch_size=16   |
+| Optimizer                  | Adam                          | Adam                      |
+| Loss Function              | NLLLoss                       | NLLLoss                   |
+| outputs                    | probability                   | probability               |
+| Speed                      | 35ms/step (1pcs)              | 200ms/step (1pcs)         |
+| Epoch Time                 | 64.4s (1pcs)                  | 361.5s (1pcs)             |
+| Loss                       | 3.86888                       |2.533958                   |
+| Params (M)                 | 21                            | 21                        |
+| Checkpoint for inference   | 272M (.ckpt file)             | 272M (.ckpt file)         |
+| Scripts                    | [gru](https://gitee.com/mindspore/mindspore/tree/master/model_zoo/official/nlp/gru) |[gru](https://gitee.com/mindspore/mindspore/tree/master/model_zoo/official/nlp/gru) |
 
 ### Inference Performance
 
-| Parameters          | Ascend                      |
-| ------------------- | --------------------------- |
-| Resource            | Ascend 910; OS Euler2.8                   |
-| Uploaded Date       | 01/18/2020 (month/day/year) |
-| MindSpore Version   | 1.1.0                       |
-| Dataset             | Multi30K                    |
-| batch_size          | 1                         |
-| outputs             | label index                 |
-| Accuracy            | BLEU: 30.30                        |
-| Model for inference | 272M (.ckpt file)           |
+| Parameters          | Ascend                      | GPU |
+| ------------------- | --------------------------- |---------------------------|
+| Resource            | Ascend 910; OS Euler2.8     | GTX1080Ti, Ubuntu 18.04   |
+| Uploaded Date       | 06/05/2021 (month/day/year) | 06/05/2021 (month/day/year)|
+| MindSpore Version   | 1.2.0                       | 1.2.0                     |
+| Dataset             | Multi30K                    | Multi30K                  |
+| batch_size          | 1                           | 1                         |
+| outputs             | label index                 | label index               |
+| Accuracy            | BLEU: 31.26                 | BLEU: 29.30               |
+| Model for inference | 272M (.ckpt file)           | 272M (.ckpt file)         |
 
 # [Random Situation Description](#content)
 
diff --git a/model_zoo/official/nlp/gru/default_config.yaml b/model_zoo/official/nlp/gru/default_config.yaml
index c8599cce906..4c1ffebb54b 100644
--- a/model_zoo/official/nlp/gru/default_config.yaml
+++ b/model_zoo/official/nlp/gru/default_config.yaml
@@ -36,6 +36,8 @@ scale_factor: 2
 scale_window: 2000
 warmup_ratio: 0.333333
 teacher_force_ratio: 0.5
+compute_type: mstype.float16
+dtype: mstype.float32
 
 run_distribute: False
 dataset_path: ""
diff --git a/model_zoo/official/nlp/gru/model_utils/config.py b/model_zoo/official/nlp/gru/model_utils/config.py
index ad0d7497a8e..42cde250dff 100644
--- a/model_zoo/official/nlp/gru/model_utils/config.py
+++ b/model_zoo/official/nlp/gru/model_utils/config.py
@@ -20,6 +20,8 @@ import ast
 import argparse
 from pprint import pformat
 import yaml
+import mindspore.common.dtype as mstype
+
 
 class Config:
     """
@@ -108,6 +110,24 @@ def merge(args, cfg):
         cfg[item] = args_var[item]
     return cfg
 
+def parse_dtype(dtype):
+    if dtype not in ["mstype.float32", "mstype.float16"]:
+        raise ValueError("Not supported dtype")
+
+    if dtype == "mstype.float32":
+        return mstype.float32
+    if dtype == "mstype.float16":
+        return mstype.float16
+    return None
+
+def extra_operations(cfg):
+    """
+    Do extra work on config
+    Args:
+        config: Object after instantiation of class 'Config'.
+    """
+    cfg.dtype = parse_dtype(cfg.dtype)
+    cfg.compute_type = parse_dtype(cfg.compute_type)
 
 def get_config():
     """
@@ -121,6 +141,8 @@ def get_config():
     default, helper, choices = parse_yaml(path_args.config_path)
     args = parse_cli_to_yaml(parser=parser, cfg=default, helper=helper, choices=choices, cfg_path=path_args.config_path)
     final_config = merge(args, default)
-    return Config(final_config)
+    final_config = Config(final_config)
+    extra_operations(final_config)
+    return final_config
 
 config = get_config()
diff --git a/model_zoo/official/nlp/gru/requirements.txt b/model_zoo/official/nlp/gru/requirements.txt
deleted file mode 100644
index 0ba33e85625..00000000000
--- a/model_zoo/official/nlp/gru/requirements.txt
+++ /dev/null
@@ -1,3 +0,0 @@
-nltk
-numpy
-pyyaml
diff --git a/model_zoo/official/nlp/gru/scripts/create_dataset.sh b/model_zoo/official/nlp/gru/scripts/create_dataset.sh
index 6d6521b9ab3..9626cd7d1d4 100644
--- a/model_zoo/official/nlp/gru/scripts/create_dataset.sh
+++ b/model_zoo/official/nlp/gru/scripts/create_dataset.sh
@@ -17,7 +17,6 @@ echo "==========================================================================
 echo "Please run the script as: "
 echo "sh create_dataset.sh DATASET_PATH OUTPUT_PATH"
 echo "for example: sh create_dataset.sh /path/multi30k/ /path/multi30k/mindrecord/"
-echo "DATASET_NAME including ag, dbpedia, and yelp_p"
 echo "It is better to use absolute path."
 echo "=============================================================================================================="
 ulimit -u unlimited
diff --git a/model_zoo/official/nlp/gru/scripts/download_dataset.sh b/model_zoo/official/nlp/gru/scripts/download_dataset.sh
new file mode 100644
index 00000000000..060d90b7bbd
--- /dev/null
+++ b/model_zoo/official/nlp/gru/scripts/download_dataset.sh
@@ -0,0 +1,33 @@
+#!/bin/bash
+# Copyright 2021 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+CUR_PATH=`pwd`
+DATA_PATH=${CUR_PATH}/../data
+TRAIN_URL=http://www.quest.dcs.shef.ac.uk/wmt16_files_mmt/training.tar.gz
+VALID_URL=http://www.quest.dcs.shef.ac.uk/wmt16_files_mmt/validation.tar.gz
+TEST_URL=http://www.quest.dcs.shef.ac.uk/wmt16_files_mmt/mmt16_task1_test.tar.gz
+
+mkdir ${DATA_PATH}
+cd ${DATA_PATH}
+wget --no-check-certificate ${TRAIN_URL} 
+wget --no-check-certificate ${VALID_URL}
+wget --no-check-certificate ${TEST_URL}
+tar xvf training.tar.gz
+tar xvf validation.tar.gz
+tar xvf mmt16_task1_test.tar.gz
+/bin/rm training.tar.gz
+/bin/rm validation.tar.gz
+/bin/rm mmt16_task1_test.tar.gz
+
diff --git a/model_zoo/official/nlp/gru/scripts/run_distribute_train_ascend.sh b/model_zoo/official/nlp/gru/scripts/run_distribute_train_ascend.sh
index bc99c693497..c5e7f87b48f 100644
--- a/model_zoo/official/nlp/gru/scripts/run_distribute_train_ascend.sh
+++ b/model_zoo/official/nlp/gru/scripts/run_distribute_train_ascend.sh
@@ -47,6 +47,7 @@ exit 1
 fi
 
 ulimit -u unlimited
+export DEVICE_TARGET="Ascend"
 export DEVICE_NUM=8
 export RANK_SIZE=8
 export RANK_TABLE_FILE=$PATH1
@@ -65,6 +66,6 @@ do
     cd ./train_parallel$i || exit
     echo "start training for rank $RANK_ID, device $DEVICE_ID"
     env > env.log
-    python train.py --run_distribute=True --dataset_path=$DATASET_PATH &> log &
+    python train.py --device_target=$DEVICE_TARGET --run_distribute=True --dataset_path=$DATASET_PATH &> log &
     cd ..
-done
\ No newline at end of file
+done
diff --git a/model_zoo/official/nlp/gru/scripts/run_distribute_train_gpu.sh b/model_zoo/official/nlp/gru/scripts/run_distribute_train_gpu.sh
new file mode 100644
index 00000000000..43c4548ce77
--- /dev/null
+++ b/model_zoo/official/nlp/gru/scripts/run_distribute_train_gpu.sh
@@ -0,0 +1,54 @@
+#!/bin/bash
+# Copyright 2021 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+if [ $# -ne 1 ]
+then
+    echo "Usage: sh run_distribute_train_gpu.sh [DATASET_PATH]"
+exit 1
+fi
+
+get_real_path(){
+  if [ "${1:0:1}" == "/" ]; then
+    echo "$1"
+  else
+    echo "$(realpath -m $PWD/$1)"
+  fi
+}
+
+DATASET_PATH=$(get_real_path $1)
+echo $DATASET_PATH
+if [ ! -f $DATASET_PATH ]
+then
+    echo "error: DATASET_PATH=$DATASET_PATH is not a file"
+exit 1
+fi
+
+ulimit -u unlimited
+export DEVICE_TARGET="GPU"
+export DEVICE_NUM=8
+
+rm -rf ./train
+mkdir ./train
+cp ../*.py ./train
+cp ../*.yaml ./train
+cp *.sh ./train
+cp -r ../src ./train
+cp -r ../model_utils ./train
+cd ./train || exit
+echo "start training for $DEVICE_NUM GPUs"
+env > env.log
+mpirun --allow-run-as-root -n $DEVICE_NUM python train.py --run_distribute=True --device_target=$DEVICE_TARGET --dataset_path=$DATASET_PATH &> log &
+cd ..
diff --git a/model_zoo/official/nlp/gru/scripts/run_eval_ascend.sh b/model_zoo/official/nlp/gru/scripts/run_eval_ascend.sh
new file mode 100644
index 00000000000..b6f87802da2
--- /dev/null
+++ b/model_zoo/official/nlp/gru/scripts/run_eval_ascend.sh
@@ -0,0 +1,62 @@
+#!/bin/bash
+# Copyright 2021 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+if [ $# -ne 2 ]
+then
+    echo "Usage: sh run_eval_ascend.sh [CKPT_FILE] [DATASET_PATH]"
+exit 1
+fi
+ulimit -u unlimited
+export DEVICE_NUM=1
+export DEVICE_ID=0
+export RANK_ID=0
+export RANK_SIZE=1
+export DEVICE_TARGET="Ascend"
+
+get_real_path(){
+  if [ "${1:0:1}" == "/" ]; then
+    echo "$1"
+  else
+    echo "$(realpath -m $PWD/$1)"
+  fi
+}
+
+CKPT_FILE=$(get_real_path $1)
+echo $CKPT_FILE
+if [ ! -f $CKPT_FILE ]
+then
+    echo "error: CKPT_FILE=$CKPT_FILE is not a file"
+exit 1
+fi
+
+DATASET_PATH=$(get_real_path $2)
+echo $DATASET_PATH
+if [ ! -f $DATASET_PATH ]
+then
+    echo "error: DATASET_PATH=$DATASET_PATH is not a file"
+exit 1
+fi
+rm -rf ./eval
+mkdir ./eval
+cp ../*.py ./eval
+cp ../*.yaml ./eval
+cp *.sh ./eval
+cp -r ../src ./eval
+cp -r ../model_utils ./eval
+cd ./eval || exit
+echo "start eval for device $DEVICE_ID"
+env > env.log
+python eval.py --device_target=$DEVICE_TARGET --ckpt_file=$CKPT_FILE --dataset_path=$DATASET_PATH &> log &
+cd ..
diff --git a/model_zoo/official/nlp/gru/scripts/run_eval.sh b/model_zoo/official/nlp/gru/scripts/run_eval_gpu.sh
similarity index 87%
rename from model_zoo/official/nlp/gru/scripts/run_eval.sh
rename to model_zoo/official/nlp/gru/scripts/run_eval_gpu.sh
index a4f8869175c..bb4babf1866 100644
--- a/model_zoo/official/nlp/gru/scripts/run_eval.sh
+++ b/model_zoo/official/nlp/gru/scripts/run_eval_gpu.sh
@@ -15,7 +15,7 @@
 # ============================================================================
 if [ $# -ne 2 ]
 then
-    echo "Usage: sh run_eval.sh [CKPT_FILE] [DATASET_PATH]"
+    echo "Usage: sh run_eval_gpu.sh [CKPT_FILE] [DATASET_PATH]"
 exit 1
 fi
 ulimit -u unlimited
@@ -23,6 +23,8 @@ export DEVICE_NUM=1
 export DEVICE_ID=0
 export RANK_ID=0
 export RANK_SIZE=1
+export DEVICE_TARGET="GPU"
+
 get_real_path(){
   if [ "${1:0:1}" == "/" ]; then
     echo "$1"
@@ -56,5 +58,5 @@ cp -r ../model_utils ./eval
 cd ./eval || exit
 echo "start eval for device $DEVICE_ID"
 env > env.log
-python eval.py --ckpt_file=$CKPT_FILE --dataset_path=$DATASET_PATH &> log &
+python eval.py --device_target=$DEVICE_TARGET --ckpt_file=$CKPT_FILE --dataset_path=$DATASET_PATH &> log &
 cd ..
diff --git a/model_zoo/official/nlp/gru/scripts/run_standalone_train_ascend.sh b/model_zoo/official/nlp/gru/scripts/run_standalone_train_ascend.sh
new file mode 100644
index 00000000000..b62836ca207
--- /dev/null
+++ b/model_zoo/official/nlp/gru/scripts/run_standalone_train_ascend.sh
@@ -0,0 +1,54 @@
+#!/bin/bash
+# Copyright 2021 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+if [ $# -ne 1 ]
+then
+    echo "Usage: sh run_standalone_train_ascend.sh [DATASET_PATH]"
+exit 1
+fi
+ulimit -u unlimited
+export DEVICE_NUM=1
+export DEVICE_ID=0
+export RANK_ID=0
+export RANK_SIZE=1
+export DEVICE_TARGET="Ascend"
+get_real_path(){
+  if [ "${1:0:1}" == "/" ]; then
+    echo "$1"
+  else
+    echo "$(realpath -m $PWD/$1)"
+  fi
+}
+
+DATASET_PATH=$(get_real_path $1)
+echo $DATASET_PATH
+if [ ! -f $DATASET_PATH ]
+then
+    echo "error: DATASET_PATH=$DATASET_PATH is not a file"
+exit 1
+fi
+
+rm -rf ./train
+mkdir ./train
+cp ../*.py ./train
+cp ../*.yaml ./train
+cp *.sh ./train
+cp -r ../src ./train
+cp -r ../model_utils ./train
+cd ./train || exit
+echo "start training for device $DEVICE_ID"
+env > env.log
+python train.py --device_target=$DEVICE_TARGET --dataset_path=$DATASET_PATH &> log &
+cd ..
diff --git a/model_zoo/official/nlp/gru/scripts/run_standalone_train.sh b/model_zoo/official/nlp/gru/scripts/run_standalone_train_gpu.sh
similarity index 86%
rename from model_zoo/official/nlp/gru/scripts/run_standalone_train.sh
rename to model_zoo/official/nlp/gru/scripts/run_standalone_train_gpu.sh
index 66e7893f9f0..78ed6113fb1 100644
--- a/model_zoo/official/nlp/gru/scripts/run_standalone_train.sh
+++ b/model_zoo/official/nlp/gru/scripts/run_standalone_train_gpu.sh
@@ -15,14 +15,15 @@
 # ============================================================================
 if [ $# -ne 1 ]
 then
-    echo "Usage: sh run_distribute_train_ascend.sh [DATASET_PATH]"
+    echo "Usage: sh run_standalone_train_gpu.sh [DATASET_PATH]"
 exit 1
 fi
 ulimit -u unlimited
 export DEVICE_NUM=1
-export DEVICE_ID=4
+export DEVICE_ID=0
 export RANK_ID=0
 export RANK_SIZE=1
+export DEVICE_TARGET="GPU"
 get_real_path(){
   if [ "${1:0:1}" == "/" ]; then
     echo "$1"
@@ -38,7 +39,6 @@ then
     echo "error: DATASET_PATH=$DATASET_PATH is not a file"
 exit 1
 fi
-
 rm -rf ./train
 mkdir ./train
 cp ../*.py ./train
@@ -49,5 +49,5 @@ cp -r ../model_utils ./train
 cd ./train || exit
 echo "start training for device $DEVICE_ID"
 env > env.log
-python train.py --dataset_path=$DATASET_PATH &> log &
+python train.py --device_target=$DEVICE_TARGET --dataset_path=$DATASET_PATH &> log &
 cd ..
diff --git a/model_zoo/official/nlp/gru/src/gru.py b/model_zoo/official/nlp/gru/src/gru.py
deleted file mode 100644
index 08199c43ccc..00000000000
--- a/model_zoo/official/nlp/gru/src/gru.py
+++ /dev/null
@@ -1,104 +0,0 @@
-# Copyright 2021 Huawei Technologies Co., Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ============================================================================
-"""GRU cell"""
-import mindspore.nn as nn
-import mindspore.ops.operations as P
-import mindspore.common.dtype as mstype
-from src.weight_init import gru_default_state
-
-class BidirectionGRU(nn.Cell):
-    '''
-    BidirectionGRU model
-
-    Args:
-        config: config of network
-    '''
-    def __init__(self, config, is_training=True):
-        super(BidirectionGRU, self).__init__()
-        if is_training:
-            self.batch_size = config.batch_size
-        else:
-            self.batch_size = config.eval_batch_size
-        self.embedding_size = config.encoder_embedding_size
-        self.hidden_size = config.hidden_size
-        self.weight_i, self.weight_h, self.bias_i, self.bias_h, self.init_h = gru_default_state(self.batch_size,
-                                                                                                self.embedding_size,
-                                                                                                self.hidden_size)
-        self.weight_bw_i, self.weight_bw_h, self.bias_bw_i, self.bias_bw_h, self.init_bw_h = \
-            gru_default_state(self.batch_size, self.embedding_size, self.hidden_size)
-        self.reverse = P.ReverseV2(axis=[1])
-        self.concat = P.Concat(axis=2)
-        self.squeeze = P.Squeeze(axis=0)
-        self.rnn = P.DynamicGRUV2()
-        self.text_len = config.max_length
-        self.cast = P.Cast()
-
-    def construct(self, x):
-        '''
-        BidirectionGRU construction
-
-        Args:
-            x(Tensor): BidirectionGRU input
-
-        Returns:
-            output(Tensor): rnn output
-            hidden(Tensor): hidden state
-        '''
-        x = self.cast(x, mstype.float16)
-        y1, _, _, _, _, _ = self.rnn(x, self.weight_i, self.weight_h, self.bias_i, self.bias_h, None, self.init_h)
-        bw_x = self.reverse(x)
-        y1_bw, _, _, _, _, _ = self.rnn(bw_x, self.weight_bw_i,
-                                        self.weight_bw_h, self.bias_bw_i, self.bias_bw_h, None, self.init_bw_h)
-        y1_bw = self.reverse(y1_bw)
-        output = self.concat((y1, y1_bw))
-        hidden = self.concat((y1[self.text_len-1:self.text_len:1, ::, ::],
-                              y1_bw[self.text_len-1:self.text_len:1, ::, ::]))
-        hidden = self.squeeze(hidden)
-        return output, hidden
-
-class GRU(nn.Cell):
-    '''
-    GRU model
-
-    Args:
-        config: config of network
-    '''
-    def __init__(self, config, is_training=True):
-        super(GRU, self).__init__()
-        if is_training:
-            self.batch_size = config.batch_size
-        else:
-            self.batch_size = config.eval_batch_size
-        self.embedding_size = config.encoder_embedding_size
-        self.hidden_size = config.hidden_size
-        self.weight_i, self.weight_h, self.bias_i, self.bias_h, self.init_h = \
-            gru_default_state(self.batch_size, self.embedding_size + self.hidden_size*2, self.hidden_size)
-        self.rnn = P.DynamicGRUV2()
-        self.cast = P.Cast()
-
-    def construct(self, x):
-        '''
-        GRU construction
-
-        Args:
-            x(Tensor): GRU input
-
-        Returns:
-            output(Tensor): rnn output
-            hidden(Tensor): hidden state
-        '''
-        x = self.cast(x, mstype.float16)
-        y1, h1, _, _, _, _ = self.rnn(x, self.weight_i, self.weight_h, self.bias_i, self.bias_h, None, self.init_h)
-        return y1, h1
diff --git a/model_zoo/official/nlp/gru/src/gru_for_train.py b/model_zoo/official/nlp/gru/src/gru_for_train.py
index b60cb2d7e9f..50e028dafab 100644
--- a/model_zoo/official/nlp/gru/src/gru_for_train.py
+++ b/model_zoo/official/nlp/gru/src/gru_for_train.py
@@ -240,3 +240,48 @@ class GRUTrainOneStepWithLossScaleCell(nn.Cell):
             succ = self.optimizer(grads)
         ret = (loss, cond, scaling_sens)
         return F.depend(ret, succ)
+
+class GRUTrainOneStepCell(nn.TrainOneStepCell):
+    """
+    Encapsulation class of GRU network training.
+    Append an optimizer to the training network after that the construct
+    function can be called to create the backward graph.
+    Args:
+        network (Cell): The training network. Note that loss function should have been added.
+        optimizer (Optimizer): Optimizer for updating the weights.
+        sens (Number): The adjust parameter. Default: 1.0.
+        enable_clip_grad (boolean): If True, clip gradients in GRUTrainOneStepCell. Default: True.
+    """
+
+    def __init__(self, network, optimizer, sens=1.0, enable_clip_grad=True):
+        super(GRUTrainOneStepCell, self).__init__(network, optimizer, sens)
+        self.cast = P.Cast()
+        self.hyper_map = C.HyperMap()
+        self.clip_gradients = ClipGradients()
+        self.enable_clip_grad = enable_clip_grad
+
+    def set_sens(self, value):
+        self.sens = value
+
+    def construct(self,
+                  encoder_inputs,
+                  decoder_inputs,
+                  teacher_force,
+                  sens=None):
+        """Defines the computation performed."""
+
+        weights = self.weights
+        loss = self.network(encoder_inputs,
+                            decoder_inputs,
+                            teacher_force)
+
+        grads = self.grad(self.network, weights)(encoder_inputs,
+                                                 decoder_inputs,
+                                                 teacher_force,
+                                                 self.cast(F.tuple_to_array((self.sens,)),
+                                                           mstype.float32))
+        if self.enable_clip_grad:
+            grads = self.clip_gradients(grads, GRADIENT_CLIP_TYPE, GRADIENT_CLIP_VALUE)
+        grads = self.grad_reducer(grads)
+        succ = self.optimizer(grads)
+        return F.depend(loss, succ)
diff --git a/model_zoo/official/nlp/gru/src/rnn_cells.py b/model_zoo/official/nlp/gru/src/rnn_cells.py
new file mode 100644
index 00000000000..6930556c89c
--- /dev/null
+++ b/model_zoo/official/nlp/gru/src/rnn_cells.py
@@ -0,0 +1,130 @@
+# Copyright 2021 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+'''RNN Cells module, include RNNCell, GRUCell, LSTMCell'''
+import math
+import numpy as np
+import mindspore.nn as nn
+import mindspore.ops as P
+from mindspore import Tensor, Parameter
+from mindspore.common.initializer import initializer, Uniform
+
+def rnn_tanh_cell(inputs, hidden, w_ih, w_hh, b_ih, b_hh):
+    '''RNN cell function with tanh activation'''
+    if b_ih is None:
+        igates = P.MatMul(False, True)(inputs, w_ih)
+        hgates = P.MatMul(False, True)(hidden, w_hh)
+    else:
+        igates = P.MatMul(False, True)(inputs, w_ih) + b_ih
+        hgates = P.MatMul(False, True)(hidden, w_hh) + b_hh
+    return P.Tanh()(igates + hgates)
+
+def rnn_relu_cell(inputs, hidden, w_ih, w_hh, b_ih, b_hh):
+    '''RNN cell function with relu activation'''
+    if b_ih is None:
+        igates = P.MatMul(False, True)(inputs, w_ih)
+        hgates = P.MatMul(False, True)(hidden, w_hh)
+    else:
+        igates = P.MatMul(False, True)(inputs, w_ih) + b_ih
+        hgates = P.MatMul(False, True)(hidden, w_hh) + b_hh
+    return P.ReLU()(igates + hgates)
+
+def lstm_cell(inputs, hidden, w_ih, w_hh, b_ih, b_hh):
+    '''LSTM cell function'''
+    hx, cx = hidden
+    if b_ih is None:
+        gates = P.MatMul(False, True)(inputs, w_ih) + P.MatMul(False, True)(hx, w_hh)
+    else:
+        gates = P.MatMul(False, True)(inputs, w_ih) + P.MatMul(False, True)(hx, w_hh) + b_ih + b_hh
+    ingate, forgetgate, cellgate, outgate = P.Split(1, 4)(gates)
+
+    ingate = P.Sigmoid()(ingate)
+    forgetgate = P.Sigmoid()(forgetgate)
+    cellgate = P.Tanh()(cellgate)
+    outgate = P.Sigmoid()(outgate)
+
+    cy = (forgetgate * cx) + (ingate * cellgate)
+    hy = outgate * P.Tanh()(cy)
+
+    return hy, cy
+
+def gru_cell(inputs, hidden, w_ih, w_hh, b_ih, b_hh):
+    '''GRU cell function'''
+    if b_ih is None:
+        gi = P.MatMul(False, True)(inputs, w_ih)
+        gh = P.MatMul(False, True)(hidden, w_hh)
+    else:
+        gi = P.MatMul(False, True)(inputs, w_ih) + b_ih
+        gh = P.MatMul(False, True)(hidden, w_hh) + b_hh
+    i_r, i_i, i_n = P.Split(1, 3)(gi)
+    h_r, h_i, h_n = P.Split(1, 3)(gh)
+
+    resetgate = P.Sigmoid()(i_r + h_r)
+    inputgate = P.Sigmoid()(i_i + h_i)
+    newgate = P.Tanh()(i_n + resetgate * h_n)
+    hy = newgate + inputgate * (hidden - newgate)
+
+    return hy
+
+class RNNCellBase(nn.Cell):
+    '''Basic class for RNN Cells'''
+    def __init__(self, input_size: int, hidden_size: int, bias: bool, num_chunks: int):
+        super().__init__()
+        self.input_size = input_size
+        self.hidden_size = hidden_size
+        self.bias = bias
+        self.weight_ih = Parameter(Tensor(np.random.randn(num_chunks * hidden_size, input_size).astype(np.float32)))
+        self.weight_hh = Parameter(Tensor(np.random.randn(num_chunks * hidden_size, hidden_size).astype(np.float32)))
+        if bias:
+            self.bias_ih = Parameter(Tensor(np.random.randn(num_chunks * hidden_size).astype(np.float32)))
+            self.bias_hh = Parameter(Tensor(np.random.randn(num_chunks * hidden_size).astype(np.float32)))
+        self.reset_parameters()
+
+    def reset_parameters(self):
+        stdv = 1 / math.sqrt(self.hidden_size)
+        for weight in self.get_parameters():
+            weight.set_data(initializer(Uniform(stdv), weight.shape))
+
+class RNNCell(RNNCellBase):
+    '''RNNCell operator class'''
+    _non_linearity = ['tanh', 'relu']
+    def __init__(self, input_size: int, hidden_size: int, bias: bool = True, nonlinearity: str = "tanh"):
+        super().__init__(input_size, hidden_size, bias, num_chunks=1)
+        if nonlinearity not in self._non_linearity:
+            raise ValueError("Unknown nonlinearity: {}".format(nonlinearity))
+        self.nonlinearity = nonlinearity
+
+    def construct(self, inputs, hx):
+        if self.nonlinearity == "tanh":
+            ret = rnn_tanh_cell(inputs, hx, self.weight_ih, self.weight_hh, self.bias_ih, self.bias_hh)
+        else:
+            ret = rnn_relu_cell(inputs, hx, self.weight_ih, self.weight_hh, self.bias_ih, self.bias_hh)
+        return ret
+
+class LSTMCell(RNNCellBase):
+    '''LSTMCell operator class'''
+    def __init__(self, input_size: int, hidden_size: int, bias: bool = True):
+        super().__init__(input_size, hidden_size, bias, num_chunks=4)
+        self.support_non_tensor_inputs = True
+
+    def construct(self, inputs, hx):
+        return lstm_cell(inputs, hx, self.weight_ih, self.weight_hh, self.bias_ih, self.bias_hh)
+
+class GRUCell(RNNCellBase):
+    '''GRUCell operator class'''
+    def __init__(self, input_size: int, hidden_size: int, bias: bool = True):
+        super().__init__(input_size, hidden_size, bias, num_chunks=3)
+
+    def construct(self, inputs, hx):
+        return gru_cell(inputs, hx, self.weight_ih, self.weight_hh, self.bias_ih, self.bias_hh)
diff --git a/model_zoo/official/nlp/gru/src/rnns.py b/model_zoo/official/nlp/gru/src/rnns.py
new file mode 100644
index 00000000000..104d0fe72a0
--- /dev/null
+++ b/model_zoo/official/nlp/gru/src/rnns.py
@@ -0,0 +1,311 @@
+# Copyright 2021 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+'''RNN operators module, include RNN, GRU, LSTM'''
+import math
+import numpy as np
+import mindspore
+import mindspore.nn as nn
+import mindspore.ops as P
+from mindspore.ops.primitive import constexpr
+from mindspore import Tensor, Parameter, ParameterTuple
+from mindspore import log as logger
+from mindspore import context
+from src.rnn_cells import rnn_relu_cell, rnn_tanh_cell, lstm_cell, gru_cell
+from src.utils import Reverse, ReverseSequence
+
+@constexpr
+def _init_state(shape, dtype, is_lstm):
+    hx = Tensor(np.zeros(shape), dtype)
+    cx = Tensor(np.zeros(shape), dtype)
+    if is_lstm:
+        return (hx, cx)
+    return hx
+
+class DynamicRNN(nn.Cell):
+    '''Dynamic RNN module to compute RNN cell by timesteps'''
+    def __init__(self, mode):
+        super().__init__()
+        if mode == "RNN_RELU":
+            cell = rnn_relu_cell
+        elif mode == "RNN_TANH":
+            cell = rnn_tanh_cell
+        elif mode == "LSTM":
+            cell = lstm_cell
+        elif mode == "GRU":
+            cell = gru_cell
+        else:
+            raise ValueError("Unrecognized RNN mode: " + mode)
+        self.cell = cell
+        self.is_lstm = mode == "LSTM"
+
+    def recurrent(self, x, h_0, w_ih, w_hh, b_ih, b_hh):
+        '''recurrent steps without sequence length'''
+        time_step = x.shape[0]
+        outputs = []
+        t = 0
+        h = h_0
+        while t < time_step:
+            x_t = x[t:t+1:1]
+            x_t = P.Squeeze(0)(x_t)
+            h = self.cell(x_t, h, w_ih, w_hh, b_ih, b_hh)
+            if self.is_lstm:
+                outputs.append(h[0])
+            else:
+                outputs.append(h)
+            t += 1
+        outputs = P.Stack()(outputs)
+        return outputs, h
+
+    def variable_recurrent(self, x, h, seq_length, w_ih, w_hh, b_ih, b_hh):
+        '''recurrent steps with sequence length'''
+        time_step = x.shape[0]
+        h_t = h
+        if self.is_lstm:
+            hidden_size = h[0].shape[-1]
+            zero_output = P.ZerosLike()(h_t[0])
+        else:
+            hidden_size = h.shape[-1]
+            zero_output = P.ZerosLike()(h_t)
+        seq_length = P.Cast()(seq_length, mindspore.float32)
+        seq_length = P.BroadcastTo((hidden_size, -1))(seq_length)
+        seq_length = P.Cast()(seq_length, mindspore.int32)
+        seq_length = P.Transpose()(seq_length, (1, 0))
+
+        outputs = []
+        state_t = h_t
+        t = 0
+        while t < time_step:
+            x_t = x[t:t+1:1]
+            x_t = P.Squeeze(0)(x_t)
+            h_t = self.cell(x_t, state_t, w_ih, w_hh, b_ih, b_hh)
+            seq_cond = seq_length > t
+            if self.is_lstm:
+                state_t_0 = P.Select()(seq_cond, h_t[0], state_t[0])
+                state_t_1 = P.Select()(seq_cond, h_t[1], state_t[1])
+                output = P.Select()(seq_cond, h_t[0], zero_output)
+                state_t = (state_t_0, state_t_1)
+            else:
+                state_t = P.Select()(seq_cond, h_t, state_t)
+                output = P.Select()(seq_cond, h_t, zero_output)
+            outputs.append(output)
+            t += 1
+        outputs = P.Stack()(outputs)
+        return outputs, state_t
+
+    def construct(self, x, h, seq_length, w_ih, w_hh, b_ih, b_hh):
+        if seq_length is None:
+            return self.recurrent(x, h, w_ih, w_hh, b_ih, b_hh)
+        return self.variable_recurrent(x, h, seq_length, w_ih, w_hh, b_ih, b_hh)
+
+class RNNBase(nn.Cell):
+    '''Basic class for RNN operators'''
+    def __init__(self, mode, input_size, hidden_size, num_layers=1, has_bias=True,
+                 batch_first=False, dropout=0, bidirectional=False):
+        super().__init__()
+        if not 0 <= dropout <= 1:
+            raise ValueError("dropout should be a number in range [0, 1] "
+                             "representing the probability of an element being "
+                             "zeroed")
+
+        if dropout > 0 and num_layers == 1:
+            logger.warning("dropout option adds dropout after all but last "
+                           "recurrent layer, so non-zero dropout expects "
+                           "num_layers greater than 1, but got dropout={} and "
+                           "num_layers={}".format(dropout, num_layers))
+        if mode == "LSTM":
+            gate_size = 4 * hidden_size
+        elif mode == "GRU":
+            gate_size = 3 * hidden_size
+        elif mode == "RNN_TANH":
+            gate_size = hidden_size
+        elif mode == "RNN_RELU":
+            gate_size = hidden_size
+        else:
+            raise ValueError("Unrecognized RNN mode: " + mode)
+
+        self.is_ascend = context.get_context("device_target") == "Ascend"
+        if self.is_ascend:
+            self.reverse = P.ReverseV2([0])
+            self.reverse_sequence = P.ReverseSequence(0, 1)
+        else:
+            self.reverse = Reverse(0)
+            self.reverse_sequence = ReverseSequence(0, 1)
+        self.hidden_size = hidden_size
+        self.batch_first = batch_first
+        self.num_layers = num_layers
+        self.dropout = dropout
+        self.dropout_op = nn.Dropout(float(1 - dropout))
+        self.bidirectional = bidirectional
+        self.has_bias = has_bias
+        self.rnn = DynamicRNN(mode)
+        num_directions = 2 if bidirectional else 1
+        self.is_lstm = mode == "LSTM"
+
+        self.w_ih_list = []
+        self.w_hh_list = []
+        self.b_ih_list = []
+        self.b_hh_list = []
+        stdv = 1 / math.sqrt(self.hidden_size)
+        for layer in range(num_layers):
+            for direction in range(num_directions):
+                layer_input_size = input_size if layer == 0 else hidden_size * num_directions
+                suffix = '_reverse' if direction == 1 else ''
+
+                self.w_ih_list.append(Parameter(
+                    Tensor(np.random.uniform(-stdv, stdv, (gate_size, layer_input_size)).astype(np.float32)),
+                    name='weight_ih_l{}{}'.format(layer, suffix)))
+                self.w_hh_list.append(Parameter(
+                    Tensor(np.random.uniform(-stdv, stdv, (gate_size, hidden_size)).astype(np.float32)),
+                    name='weight_hh_l{}{}'.format(layer, suffix)))
+                if has_bias:
+                    self.b_ih_list.append(Parameter(
+                        Tensor(np.random.uniform(-stdv, stdv, (gate_size)).astype(np.float32)),
+                        name='bias_ih_l{}{}'.format(layer, suffix)))
+                    self.b_hh_list.append(Parameter(
+                        Tensor(np.random.uniform(-stdv, stdv, (gate_size)).astype(np.float32)),
+                        name='bias_hh_l{}{}'.format(layer, suffix)))
+        self.w_ih_list = ParameterTuple(self.w_ih_list)
+        self.w_hh_list = ParameterTuple(self.w_hh_list)
+        self.b_ih_list = ParameterTuple(self.b_ih_list)
+        self.b_hh_list = ParameterTuple(self.b_hh_list)
+
+    def _stacked_bi_dynamic_rnn(self, x, h, seq_length):
+        """stacked bidirectional dynamic_rnn"""
+        pre_layer = x
+        h_n = ()
+        c_n = ()
+        output = 0
+        for i in range(self.num_layers):
+            offset = i * 2
+            if self.has_bias:
+                w_f_ih, w_f_hh, b_f_ih, b_f_hh = \
+                    self.w_ih_list[offset], self.w_hh_list[offset], \
+                    self.b_ih_list[offset], self.b_hh_list[offset]
+                w_b_ih, w_b_hh, b_b_ih, b_b_hh = \
+                    self.w_ih_list[offset + 1], self.w_hh_list[offset + 1], \
+                    self.b_ih_list[offset + 1], self.b_hh_list[offset + 1]
+            else:
+                w_f_ih, w_f_hh = self.w_ih_list[offset], self.w_hh_list[offset]
+                w_b_ih, w_b_hh = self.w_ih_list[offset + 1], self.w_hh_list[offset + 1]
+                b_f_ih, b_f_hh, b_b_ih, b_b_hh = None, None, None, None
+            if self.is_lstm:
+                h_f_i = (h[0][offset], h[1][offset])
+                h_b_i = (h[0][offset + 1], h[1][offset + 1])
+            else:
+                h_f_i = h[offset]
+                h_b_i = h[offset + 1]
+            if seq_length is None:
+                x_b = self.reverse(pre_layer)
+            else:
+                x_b = self.reverse_sequence(pre_layer, seq_length)
+            output_f, h_t_f = self.rnn(pre_layer, h_f_i, seq_length, w_f_ih, w_f_hh, b_f_ih, b_f_hh)
+            output_b, h_t_b = self.rnn(x_b, h_b_i, seq_length, w_b_ih, w_b_hh, b_b_ih, b_b_hh)
+            if seq_length is None:
+                output_b = self.reverse(output_b)
+            else:
+                output_b = self.reverse_sequence(output_b, seq_length)
+            output = P.Concat(2)((output_f, output_b))
+            pre_layer = self.dropout_op(output) if (self.dropout != 0 and i < self.num_layers - 1) else output
+            if self.is_lstm:
+                h_n += (h_t_f[0], h_t_b[0],)
+                c_n += (h_t_f[1], h_t_b[1],)
+            else:
+                h_n += (h_t_f, h_t_b,)
+        if self.is_lstm:
+            h_n = P.Concat(0)(h_n)
+            c_n = P.Concat(0)(c_n)
+            h_n = h_n.view(h[0].shape)
+            c_n = c_n.view(h[1].shape)
+            return output, (h_n.view(h[0].shape), c_n.view(h[1].shape))
+        h_n = P.Concat(0)(h_n)
+        return output, h_n.view(h.shape)
+
+    def _stacked_dynamic_rnn(self, x, h, seq_length):
+        """stacked mutil_layer dynamic_rnn"""
+        pre_layer = x
+        h_n = ()
+        c_n = ()
+        output = 0
+        for i in range(self.num_layers):
+            if self.has_bias:
+                w_ih, w_hh, b_ih, b_hh = self.w_ih_list[i], self.w_hh_list[i], self.b_ih_list[i], self.b_hh_list[i]
+            else:
+                w_ih, w_hh = self.w_ih_list[i], self.w_hh_list[i]
+                b_ih, b_hh = None, None
+            if self.is_lstm:
+                h_i = (h[0][i], h[1][i])
+            else:
+                h_i = h[i]
+            output, h_t = self.rnn(pre_layer, h_i, seq_length, w_ih, w_hh, b_ih, b_hh)
+            pre_layer = self.dropout_op(output) if (self.dropout != 0 and i < self.num_layers - 1) else output
+            if self.is_lstm:
+                h_n += (h_t[0],)
+                c_n += (h_t[1],)
+            else:
+                h_n += (h_t,)
+        if self.is_lstm:
+            h_n = P.Concat(0)(h_n)
+            c_n = P.Concat(0)(c_n)
+            h_n = h_n.view(h[0].shape)
+            c_n = c_n.view(h[1].shape)
+            return output, (h_n.view(h[0].shape), c_n.view(h[1].shape))
+        h_n = P.Concat(0)(h_n)
+        return output, h_n.view(h.shape)
+
+    def construct(self, x, h=None, seq_length=None):
+        '''Defines the RNN like operators performed'''
+        max_batch_size = x.shape[0] if self.batch_first else x.shape[1]
+        num_directions = 2 if self.bidirectional else 1
+        if h is None:
+            h = _init_state((self.num_layers * num_directions, max_batch_size, self.hidden_size), x.dtype, self.is_lstm)
+        if self.batch_first:
+            x = P.Transpose()(x, (1, 0, 2))
+        if self.bidirectional:
+            x, h = self._stacked_bi_dynamic_rnn(x, h, seq_length)
+        else:
+            x, h = self._stacked_dynamic_rnn(x, h, seq_length)
+        if self.batch_first:
+            x = P.Transpose()(x, (1, 0, 2))
+        return x, h
+
+class RNN(RNNBase):
+    '''RNN operator class'''
+    def __init__(self, *args, **kwargs):
+        if 'nonlinearity' in kwargs:
+            if kwargs['nonlinearity'] == 'tanh':
+                mode = 'RNN_TANH'
+            elif kwargs['nonlinearity'] == 'relu':
+                mode = 'RNN_RELU'
+            else:
+                raise ValueError("Unknown nonlinearity '{}'".format(
+                    kwargs['nonlinearity']))
+            del kwargs['nonlinearity']
+        else:
+            mode = 'RNN_TANH'
+
+        super(RNN, self).__init__(mode, *args, **kwargs)
+
+class GRU(RNNBase):
+    '''GRU operator class'''
+    def __init__(self, *args, **kwargs):
+        mode = 'GRU'
+        super(GRU, self).__init__(mode, *args, **kwargs)
+
+class LSTM(RNNBase):
+    '''LSTM operator class'''
+    def __init__(self, *args, **kwargs):
+        mode = 'LSTM'
+        super(LSTM, self).__init__(mode, *args, **kwargs)
+        self.support_non_tensor_inputs = True
diff --git a/model_zoo/official/nlp/gru/src/seq2seq.py b/model_zoo/official/nlp/gru/src/seq2seq.py
index 06ef8daa15f..97c117d0101 100644
--- a/model_zoo/official/nlp/gru/src/seq2seq.py
+++ b/model_zoo/official/nlp/gru/src/seq2seq.py
@@ -18,8 +18,8 @@ from mindspore import Tensor
 import mindspore.nn as nn
 import mindspore.ops.operations as P
 import mindspore.common.dtype as mstype
-from src.gru import BidirectionGRU, GRU
 from src.weight_init import dense_default_state
+from src.rnns import GRU
 
 class Attention(nn.Cell):
     '''
@@ -29,8 +29,8 @@ class Attention(nn.Cell):
         super(Attention, self).__init__()
         self.text_len = config.max_length
         self.attn = nn.Dense(in_channels=config.hidden_size * 3,
-                             out_channels=config.hidden_size).to_float(mstype.float16)
-        self.fc = nn.Dense(config.hidden_size, 1, has_bias=False).to_float(mstype.float16)
+                             out_channels=config.hidden_size).to_float(config.compute_type)
+        self.fc = nn.Dense(config.hidden_size, 1, has_bias=False).to_float(config.compute_type)
         self.expandims = P.ExpandDims()
         self.tanh = P.Tanh()
         self.softmax = P.Softmax()
@@ -39,6 +39,9 @@ class Attention(nn.Cell):
         self.concat = P.Concat(axis=2)
         self.squeeze = P.Squeeze(axis=2)
         self.cast = P.Cast()
+        self.dtype = config.dtype
+        self.compute_type = config.compute_type
+
     def construct(self, hidden, encoder_outputs):
         '''
         Attention construction
@@ -58,9 +61,9 @@ class Attention(nn.Cell):
         energy = self.tanh(out)
         attention = self.fc(energy)
         attention = self.squeeze(attention)
-        attention = self.cast(attention, mstype.float32)
+        attention = self.cast(attention, self.dtype)
         attention = self.softmax(attention)
-        attention = self.cast(attention, mstype.float16)
+        attention = self.cast(attention, self.compute_type)
         return attention
 
 class Encoder(nn.Cell):
@@ -76,8 +79,9 @@ class Encoder(nn.Cell):
         self.vocab_size = config.src_vocab_size
         self.embedding_size = config.encoder_embedding_size
         self.embedding = nn.Embedding(self.vocab_size, self.embedding_size)
-        self.rnn = BidirectionGRU(config, is_training=is_training).to_float(mstype.float16)
-        self.fc = nn.Dense(2*self.hidden_size, self.hidden_size).to_float(mstype.float16)
+        self.rnn = GRU(input_size=self.embedding_size, \
+            hidden_size=self.hidden_size, bidirectional=True).to_float(config.compute_type)
+        self.fc = nn.Dense(2*self.hidden_size, self.hidden_size).to_float(config.compute_type)
         self.shape = P.Shape()
         self.transpose = P.Transpose()
         self.p = P.Print()
@@ -85,6 +89,8 @@ class Encoder(nn.Cell):
         self.text_len = config.max_length
         self.squeeze = P.Squeeze(axis=0)
         self.tanh = P.Tanh()
+        self.concat = P.Concat(2)
+        self.dtype = config.dtype
 
     def construct(self, src):
         '''
@@ -99,8 +105,10 @@ class Encoder(nn.Cell):
         '''
         embedded = self.embedding(src)
         embedded = self.transpose(embedded, (1, 0, 2))
-        embedded = self.cast(embedded, mstype.float16)
+        embedded = self.cast(embedded, self.dtype)
         output, hidden = self.rnn(embedded)
+        hidden = self.transpose(hidden, (1, 0, 2))
+        hidden = hidden.view(hidden.shape[0], -1)
         hidden = self.fc(hidden)
         hidden = self.tanh(hidden)
         return output, hidden
@@ -118,7 +126,8 @@ class Decoder(nn.Cell):
         self.vocab_size = config.trg_vocab_size
         self.embedding_size = config.decoder_embedding_size
         self.embedding = nn.Embedding(self.vocab_size, self.embedding_size)
-        self.rnn = GRU(config, is_training=is_training).to_float(mstype.float16)
+        self.rnn = GRU(input_size=self.embedding_size + self.hidden_size*2, \
+            hidden_size=self.hidden_size).to_float(config.compute_type)
         self.text_len = config.max_length
         self.shape = P.Shape()
         self.transpose = P.Transpose()
@@ -130,11 +139,13 @@ class Decoder(nn.Cell):
         self.log_softmax = P.LogSoftmax(axis=1)
         weight, bias = dense_default_state(self.embedding_size+self.hidden_size*3, self.vocab_size)
         self.fc = nn.Dense(self.embedding_size+self.hidden_size*3, self.vocab_size,
-                           weight_init=weight, bias_init=bias).to_float(mstype.float16)
+                           weight_init=weight, bias_init=bias).to_float(config.compute_type)
         self.attention = Attention(config)
         self.bmm = P.BatchMatMul()
         self.dropout = nn.Dropout(0.7)
         self.expandims = P.ExpandDims()
+        self.dtype = config.dtype
+
     def construct(self, inputs, hidden, encoder_outputs):
         '''
         Decoder construction
@@ -150,21 +161,22 @@ class Decoder(nn.Cell):
         '''
         embedded = self.embedding(inputs)
         embedded = self.transpose(embedded, (1, 0, 2))
-        embedded = self.cast(embedded, mstype.float16)
+        embedded = self.cast(embedded, self.dtype)
         attn = self.attention(hidden, encoder_outputs)
         attn = self.expandims(attn, 1)
         encoder_outputs = self.transpose(encoder_outputs, (1, 0, 2))
         weight = self.bmm(attn, encoder_outputs)
         weight = self.transpose(weight, (1, 0, 2))
+        weight = self.cast(weight, self.dtype)
         emd_con = self.concat((embedded, weight))
         output, hidden = self.rnn(emd_con)
+        output = self.cast(output, self.dtype)
         out = self.concat((embedded, output, weight))
         out = self.squeeze(out)
         hidden = self.squeeze(hidden)
         prediction = self.fc(out)
         prediction = self.dropout(prediction)
-        prediction = self.cast(prediction, mstype.float32)
-        prediction = self.cast(prediction, mstype.float32)
+        prediction = self.cast(prediction, self.dtype)
         pred_prob = self.log_softmax(prediction)
         pred_prob = self.expandims(pred_prob, 0)
         return pred_prob, hidden
diff --git a/model_zoo/official/nlp/gru/src/utils.py b/model_zoo/official/nlp/gru/src/utils.py
new file mode 100644
index 00000000000..27a4b5872b6
--- /dev/null
+++ b/model_zoo/official/nlp/gru/src/utils.py
@@ -0,0 +1,76 @@
+# Copyright 2021 Huawei Technologies Co., Ltd
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+'''Utils for GPU version GRU, like Reverse operators'''
+import mindspore
+import mindspore.nn as nn
+import mindspore.ops as ops
+import mindspore.numpy as np
+
+class Reverse(nn.Cell):
+    """Reverse operator, like Reverse in mindspore"""
+    def __init__(self, dim):
+        super().__init__()
+        self.dim = dim
+
+    def construct(self, input_x):
+        shape = input_x.shape
+        dim_size = shape[self.dim]
+        reversed_indexes = np.arange(dim_size-1, -1, -1)
+        output = ops.Gather()(input_x, reversed_indexes, self.dim)
+        return output
+
+class ReverseSequence(nn.Cell):
+    """Reverse sequence operator, like ReverseSequenceV2 in mindspore"""
+    def __init__(self, seq_dim, batch_dim=0):
+        super().__init__()
+        self.seq_dim = seq_dim
+        self.batch_dim = batch_dim
+
+    def construct(self, x, seq_lengths):
+        """Defines the ReverseSequence operator computation performed."""
+        batch_size = x.shape[self.batch_dim]
+        max_seq_len = x.shape[self.seq_dim]
+        seq_lens_type = seq_lengths.dtype
+
+        back = ops.Sub()(seq_lengths, ops.OnesLike()(seq_lengths))
+
+        batch_idx = self.make_shape((batch_size, max_seq_len), seq_lens_type, 0)
+        forward_idx = self.make_shape((batch_size, max_seq_len), seq_lens_type, 1)
+
+        back = back.view(-1, 1)
+        reverse_idx = ops.Sub()(back, forward_idx)
+
+        condition = ops.Less()(reverse_idx, ops.ZerosLike()(reverse_idx))
+        reverse_idx = ops.Select()(condition, forward_idx, reverse_idx)
+
+        reverse_idx = ops.ExpandDims()(reverse_idx, 2)
+        batch_idx = ops.ExpandDims()(batch_idx, 2)
+
+        if self.batch_dim > self.seq_dim:
+            batch_idx = ops.Transpose()(batch_idx, (1, 0, 2))
+            reverse_idx = ops.Transpose()(reverse_idx, (1, 0, 2))
+            x = ops.Transpose()(x, (1, 0, 2))
+        start_indices = ops.Concat(2)((batch_idx, reverse_idx))
+
+        output = ops.GatherNd()(x, start_indices)
+
+        return output
+
+    def make_shape(self, shape, dtype, range_dim):
+        output = ops.Ones()(shape, mindspore.float32)
+        output = ops.CumSum()(output, range_dim)
+        output = ops.Cast()(output, dtype)
+        output = output - 1
+        return output
diff --git a/model_zoo/official/nlp/gru/src/weight_init.py b/model_zoo/official/nlp/gru/src/weight_init.py
index 48a1ad2460e..1f92efc14f0 100644
--- a/model_zoo/official/nlp/gru/src/weight_init.py
+++ b/model_zoo/official/nlp/gru/src/weight_init.py
@@ -15,21 +15,7 @@
 """weight init"""
 import math
 import numpy as np
-from mindspore import Tensor, Parameter
-
-def gru_default_state(batch_size, input_size, hidden_size, num_layers=1, bidirectional=False):
-    '''Weight init for gru cell'''
-    stdv = 1 / math.sqrt(hidden_size)
-    weight_i = Parameter(Tensor(
-        np.random.uniform(-stdv, stdv, (input_size, 3*hidden_size)).astype(np.float32)), name='weight_i')
-    weight_h = Parameter(Tensor(
-        np.random.uniform(-stdv, stdv, (hidden_size, 3*hidden_size)).astype(np.float32)), name='weight_h')
-    bias_i = Parameter(Tensor(
-        np.random.uniform(-stdv, stdv, (3*hidden_size)).astype(np.float32)), name='bias_i')
-    bias_h = Parameter(Tensor(
-        np.random.uniform(-stdv, stdv, (3*hidden_size)).astype(np.float32)), name='bias_h')
-    init_h = Tensor(np.zeros((batch_size, hidden_size)).astype(np.float16))
-    return weight_i, weight_h, bias_i, bias_h, init_h
+from mindspore import Tensor
 
 def dense_default_state(in_channel, out_channel):
     '''Weight init for dense cell'''
diff --git a/model_zoo/official/nlp/gru/train.py b/model_zoo/official/nlp/gru/train.py
index de219f93a4b..2d795e9ff77 100644
--- a/model_zoo/official/nlp/gru/train.py
+++ b/model_zoo/official/nlp/gru/train.py
@@ -15,17 +15,19 @@
 """train script"""
 import os
 import time
+import mindspore.common.dtype as mstype
 from mindspore.context import ParallelMode
 from mindspore import context
-from mindspore.communication.management import init
+from mindspore.communication.management import init, get_rank
 from mindspore.train.callback import Callback, CheckpointConfig, ModelCheckpoint, TimeMonitor
 from mindspore.train import Model
 from mindspore.common import set_seed
 from mindspore.train.loss_scale_manager import DynamicLossScaleManager
 from mindspore.nn.optim import Adam
+from mindspore import log as logger
 
 from src.seq2seq import Seq2Seq
-from src.gru_for_train import GRUWithLossCell, GRUTrainOneStepWithLossScaleCell
+from src.gru_for_train import GRUWithLossCell, GRUTrainOneStepWithLossScaleCell, GRUTrainOneStepCell
 from src.dataset import create_gru_dataset
 from src.lr_schedule import dynamic_lr
 
@@ -72,13 +74,20 @@ class LossCallBack(Callback):
                                                                      cb_params.cur_step_num,
                                                                      str(cb_params.net_outputs)))
         with open("./loss_{}.log".format(self.rank_id), "a+") as f:
-            f.write("time: {}, epoch: {}, step: {}, loss: {}, overflow: {}, loss_scale: {}".format(
-                time_stamp_current - time_stamp_first,
-                cb_params.cur_epoch_num,
-                cb_params.cur_step_num,
-                str(cb_params.net_outputs[0].asnumpy()),
-                str(cb_params.net_outputs[1].asnumpy()),
-                str(cb_params.net_outputs[2].asnumpy())))
+            if context.get_context("device_target") == "Ascend":
+                f.write("time: {}, epoch: {}, step: {}, loss: {}, overflow: {}, loss_scale: {}".format(
+                    time_stamp_current - time_stamp_first,
+                    cb_params.cur_epoch_num,
+                    cb_params.cur_step_num,
+                    str(cb_params.net_outputs[0].asnumpy()),
+                    str(cb_params.net_outputs[1].asnumpy()),
+                    str(cb_params.net_outputs[2].asnumpy())))
+            else:
+                f.write("time: {}, epoch: {}, step: {}, loss: {}".format(
+                    time_stamp_current - time_stamp_first,
+                    cb_params.cur_epoch_num,
+                    cb_params.cur_step_num,
+                    str(cb_params.net_outputs.asnumpy())))
             f.write('\n')
 
 
@@ -139,13 +148,32 @@ def modelarts_pre_process():
 @moxing_wrapper(pre_process=modelarts_pre_process)
 def run_train():
     """run train."""
-    context.set_context(mode=context.GRAPH_MODE, device_target="Ascend", device_id=get_device_id(), save_graphs=False)
-    rank = get_rank_id()
+    context.set_context(mode=context.GRAPH_MODE, device_target=config.device_target,
+                        device_id=get_device_id(), save_graphs=False)
+    if config.device_target == "GPU":
+        if config.compute_type != mstype.float32:
+            logger.warning('GPU only support fp32 temporarily, run with fp32.')
+            config.compute_type = mstype.float32
+
     device_num = get_device_num()
     if config.run_distribute:
-        context.set_auto_parallel_context(device_num=device_num, parallel_mode=ParallelMode.DATA_PARALLEL,
-                                          gradients_mean=True)
-        init()
+        if config.device_target == "Ascend":
+            rank = get_rank_id()
+            context.set_auto_parallel_context(device_num=device_num,
+                                              parallel_mode=ParallelMode.DATA_PARALLEL,
+                                              gradients_mean=True)
+            init()
+        elif config.device_target == "GPU":
+            rank = get_rank()
+            init("nccl")
+            context.set_auto_parallel_context(parallel_mode=ParallelMode.DATA_PARALLEL,
+                                              gradients_mean=True)
+        else:
+            raise ValueError(config.device_target)
+    else:
+        rank = 0
+        device_num = 1
+
     mindrecord_file = config.dataset_path
     if not os.path.exists(mindrecord_file):
         print("dataset file {} not exists, please check!".format(mindrecord_file))
@@ -162,8 +190,10 @@ def run_train():
                                             scale_factor=config.scale_factor,
                                             scale_window=config.scale_window)
     update_cell = scale_manager.get_update_cell()
-    netwithgrads = GRUTrainOneStepWithLossScaleCell(network, opt, update_cell)
-
+    if config.device_target == "Ascend":
+        netwithgrads = GRUTrainOneStepWithLossScaleCell(network, opt, update_cell)
+    else:
+        netwithgrads = GRUTrainOneStepCell(network, opt)
     time_cb = TimeMonitor(data_size=dataset_size)
     loss_cb = LossCallBack(rank_id=rank)
     cb = [time_cb, loss_cb]
@@ -171,10 +201,10 @@ def run_train():
     if config.save_checkpoint:
         ckpt_config = CheckpointConfig(save_checkpoint_steps=config.ckpt_epoch * dataset_size,
                                        keep_checkpoint_max=config.keep_checkpoint_max)
-        save_ckpt_path = os.path.join(config.outputs_dir, 'ckpt_' + str(get_rank_id()) + '/')
+        save_ckpt_path = os.path.join(config.outputs_dir, 'ckpt_' + str(rank) + '/')
         ckpt_cb = ModelCheckpoint(config=ckpt_config,
                                   directory=save_ckpt_path,
-                                  prefix='{}'.format(get_rank_id()))
+                                  prefix='{}'.format(rank))
         cb += [ckpt_cb]
     netwithgrads.set_train(True)
     model = Model(netwithgrads)