mindspore2022/model_zoo/official/cv/ctpn

CTPN

• CTPN Description
• Model Architecture
• Dataset
• Features
  • Mixed Precision
• Environment Requirements
• Script Description
  • Script and Sample Code
  • Training Process
  • Evaluation Process
    • Evaluation
• Model Description
  • Performance
    • Training Performance
    • Inference Performance
• Description of Random Situation
• ModelZoo Homepage

CTPN Description

CTPN is a text detection model based on object detection method. It improves Faster R-CNN and combines with bidirectional LSTM, so ctpn is very effective for horizontal text detection. Another highlight of ctpn is to transform the text detection task into a series of small-scale text box detection.This idea was proposed in the paper "Detecting Text in Natural Image with Connectionist Text Proposal Network".

Paper Zhi Tian, Weilin Huang, Tong He, Pan He, Yu Qiao, "Detecting Text in Natural Image with Connectionist Text Proposal Network", ArXiv, vol. abs/1609.03605, 2016.

Model architecture

The overall network architecture contains a VGG16 as backbone, and use bidirection lstm to extract context feature of the small-scale text box, then it used the RPN(RegionProposal Network) to predict the boundding box and probability.

Link

Dataset

Here we used 6 datasets for training, and 1 datasets for Evaluation.

• Dataset1: ICDAR 2013: Focused Scene Text
  • Train: 142MB, 229 images
  • Test: 110MB, 233 images
• Dataset2: ICDAR 2011: Born-Digital Images
  • Train: 27.7MB, 410 images
• Dataset3: ICDAR 2015:
  • Train：89MB, 1000 images
• Dataset4: SCUT-FORU: Flickr OCR Universal Database
  • Train: 388MB, 1715 images
• Dataset5: CocoText v2(Subset of MSCOCO2017):
  • Train: 13GB, 63686 images
• Dataset6: SVT(The Street View Dataset)
  • Train: 115MB, 349 images

Features

Environment Requirements

• Hardware（Ascend）
  • Prepare hardware environment with Ascend processor.
• Framework
  • MindSpore
• For more information, please check the resources below：
  • MindSpore Tutorials
  • MindSpore Python API

Script description

Script and sample code

└─ctpn
  ├── README.md                             # network readme
  ├──ascend310_infer                        #application for 310 inference
  ├── eval.py                               # eval net
  ├── scripts
  │   ├── eval_res.sh                       # calculate precision and recall
  │   ├── run_distribute_train_ascend.sh    # launch distributed training with ascend platform(8p)
  │   ├── run_eval_ascend.sh                # launch evaluating with ascend platform
  │   ├──run_infer_310.sh                   # shell script for 310 inference
  │   └── run_standalone_train_ascend.sh    # launch standalone training with ascend platform(1p)
  ├── src
  │   ├── CTPN
  │   │   ├── BoundingBoxDecode.py          # bounding box decode
  │   │   ├── BoundingBoxEncode.py          # bounding box encode
  │   │   ├── __init__.py                   # package init file
  │   │   ├── anchor_generator.py           # anchor generator
  │   │   ├── bbox_assign_sample.py         # proposal layer
  │   │   ├── proposal_generator.py         # proposla generator
  │   │   ├── rpn.py                        # region-proposal network
  │   │   └── vgg16.py                      # backbone
  │   ├── model_utils
  │   │   ├──config.py             // Parameter config
  │   │   ├──moxing_adapter.py     // modelarts device configuration
  │   │   ├──device_adapter.py     // Device Config
  │   │   ├──local_adapter.py      // local device config
  │   ├── convert_icdar2015.py              # convert icdar2015 dataset label
  │   ├── convert_svt.py                    # convert svt label
  │   ├── create_dataset.py                 # create mindrecord dataset
  │   ├── ctpn.py                           # ctpn network definition
  │   ├── dataset.py                        # data proprocessing
  │   ├── eval_callback.py                  # evaluation callback while training
  │   ├── eval_utils.py                     # evaluation function
  │   ├── lr_schedule.py                    # learning rate scheduler
  │   ├── network_define.py                 # network definition
  │   └── text_connector
  │       ├── __init__.py                   # package init file
  │       ├── connect_text_lines.py         # connect text lines
  │       ├── detector.py                   # detect box
  │       ├── get_successions.py            # get succession proposal
  │       └── utils.py                      # some functions which is commonly used
  ├──postprogress.py                        # post process for 310 inference
  ├──export.py                              # script to export AIR,MINDIR model
  └── train.py                              # train net
  ├── default_config.yaml                   #  config file

Training process

Dataset

To create dataset, download the dataset first and deal with it.We provided src/convert_svt.py and src/convert_icdar2015.py to deal with svt and icdar2015 dataset label.For svt dataset, you can deal with it as below:

    python convert_svt.py --dataset_path=/path/img --xml_file=/path/train.xml --location_dir=/path/location

For ICDAR2015 dataset, you can deal with it

    python convert_icdar2015.py --src_label_path=/path/train_label --target_label_path=/path/label

Then modify the src/config.py and add the dataset path.For each path, add IMAGE_PATH and LABEL_PATH into a list in config.An example is show as blow:

    # create dataset
    "coco_root": "/path/coco",
    "coco_train_data_type": "train2017",
    "cocotext_json": "/path/cocotext.v2.json",
    "icdar11_train_path": ["/path/image/", "/path/label"],
    "icdar13_train_path": ["/path/image/", "/path/label"],
    "icdar15_train_path": ["/path/image/", "/path/label"],
    "icdar13_test_path": ["/path/image/", "/path/label"],
    "flick_train_path": ["/path/image/", "/path/label"],
    "svt_train_path": ["/path/image/", "/path/label"],
    "pretrain_dataset_path": "",
    "finetune_dataset_path": "",
    "test_dataset_path": "",

Then you can create dataset with src/create_dataset.py with the command as below:

python src/create_dataset.py

Usage

• Ascend:

if pretraining set pretraining_dataset_file: /home/DataSet/ctpn_dataset/pretrain/ctpn_pretrain.mindrecord0
if finetune set pretraining_dataset_file: /home/DataSet/ctpn_dataset/finetune/ctpn_finetune.mindrecord0
img_dir:/home/DataSet/ctpn_dataset/ICDAR2013/test

Modify the parameters according to the actual path

# distribute training
bash scripts/run_distribute_train_ascend.sh [RANK_TABLE_FILE] [TASK_TYPE] [PRETRAINED_PATH]
# example: bash scripts/run_distribute_train_ascend.sh /home/hccl_8p_01234567_10.155.170.71.json Pretraining(or Finetune) \
# /home/DataSet/ctpn_dataset/backbone/0-150_5004.ckpt

# standalone training
bash scrpits/run_standalone_train_ascend.sh [TASK_TYPE] [PRETRAINED_PATH] [DEVICE_ID]
example: bash scrpits/run_standalone_train_ascend.sh Pretraining(or Finetune) /home/DataSet/ctpn_dataset/backbone/0-150_5004.ckpt 0

# evaluation:
bash scripts/run_eval_ascend.sh [IMAGE_PATH] [DATASET_PATH] [CHECKPOINT_PATH]
# example: bash script/run_eval_ascend.sh /home/DataSet/ctpn_dataset/ICDAR2013/test \
# /home/DataSet/ctpn_dataset/ctpn_final_dataset/test/ctpn_test.mindrecord /home/model/cv/ctpn/train_parallel0/ckpt_0/

The pretrained_path should be a checkpoint of vgg16 trained on Imagenet2012. The name of weight in dict should be totally the same, also the batch_norm should be enabled in the trainig of vgg16, otherwise fails in further steps.COCO_TEXT_PARSER_PATH coco_text.py can refer to Link.To get the vgg16 backbone, you can use the network structure defined in src/CTPN/vgg16.py.To train the backbone, copy the src/CTPN/vgg16.py under modelzoo/official/cv/vgg16/src/, and modify the vgg16/train.py to suit the new construction.You can fix it as below:

...
from src.vgg16 import VGG16
...
network = VGG16(num_classes=cfg.num_classes)
...

To train a better model, you can modify some parameter in modelzoo/official/cv/vgg16/src/config.py, here we suggested you modify the "warmup_epochs" just like below, you can also try to adjust other parameter.

imagenet_cfg = edict({
    ...
    "warmup_epochs": 5
    ...
})

Then you can train it with ImageNet2012.

Notes: RANK_TABLE_FILE can refer to Link , and the device_ip can be got as Link. For large models like InceptionV4, it's better to export an external environment variable export HCCL_CONNECT_TIMEOUT=600 to extend hccl connection checking time from the default 120 seconds to 600 seconds. Otherwise, the connection could be timeout since compiling time increases with the growth of model size.

This is processor cores binding operation regarding the device_num and total processor numbers. If you are not expect to do it, remove the operations taskset in scripts/run_distribute_train.sh

TASK_TYPE contains Pretraining and Finetune. For Pretraining, we use ICDAR2013, ICDAR2015, SVT, SCUT-FORU, CocoText v2. For Finetune, we use ICDAR2011, ICDAR2013, SCUT-FORU to improve precision and recall, and when doing Finetune, we use the checkpoint training in Pretrain as our PRETRAINED_PATH. COCO_TEXT_PARSER_PATH coco_text.py can refer to Link.

Launch

# training example
  shell:
    Ascend:
      # distribute training example(8p)
      bash run_distribute_train_ascend.sh [RANK_TABLE_FILE] [TASK_TYPE] [PRETRAINED_PATH]
      # example: bash scripts/run_distribute_train_ascend.sh /home/hccl_8p_01234567_10.155.170.71.json Pretraining(or Finetune) /home/DataSet/ctpn_dataset/backbone/0-150_5004.ckpt

      # standalone training
      bash run_standalone_train_ascend.sh [TASK_TYPE] [PRETRAINED_PATH]
      # example: bash scrpits/run_standalone_train_ascend.sh Pretraining(or Finetune) /home/DataSet/ctpn_dataset/backbone/0-150_5004.ckpt 0

Result

Training result will be stored in the example path. Checkpoints will be stored at ckpt_path by default, and training log will be redirected to ./log, also the loss will be redirected to ./loss_0.log like followings.

377 epoch: 1 step: 229 ,rpn_loss: 0.00355
399 epoch: 2 step: 229 ,rpn_loss: 0.00327
424 epoch: 3 step: 229 ,rpn_loss: 0.00910

• running on ModelArts
• If you want to train the model on modelarts, you can refer to the [official guidance document] of modelarts (https://support.huaweicloud.com/modelarts/)

#  Example of using distributed training dpn on modelarts :
#  Data set storage method

#  ├── ctpn_dataset              # dir
#    ├──train                         # train dir
#      ├── pretrain               # pretrain dataset dir
#      ├── finetune               # finetune dataset dir
#      ├── backbone               # predtrained dir if exists
#    ├── eval                    # eval dir
#      ├── ICDAR2013              # ICDAR2013 img dir
#      ├── checkpoint           # ckpt files dir
#      ├── test                  # ckpt files dir
#          ├── ctpn_test.mindrecord       # test img of mindrecord
#          ├── ctpn_test.mindrecord.db    # test img of mindrecord.db

# (1) Choose either a (modify yaml file parameters) or b (modelArts create training job to modify parameters) 。
#       a. set "enable_modelarts=True" 。
#          set "run_distribute=True"
#          set "save_checkpoint_path=/cache/train/checkpoint/"
#          set "finetune_dataset_file=/cache/data/finetune/ctpn_finetune.mindrecord0"
#          set "pretrain_dataset_file=/cache/data/finetune/ctpn_pretrain.mindrecord0"
#          set "task_type=Pretraining" or task_type=Finetune
#          set "pre_trained=/cache/data/backbone/pred file name" Without pre-training weights  pre_trained=""
#
#       b. add "enable_modelarts=True" Parameters are on the interface of modearts。
#          Set the parameters required by method a on the modelarts interface
#          Note: The path parameter does not need to be quoted

# (2) Set the path of the network configuration file  "_config_path=/The path of config in default_config.yaml/"
# (3) Set the code path on the modelarts interface "/path/ctpn"。
# (4) Set the model's startup file on the modelarts interface "train.py" 。
# (5) Set the data path of the model on the modelarts interface ".../ctpn_dataset/train"(choices ctpn_dataset/train Folder path) ,
# The output path of the model "Output file path" and the log path of the model "Job log path" 。
# (6) start trainning the model。

# Example of using model inference on modelarts
# (1) Place the trained model to the corresponding position of the bucket。
# (2) chocie a or b。
#       a. set "enable_modelarts=True" 。
#          set "dataset_path=/cache/data/test/ctpn_test.mindrecord"
#          set "img_dir=/cache/data/ICDAR2013/test"
#          set "checkpoint_path=/cache/data/checkpoint/checkpoint file name"

#       b. Add "enable_modelarts=True" parameter on the interface of modearts。
#          Set the parameters required by method a on the modelarts interface
#          Note: The path parameter does not need to be quoted

# (3) Set the path of the network configuration file "_config_path=/The path of config in default_config.yaml/"
# (4) Set the code path on the modelarts interface "/path/ctpn"。
# (5) Set the model's startup file on the modelarts interface "eval.py" 。
# (6) Set the data path of the model on the modelarts interface ".../ctpn_dataset/eval"(choices FSNS/eval Folder path) ,
# The output path of the model "Output file path" and the log path of the model "Job log path"  。
# (7) Start model inference。

Eval process

Usage

You can start training using python or shell scripts. The usage of shell scripts as follows:

• Ascend:

  bash run_eval_ascend.sh [IMAGE_PATH] [DATASET_PATH] [CHECKPOINT_PATH]
  # example: bash script/run_eval_ascend.sh /home/DataSet/ctpn_dataset/ICDAR2013/test /home/DataSet/ctpn_dataset/ctpn_final_dataset/test/ctpn_test.mindrecord /home/model/cv/ctpn/train_parallel0/ckpt_0/

After eval, you can get serval archive file named submit_ctpn-xx_xxxx.zip, which contains the name of your checkpoint file.To evalulate it, you can use the scripts provided by the ICDAR2013 network, you can download the Deteval scripts from the link After download the scripts, unzip it and put it under ctpn/scripts and use eval_res.sh to get the result.You will get files as below:

gt.zip
readme.txt
rrc_evalulation_funcs_1_1.py
script.py

Then you can run the scripts/eval_res.sh to calculate the evalulation result.

bash eval_res.sh

Evaluation while training

You can add run_eval to start shell and set it True, if you want evaluation while training. And you can set argument option: eval_dataset_path, save_best_ckpt, eval_start_epoch, eval_interval when run_eval is True.

Result

Evaluation result will be stored in the example path, you can find result like the followings in log.

{"precision": 0.90791, "recall": 0.86118, "hmean": 0.88393}

Model Export

python export.py --ckpt_file [CKPT_PATH] --file_format[EXPORT_FORMAT]

• Export MindIR on Modelarts

Export MindIR example on ModelArts
Data storage method is the same as training
# (1) Choose either a (modify yaml file parameters) or b (modelArts create training job to modify parameters)。
#       a. set "enable_modelarts=True"
#          set "file_name=ctpn"
#          set "file_format=MINDIR"
#          set "ckpt_file=/cache/data/checkpoint file name"

#       b. Add "enable_modelarts=True" parameter on the interface of modearts。
#          Set the parameters required by method a on the modelarts interface
#          Note: The path parameter does not need to be quoted
# (2)Set the path of the network configuration file "_config_path=/The path of config in default_config.yaml/"
# (3) Set the code path on the modelarts interface "/path/ctpn"。
# (4) Set the model's startup file on the modelarts interface "export.py" 。
# (5) Set the data path of the model on the modelarts interface ".../ctpn_dataset/eval/checkpoint"(choices CNNCTC_Data/eval/checkpoint Folder path) ,
# The output path of the model "Output file path" and the log path of the model "Job log path"  。

EXPORT_FORMAT should be in ["AIR", "MINDIR"]

Inference process

Usage

Before performing inference, the air file must bu exported by export script on the Ascend910 environment.

# Ascend310 inference
bash run_infer_310.sh [MODEL_PATH] [DATA_PATH] [ANN_FILE_PATH] [DEVICE_ID]]

After inference, you can get a archive file named submit.zip.To evalulate it, you can use the scripts provided by the ICDAR2013 network, you can download the Deteval scripts from the link After download the scripts, unzip it and put it under ctpn/scripts and use eval_res.sh to get the result.You will get files as below:

gt.zip
readme.txt
rrc_evalulation_funcs_1_1.py
script.py

Then you can run the scripts/eval_res.sh to calculate the evalulation result.

bash eval_res.sh

Result

Evaluation result will be stored in the example path, you can find result like the followings in log.

{"precision": 0.88913, "recall": 0.86082, "hmean": 0.87475}

Model description

Performance

Training Performance

Parameters	Ascend
Model Version	CTPN
Resource	Ascend 910; cpu 2.60GHz, 192cores; memory 755G; OS Euler2.8
uploaded Date	02/06/2021
MindSpore Version	1.1.1
Dataset	16930 images
Batch_size	2
Training Parameters	src/config.py
Optimizer	Momentum
Loss Function	SoftmaxCrossEntropyWithLogits for classification, SmoothL2Loss for bbox regression
Loss	~0.04
Total time (8p)	6h
Scripts	ctpn script

Inference Performance

Parameters	Ascend
Model Version	CTPN
Resource	Ascend 910; cpu 2.60GHz, 192cores; memory 755G; OS Euler2.8
Uploaded Date	02/06/2020
MindSpore Version	1.1.1
Dataset	229 images
Batch_size	1
Accuracy	precision=0.9079, recall=0.8611 F-measure:0.8839
Total time	1 min
Model for inference	135M (.ckpt file)

Training performance results

Ascend	train performance
1p	10 img/s

Ascend	train performance
8p	84 img/s

Description of Random Situation

We set seed to 1 in train.py.

ModelZoo Homepage

Please check the official homepage.