Skip to content

Latest commit

 

History

History

smtr

readme.md

SMTR

1. Introduction

Paper:

Out of Length Text Recognition with Sub-String Matching Yongkun Du, Zhineng Chen*, Caiyan Jia, Xieping Gao, Yu-Gang Jiang

Scene Text Recognition (STR) methods have demonstrated robust performance in word-level text recognition. However, in applications the text image is sometimes long due to detected with multiple horizontal words. It triggers the requirement to build long text recognition models from readily available short word-level text datasets, which has been less studied previously. In this paper, we term this the Out of Length (OOL) text recognition. We establish the first Long Text Benchmark (LTB) to facilitate the assessment of different methods in long text recognition. Meanwhile, we propose a novel method called OOL Text Recognition with sub-String Matching (SMTR). SMTR comprises two cross-attention-based modules: one encodes a sub-string containing multiple characters into next and previous queries, and the other employs the queries to attend to the image features, matching the sub-string and simultaneously recognizing its next and previous character. SMTR can recognize text of arbitrary length by iterating the process above. To avoid being trapped in recognizing highly similar sub-strings, we introduce a regularization training to compel SMTR to effectively discover subtle differences between similar sub-strings for precise matching. In addition, we propose an inference augmentation to alleviate confusion caused by identical sub-strings and improve the overall recognition efficiency. Extensive experimental results reveal that SMTR, even when trained exclusively on short text, outperforms existing methods in public short text benchmarks and exhibits a clear advantage on LTB.

The accuracy (%) and model files of SMTR on the public dataset of scene text recognition are as follows:

Model Training Data LTB Config&Model&Log
SMTR Syn 39.6 link
SMTR U14M 51.0 link
FocalSVTR U14M 42.1 link
  • Test on Common Benchmarks from PARSeq:
Model Training Data IC13
857		 SVT IIIT5k
3000		 IC15
1811		 SVTP CUTE80 Avg Config&Model&Log
SMTR Syn 97.4 94.9 97.4 88.4 89.9 96.2 94.02 Same as the above table
SMTR U14M 98.3 97.4 99.0 90.1 92.7 97.9 95.90 Same as the above table
FocalSVTR U14M 97.3 96.3 98.2 87.4 88.4 96.2 93.97 Same as the above table
  • Test on Union14M-L benchmark from Union14M.
Model Traing Data Curve Multi-
Oriented		 Artistic Contextless Salient Multi-
word		 General Avg Config&Model&Log
SMTR Syn 74.2 30.6 58.5 67.6 79.6 75.1 67.9 64.79 Same as the above table
SMTR U14M 89.1 87.7 76.8 83.9 84.6 89.3 83.7 85.00 Same as the above table
FocalSVTR U14M 77.7 62.4 65.7 78.6 71.6 81.3 79.2 73.80 Same as the above table
Model Scene Web Document Handwriting Avg Config&Model&Log
SMTR w/o Aug 79.8 80.6 99.1 61.9 80.33 link
SMTR w/ Aug 83.4 83.0 99.3 65.1 82.68 link

Download all Configs, Models, and Logs from Google Drive.

2. Environment

  • PyTorch version >= 1.13.0
  • Python version >= 3.7
git clone -b develop https://github.com/Topdu/OpenOCR.git
cd OpenOCR
# A100 Ubuntu 20.04 Cuda 11.8
conda create -n openocr python==3.8
conda activate openocr
conda install pytorch==2.2.0 torchvision==0.17.0 torchaudio==2.2.0 pytorch-cuda=11.8 -c pytorch -c nvidia
pip install -r requirements.txt

Dataset Preparation

The expected filesystem structure is as follows:

benchmark_bctr
├── benchmark_bctr_test
│   ├── document_test
│   ├── handwriting_test
│   ├── scene_test
│   └── web_test
└── benchmark_bctr_train
    ├── document_train
    ├── handwriting_train
    ├── scene_train
    └── web_train
evaluation
├── CUTE80
├── IC13_857
├── IC15_1811
├── IIIT5k
├── SVT
└── SVTP
OpenOCR
synth
├── MJ
│   ├── test
│   ├── train
│   └── val
└── ST
test # from PARSeq
├── ArT
├── COCOv1.4
├── CUTE80
├── IC13_1015
├── IC13_1095  
├── IC13_857
├── IC15_1811
├── IC15_2077
├── IIIT5k
├── SVT
├── SVTP
└── Uber
u14m # lmdb format
├── artistic
├── contextless
├── curve
├── general
├── multi_oriented
├── multi_words
└── salient
ltb # download link: https://drive.google.com/drive/folders/1NChdlw7ustbXtlFBmh_0xnHvRkffb9Ge?usp=sharing
Union14M-LMDB-L # lmdb format
├── train_challenging
├── train_easy
├── train_hard
├── train_medium
└── train_normal

3. Model Training / Evaluation

Training:

# Multi GPU training
CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --nproc_per_node=4 tools/train_rec.py --c configs/rec/smtr/focalsvtr_smtr.yml
# For RTX 4090
NCCL_P2P_DISABLE=1 CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --nproc_per_node=4 tools/train_rec.py --c configs/rec/smtr/focalsvtr_smtr.yml

Evaluation:

# en
python tools/eval_rec_all_ratio.py --c configs/rec/smtr/focalsvtr_smtr.yml
# long text
python tools/eval_rec_all_long_simple.py --c configs/rec/smtr/focalsvtr_smtr_long.yml
# ch
python tools/eval_rec_all_ch.py --c configs/rec/smtr/focalsvtr_smtr_ch.yml

引用

@article{Du2024SMTR,
  title     = {Out of Length Text Recognition with Sub-String Matching},
  author    = {Yongkun Du, Zhineng Chen, Caiyan Jia, Xieping Gao, Yu-Gang Jiang},
  journal   = {CoRR},
  eprinttype = {arXiv},
  primaryClass={cs.CV},
  volume    = {abs/2407.12317},
  year      = {2024},
  url       = {https://arxiv.org/abs/2407.12317}
}