Data pipeline refactoring #416

hadipash · 2023-06-16T06:44:38Z

Thank you for your contribution to the MindOCR repo.
Before submitting this PR, please make sure:

You have read the Contributing Guidelines on pull requests
Your code builds clean without any errors or warnings
You are using approved terminology
You have added unit tests

Motivation

Refactored data pipeline to match best MindData practices, including:

Use GeneratorDataset for data loading only.
Use dataset.map operation to apply data transformations and augmentations.
Reduce number of Python transformations by grouping them into a single operation.
Group MindSpore operations as well.
Move to MindSpore operations where it is possible (Decode, Normalize, HWC2CHW).
Integrate MindRecord support.

mindocr/data/predict_dataset.py

mindocr/data/rec_lmdb_dataset.py

* Refactor recogniton data pipeline * Fix deterioration args * Fix transform * remove duplicated * merge PR #437 * Better name * better name * Fix SVTR transform * fix svtr transform * best setting for speed * Fix the random in CVRescale * better speed in inference * remove _Compose class from augmentations * move type checking inside `__init__` * remove non data related change --------- Co-authored-by: Rustam Khadipash <[email protected]>

refactor online prediction can run refactor PredictDataset finish refactor online inference data pipeline small fix

hadipash · 2023-07-07T07:59:22Z

Rebased onto the main branch to resolve conflicts.

# Conflicts: # configs/det/psenet/README.md # configs/det/psenet/README_CN.md # mindocr/data/transforms/svtr_transform.py # tests/ut/_common.py

# Conflicts: # configs/rec/visionlan/visionlan_resnet45_LA.yaml # configs/rec/visionlan/visionlan_resnet45_LF_1.yaml # configs/rec/visionlan/visionlan_resnet45_LF_2.yaml # deploy/py_infer/src/data_process/preprocess/preprocess_mapping.py # deploy/py_infer/src/data_process/preprocess/transforms/general_transforms.py # mindocr/data/transforms/svtr_transform.py # mindocr/losses/det_loss.py # tests/ut/test_infer_datasets.py

# Conflicts: # deploy/py_infer/src/data_process/preprocess/builder.py # tests/ut/test_infer_datasets.py

This reverts commit 7e3f3e1.

# Conflicts: # configs/det/dbnet/README.md # configs/det/dbnet/README_CN.md # configs/det/dbnet/db_r18_mlt2017.yaml # configs/det/dbnet/db_r18_td500.yaml # configs/det/dbnet/db_r18_totaltext.yaml # configs/det/dbnet/db_r50_mlt2017.yaml # configs/det/dbnet/db_r50_td500.yaml # configs/det/dbnet/db_r50_totaltext.yaml # requirements.txt

hadipash mentioned this pull request Jun 16, 2023

Data pipeline refactoring #300

Closed

4 tasks

zhtmike force-pushed the refactor_dp branch from 4524302 to 136f15c Compare June 20, 2023 02:37

hadipash commented Jul 6, 2023

View reviewed changes

mindocr/data/predict_dataset.py Outdated Show resolved Hide resolved

hadipash commented Jul 6, 2023

View reviewed changes

mindocr/data/rec_lmdb_dataset.py Outdated Show resolved Hide resolved

zhtmike and others added 8 commits July 7, 2023 15:37

switch to MindSpore native data pipeline

31f96d8

MindRecord loading bug fix

2d4c8aa

New dataset pipeline for PSENet upload (#438)

8f480de

update detection configs to match the new pipeline

85d3243

new dataset pipeline for east upload

47d11f6

Refactor online inference to adapt new data pipeline

3c85542

refactor online prediction can run refactor PredictDataset finish refactor online inference data pipeline small fix

small fix in parallel online inference

4864f99

hadipash force-pushed the refactor_dp branch from e7e0133 to 4864f99 Compare July 7, 2023 07:58

hadipash added 8 commits July 7, 2023 16:03

fix spelling

c4d3f4d

small fixes

16d47c6

tune OpenCV performance

b7727a1

adjust FCENet and ABINet to the new data pipeline

a724ed0

update docs

2dd3133

quick fix to the deployment pipeline

bf43ff0

fix unit tests

0589d47

fix unit tests

52d4ac1

hadipash marked this pull request as ready for review July 10, 2023 06:20

hadipash requested review from zhtmike, HaoyangLee, SamitHuang, liangxhao, Songyuanwei, VictorHe-1 and CaitinZhao July 10, 2023 06:45

zhtmike and others added 8 commits July 18, 2023 18:05

consistent batch size (#513)

eda3c80

Update east_mobilenetv3_icdar15.yaml for new data pipline

1059495

update test configuration to avoid mistakes in the future

f326aec

adjust VisionLan to the new pipeline

5e195c1

adjust EAST and PSENet to the new pipeline

51ee2c4

Merge branch 'main' into dataset_refactor

c18d049

# Conflicts: # configs/det/psenet/README.md # configs/det/psenet/README_CN.md # mindocr/data/transforms/svtr_transform.py # tests/ut/_common.py

ensure transform output tuple order by output_cols (#523)

39af076

Update test_models.py

b79b6ec

zhtmike force-pushed the refactor_dp branch from 54441ce to b79b6ec Compare July 24, 2023 09:06

zhtmike and others added 20 commits July 24, 2023 17:07

Merge branch 'main' into refactor_dp

999a951

merge fix

c9a0c1f

data pipeline for master (#525)

6ffaa08

keep accuracy of the ckpt and turn on test

f557c71

Merge branch 'main' into refactor_dp

71fe3a8

data pipeline for SVTR (#532)

9347437

small fix to IC15 converter

0e7f2bc

visionlan aug edits: remove CVRescale (#541)

104b93d

SVTR and RARE data pipeline (#548)

d4a1ec1

rename parameter

3a20051

adjust transformation pipeline for deployment

9d68033

Merge branch 'main' into dataset_refactor

680b1c3

# Conflicts: # deploy/py_infer/src/data_process/preprocess/builder.py # tests/ut/test_infer_datasets.py

check for MindOCR augmentations first (#555)

6833fb3

linting

903b72d

Update _common.py

7e52f00

change the accuracy back to the check point

73606a7

remove Decode to read images in correct orientation (#579)

7e3f3e1

Revert PR#579 as further improvements needed

e5d1ee4

This reverts commit 7e3f3e1.

hadipash mentioned this pull request Jan 18, 2024

improve data pipeline by adding MindData native support mindspore-lab/mindone#301

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Data pipeline refactoring #416

Data pipeline refactoring #416

hadipash commented Jun 16, 2023

hadipash commented Jul 7, 2023 •

edited

Loading

Data pipeline refactoring #416

Are you sure you want to change the base?

Data pipeline refactoring #416

Conversation

hadipash commented Jun 16, 2023

Motivation

hadipash commented Jul 7, 2023 • edited Loading

hadipash commented Jul 7, 2023 •

edited

Loading