Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data pipeline refactoring #416

Open
wants to merge 56 commits into
base: main
Choose a base branch
from
Open

Data pipeline refactoring #416

wants to merge 56 commits into from

Commits on Jul 7, 2023

  1. switch to MindSpore native data pipeline

    @zhtmike @hadipash
    zhtmike authored and hadipash committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    31f96d8 View commit details
    Browse the repository at this point in the history

  2. MindRecord loading bug fix

    @hadipash
    hadipash committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    2d4c8aa View commit details
    Browse the repository at this point in the history

  3. New dataset pipeline for PSENet upload (#438)

    @VictorHe-1 @hadipash
    VictorHe-1 authored and hadipash committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    8f480de View commit details
    Browse the repository at this point in the history

  4. update detection configs to match the new pipeline

    @hadipash
    hadipash committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    85d3243 View commit details
    Browse the repository at this point in the history

  5. new dataset pipeline for east upload

    @hadipash
    songyuanwei authored and hadipash committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    47d11f6 View commit details
    Browse the repository at this point in the history

  6. Refactor recogniton data pipeline (#430) 

    * Refactor recogniton data pipeline

* Fix deterioration args

* Fix transform

* remove duplicated

* merge PR #437

* Better name

* better name

* Fix SVTR transform

* fix svtr transform

* best setting for speed

* Fix the random in CVRescale

* better speed in inference

* remove _Compose class from augmentations

* move type checking inside `__init__`

* remove non data related change

---------

Co-authored-by: Rustam Khadipash <[email protected]>
    @zhtmike @hadipash
    zhtmike and hadipash committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    ebc7b70 View commit details
    Browse the repository at this point in the history

  7. Refactor online inference to adapt new data pipeline 

    refactor online prediction can run

refactor PredictDataset

finish refactor online inference data pipeline

small fix
    @HaoyangLee @hadipash
    HaoyangLee authored and hadipash committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    3c85542 View commit details
    Browse the repository at this point in the history

  8. small fix in parallel online inference

    @HaoyangLee @hadipash
    HaoyangLee authored and hadipash committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    4864f99 View commit details
    Browse the repository at this point in the history

  9. fix spelling

    @hadipash
    hadipash committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    c4d3f4d View commit details
    Browse the repository at this point in the history

  10. small fixes

    @hadipash
    hadipash committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    16d47c6 View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2023

  1. tune OpenCV performance

    @hadipash
    hadipash committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    b7727a1 View commit details
    Browse the repository at this point in the history

  2. adjust FCENet and ABINet to the new data pipeline

    @hadipash
    hadipash committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    a724ed0 View commit details
    Browse the repository at this point in the history

  3. update docs

    @hadipash
    hadipash committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    2dd3133 View commit details
    Browse the repository at this point in the history

  4. quick fix to the deployment pipeline

    @hadipash
    hadipash committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    bf43ff0 View commit details
    Browse the repository at this point in the history

  5. fix unit tests

    @hadipash
    hadipash committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    0589d47 View commit details
    Browse the repository at this point in the history

  6. fix unit tests

    @hadipash
    hadipash committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    52d4ac1 View commit details
    Browse the repository at this point in the history

Commits on Jul 13, 2023

  1. update throughputs in README

    @hadipash
    hadipash committed Jul 13, 2023
    Configuration menu
    Copy the full SHA
    1b4355c View commit details
    Browse the repository at this point in the history

Commits on Jul 14, 2023

  1. Merge branch 'main' into dataset_refactor 

    # Conflicts:
#	configs/det/psenet/README.md
#	configs/det/psenet/README_CN.md
#	configs/rec/svtr/svtr_tiny.yaml
#	mindocr/data/builder.py
#	mindocr/data/rec_lmdb_dataset.py
#	mindocr/data/transforms/svtr_transform.py
    @hadipash
    hadipash committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    0f4409c View commit details
    Browse the repository at this point in the history

  2. unpdate readme

    songyuanwei committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    06488f7 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'refactor_dp' of https://github.com/mindspore-lab/mindocr … 

    …into refactor_dp
    songyuanwei committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    11daea6 View commit details
    Browse the repository at this point in the history

  4. fix spelling

    @hadipash
    hadipash committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    d914f6e View commit details
    Browse the repository at this point in the history

  5. loss fix to pass tests

    @hadipash
    hadipash committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    ffc4a59 View commit details
    Browse the repository at this point in the history

  6. quick fix to online inference

    @hadipash
    hadipash committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    273cdb9 View commit details
    Browse the repository at this point in the history

Commits on Jul 18, 2023

  1. temporary turn the CI partially to identify the problem

    @hadipash
    hadipash committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    8bb302d View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into dataset_refactor 

    # Conflicts:
#	configs/det/psenet/README.md
#	configs/det/psenet/README_CN.md
    @hadipash
    hadipash committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    45651d0 View commit details
    Browse the repository at this point in the history

  3. revert CI changes in 8bb302d

    @hadipash
    hadipash committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    058564a View commit details
    Browse the repository at this point in the history

  4. update performance

    @hqkate
    hqkate committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    146a1c8 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'refactor_dp' of https://github.com/mindspore-lab/mindocr … 

    …into refactor_dp
    @hqkate
    hqkate committed Jul 18, 2023
    Configuration menu
    Copy the full SHA
    e7e0d1c View commit details
    Browse the repository at this point in the history

  6. consistent batch size (#513)

    @zhtmike
    zhtmike authored Jul 18, 2023
    Configuration menu
    Copy the full SHA
    eda3c80 View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2023

  1. Update east_mobilenetv3_icdar15.yaml for new data pipline

    @Songyuanwei
    Songyuanwei authored Jul 19, 2023
    Configuration menu
    Copy the full SHA
    1059495 View commit details
    Browse the repository at this point in the history

  2. update test configuration to avoid mistakes in the future

    @hadipash
    hadipash committed Jul 19, 2023
    Configuration menu
    Copy the full SHA
    f326aec View commit details
    Browse the repository at this point in the history

Commits on Jul 21, 2023

  1. adjust VisionLan to the new pipeline

    @hadipash
    hadipash committed Jul 21, 2023
    Configuration menu
    Copy the full SHA
    5e195c1 View commit details
    Browse the repository at this point in the history

Commits on Jul 24, 2023

  1. adjust EAST and PSENet to the new pipeline

    @hadipash
    hadipash committed Jul 24, 2023
    Configuration menu
    Copy the full SHA
    51ee2c4 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into dataset_refactor 

    # Conflicts:
#	configs/det/psenet/README.md
#	configs/det/psenet/README_CN.md
#	mindocr/data/transforms/svtr_transform.py
#	tests/ut/_common.py
    @hadipash
    hadipash committed Jul 24, 2023
    Configuration menu
    Copy the full SHA
    c18d049 View commit details
    Browse the repository at this point in the history

  3. ensure transform output tuple order by output_cols (#523)

    @wtomin
    wtomin authored Jul 24, 2023
    Configuration menu
    Copy the full SHA
    39af076 View commit details
    Browse the repository at this point in the history

  4. Update test_models.py

    @Songyuanwei
    Songyuanwei authored Jul 24, 2023
    Configuration menu
    Copy the full SHA
    b79b6ec View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into refactor_dp

    @zhtmike
    zhtmike committed Jul 24, 2023
    Configuration menu
    Copy the full SHA
    999a951 View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2023

  1. merge fix

    @hadipash
    hadipash committed Jul 25, 2023
    Configuration menu
    Copy the full SHA
    c9a0c1f View commit details
    Browse the repository at this point in the history

Commits on Jul 26, 2023

  1. data pipeline for master (#525)

    @zhtmike
    zhtmike authored Jul 26, 2023
    Configuration menu
    Copy the full SHA
    6ffaa08 View commit details
    Browse the repository at this point in the history

  2. keep accuracy of the ckpt and turn on test

    @zhtmike
    zhtmike committed Jul 26, 2023
    Configuration menu
    Copy the full SHA
    f557c71 View commit details
    Browse the repository at this point in the history

Commits on Jul 27, 2023

  1. Merge branch 'main' into refactor_dp

    @zhtmike
    zhtmike committed Jul 27, 2023
    Configuration menu
    Copy the full SHA
    71fe3a8 View commit details
    Browse the repository at this point in the history

Commits on Jul 28, 2023

  1. data pipeline for SVTR (#532)

    @zhtmike
    zhtmike authored Jul 28, 2023
    Configuration menu
    Copy the full SHA
    9347437 View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2023

  1. small fix to IC15 converter

    @hadipash
    hadipash committed Aug 2, 2023
    Configuration menu
    Copy the full SHA
    0e7f2bc View commit details
    Browse the repository at this point in the history

Commits on Aug 3, 2023

  1. visionlan aug edits: remove CVRescale (#541)

    @wtomin
    wtomin authored Aug 3, 2023
    Configuration menu
    Copy the full SHA
    104b93d View commit details
    Browse the repository at this point in the history

Commits on Aug 4, 2023

  1. SVTR and RARE data pipeline (#548)

    @zhtmike
    zhtmike authored Aug 4, 2023
    Configuration menu
    Copy the full SHA
    d4a1ec1 View commit details
    Browse the repository at this point in the history

  2. rename parameter

    @hadipash
    hadipash committed Aug 4, 2023
    Configuration menu
    Copy the full SHA
    3a20051 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into dataset_refactor 

    # Conflicts:
#	configs/rec/visionlan/visionlan_resnet45_LA.yaml
#	configs/rec/visionlan/visionlan_resnet45_LF_1.yaml
#	configs/rec/visionlan/visionlan_resnet45_LF_2.yaml
#	deploy/py_infer/src/data_process/preprocess/preprocess_mapping.py
#	deploy/py_infer/src/data_process/preprocess/transforms/general_transforms.py
#	mindocr/data/transforms/svtr_transform.py
#	mindocr/losses/det_loss.py
#	tests/ut/test_infer_datasets.py
    @hadipash
    hadipash committed Aug 4, 2023
    Configuration menu
    Copy the full SHA
    dafbcfe View commit details
    Browse the repository at this point in the history

  4. adjust transformation pipeline for deployment

    @hadipash
    hadipash committed Aug 4, 2023
    Configuration menu
    Copy the full SHA
    9d68033 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'main' into dataset_refactor 

    # Conflicts:
#	deploy/py_infer/src/data_process/preprocess/builder.py
#	tests/ut/test_infer_datasets.py
    @hadipash
    hadipash committed Aug 4, 2023
    Configuration menu
    Copy the full SHA
    680b1c3 View commit details
    Browse the repository at this point in the history

  6. check for MindOCR augmentations first (#555)

    @hadipash
    hadipash authored Aug 4, 2023
    Configuration menu
    Copy the full SHA
    6833fb3 View commit details
    Browse the repository at this point in the history

  7. linting

    @hadipash
    hadipash committed Aug 4, 2023
    Configuration menu
    Copy the full SHA
    903b72d View commit details
    Browse the repository at this point in the history

  8. Update _common.py

    @Songyuanwei
    Songyuanwei authored Aug 4, 2023
    Configuration menu
    Copy the full SHA
    7e52f00 View commit details
    Browse the repository at this point in the history

  9. change the accuracy back to the check point

    @zhtmike
    zhtmike committed Aug 4, 2023
    Configuration menu
    Copy the full SHA
    73606a7 View commit details
    Browse the repository at this point in the history

Commits on Aug 11, 2023

  1. remove Decode to read images in correct orientation (#579)

    @hadipash
    hadipash authored Aug 11, 2023
    Configuration menu
    Copy the full SHA
    7e3f3e1 View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2023

  1. Revert PR#579 as further improvements needed 

    This reverts commit 7e3f3e1.
    @hadipash
    hadipash committed Aug 16, 2023
    Configuration menu
    Copy the full SHA
    e5d1ee4 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into refactor_dp 

    # Conflicts:
#	configs/det/dbnet/README.md
#	configs/det/dbnet/README_CN.md
#	configs/det/dbnet/db_r18_mlt2017.yaml
#	configs/det/dbnet/db_r18_td500.yaml
#	configs/det/dbnet/db_r18_totaltext.yaml
#	configs/det/dbnet/db_r50_mlt2017.yaml
#	configs/det/dbnet/db_r50_td500.yaml
#	configs/det/dbnet/db_r50_totaltext.yaml
#	requirements.txt
    @hadipash
    hadipash committed Aug 16, 2023
    Configuration menu
    Copy the full SHA
    f7db655 View commit details
    Browse the repository at this point in the history