Translate texts in manga/images.
中文说明 | Change Log
Some manga/images will never be translated, therefore this project is born.
Primarily designed for translating Japanese text, but also support Chinese, English and Korean.
Support inpainting and text rendering.
Successor to https://github.com/PatchyVideo/MMDOCR-HighPerformance
This is a hobby project, you are welcome to contribute!
Currently this only a simple demo, many imperfections exist, we need your support to make this project better!
GPU server is not cheap, please consider to donate to us.
- Ko-fi: https://ko-fi.com/voilelabs
- Patreon: https://www.patreon.com/voilelabs
- 爱发电: https://afdian.net/@voilelabs
Official Demo (by zyddnys): https://touhou.ai/imgtrans/
Browser Userscript (by QiroNT): https://greasyfork.org/scripts/437569
- Note this may not work sometimes due to stupid google gcp kept restarting my instance. In that case you can wait for me to restart the service, which may take up to 24 hrs.
- Note this online demo is using the current main branch version.
# First, you need to have Python(>=3.8) installed on your system.
$ python --version
Python 3.8.13
# Clone this repo
$ git clone https://github.com/zyddnys/manga-image-translator.git
# Install the dependencies
$ pip install -r requirements.txt
However, pydensecrf
isn't listed as a dependency, so you need to install it manually.
On Windows you can download the pre-compiled wheels from https://www.lfd.uci.edu/~gohlke/pythonlibs/#_pydensecrf
according to your python version and install it with pip.
On other platforms, you should be able to install it via pip install git+https://github.com/lucasb-eyer/pydensecrf.git
.
Then, download ocr.ckpt
, ocr-ctc.ckpt
, detect.ckpt
, comictextdetector.pt
, comictextdetector.pt.onnx
and inpainting_lama_mpe.ckpt
from https://github.com/zyddnys/manga-image-translator/releases/, put them in the root directory of this repo.
[Optional if using Google translate]
Apply for Youdao or DeepL translate API, put your APP_KEY
and APP_SECRET
or AUTH_KEY
in translators/key.py
Used by the --target-lang
argument.
CHS: Chinese (Simplified)
CHT: Chinese (Traditional)
CSY: Czech
NLD: Dutch
ENG: English
FRA: French
DEU: German
HUN: Hungarian
ITA: Italian
JPN: Japanese
KOR: Korean
PLK: Polish
PTB: Portuguese (Brazil)
ROM: Romanian
RUS: Russian
ESP: Spanish
TRK: Turkish
VIN: Vietnames
# `--use-cuda` is optional, if you have a compatible NVIDIA GPU, you can use it.
# use `--use-inpainting` to enable inpainting.
# use `--translator=<translator>` to specify a translator.
# use `--target-lang=<languge_code>` to specify a target language.
# replace <path_to_image_file> with the path to the image file.
$ python translate_demo.py --verbose --use-inpainting --use-cuda --translator=google --target-lang=ENG --image <path_to_image_file>
# result can be found in `result/`.
# same options as above.
# use `--mode batch` to enable batch translation.
# replace <path_to_image_folder> with the path to the image folder.
$ python translate_demo.py --verbose --mode batch --use-inpainting --use-cuda --translator=google --target-lang=ENG --image <path_to_image_folder>
# results can be found in `<path_to_image_folder>-translated/`.
# same options as above.
# use `--mode web` to start a web server.
$ python translate_demo.py --verbose --mode web --use-inpainting --use-cuda
# the demo will be serving on http://127.0.0.1:5003>
Two modes of translation service are provided by the demo: synchronous mode and asynchronous mode.
In synchronous mode your HTTP POST request will finish once the translation task is finished.
In asynchronous mode your HTTP POST request will respond with a task_id
immediately, you can use this task_id
to poll for translation task state.
- POST a form request with form data
file:<content-of-image>
to http://127.0.0.1:5003/run - Wait for response
- Use the resultant
task_id
to find translation result inresult/
directory, e.g. using Nginx to exposeresult/
- POST a form request with form data
file:<content-of-image>
to http://127.0.0.1:5003/submit - Acquire translation
task_id
- Poll for translation task state by posting JSON
{"taskid": <task-id>}
to http://127.0.0.1:5003/task-state - Translation is finished when the resultant state is either
finished
,error
orerror-lang
- Find translation result in
result/
directory, e.g. using Nginx to exposeresult/
Manual translation replace machine translation with human translators. Basic manual translation demo can be found at http://127.0.0.1:5003/manual
POST a form request with form data file:<content-of-image>
to http://127.0.0.1:5003/manual-translate
and wait for response.
You will obtain a JSON response like this:
{
"task_id": "12c779c9431f954971cae720eb104499",
"status": "pending",
"trans_result": [
{
"s": "☆上司来ちゃった……",
"t": ""
}
]
}
Fill in translated texts:
{
"task_id": "12c779c9431f954971cae720eb104499",
"status": "pending",
"trans_result": [
{
"s": "☆上司来ちゃった……",
"t": "☆Boss is here..."
}
]
}
Post translated JSON to http://127.0.0.1:5003/post-translation-result and wait for response.
Then you can find the translation result in result/
directory, e.g. using Nginx to expose result/
.
A list of what needs to be done next, you're welcome to contribute.
- Inpainting is based on Aggregated Contextual Transformations for High-Resolution Image Inpainting
IMPORTANT!!!HELP NEEDED!!! The current text rendering engine is barely usable, we need your help to improve text rendering!- Text rendering area is determined by detected text lines, not speech bubbles.
This works for images without speech bubbles, but making it impossible to decide where to put translated English text. I have no idea how to solve this. - Ryota et al. proposed using multimodal machine translation, maybe we can add ViT features for building custom NMT models.
- Make this project works for video(rewrite code in C++ and use GPU/other hardware NN accelerator).
Used for detecting hard subtitles in videos, generting ass file and remove them completetly. Mask refinement based using non deep learning algorithms, I am currently testing out CRF based algorithm.Angled text region merge is not currently supported
The following samples are from the original version, they do not represent the current main branch version.
Original | Translated |
---|---|