HDR setting applied to autoProcess.ini but not being carried over during Transcoding\Remux #1285
Comments
|
This will be very challenging to fix due to limitations with FFMPEG and FFPROBE But unless you can figure out a way to read those values via FFPROBE and reliable include them in the params for the command generated for the output file I'm not sure how else to fix it. If you can find where to read that info and post some resulting FFMPEG commands that generate what you're after I can try and build it |
|
I know others have been using the --master-display option and the Maxcll\fall options in a CLI script, I just dont know were to add it to you autoprocess.ini file. http://x265.readthedocs.io/en/default/cli.html --master-display Example for a P3D65 1000-nits monitor, where G(x=0.265, y=0.690), B(x=0.150, y=0.060), R(x=0.680, y=0.320), WP(x=0.3127, y=0.3290), L(max=1000, min=0.0001): G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,1) --max-cll Specified as a string which is parsed when the stream header SEI are emitted. The string format is “%hu,%hu” where %hu are unsigned 16bit integers. The first value is the max content light level (or 0 if no maximum is indicated), the second value is the maximum picture average light level (or 0). Applicable for HDR content. Example for MaxCLL=1000 candela per square meter, MaxFALL=400 candela per square meter: –max-cll ?1000,400? people have been doing this in CLI without issues. so how do i convert this to work in your script and were do I place it? if I add this to postopt =x265-parm, -master-display "G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,1)", –max-cll ?1000,400? I get an error. If I add it to the HDR filter= x265-parm, -master-display "G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,1)", –max-cll "1000,400" I get error saying not a corret filter or something like that. More Research done here: DCI-P3: --master-display G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,1) --max-cll 1000,1 Rec.709: --master-display G(15000,30000)B(7500,3000)R(32000,16500)WP(15635,16450)L(10000000,1) --max-cll 1000,1 Rec.2020: --master-display G(8500,39850)B(6550,2300)R(35400,14600)WP(15635,16450)L(10000000,1) --max-cll 1000,1 To convert, simply divide the RGB values by 0.00002 Information: https://www.smpte.org/sites/default/files/section-files/HDR.pdf |
|
Definitely formatted things incorrectly above but there's also some additional challenge here because postopts is normally split by commas which are included in your master-display parameter I went ahead and added an option called This will work for h264 and h265 for your HDR profile set it like this that will create an -x265-params argument with those options when generating the command |
|
Any updates on if this solution is working for you? |
|
Closing for now, feel free to continue discussion or reopen if there are bugs |
|
sorry got called out to travel for work to a remote location and didn't have access to my server that runs all the Transcoding scripts. will check this out further and post update.. Thank you for the quick response. |
|
No problem just update when you can if this helps |
|
Ok, so I had some time to run the script with the new changes. I deleted install directory and use GIT to pull down the new source. So I would have all the new changes with the correct autoProcess.ini. I put in all the modification I wanted in the autoProcess.ini. I have tried 2 different ways to add the HDR metadata or copy over the HDR Metadata as stated in your updated WIKI. Here is my SMA.Log of this conversation: My AutoProcess.ini: First option: ** Second option: Third Option with both of them added: All of these options are NOT copying over the HDR Metadata: Source file are: After Conversion File is not having HDR Metadata, This file is not the same name as I forgot to set Delete original=false, but the metadata informations is the same just with a different episode: I just noticed this from the source file:Could this be an Issue? A Document on SMPTE ST 2094 Found this with some great info and Options: So I am at a loss, on this. Am I forgetting something? Or are my settings to low to allow for the extra metadata, Or could it be the h265_nvenc option not allowing for the metadata transfer? Should I only use "h265, x265, hevc" My Video card is a Nvidia RTX2070 Super, So I know that its capable to produce just about anything. I have installed MS Visual studio 2019, Cuda Toolkit 11 and python 3.8.5 which are all required for the "h265_nvenc" to provide gpu hwaccels and Nvidia\Nvenc support. 👍Please dont take this a a troll or a slam. I am admire your skills and wish I had them to be more supportive of things like this to were I would be more of an contributor than a pain in the ass "saying this dont work". 😎Again Please DON'T take any of this a a Slam\Flame\Trolling it is not intended that way at all. I am sincerely wanting to help in this growth of you source\platform, by providing feedback that I hope will stimulate growth\ progression👍 |
|
So from your conversion log everything looks like its working as intended. Its detecting HDR and its applying the -x265-params that you have set in your config. If there's any issue beyond that its probably with the formulation of the -x265-params. You're doing a little comparison of apples to oranges here since you're using 2 different sets of -x265-params between the two scripts which may be where the problem lies. Do things change when you using the exact same -x265-params? Ultimately both scripts are just creating FFMPEG commands so if you have a working FFMPEG command that's generated by the other script just post that so we can compare the difference between the two commands generated and sort out why things aren't working cause its probably some subtle difference that's causing FFMPEG to not behave. It may be that there's just a missing NVENC option that's required to get FFMPEG to write those tags. I will say this script should have full hardware acceleration support for encoding and decoding and that includes nvidia based codecs, though not nearly as many specific options as Collisionc's implementation but I have improved the hw-acceleration options substantially recently |
|
Sorry to jump on this one, is the problem just that the HDR metadata isn't being copied over as from what I can tell the files plays just fine with HDR if I manually set TV to HDR mode, its just not being correctly labelled or picked up which obviously would be helpful. |
|
I'm not sure the x265-params option is applied when you use hardware encoding, believe that option is exclusive to software encoding |
|
Does the option work if you use software encoding? If you want support for hardware encoding with tags please provide a sample FFmpeg command that does that and I'll find a way to incorporate it |
|
Ahh ok, well I found this if its any help https://codecalamity.com/encoding-uhd-4k-hdr10-videos-with-ffmpeg/ |
|
Just give me a sample ffmpeg command that does what you want and I'll incorporate it, preferably with a comparison of the command the script is currently generating |
|
Sorry If I wasn't clear, I don't know what the command it is I'm currently trying to figure it out using the above link. Sorry about the miss understanding that was my bad :D |
|
Try setting your params to
|
|
Yes you are right that copying the HDR metadata is software only, have just found some forums post to confirm this (although there 2 years old so HW may support this now but I doubt it) Ok will do and will run another test, also I have just run across a script that seems to add the metadata in to HW encoded files not sure if its something that can be cannibalised and added in as a postprocess e.g once your scripts has finished with the encoding this runs quick before the file is finished with? |
|
Quick update ran the test using the params you said, and it work great in software as expected, while slow it gave be the desired results and was picked up fine 👍 |
|
See if those params work with nvenc cause they might. I do see some updates that indicate maybe they'll work it's probably worth a shot. Apparently it was working for @Weezzel using nvenc on collisions fork |
|
Unfortunately they don't while yes it most definitely dose encode keeping HDR it doesn't add HDR metadata to the stream so mediainfo, and my TV don't pick it up, however if I set my TV to HDR while play said file it looks just like the original source file. Hope that makes since Its all good though as I don't have or plan on having loads of stuff in HDR so doing it slowly for the few things I want is no big deal :D |
|
@Weezzel If you could post the command that works for nvenc from Collisions fork it would be very much appreciated cause it would be nice to preserve the metadata and that command is probably all I need to make it work |
|
It actually looks like he had posted a few things in this regard years ago https://github.com/Collisionc/sickbeard_mp4_automator/issues/21 Very strange since this seems to be the same question that was asked a year ago And was basically solved by inserting post opts which should be fine, only difference was that the level was included which who knows is maybe required. The params option should work instead of post opts but I think maybe we just need to tweak the param itself Try
but it was definitely working with nvenc at some point unless there was a regression with FFmpeg that broke things. If you could post the FFmpeg command that's generated even if it fails I would appreciate it |
|
Ok will give it a try now |
|
Ok using them param don't work lol |
|
Can you post the command that gets generated? |
|
Yep sorry. |
|
So it looks like those aren't the right params! You sure you updated things in autoprocess? Just FYI the params in the HDR section only apply if the source material is HDR so maybe those are the params from the video section but please try the other params listed before |
|
🤦 I updated the wrong params, this is the new output, still no copy over Params first mention Params above |
|
Does setting an HDR pix-fmt make any difference? I don't see that in the options |
|
|
TrueHD does work on some very new builds of FFMPEG though it looks like you're not using the trueHD streams It does appear cuda/cuvid and nvdec are being initialized which I wonder if they are maybe the problem I would try removing the hwaccel options from autoprocess and see if that changes things |
Even if a File plays ok dosn't mean its not bad, all it takes is for just 1bit to be wrong and ffmpeg has a mental brake down despite the file still playing fine on devices etc as they have built in error correction where as ffmpeg doesn't as it deals with the raw data. |
|
"I would try removing the hwaccel options from autoprocess and see if that changes things" |
|
Actually, this might just be a queue size issue Try adding
|
|
Its defo a setting somewhere as I have done the whole of Season one of Picard and kept the HDR meta and its halfway through doing the Avatar 4k HDR upscale thats kicking about |
|
Different Movie Same outcome: [AVIOContext @ 0000020d1ee45540] Statistics: 0 seeks, 0 writeouts [hevc @ 0000020d1e79b140] Mastering Display Metadata: [hevc @ 0000020d1e79b140] r(0.6800,0.3200) g(0.2650,0.6900) b(0.1500 0.0600) wp(0.3127, 0.3290) [hevc @ 0000020d1e79b140] min_luminance=0.001000, max_luminance=1000.000000 [hevc @ 0000020d1e79b140] Content Light Level Metadata: [hevc @ 0000020d1e79b140] MaxCLL=1000, MaxFALL=358 [aac @ 0000020d1e795340] Qavg: 52387.383 [aac @ 0000020d1e795340] 2 frames left in the queue on closing [AVIOContext @ 0000020d1e76ae40] Statistics: 1245445 bytes read, 2 seeks [hevc @ 0000020d1e79b140] Decoded frame with POC 0. Conversion failed! |
|
Definitely try the queue size fix I bet that's it
|
|
Also out of interest can you see how much ram it uses when it try's to run? |
|
I actually think this error is because its using eAC3 for stream 0:1 and has nothing to do with the video stream or HDR, there's a few closed issues on here with the same problem and the queue size thing fixed it |
|
Dell Power edge server 192GB not memory issues. [Converter] [Permissions] [Metadata] [Video] [HDR] [Audio] [Universal Audio] [Subtitle] |
|
but there is a typo in your preopts that is preventing that from working
Looks like there's a colon in there somewhere that shouldn't be |
|
had to put a ":" in there or else these comands don't work . |
|
if I remove -loglevel,debug,-probesize,1000000,-analyzeduration,10000000 I get this |
|
Shit I'm stupid Its supposed to be in postopts not preopts |
|
lol. 0-09-09 17:15:51 - MANUAL - INFO - Preopts |
|
So is it running now? |
|
yes working slow as hell though, Quadro P2000 of no use now.. |
|
Good news though |
|
Here it is running on my desktop |
|
But very good progress Gentleman. Thank you guys for all your help. |
|
hwaccel decoding should help some, but that is a massive amount of data There's a bunch of patches proposed to fix ffmpeg in the future with hardware encoding and framedata so hopefully one of them will get merged. For such a large job it might even be worth it to try and build from source using those patches Alternatively you could try nvenc and just confirm that the metadata doesn't work, if collisionincs solution ever worked with metadata and nvenc then maybe there's some way out there I'm not aware of You could also just skip it and then use this tool to patch the metadata |
|
I have HW Decoding just fine but the moment you use HW encoding you loose all metadata to do with HDR. |
|
whats the best CRF Format to have on the top end for HDR?
|
|
Hm just noticed I have those crf-profile values backwards on that sample, and the crf should be going down as the source bitrate goes up. I just fixed that on the wiki
What CRF to use will depend on what kind of quality you want and how much storage you want. You can combine it with a Given that 4k HDR content will generally be high bitrate, then the highest profile will likely be used, so Here is a site with some info |
|
Preset=Fast |
|
I'll add a preset option but for now you can use preopts |
|
dedicated preset option for standard or hdr wiki updated |
HDR setting applied to autoProcess.ini and seem to be pushed to the manual.py. They are being Identified from the original file and processing is successful but transcoded\remuxed copy does not have the correct Metadata
I don't think that the Master display & Max cll\Max fall needed for HDR10 is being carried over.
sma.log
2020-07-24` 09:08:33 - MANUAL - INFO - Reading video stream.
2020-07-24 09:08:33 - MANUAL - INFO - Video codec detected: hevc.
2020-07-24 09:08:33 - MANUAL - INFO - Pix Fmt: yuv420p10le.
2020-07-24 09:08:33 - MANUAL - INFO - Profile: main 10.
2020-07-24 09:08:33 - MANUAL - INFO - HDR video stream detected for 0.
2020-07-24 09:08:33 - MANUAL - INFO - Reading audio streams.
2020-07-24 09:08:33 - MANUAL - INFO - Audio detected for stream 1 - eac3 eng 6 channel.
2020-07-24 09:08:33 - MANUAL - INFO - Creating aac audio stream source audio stream 1 [universal-audio].
2020-07-24 09:08:33 - MANUAL - INFO - Creating copy audio stream from source stream 1.
2020-07-24 09:08:33 - MANUAL - INFO - Default audio stream set to eng copy 6 channel stream [default-more-channels: True].
2020-07-24 09:08:33 - MANUAL - INFO - Reading subtitle streams.
2020-07-24 09:08:34 - MANUAL - INFO - Text-based subtitle detected for stream 2 - subrip eng.
2020-07-24 09:08:34 - MANUAL - INFO - Creating mov_text subtitle stream from source stream 2.
2020-07-24 09:08:34 - MANUAL - INFO - Text-based subtitle detected for stream 3 - subrip eng.
2020-07-24 09:08:35 - MANUAL - INFO - Text-based subtitle detected for stream 4 - subrip spa.
2020-07-24 09:08:35 - MANUAL - INFO - Text-based subtitle detected for stream 5 - subrip spa.
2020-07-24 09:08:35 - MANUAL - INFO - Text-based subtitle detected for stream 6 - subrip fra.
2020-07-24 09:08:35 - MANUAL - INFO - Text-based subtitle detected for stream 7 - subrip deu.
2020-07-24 09:08:36 - MANUAL - INFO - Text-based subtitle detected for stream 8 - subrip ita.
2020-07-24 09:08:36 - MANUAL - INFO - Text-based subtitle detected for stream 9 - subrip por.
2020-07-24 09:08:37 - MANUAL - INFO - Text-based subtitle detected for stream 10 - subrip por.
2020-07-24 09:08:38 - MANUAL - INFO - Text-based subtitle detected for stream 11 - subrip pol.
2020-07-24 09:08:38 - MANUAL - INFO - Text-based subtitle detected for stream 12 - subrip tur.
2020-07-24 09:08:38 - MANUAL - INFO - Text-based subtitle detected for stream 13 - subrip swe.
2020-07-24 09:08:39 - MANUAL - INFO - Text-based subtitle detected for stream 14 - subrip dan.
2020-07-24 09:08:39 - MANUAL - INFO - Text-based subtitle detected for stream 15 - subrip fin.
2020-07-24 09:08:39 - MANUAL - INFO - Text-based subtitle detected for stream 16 - subrip nld.
2020-07-24 09:08:40 - MANUAL - INFO - Text-based subtitle detected for stream 17 - subrip nor.
2020-07-24 09:08:40 - MANUAL - INFO - Text-based subtitle detected for stream 18 - subrip rus.
2020-07-24 09:08:40 - MANUAL - INFO - Text-based subtitle detected for stream 19 - subrip hun.
2020-07-24 09:08:41 - MANUAL - INFO - Text-based subtitle detected for stream 20 - subrip ell.
2020-07-24 09:08:41 - MANUAL - INFO - Text-based subtitle detected for stream 21 - subrip heb.
2020-07-24 09:08:41 - MANUAL - INFO - Text-based subtitle detected for stream 22 - subrip ara.
2020-07-24 09:08:42 - MANUAL - INFO - Text-based subtitle detected for stream 23 - subrip ron.
2020-07-24 09:08:42 - MANUAL - INFO - Text-based subtitle detected for stream 24 - subrip ces.
2020-07-24 09:08:42 - MANUAL - INFO - Text-based subtitle detected for stream 25 - subrip ind.
2020-07-24 09:08:43 - MANUAL - INFO - Text-based subtitle detected for stream 26 - subrip vie.
2020-07-24 09:08:43 - MANUAL - INFO - Text-based subtitle detected for stream 27 - subrip tha.
2020-07-24 09:08:43 - MANUAL - INFO - Text-based subtitle detected for stream 28 - subrip kor.
2020-07-24 09:08:44 - MANUAL - INFO - Text-based subtitle detected for stream 29 - subrip zho.
2020-07-24 09:08:44 - MANUAL - INFO - Text-based subtitle detected for stream 30 - subrip zho.
2020-07-24 09:08:44 - MANUAL - INFO - Text-based subtitle detected for stream 31 - subrip jpn.
2020-07-24 09:08:45 - MANUAL - INFO - cuda hwaccel is supported by this ffmpeg build and will be used [hwaccels].
2020-07-24 09:08:45 - MANUAL - INFO - Output Data
2020-07-24 09:08:45 - MANUAL - INFO - {
"source": [
"D:\TV\13 Reasons Why (2017)\Season 03\13 Reasons Why.S03E01.Yeah. I'm the New Girl.WEBDL-1080p.HEVC.EAC3-5.1.HDR-MZABI.mkv"
],
"format": "mp4",
"video": {
"codec": "h265_nvenc",
"map": 0,
"bitrate": 5787.0,
"crf": -18,
"maxrate": null,
"bufsize": null,
"level": 5.1,
"profile": "main10",
"pix_fmt": null,
"field_order": "unknown",
"width": null,
"filter": null,
"title": "FHD HDR",
"debug": "video.max-level.profile"
},
"audio": [
{
"map": 1,
"codec": "aac",
"channels": 2,
"bitrate": 256,
"samplerate": null,
"filter": null,
"language": "eng",
"disposition": "-default-dub-original-comment-lyrics-karaoke-forced-hearing_impaired-visual_impaired",
"title": "Stereo",
"debug": "universal-audio"
},
{
"map": 1,
"codec": "copy",
"channels": 6,
"bitrate": 1536,
"filter": null,
"samplerate": null,
"language": "eng",
"disposition": "+default-dub-original-comment-lyrics-karaoke-forced-hearing_impaired-visual_impaired",
"bsf": null,
"title": "5.1 Channel",
"debug": "audio"
}
],
"subtitle": [
{
"map": 2,
"codec": "mov_text",
"language": "eng",
"encoding": "",
"disposition": "+default-dub-original-comment-lyrics-karaoke-forced-hearing_impaired-visual_impaired",
"title": "",
"debug": "subtitle.embed-subs"
}
],
"attachment": []
}
2020-07-24 09:08:45 - MANUAL - INFO - Preopts
2020-07-24 09:08:45 - MANUAL - INFO - [
"-fix_sub_duration",
"-hwaccel",
"cuda",
"-loglevel",
"debug",
"-probesize",
"1000000",
"-analyzeduration",
"10000000"
]
2020-07-24 09:08:45 - MANUAL - INFO - Postopts
2020-07-24 09:08:45 - MANUAL - INFO - [
"-threads",
"0",
"-metadata:g",
"encoding_tool=SMA"
]
2020-07-24 09:08:45 - MANUAL - INFO - Starting conversion.
2020-07-24 09:16:55 - MANUAL - INFO - D:\TV\13 Reasons Why (2017)\Season 03\13 Reasons Why.S03E01.Yeah. I'm the New Girl.WEBDL-1080p.HEVC.EAC3-5.1.HDR-MZABI.mp4 created.
2020-07-24 09:16:56 - MANUAL - INFO - Tagging file: D:\TV\13 Reasons Why (2017)\Season 03\13 Reasons Why.S03E01.Yeah. I'm the New Girl.WEBDL-1080p.HEVC.EAC3-5.1.HDR-MZABI.mp4.
2020-07-24 09:16:56 - MANUAL - INFO - Trying to write tags.
2020-07-24 09:16:56 - MANUAL - INFO - Tags written successfully using mutagen.
2020-07-24 09:16:56 - MANUAL - INFO - Relocating MOOV atom to start of file.
2020-07-24 09:20:04 - MANUAL - INFO - Manual processor started.
2020-07-24 09:20:04 - MANUAL - INFO - C:\Python38\python.exe
2020-07-24 09:20:04 - MANUAL - INFO - Loading config file C:\Sick_MP4\config\autoProcess.ini.
2020-07-24 09:20:04 - MANUAL - INFO - Processing file D:\TV\13 Reasons Why (2017)\Season 03\13 Reasons Why.S03E01.Yeah. I'm the New Girl.WEBDL-1080p.HEVC.EAC3-5.1.HDR-MZABI.mp4
2020-07-24 09:20:06 - MANUAL - INFO - Matched TV episode as 13 Reasons Why (TMDB ID: 66788) S03E01
2020-07-24 09:20:06 - MANUAL - INFO - Processing 13 Reasons Why Season 03 Episode 01 - Yeah. I'm the New Girl
2020-07-24 09:20:06 - MANUAL - INFO - Input and output extensions are the same so passing back the original file [process-same-extensions: False].
2020-07-24 09:20:06 - MANUAL - INFO - Bypassing conversion and setting outputfile to inputfile.
2020-07-24 09:20:06 - MANUAL - INFO - Tagging file: D:\TV\13 Reasons Why (2017)\Season 03\13 Reasons Why.S03E01.Yeah. I'm the New Girl.WEBDL-1080p.HEVC.EAC3-5.1.HDR-MZABI.mp4.
2020-07-24 09:20:07 - MANUAL - INFO - Trying to write tags.
2020-07-24 09:20:09 - MANUAL - INFO - Tags written successfully using mutagen.
2020-07-24 09:20:11 - MANUAL - INFO - Relocating MOOV atom to start of file.
2020-07-24 09:20:11 - MANUAL - WARNING - QT FastStart did not run - perhaps moov atom was at the start already or file is in the wrong format.
autoProcess.zip
[Converter]
ffmpeg = ffmpeg.exe
ffprobe = ffprobe.exe
threads = 0
hwaccels = hevc_cuvid,cuda,dxva2,d3d11va,qsv
hwaccel-decoders = h265_nvenc,hevc_cuvid,h265_cuvid,hevc_nvenc
output-directory =
output-format = mp4
output-extension = mp4
temp-extension =
minimum-size = 0
ignored-extensions = nfo, ds_store
copy-to =
move-to =
delete-original = True
sort-streams = True
process-same-extensions = False
force-convert = False
post-process = False
preopts = -loglevel,debug,-probesize,1000000,-analyzeduration,10000000
postopts =
[Permissions]
chmod = 0644
uid = -1
gid = -1
[Metadata]
relocate-moov = True
full-path-guess = True
tag = True
tag-language = eng
download-artwork = poster
sanitize-disposition =
[Video]
codec = h265_nvenc,h265,x265
max-bitrate = 0
crf = -18
crf-profiles =
max-width = 0
profile = Main10
max-level = 5.1
pix-fmt = yuv420p10le,p010le
filter =
force-filter = False
[HDR]
space = bt2020nc
transfer = smpte2084
primaries = bt2020
filter =
force-filter = False
[Audio]
codec = eac3,TrueHD,dts,ac3,flac
languages = eng
default-language = eng
first-stream-of-language = False
allow-language-relax = True
channel-bitrate = 256
max-bitrate = 0
max-channels = 0
prefer-more-channels = True
default-more-channels = True
filter =
force-filter = False
sample-rates =
copy-original = False
aac-adtstoasc = False
ignore-truehd = false
[Universal Audio]
codec = aac
channel-bitrate = 128
first-stream-only = False
move-after = True
filter =
force-filter = False
[Subtitle]
codec = mov_text
codec-image-based = copy,sup,subrip,dvdsub,ass,pgs,pgssub,hdmv_pgs_subtitle
languages = eng
default-language = eng
first-stream-of-language = True
encoding =
burn-subtitles = False
burn-dispositions =
download-subs = False
download-hearing-impaired-subs = False
download-providers =
embed-subs = True
embed-image-subs = True
embed-only-internal-subs = True
filename-dispositions = forced
ignore-embedded-subs = False
attachment-codec =
Pictures of File Before script:
After running the manual.py script:
The text was updated successfully, but these errors were encountered: