issue with MIDI_0088.MID and MIDI_0089.HMI from Daggerfall

[sezero]

MIDI_0089.HMI from http://www.vgmpf.com/Rips/Daggerfall-DOS.zip
reports 166m 34s with wildmidi.

As one would expect, wav output from the latter is about 30MB whereas
the wav output from the former is about 1.3GB. Something is certainly
broken in f_hmi.c.

[sezero]

I was wrong to mark this as an HMI issue: it is not. There is a confusion of
names in that Daggerfall zip. MIDI_0088.MID (167m 9s, as reported by the
wildmidi player) corresponds to MIDI_0089.HMI. So it is a midi issue in general
not just hmi.

[sezero]

For the record, libtimidity reports 0m 56s for MIDI_0088.MID.
Something is crazy with libwildmidi.

[sezero]

Interestingly, wildmidi-0.3.12 is OK: it reports 0m 56s just like libtimidity.
wildmidi-0.4.x is broken for it.

[psi29a]

That is interesting... @chrisisonwildcode this needs to be fixed before we ship 0.4.2 since it is likely the cause to other such midi rendering problems we've had since going to 0.4

any idea?

[chrisisonwildcode]

Looking into this issue with extra long times being reported.

…

Sent from my iPhone

On 3 Aug 2017, at 7:24 am, sezero ***@***.***> wrote: For the record, libtimidity reports 0m 56s for MIDI_0088.MID. Something is crazy with libwildmidi. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[sezero]

PING?

[psi29a]

@chrisisonwildcode any results?

[sezero]

PING ?

[sezero]

PING ?

[sezero]

PING: Any chance that anyone will have a look at this?

[psi29a]

Unless this can be properly debugged, we can always have the xmi converted to midi using xmi2mid.c which should give us the correct results. Is this an acceptable solution @sezero ?

[sezero]

No: This issue is not restricted to hmi or xmi files, but seen with plain midis too.
See #176 (comment)

[sezero]

To re-summarize the issue:

• MIDI_0088.MID from Daggerfall: Crazy with 0.4.x, good with 0.3.x
• MIDI_0089.HMI from Daggerfall: Crazy the same way as MIDI_0088.MID
  with 0.4.x. Cannot be tested with 0.3.x.

[sezero]

devtest_d1.txt

Attached output of wildmidi-devtest -d 1 MIDI_0088.MID
maybe it helps.

[psi29a]

If you count up the time elapsed on each track, does that match up with the known 'good' time or does devtest behave like wildmidi now? (sorry, didn't check, at work)

[sezero]

Didin't look closely either. (I'm at work too :)

[lieff]

approx_total_samples corrupted after load loop in f_midi.c

approx_total_samples=559212 sample_count=805
approx_total_samples=559748 sample_count=536
approx_total_samples=321663673

[sezero]

Also see: #149 (comment)

[chrisisonwildcode]

Working on DevTest first to fix reported midi time length issue then will port to f_midi.c. I know what the issue is, and am reworking it right now. I expect to have a solution by the end of today for DevTest and f_midi.c.

[sezero]

[...] I know what the issue is, and ...

What is the issue?

[chrisisonwildcode]

This issue seems to be resolved in DetTest for other HMI/MIDI files but the particular HMI reported still showing 166min on a particular track.

Time 00:48.00000    Note On: chan(5) note(76) vel(80)
Note Length: 0.041667 secs
Time 00:56.25000    Note On: chan(5) note(69) vel(1)
Note Length: 0.425000 secs
Time 166:34.24902    Controller  Unknown(91): chan(5) set(76)
Time 166:34.24902    Controller  Unknown(105): chan(0) set(0)
Time 166:34.24902    Meta Event: End Of Track

Exploring further

[chrisisonwildcode]

So I've looked at the raw MIDI-0089.HMI file and this is what I found.

in the fifth track, just before the 3rd last event, there appears to be a huge delta (aka timing data) causing the track length (at that point) to go from just 56secs to 166mins. Extending the output of DevTest shows this as well.

00001800  47 50 07 1e 4c 50 05 87  5e 45 01 33
 massive delta before controller event   ->    c8 e4 70 b5  |GP..LP..^E.3..p.|
00001810  5b 4c 
		00 b0 69 00 
			    00 ff  2f 00 

and

Time 00:56.25000    Note On: chan(5) note(69) vel(1)
Running Event: 0x95
Note Length: 0.425000 secs
DEBUG @ 0x0000180c: c8 e4 70 b5 5b 4c 00 b0 69 00 00 ff 2f 00 48 4d 
DEBUG: DELTA c8 e4 70  = 00123270 (1192560)
Time 166:34.24902    Controller  Unknown(91): chan(5) set(76)
Running Event: 0xb5
DEBUG @ 0x00001812: 00 b0 69 00 00 ff 2f 00 48 4d 49 2d 4d 49 44 49 
DEBUG: DELTA 00  = 00000000 (0)
Time 166:34.24902    Controller  Unknown(105): chan(0) set(0)
Running Event: 0xb0
DEBUG @ 0x00001816: 00 ff 2f 00 48 4d 49 2d 4d 49 44 49 54 52 41 43 
DEBUG: DELTA 00  = 00000000 (0)
Time 166:34.24902    Meta Event: End Of Track

Both DevTest and f_hmi.c behave the same way with this file. Unless there is a fundamental issue with how we interpreted the HMI format I gotta put this particular issue down to a bad file.

[chrisisonwildcode]

MIDI-0088.MID has a similar issue, massive delta before the last few events within one of its tracks. Interestingly enough the bad delta is almost the same length as the bad delta in MIDI-0089.HMI

f_midi.c parses midi files slightly differently to DevTest but both have the same result.

Unless someone comes up with a fundamental flaw in how we parse the files in relation to their respective formats I'm gonna have to put it down to bad data within the files mentioned that results in extra long times.

I could look at adding work arounds for files that appear extra long by looking at where the lat event that effects notes is in relation to the reported length and snip off the dead space. But that would need to be up for discussion first.

P.S. Looking for midi files that report extra long times within wildmidi …

[sezero]

Well. there is a fundamental flaw somewhere, because neither wildmidi-0.3 nor libtimidity has any problems with this file.

[psi29a]

that's at least something to go on.

[chrisisonwildcode]

I will have a good look at 0.3 tomorrow. I just know that after looking at the hexdump of MIDI_0088.MID that there is an issue similar to MIDI-0089.HMI in that a delta is excessively large. You can see from the hexdump above that I tend to separate out the commands within the file to trace whats going on. its a long process when all else fails. And I only resorted to the hexdump because I thought we may be reading the file wrong.

[chrisisonwildcode]

Sorry for the delay. Had to send new computer back for replacement after it failed.

On my radar for this weekend after #186. This is because I'm going to have to work through the midi file byte by byte to ensure we are reading the data in properly, and that we are processing it correctly with running events and stuff. I will be starting off by quickly seeing what 0.3 does with it including the 0.3 DevTest (which probably needs some lovin as well but will leave that for a later date).

Since I already know that its track 5 of the midi file thats creating the issue I will either be able to find the issue with 0.4 branch or discover 0.3 does something extra with event timing after the last note off within a track (which sounds like something I might have done in my original versions of wildmidi).

[sezero]

PING

[sezero]

This is the only serious issue to block a new 0.4.3 release.

[sezero]

PING?

[psi29a]

@chrisisonwildcode ping

[fawtytoo]

This is a bit old now, but I'd like to add that the music can be found at https://images.uesp.net/d/df/Dfmusics2.zip
File FM_SUNNY.MID is the same music but Wildmidi reports a length of 54 seconds.
I have to agree this issue is down to a corrupt file.

[sezero]

This is a bit old now,

Old, maybe, but still not resolved (maybe I haven't tried hard enough.)

I have to agree this issue is down to a corrupt file.

That does not explain why the wildmidi-0.3 branch (or libtimidity which
is my main focus now) have no problems with the file.

[fawtytoo]

Timidity makes allowances, and tries to handle corruptions. I believe that @chrisisonwildcode had mentioned wildmidi-0.3 (and earlier) had done a similar thing, and 0.4 no longer does it (for whatever reason).
I don't think it's the job of a player to handle corrupted files anyway. I've already proved that the same music can be obtained that has no playback issues. I think that should be the solution.
If you REALLY want to fix it, I'll take a look myself and see if I can spot something.

[fawtytoo]

A thought for you to consider.
I have a collection of classical music as MIDI files. Generally, classical music is in movements, and those that sequence the music into MIDI tend to create a separate file for each movement. It's quite common for pauses (rests) to occur between movements, consequently, those pauses are at the beginning and ends of the MIDI files.

The "fix" you are looking for will most likely eliminate the pause at the end of the file, which some have naturally as I've described. WildMidi already skips pauses at the beginning of files.

Sometimes, what fixes one file can break others.

The BEST solution is to fix broken MIDI files using an editor rather than have the MIDI player work around it.

[sezero]

OK, closing this as wontfix