Flashcarts have cart-unique data, which ntrboot_flasher will happily modify and causes dumps to have varying checksums.

[DeadSkullzJr]

Hello again, I am unsure if I stumbled on an issue that was either overlooked or something nobody realized was happening with flash dumps. However, I am still going to post it here because this seems to be happening with every flashcart that works on NTRboot Flasher. For a few months I have been gathering some flash backups for restore purposes in case I find a flashcart in the future that needs a restore, I have also been dumping my own flashcarts as well. One thing that struck me very odd though was the hash information differences between the dumps. One could assume that people just have dirty contacts or something, ruining dump information, however, I decided to do some testing of my own. I recently got my hands on another R4i Gold 3DS Plus to test with to narrow down the possible issue. I dumped the flash of the new card with the latest NTRboot Flasher (v0.5.0) and compared it to my dump of my other R4i Gold 3DS Plus (also dumped with the same version of NTRboot Flasher). Mind you that I have tested this on all of my CFW devices and the contacts on both cards and the consoles themselves aren't dirty. Both flashcarts are the same hardware type, so that can be ruled out, however the dumps still had different hash information (dumping the same card twice doesn't render different information each time either, that's the other baffling part here). I made a video though so I can show you more clearly what I am talking about:
https://www.youtube.com/watch?v=5FHIJRwiawc

I haven't dabbled in the 3DS homebrew scene yet, I don't know how NTRFlasher really checks for this information, I just know that visually speaking the issue I encountered does seem to be out of place, thus causing the weird differences in dump information.

[kitlith]

dumping the same card twice doesn't render different information each time

It's probably some sort of unique per-cart information written to the flash at factory, then. they seem to be different at the same place consistently, at very round address numbers, 0x30000 and 0x40000. 0x30000 could store some kind of unique id, while 0x40000 could be some sort of test pattern they never bothered to erase. I don't really know.

If it's an issue, what we could do is prevent writes to those cart-unique areas. Honestly though, this is why we try to have everyone take backups of their own carts, just in case. >_>

[hedgeberg]

Dude that absolutely is bizarre, first time we've had a real issue here in a very long time. I don't know if it's necessarily an ntrflasher bug or not, but it's a good idea to test to be sure.

Have you run a binary diff on the different flashes to see what's different? And can you flash the mismatched flashes to each other (restore should be possible even if you soft brick the cart if you have the flash dumped, if you're willing to test that)? Its possible the difference is some sort of calibration value, like clock max speeds? Or maybe some sort of chip ID? So we need to pin that down. If you could send both the flash dumps my way, that would be a good step too. If you're willing to risk destroying the casing, pictures of the PCBs would be great too.

[hedgeberg]

@kitlith fwiw, if they are differing in specific flash regions, we should stub out that information to prevent panics due to differing checksums. Just from an archival perspective, knowing what differs where is a good idea.

[kitlith]

i wish we had a good way to dump out multiple files, because then we could have a scrubbed file and a file with just the cart unique information, or something like that.

honestly, I just want to continue dumping out the full flash because it's nice for anyone who wants to inspect and, say, find out that there's this cart-unique data lurking.

[hedgeberg]

Why can't we dump out multiple files? One file scrubbed, and then a second which describes the cart-unique info and where it goes, a-la Intel hex? Lump in a simple script to merge the 2 files into one for those who want to?

[hedgeberg]

At the absolute very least, we should find a way to check just the non-unique info when calculating checksums.

[kitlith]

we can't currently dump out multiple files because the interface between flashcart_core and ntrboot_flasher does not have a way to describe this currently. This could be changed, but if it were changed I'd like to identify these areas on all currently supported carts... which feels like more hassle than it's worth at the moment.

As for checksums... we could provide a script to scrub out unique info for sharing and checksumming or something?

[DeadSkullzJr]

Dude that absolutely is bizarre, first time we've had a real issue here in a very long time. I don't know if it's necessarily an ntrflasher bug or not, but it's a good idea to test to be sure.

Have you run a binary diff on the different flashes to see what's different? And can you flash the mismatched flashes to each other (restore should be possible even if you soft brick the cart if you have the flash dumped, if you're willing to test that)? Its possible the difference is some sort of calibration value, like clock max speeds? Or maybe some sort of chip ID? So we need to pin that down. If you could send both the flash dumps my way, that would be a good step too. If you're willing to risk destroying the casing, pictures of the PCBs would be great too.

Have you run a binary diff on the different flashes to see what's different?
I have indeed, the problem exists for EVERY flashcart compatible with NTRBoot Flasher, I actually have a few flash dumps missing data as well that I found over the net.

And can you flash the mismatched flashes to each other (restore should be possible even if you soft brick the cart if you have the flash dumped, if you're willing to test that)?
As long as the flash belongs to the flashcart hardware, yes, if you want to try and use a flash from say, the Acekard, then no, it won't work at all. It will work visually speaking, people won't notice anything at all, everything noticeable is under the hood. The problem is when you restore with another flash, you overwrite the bytes that the original dump presented (if you made one) and basically permanently turn your card into an actual clone of someone else's flashcart. At this current time, everyone's flashes are not identical, meaning you can literally overwrite your flash to match someone else's. Sure that doesn't seem like much of an issue for most people, but the point is, an issue like this can potentially lead to issues later on if people aren't careful.

Its possible the difference is some sort of calibration value, like clock max speeds? Or maybe some sort of chip ID?
Doubt it, the pattern of these weird bytes suggests it is either inaccurately read data or data that shouldn't be read for the sake of the dump.

If you could send both the flash dumps my way, that would be a good step too.
I can do that:
https://www.dropbox.com/s/nzenis11duy1rnb/R4i%20Gold%203DS%20Backups.7z?dl=0

Hardware wise you won't find any differences, I already checked under the shell, they look identical and are marked the same way and all, there isn't anything on the PC board that shows differences between the two.

@kitlith you should change the original title back, this effects ALL flashcarts that can be read by NTRBoot Flasher. I tested other cards outside the R4i Gold 3DS line, I can confirm that no dumps are the same, all of them are different.

[kitlith]

I do not consider this to be a kind of corruption. I do not know why these values are present on flash, but I'm pretty sure that if you used a hardware flash reader that you'd find those exact bytes in these exact locations. e.g., ntrboot_flasher/flashcart_core are accurately dumping the flash.

[kitlith]

If you can provide differing locations for all carts, then we can start by considering an update which prevents writes to these areas when restoring backups. (iirc one cart already does this, but we were not aware of others having the same type of issue)

[hedgeberg]

@DeadSkullzJr Unless we have a way to check if the regions that differ are consistent or not, were going to need to assume that it's not a corruption thing. The fact that a dumped flashcart can be written from a dump with a different checksum would insinuate that at the very worst case these are uninitialized flash regions with unpredictable data.

The dump is, for the purposes of ntrflasher, functionally "clean" if it can be flashed back and continues to run as normal. The reason I said this was interesting is the fact that there are regions that arent consistent across devices, and that can cause checksum mismatches. That does not however mean that ntrflasher is dumping incorrectly, it just means we're not feeding certain important safeguard information to the user.

Also, side note, I know you're trying to be helpful, but it's bizarre that you'd assume I didn't know that you can't run acekard firmware on an r4i etc.

[DeadSkullzJr]

@DeadSkullzJr Unless we have a way to check if the regions that differ are consistent or not, were going to need to assume that it's not a corruption thing. The fact that a dumped flashcart can be written from a dump with a different checksum would insinuate that at the very worst case these are uninitialized flash regions with unpredictable data.

The dump is, for the purposes of ntrflasher, functionally "clean" if it can be flashed back and continues to run as normal. The reason I said this was interesting is the fact that there are regions that arent consistent across devices, and that can cause checksum mismatches. That does not however mean that ntrflasher is dumping incorrectly, it just means we're not feeding certain important safeguard information to the user.

Also, side note, I know you're trying to be helpful, but it's bizarre that you'd assume I didn't know that you can't run acekard firmware on an r4i etc.

but it's bizarre that you'd assume I didn't know that you can't run acekard firmware on an r4i etc.
Wasn't assuming anything actually, I was merely stating that it can't be done in general, you took the quote to heart lol.

[DeadSkullzJr]

R4i Gold 3DS flash addresses to check:

• 0x30000 through 0x30010
• 0x40000 through 0x4FFF0

R4i SDHC Family flash addresses to check:

• 0x3060 through 0x3080

[lpla]

R4i SDHC Family also mark 0x3000 as '01' or '02', after comparing several dumped flashcarts data. This is, probably, a hardware revision.

[DeadSkullzJr]

R4i SDHC Family also mark 0x3000 as '01' or '02', after comparing several dumped flashcarts data. This is, probably, a hardware revision.

While that seems plausible, that doesn't explain why that data is being dumped in the first place. If I was to dump the original dump from the freshly purchased flashcart, restore the flash using a different flash backup from another model in the same family, then dump that flash again, it's going to match the flash dump information of the tested flash you restored rather than the original dump that the flashcart had.

That data shouldn't be dumped if it's not relevant to the model of flashcart you are flashing back to normal. Overwriting important data isn't exactly a nice concept to think about knowing that some of the data is being permanently overwritten, the only way of getting it back is if you still had an original flash dump that belonged to that respected flashcart model/revision.

[kitlith]

that doesn't explain why that data is being dumped in the first place

It's because we wrote it to do full flash backups and restores and never fiddled around with avoiding cart specific areas. There was already enough per-model stuff, and oftentimes during initial testing we only had one cart being tested against. Avoiding it was, tbh, never very important to me, and also allows you to take a backup, modidy it however you wish, and flash it back for experimentation. And my policy has always been that you shoud take a backup of your own cart rather than relying on other backups.

Nevertheless, ntrboot_flasher and flashcart_core are unlikely to change at this point, as they're unmaintained, and I have other projects on my mind. If I did ever come back to ntrboot_flasher and such, it'd probably be to completely rewrite it in Rust or something. (I'm not really happy with the state of the flashcart_core api, though I suppose it worked okay.)