Batch Removing/Replacing/Deleting DataTypes

[astrelsky]

Version tracking decided that it wanted to make 1000s of .conflict types for equivalent datatypes. Now I am stuck with 1082866 types with a .conflict suffix which should have just used the existing equivalent type. Ghidra is completely unusable this way and constantly lags. I've tried just deleting every datatype but Ghidra seems to soft lock. I'm currently taking the naive approach and using the script below as I have some progress (3% after about an hour which will exponentially get faster).

public class FixConflictsScript extends GhidraScript {

	@Override
	protected void run() throws Exception {
		if (!askYesNo("", "ready")) {
			return;
		}
		DataTypeManager dtm = currentProgram.getDataTypeManager();
		List<DataType> conflicts = CollectionUtils.asStream(dtm.getAllDataTypes())
			.filter(dt -> dt.getName().contains(".conflict"))
			.collect(Collectors.toList());
		int id = dtm.startTransaction("Fixing Conflicts");
		boolean success = false;
		try {
			monitor.initialize(conflicts.size());
			monitor.setMessage("Fixing Conflicts");
			for (DataType dt : conflicts) {
				monitor.checkCanceled();
				if (dt.isDeleted()) {
					monitor.incrementProgress(1);
					continue;
				}
				String name = dt.getName();
				int i = name.indexOf(".conflict");
				if (i == -1) {
					monitor.incrementProgress(1);
					continue;
				}
				name = name.substring(0, i);
				DataType good = dtm.getDataType(dt.getCategoryPath(), name);
				if (good == null) {
					monitor.incrementProgress(1);
					continue;
				}
				dtm.replaceDataType(dt, good, false);
				monitor.incrementProgress(1);
			}
			success = true;
		} finally {
			dtm.endTransaction(id, success);
		}
	}
	
}

I don't understand why this happened in the first place but if there is a better way to deal with this situation that'd be great.

[astrelsky]

Oh I'm going to be real happy when I come back in the morning to find out something threw an exception aren't I 😅

[astrelsky]

Well it only got to 25% by the morning so I killed it.

[dev747368]

Maybe sorting the list of conflicts by some kind of metric that improved the speed of the replaceDataType? My first idea would be complexity / dependency count so that the simpler data types are processed first, but idk.

[astrelsky]

Maybe sorting the list of conflicts by some kind of metric that improved the speed of the replaceDataType? My first idea would be complexity / dependency count so that the simpler data types are processed first, but idk.

I had to completely delete the program from the ghidra project as I could do almost nothing due to the lag. Either way I would of ended up starting over because I found a different version of what I was reversing and wanted to do that one instead.

[astrelsky]

Maybe sorting the list of conflicts by some kind of metric that improved the speed of the replaceDataType? My first idea would be complexity / dependency count so that the simpler data types are processed first, but idk.

Thank you for your response. I decided that it wasn't worth the hassle.

Lesson learned. When using version tracking you better associate all the datatypes in the source program with a project archive or your going to have a bad time.

[ghidra1]

Unfortunately, the replaceDataType method is extremely slow since in addition to fixing-up references by other datatypes it searches all Data code units and Function Variables for use of the datatype. We do not employ any indexing on these elements making it a very brute force approach. Avoiding conflicts in the first place is preferable but easier-said-than-done in many cases. I am not really familiar with Version Tracking so I don't know how it handles datatypes but I can see how it could drag in a bunch from a different source. Not an easy problem to solve.