Find Uses of Field - Automation

[bdemick]

The Find Uses of Field capability is incredibly useful for some automation that I'm working on to examine types in a large program with DWARF info. After quite a bit of following the GUI code and realizing I couldn't easily reuse the API invoked there, I discovered ReferenceUtils.findDataTypeFieldReferences which accomplishes what I'm looking for. However, each invocation of this function, when setting discoverTypes to true, takes about 1.25hrs on this particular binary with 57,000 functions. To be fair, this is documented as a known limitation.

It seems to be bottlenecked by how many decompile instances Ghidra will run at once (~10 on my machine), and not compute power. I tried different values of the Cache Size (Functions) which didn't seem to impact adjacent invocations at all.

Are there any other things I can try here? Sure would be nice if I had the option to cache decompile results for all functions, even in a limited capacity for cases like this.

[dragonmacher]

That is a lot of functions. I imagine if we naively cached all decompilation, that memory would be exhausted.

It seems like a different approach to the issue would better fit this use case. I'm envisioning a pass through all functions in which every data type use and field use is recorded. Once you have that info, you can later have any query check the pre-recorded data.

[bdemick]

That definitely sounds like a more reasonable approach than caching all the decompilation data. Perhaps as an analysis pass that can be enabled or independently run to discoverTypes? Would it be useful to create a feature issue for this? I'd also be happy to take a stab at implementing it, even more so if pointed in the general direction of the type discovery code and the APIs for adding that to wherever the type references are cached.

[dragonmacher]

This is a feature that we are unlikely to work on any time soon. This is something you can explore if you have the inclination to get your hands dirtier than most. The way the type usage is processed now is a bit of a conceptual 'exploration', not as polished as our framework APIs.

The code is mostly well-written and tested, but is admittedly hard to understand. I will provide a course grained overview of the classes involved in the process here.

You have started your path an a relatively good place by looking at ReferenceUtils. That class will find applied types and discovered types (using the Decompiler). The code for finding the discovered types is this:

private static void findDataTypeMatchesOutsideOfListing(
			Accumulator<LocationReference> accumulator, Program program, DataType dataType,
			FieldMatcher fieldMatcher, TaskMonitor monitor) throws CancelledException {

		List<DataTypeReferenceFinder> finders =
			ClassSearcher.getInstances(DataTypeReferenceFinder.class);

		Consumer<DataTypeReference> callback = ref -> {

			LocationReferenceContext context = ref.getContext();
			LocationReference locationReference = new LocationReference(ref.getAddress(), context);
			accumulator.add(locationReference);
		};

		if (finders.isEmpty()) {
			Msg.debug(ReferenceUtils.class, "Unable to find any implementations of " +
				DataTypeReferenceFinder.class.getSimpleName());
			return;
		}

		for (DataTypeReferenceFinder finder : finders) {
			finder.findReferences(program, fieldMatcher, callback, monitor);
		}
	}

That code is using our Extension Point mechanism to find an implementation of DataTypeReferenceFinder. Our only real implementation of that is DecompilerDataTypeReferenceFinder. If you read the code for that, you will see that it uses parallel decompilers to load all functions in the program, processing the decompilation results for each. That delegates to an inner class called DecompilerDataTypeFinder to process each line of decompilation, inside of findVariablesInLine().

That method builds up a collection of objects for the various types of function pieces it finds (return type, parameter, etc). Each of those objects extends from DecompilerReference. They know how to figure out what data type is being referenced and if there is a structure field being referenced. How those objects work internally can be somewhat hard to follow. Those objects are later queried to get the results that are displayed in the UI via a call to getContext(). The 'context' currently displays the function that uses the data, the line number of use and the c code using the data and typically an associated address.

It seems like manipulating those objects to change the type of output they generate would be an easy place to start. You can mimic the ReferenceUtils code to generate the information you need, storing the results in a format that you can later reuse to avoid having to search again. I know this is quite 'hand wavy', but without building this myself, I can't offer a much better suggestion.

Also, @d-millar has suggested that more advanced implementations of such a feature are possible by working directly with pcode, but we don't have any examples of that which map as cleanly to your problem space as what I described above.

[bdemick]

Much appreciated! I'm going to mark this as answered for now, but will definitely take this as a starting point and see what I can do.