Extracting the self similarity matrix from novelty

[jamesb93]

After the most recent Audible Edge workshop, many participants asked if it was possible to get the self-similarity matrix out of the novelty slice object. Potential applications were suggested such as visualisation, using it as a feature for a neural network, or for trying to inspect the structure of some sound. I think the same question arose in the NOTAM workshop too.

I'm not sure if in the particular FLuCoMa implementation it is that straightforward to get it out but it might be a useful strategy for deriving a temporally sensitive feature (which are somewhat lacking ATM in the toolkit).

What are your thoughts @tremblap, @weefuzzy, and @tedmoore?

[tremblap]

a few thoughts:

• I always love more stuff. FluidBufSSM could be fun but
• more maintenance
• how do we extract anything valid from it needs postprocessing - temporality is definitely not trivial to extract in a meaningful way (hence what we currently have, a rolling window/kernel which is giving one way of reading meaning into it)

so it is a good idea but still not clear in my head how it would really be useful without some other ways of extracting meaning from it.

[weefuzzy]

We never make a full SSM, only a rolling kernel's worth.

👎 Full SSMs are very memory intensive and pretty CPU intensive
👎 It's not clear what people would be able to do with that data that would be useful

[jamesb93]

We never make a full SSM, only a rolling kernel's worth.

👎 Full SSMs are very memory intensive and pretty CPU intensive 👎 It's not clear what people would be able to do with that data that would be useful

These are good reasons to not have it. IIRC doing it in Python with accelerated numpy magic was slow, so I can't imagine it being that fun to play with in a musical context. Thanks for clarifying and giving input :)

[tedmoore]

This is the request that I was predicting you'd get. I'm not convinced that it isn't useful. At least I think it's another potential angle into "know your data", just the chance to see it in another way might lend some insight into what is there. Perhaps we should have a demo of doing it because people are gonna ask and then we can let them wrestle with if it is artistically valuable to them.

Doing every FFT frame as a point would get too memory intensive quite quickly (a la in NoveltySlice), but the audio could be segmented differently, here's the Golcar recording broken into 1000 segments and shows the dog barks as being both more different from the other material in the recording and similar to each other (I'm riffing a bit here, but the idea is there).

So, doing it in NoveltySlice, maybe not. Being able to make these matricies, and showing users how they could, might be nice.

Here's the 30 minute improv set with Ben from Pittsburgh broken into 2000 segments (obviously not every FFT frame, but the time resolution here is about 0.93 seconds. BufStats gets the mean MFCC analyses over the time window, startCoeff=1).

This could reveal some formal outlines or how different sections or moments relate to each other.

(
Window.closeAll;

s.waitForBoot{
	var cond = CondVar.new;
	var simMatrix, img, analysis_frame_samps, features_fa;
	// var src = Buffer.readChannel(s,FluidFilesPath("Tremblay-BaB-SoundscapeGolcarWithDog.wav"),channels:[0]);
	//var src = Buffer.readChannel(s,"/Users/macprocomputer/Desktop/_ben_opie/_media/flucoma_TedBen.wav",channels:[0]);
	var src = Buffer.readChannel(s,"/Users/macprocomputer/Desktop/_ben_opie/renders/20220418_122354/20220418_122354_ted_moore_and_ben_opie.wav",channels:[1]);
	// var src = Buffer.readChannel(s,"/Users/macprocomputer/Desktop/test sounds/noisy_synth/noisy_synth_mono.wav",channels:[0]);
	var features = Buffer(s);
	var descriptor = Buffer(s);
	var stats = Buffer(s);

	var total_analyses = 2000;

	s.sync;

	analysis_frame_samps = (src.numFrames / total_analyses).asInteger;

	total_analyses.do{
		arg i;
		FluidBufMFCC.processBlocking(s,src,i * analysis_frame_samps,analysis_frame_samps,features:descriptor,startCoeff:1);
		FluidBufStats.processBlocking(s,descriptor,stats:stats);
		FluidBufCompose.processBlocking(s,stats,numFrames:1,destination:features,destStartFrame:i);
		if((i % 100) == 99) {
			s.sync;
			i.postln;
		}
	};

	s.sync;

	"analysis done".postln;
	features.postln;

	features.loadToFloatArray(action:{
		arg fa;
		features_fa = fa.clump(features.numChannels);
		cond.signalOne;
	});

	cond.wait({ features.notNil });

	features_fa.size.postln;

	simMatrix = (features_fa.size-1).collect{
		arg i;
		var frame = features_fa[i];
		features_fa[(i+1)..].collect{
			arg other;
			pow(frame - other,2).sum.sqrt;
		};
	};

	simMatrix = simMatrix / simMatrix.collect{arg line; line.maxItem;}.maxItem;

	img = Image(simMatrix.size,simMatrix.size);

	simMatrix.do{
		arg line, y;
		line.do{
			arg sim, x;
			var xpos = y + x;
			var ypos = (img.height-1) - y;
			img.setColor(Color.gray(1-sim),xpos,ypos); // whiter = more similar
			img.setColor(Color.gray(1-sim),(img.height-1) - ypos,(img.width-1) - xpos); // whiter = more similar
		}
	};

	img.plot;

	FluidWaveform(src);
}
)

[tremblap]

This could reveal some formal outlines or how different sections or moments relate to each other.

Graphically at best, and even that. for gesture similarity, we'd need to look for diagonals that are parallel-ish (faster or slower will be steeper) but again doing that programmatically is not trivial... let's keep discussing with long-term agenda so we might inspire our late summer or someone somewhere.

[tedmoore]

Perhaps we can make a lower-priority help file & learn article showing how one might approach this so when users inquire about it we have a place to share some information with them.