ENH: convert classify-samples-ncv to a pipeline, or add NCV pipeline

[nbokulich]

Improvement Description
convert classify-samples-ncv to a pipeline, or add NCV pipeline. This will reduce user effort while utilizing these methods.

Current Behavior
The *-ncv actions are methods, not pipelines. Downstream visualizers are run separately by users.

Proposed Behavior
Stitch together *-ncv pipelines, either by converting the existing methods or by adding new pipelines.

[Oddant1]

So we have regress_samples and classify_samples, presumably these shouldn't both be called as part of the same pipeline? So would you choose which of those you're using in the pipeline and then the rest of the pipeline will call visualizers and such? What other methods/visualizers should be parts of this pipeline?

[nbokulich]

there are methods classify_samples_ncv and regress_samples_ncv that would be used here.

the main question is whether to convert the existing methods into pipelines, or create new pipelines that use those methods.

[Oddant1]

Yeah, I was referring to the ncv methods, my bad, I'm not sure I entirely understand the distinction. Would converting the current methods into pipelines make it impossible to run those methods independently because they would now be integrated into the pipelines they became? Whereas creating new pipelines that include those methods would still allow those methods to be used independently of the pipelines? Regardless, I suppose I don't really have a horse in that race.

[nbokulich]

yes, that is correct.

ultimately I think we want new pipelines, but then it is a question of the names (it would be nice to keep the naming conventions consistent).

[Oddant1]

So this would be two new pipelines one with regress_samples_ncv and one with classify_samples_ncv, what else would be in these pipelines? Just a visualizer? What visualizer? And we could always be kinda lazy and just call them regress_samples_ncv_pipeline and so on.

[nbokulich]

yep, two pipelines one with regress_samples_ncv and one with classify_samples_ncv.

the rest of the pipelines would look a lot like what regress_samples and classify_samples do after training. Something like this:

1. *_samples_ncv
2. send predictions to the appropriate visualizer: scatterplot or confusion_matrix
3. classify_samples_ncv could make a heatmap of the top features.

[Oddant1]

@nbokulich The more I read your comment, the more I feel like I'm missing something. Let's say we're working with classify_samples_ncv. I read your above comment as

1. call the current classify_samples_ncv method from the new classify_samples_ncv_pipeline or whatever we call it pipeline
2. use the results of the call to classify_samples_ncv to call the appropriate viz from the pipeline
3. use the results of the prior two calls to generate a heatmap from the pipeline
4. return all relevant results from the pipeline

[nbokulich]

yes. I think all you are missing is that there are some pipeline differences between classify_samples and regress_samples that we would want to replicate in a potential NCV pipeline. So, e.g., the heatmap would only be generated for classify_samples_ncv_pipeline, not regress.... But otherwise it sounds like we are on the same page.