Pandas In, Pandas Out? `.inverse_transform()` method

[naught101]

It would be really nice to have the ability to put pandas dataframes into sklearn pipelines, and to have equivalent pandas dataframes returned afterwards. I think that this module would be the place for that - probably all that would be required is a .inverse_transform method on the DataFrameMapper.

Would something like this be wanted in this module? I can make a pull request, if so.

Before I do, why is all the code in __init__.py? Seems like it'll get hard to maintain after a while...

[dukebody]

Hi Naught101!

You can already put pandas dataframes into sklearn pipelines. Just create a pipeline where the first step is the DataFrameMapper.

Regarding the proposal "to have equivalent dataframes returned afterwards", you mean to make the pipeline return a pandas DataFrame? Sklearn pipelines usually return numpy arrays, with either classification probabilities for each class (predict_proba), directly class predictions or regression values. How could you inverse transform that with the initial DataFrameMapper? The output and the input have different shapes and useful transforms.

I believe you can do the indexing thing you proposed at scikit-learn/scikit-learn#5523 (comment) just wrapping the numpy array output into a DataFrame passing as index the one from the original DataFrame you got into the pipe. Am I wrong?

Regarding the reason why all the code is in __init__.py, I guess it is because it was a very small module at first and didn't make a lot of sense to scatter the code along multiple files, although clearly we would need to go that way if the codebase grows, for clarity.

One issue we have one is that the original maintainer of the package (paulgb) is no longer working on it at all, and the second mantainer (Cal Paterson) has been quite irresponsive in the last few months as well. So it's becoming hard to get new code into this repo, and harder to get it into a release. :(

[naught101]

Aha.. I wasn't thinking clearly, but now I can: DataFrameMappers can also be useful for generating the y value passed to a fit method. The inverse_transform would then be useful to get back a suitable dataframe. But yes, this would be a different DataFrameMapper to the one used for X.

I guess that would all happen outside the pipeline though..

Has anyone working on the code asked @paulgb for push access?

On 22 October 2015 6:52:05 pm AEDT, "Israel Saeta Pérez" notifications@github.com wrote:

Hi Naught101!

You can already put pandas dataframes into sklearn pipelines. Just
create a pipeline where the first step is the DataFrameMapper.

Regarding the proposal "to have equivalent dataframes returned
afterwards", you mean to make the pipeline return a pandas DataFrame?
Sklearn pipelines usually return numpy arrays, with either
classification probabilities for each class (predict_proba), directly
class predictions or regression values. How could you inverse transform
that with the initial DataFrameMapper? The output and the input have
different shapes and useful transforms.

I believe you can do the indexing thing you proposed at
scikit-learn/scikit-learn#5523 (comment)
just wrapping the numpy array output into a DataFrame passing as
index the one from the original DataFrame you got into the pipe. Am I
wrong?

Regarding the reason why all the code is in __init__.py, I guess it
is because it was a very small module at first and didn't make a lot of
sense to scatter the code along multiple files, although clearly we
would need to go that way if the codebase grows, for clarity.

One issue we have one is that the original maintainer of the package
(paulgb) is no longer working on it at all, and the second mantainer
(Cal Paterson) has been quite irresponsive in the last few months as
well. So it's becoming hard to get new code into this repo, and harder
to get it into a release. :(

---

Reply to this email directly or view it on GitHub:
#41 (comment)

Sent from my Android device with K-9 Mail. Please excuse my brevity.

[dukebody]

@calpaterson got write access to this repo, but he's not answering my mails. :S

[naught101]

Hrm. Is there any reason you couldn't expand the current behaviour to also map the y dataframe? e.g. the call would be mapper = DataFrameMapper(X_features = [(blah...)], y_features = [(blergh)]), and then .fit(), .transform() and .predict() all call whichever transforms are relevant on X and/or y.

[dukebody]

Sounds reasonable. Could you come up with some examples where this y transformation would be useful?

[dukebody]

@naught101 I have write access now to this repo so we can work this out if you come out with useful use cases. :)

[dukebody]

@naught101 you might want something similar to what is discussed in #13 ?

[naught101]

Yeah, I suspect that #13 is a prerequisite for this issue..

[ethanluoyc]

If say the transformed dataframe has exactly the same shape as the dataframe before the transformation. Can we pass in the columns to regenerate the predicted results in a DataFrame format?

[dukebody]

@ethanluoyc Could you provide a code example of how that feature would work? Not the implementation, but how one would use it.

[ethanluoyc]

I am doing something on basketball so I will just give an exmaple on this
say I have this dataframe,

after the conversion I will get something like this.

Which basically did substitution on based on the position of the keyword (which is the name) I have in a text string, for example,

"Jumpball: (Zydrunas Ilgauskas)\PN vs. (Kendrick Perkins)\PN ((Mo Williams)\PN gains possession)"

So the two dataframes actually has the same shape. I don't know whether I can do such inverse transformation.

I checked out #13 and I think the approach can work, however, as I referenced on the documentation on sklearn I stumble about their docs on the attribute active_features_, I decided to look into that in more details once I figure out what teh active_features_ attribute does.

[dukebody]

I believe we can do the inverse transformation if we:
* Track which array columns correspond to each dataframe columns.
* Every transformer used has an inverse_transform(X) method.

It shouldn't be too hard to do. Any takers? :)