Schema transform

[rolyp]

See notebook. To do:

• set the scene better for the error when calling lm.fit
• have Ptype report on how it automates this step (i.e. summarise anomalous/erroneous rows)
• rewrite bullet points at beginning into proper text
• interweave explanatory text into Part 2

Done/dropped:

• Part 2 seems to improve on Part 1 only in saving the user from converting ‘?’ to pd.na
• maybe duplication between Part 2 and Part 1 isn’t necessary – maybe present Ptype usage within a larger scenario?
• better name for use case
• rename transform_schema
• show_schema should be a method on schema object – see Move cols property to Schema object #118
• Ptype should describe inferred type as Int64, not integer – see Use Pandas names for Pandas datatypes #117
• remove unnecessary imports
• move summary above to beginning of notebook?
• move Ptype imports to where they’re first used
• rather than showing df.loc[130], point out that it’s not obvious where the problem is in the dataset
• run query to show occurrences of ‘?’
• remove metrics such as mean_absolute_error
• instead of build_table_schema, why not just show dtypes again?
• dtypes are not automatically changed after replacing ‘?’ in part 1. add another step to change it
• explain dtype=’str’ step
• inline scatter_plot helper

[tahaceritli]

fixed some of the items above in #99

[rolyp]

Summary:

Part 1 (solve data cleaning problem without Ptype):

• import dataset using Pandas read_csv
• run linear regression on data
• error occurs because of missing data: could not convert string to float: ‘?’
• inspect dtypes property of dataframe to see the problem
• use Pandas to change encoding of missing data, remove relevant rows
• run linear regression again (no errors), plot results
• use dtypes to verify that we now have appropriate column types

Part 2 (how Ptype makes this problem easier):

• import dataset using Pandas read_csv, but this time with dtype=’str’
• instantiate Ptype
• ask Ptype to infer schema; show inferred types
• ask Ptype to adjust type of dataframe to match schema
• inspect transformed dataframe to verify types as expected
• then as per Part 1 to remove missing data and continue