- Rename
selectbox.py
toform_fields.py
andSelectBox
class toFormField
. - Rename
options.py
toform_options.py
and corresponding. - Rename
reader.py
toschema_reader.py
. - Make a constant in
constants.py
asDATA_SCHEMA_PATH = Path("src/database/schema.json")
. - Make constants for
LOCALITY_NAME.csv
andCITY.csv
inconstants.py
. - Add
LUXURY_SCORE
feature intoschema.json
, streamlit form and everywhere. - How can you validate the
LOCALITY_NAME
belongs to the particularCITY
? - How can you add new city dataset into the picture?
- Store user's new data into different directory to differentiate between main data and user's data. And show analysis on it.
- Use
functions
with@st.cache
decorator to optimize the Analytics Page. - Make a page to show Model Evaluation Metrics. Which are useful for other developers.
-
Add logging in the app.(Not for now.) - Fetch the new discussed Gurgaon data and perform EDA on it. Then, use that data in the project for better analysis and prediction.
- Do something about Recommender system.
- Add selectbox to select CITY and also refresh the map which recenter the map to the city.
- Add BHK radio button in
scatter_mapbox
plot. - Create new page and add better plots in new page.
-
Usest.tabs()
instead ofst.radio()
for good UI. -
Add a custom streamlit theme. -
Enhance the radio buttons in the page or you can use
st.selectbox
instead. -
Add summary for Newly Added Data by user in Add New City Page
- Insights about:
- Shape of the Data
- No. of property in each CITY
- No. of property in each PROPERTY_TYPE
-
Show Distribution of data. Plot graphs for this. (inst.expander
) - Write that for more insights go to Analytics Page
- Insights about:
- Fetch data from database.
- Clean data at step first to clean further for specific
PROPERTY_TYPE
. - Clean data at step second with every property entity class's method
PropertyType.clean_dataset()
. - Now we have 6 dataset in total to train models.
- Now here raises a question that how do I train models using these many dataset at once?
Things to Remember
- First do all the things with
gurgaon_10k.csv
dataset, then replicate the process for MongoDB.- Dump the cleaned data (only with step first cleaning) in the MongoDB and clean the data further with each property entity method
PropertyType.clean_dataset()
and train the model.- You need to take care of the CITY and LOCALITY_NAME options because they are fetched from a
.csv
file. You need to update them regularly when the user fetches new city data.
Questions & Doubts
- Should fetch data every time to show Analysis?
- I have to save the trained model into a directory.
- I have to create a button to fetch new city data.
- I have to create a button to re-train the model on new dataset (which contains the data of the new city fetched by user.)
- As you fetching, cleaning, training the model on the newly fetched data take a lot of time. How do I tackle this.
- Select columns for model building and analytics.
- Decode the encoded features like
FACING, FURNISH, FLOOR_NUM, FEATURES, FORMATTING_LANDMARK_DETAILS
, etc. - After cleaning at first step dump the data to MongoDB to process further.
I added a form from where users can fetch the data of a particular CITY.
Make a
streamlit
secret string from where I can disable this feature to avoid the request blocking.
Now I fetched data from website then I have to preprocess the data to dump it into the database. For that I have to create two-step function to clean the data for each PROPERTY_TYPE
.
- I need to clean the whole fetched dataset with the first function which ready the dataset to clean further for each
PROPERTY_TYPE
. - Now the cleaned datasets being cleaned further by the specific property entity's method
clean_data()
.
And then after cleaning the dataset we are ready to dump them into the database.
- User can choose whether they want to train a new model on their data or train a new model on new data and old data combined.
- Display all the CITY and No. of data point which are already present in the dataset. Update it on daily basis by reading the dataset.
- DISCUSS: If the user added a new CITY data then how do you add that city's
LOCALITY_NAME
andCITY
intoform_options
.