Circular encoding for dates

[PeterPirog]

I wonder if circular encoder is used to convert dates.
Of course years increases lineary but for days period is 7, for months period is 12, for hours period is 12 or 24

• distance between January and December is 1 not 11
• distance between Monday and Sunday id 1 not 6

In my preprocessing models I like to use circular encoding because sometimes it improve accuracy.
Link: https://www.kaggle.com/avanwyk/encoding-cyclical-features-for-deep-learning

By the way I'm not sure how to define date 2021-01-15 10:37:17.369573 in YAML

I try to use code below but there is some problem with it:

        name: Registration_date #TIME
        type: date
        preprocessing:
          missing_value_strategy: fill_with_const
          fill_value: ''
          datetime_format: "%Y-%b-%d %H:%M:%S"

I'm very satisfy that I can test ludwig framework very intensively for last week
I focused how can I use my domain knowledge to use it in YAML configuration. Typicaly I use tensorflow models (lastly I try tensorflow probability layers) but in ludwig time from configuartion YAML file to model serving is impressing.
I'm layman in pytorch so I try to use code in branch tf-legacy

[tgaddair]

Hey @PeterPirog, we have something very similar, which is the wave encoder, which you can read the docs for here:

https://ludwig-ai.github.io/ludwig-docs/user_guide/configuration/#wave-encoder

It's functionally very similar, in that every dimension (days, months, etc.) follow a cyclical period. The only difference I am aware of is that the wave encoder uses the sine function, whereas the one described in the kaggle notebook uses both sine and cosine.

If you would like to add the variant in the notebook, that would be much appreciated!

cc @w4nderlust in case you'd like to add anything.

[PeterPirog]

@tgaddair Maybe my question is little naive but do you mean to prepare modified class DateWave in notebook with sin and cos transformation? If yes, I will try to do it in kaggle notebook.
Could You suggest me how to define preprocessing datetime format for values like "2021-01-15 10:37:17.369573" because there is something wrong with datetime_format: "%Y-%b-%d %H:%M:%S" ?

[PeterPirog]

@tgaddair I found correct datetime format for "2021-01-15 10:37:17.369573":

        name: Publication_datetime 
        type: date
        preprocessing:
            datetime_format: '%Y-%m-%d %H:%M:%S.%f'

I prepared proposition of circular encoder here: 25ef562

[w4nderlust]

Thanks for the contribution @PeterPirog , looks great!

[PeterPirog]

@w4nderlust , I will try to better understand structure of the project to add some features for numerical and categorical feature engineering.
I would like to learn more how to cooperate in projects because I have no experience with that. As far I made some tensorflow models and feature preprocessing for own purposes only, but sometimes I'm exhausted to copy the same code for next projects again and again, so I would like to use ludwig in own projects.

I started to prepare table with explanation what is the equivalent of sklearn or feature-engine in ludwig yaml file.