Directional Accuracy (up/down prediction)

[ericleonardo]

Hi! Please, can you tell me the Directional Accuracy metric result?
That is one of most important metrics (but most ignored in academic papers).
Predicting if the price will rise of fall is of practical importance when trading.

Regression metrics (MAE, RMSE, etc) can be illusory in financial market.
Make a test: instead of using the model to predict next price, just repeat last known price as response and calculate MAE/RMSE.
If your model doesn't beat the RMSE resulted from just repeating last known value, the model have low predictive power.
Among many models (LSTM etc) you use to compare CryptoMamba MAE/RMSE, you could also include last known value for comparison.

But Directional accuracy is different. No matter how good or bad regression metric performs.
If the model can predict next up/down movement with more than 55% accuracy, it may be profitable and useful for trading.

Thank you and congratulations for the work!

[AsalMehradfar]

Thank you for your question and interest in CryptoMamba!

Our model is specifically trained for regression, not classification. Predicting whether the price will rise or fall is a classification problem, which would require a different training approach. However, our trading algorithms are designed based on regression and incorporate mechanisms to account for possible errors in trading decisions.

For example, in our vanilla trading algorithm, we avoid executing trades if the predicted increase or decrease is small—where "small" is defined based on the model's accuracy (as discussed in our paper). This ensures that our trading strategy remains robust and minimizes unnecessary trades due to minor fluctuations.

Ultimately, the goal of our models and trading algorithms is to generate profit. Of course, no model is perfect, and we are actively exploring new techniques to improve performance in future work.

That being said, if you wish to use our approach as a classification model—simply predicting whether the price will rise or fall—it still outperforms baseline techniques, achieving 55.43% directional accuracy.

We appreciate your feedback and look forward to any further discussions!

[ericleonardo]

Very interesting, Asal!
Please, can you share the file with all Target and Predicted values?

I want to analyse if filtering the small predicted movements, the directional accuracy may increase.
Like you said, considering only significant fluctuations may become more robust/confident.

Regression can also be interpreted as up/down prediction, don't need to use classification model.
If predicted value is greater than current input (above specified filter threshold), consider as Up.
But it is very interesting the 55.43% directional accuracy by classification.

Another option is making Regression to 0 (down) and 1 (up) as target.
So, the more the predicted value is closer to 1, more is the model confidence for Up.
Predicted values around 0.5 should be avoided as the model is not confident in this 50% up/down region.

Thank you!