Question about optimal mask ratio

[Jimmy-7664]

Hi, Thank you for your awesome work. I am trying to reproduce the results of your Hyper-parameter Study experiment, but when I changed the mask ratio, I got different results from the paper on all three datasets.
In order to reproduce the problem, I clone STEP locally, use the dataset you provided, and then I only change the mask ratio to 0.25 and GPU num to 4 and then re-train the TSFormer and STEP models. Here are the results.
Result for PEMS04 dataset,

"2023-04-06 11:44:21,101 - easytorch-training - INFO - Epoch 89 / 100
2023-04-06 11:48:05,343 - easytorch-training - INFO - Result : [train_time: 224.24 (s), lr: 6.25e-05, train_MAE: 17.7495, train_RMSE: 28.6461, train_MAPE: 0.1334]
2023-04-06 11:48:05,345 - easytorch-training - INFO - Start validation.
2023-04-06 11:50:55,722 - easytorch-training - INFO - Result : [val_time: 170.38 (s), val_MAE: 18.1383, val_RMSE: 28.1314, val_MAPE: 0.1222]
2023-04-06 11:50:57,130 - easytorch-training - INFO - Checkpoint checkpoints/STEP_100/e87a2127b73d6ba8cda30085f7beb2b3/STEP_best_val_MAE.pt saved
2023-04-06 11:53:47,204 - easytorch-training - INFO - Evaluate best model on test data for horizon 1, Test MAE: 16.3986, Test RMSE: 26.5760, Test MAPE: 0.1097
2023-04-06 11:53:47,209 - easytorch-training - INFO - Evaluate best model on test data for horizon 2, Test MAE: 16.8978, Test RMSE: 27.5110, Test MAPE: 0.1138
2023-04-06 11:53:47,213 - easytorch-training - INFO - Evaluate best model on test data for horizon 3, Test MAE: 17.3182, Test RMSE: 28.1838, Test MAPE: 0.1168
2023-04-06 11:53:47,216 - easytorch-training - INFO - Evaluate best model on test data for horizon 4, Test MAE: 17.6034, Test RMSE: 28.6846, Test MAPE: 0.1193
2023-04-06 11:53:47,219 - easytorch-training - INFO - Evaluate best model on test data for horizon 5, Test MAE: 17.8579, Test RMSE: 29.0966, Test MAPE: 0.1208
2023-04-06 11:53:47,222 - easytorch-training - INFO - Evaluate best model on test data for horizon 6, Test MAE: 18.0855, Test RMSE: 29.4533, Test MAPE: 0.1219
2023-04-06 11:53:47,226 - easytorch-training - INFO - Evaluate best model on test data for horizon 7, Test MAE: 18.2933, Test RMSE: 29.7824, Test MAPE: 0.1234
2023-04-06 11:53:47,229 - easytorch-training - INFO - Evaluate best model on test data for horizon 8, Test MAE: 18.4903, Test RMSE: 30.0777, Test MAPE: 0.1249
2023-04-06 11:53:47,232 - easytorch-training - INFO - Evaluate best model on test data for horizon 9, Test MAE: 18.6646, Test RMSE: 30.3532, Test MAPE: 0.1263
2023-04-06 11:53:47,235 - easytorch-training - INFO - Evaluate best model on test data for horizon 10, Test MAE: 18.8327, Test RMSE: 30.6055, Test MAPE: 0.1280
2023-04-06 11:53:47,238 - easytorch-training - INFO - Evaluate best model on test data for horizon 11, Test MAE: 19.0170, Test RMSE: 30.8598, Test MAPE: 0.1294
2023-04-06 11:53:47,241 - easytorch-training - INFO - Evaluate best model on test data for horizon 12, Test MAE: 19.2633, Test RMSE: 31.1607, Test MAPE: 0.1311
2023-04-06 11:53:47,286 - easytorch-training - INFO - Result : [test_time: 170.16 (s), test_MAE: 18.0602, test_RMSE: 29.3930, test_MAPE: 0.1221]"

Result for METR-LA dataset,

2023-04-10 06:42:39,345 - easytorch-training - INFO - Epoch 96 / 100
2023-04-10 06:45:14,351 - easytorch-training - INFO - Result : [train_time: 155.01 (s), lr: 1.56e-04, train_MAE: 2.7075, train_RMSE: 5.3461, train_MAPE: 0.0707]
2023-04-10 06:45:14,354 - easytorch-training - INFO - Start validation.
2023-04-10 06:45:56,978 - easytorch-training - INFO - Result : [val_time: 42.62 (s), val_MAE: 2.6742, val_RMSE: 5.1144, val_MAPE: 0.0727]
2023-04-10 06:45:59,152 - easytorch-training - INFO - Checkpoint checkpoints/STEP_100/aa28a6aab40136a6af5885691c378f94/STEP_best_val_MAE.pt saved
2023-04-10 06:47:22,991 - easytorch-training - INFO - Evaluate best model on test data for horizon 1, Test MAE: 2.1564, Test RMSE: 3.7474, Test MAPE: 0.0513
2023-04-10 06:47:22,994 - easytorch-training - INFO - Evaluate best model on test data for horizon 2, Test MAE: 2.4244, Test RMSE: 4.4858, Test MAPE: 0.0598
2023-04-10 06:47:22,998 - easytorch-training - INFO - Evaluate best model on test data for horizon 3, Test MAE: 2.6010, Test RMSE: 4.9821, Test MAPE: 0.0661
2023-04-10 06:47:23,002 - easytorch-training - INFO - Evaluate best model on test data for horizon 4, Test MAE: 2.7384, Test RMSE: 5.3681, Test MAPE: 0.0713
2023-04-10 06:47:23,005 - easytorch-training - INFO - Evaluate best model on test data for horizon 5, Test MAE: 2.8518, Test RMSE: 5.6969, Test MAPE: 0.0758
2023-04-10 06:47:23,009 - easytorch-training - INFO - Evaluate best model on test data for horizon 6, Test MAE: 2.9472, Test RMSE: 5.9612, Test MAPE: 0.0793
2023-04-10 06:47:23,012 - easytorch-training - INFO - Evaluate best model on test data for horizon 7, Test MAE: 3.0293, Test RMSE: 6.1882, Test MAPE: 0.0824
2023-04-10 06:47:23,015 - easytorch-training - INFO - Evaluate best model on test data for horizon 8, Test MAE: 3.1039, Test RMSE: 6.3804, Test MAPE: 0.0852
2023-04-10 06:47:23,019 - easytorch-training - INFO - Evaluate best model on test data for horizon 9, Test MAE: 3.1690, Test RMSE: 6.5411, Test MAPE: 0.0877
2023-04-10 06:47:23,023 - easytorch-training - INFO - Evaluate best model on test data for horizon 10, Test MAE: 3.2300, Test RMSE: 6.6843, Test MAPE: 0.0900
2023-04-10 06:47:23,026 - easytorch-training - INFO - Evaluate best model on test data for horizon 11, Test MAE: 3.2900, Test RMSE: 6.8177, Test MAPE: 0.0923
2023-04-10 06:47:23,030 - easytorch-training - INFO - Evaluate best model on test data for horizon 12, Test MAE: 3.3564, Test RMSE: 6.9536, Test MAPE: 0.0946
2023-04-10 06:47:23,075 - easytorch-training - INFO - Result : [test_time: 83.92 (s), test_MAE: 2.9081, test_RMSE: 5.8956, test_MAPE: 0.0780]

Result for PEMS-BAY dataset,

2023-04-12 21:09:24,701 - easytorch-training - INFO - Epoch 99 / 100
2023-04-12 21:17:14,073 - easytorch-training - INFO - Result : [train_time: 469.37 (s), lr: 3.13e-05, train_MAE: 1.4102, train_RMSE: 3.1304, train_MAPE: 0.0307]
2023-04-12 21:17:14,076 - easytorch-training - INFO - Start validation.
2023-04-12 21:19:04,273 - easytorch-training - INFO - Result : [val_time: 110.20 (s), val_MAE: 1.4673, val_RMSE: 3.0659, val_MAPE: 0.0333]
2023-04-12 21:19:07,612 - easytorch-training - INFO - Checkpoint checkpoints/STEP_100/88e31eeaa70eda996a9b04849238f61f/STEP_best_val_MAE.pt saved
2023-04-12 21:22:46,004 - easytorch-training - INFO - Evaluate best model on test data for horizon 1, Test MAE: 0.8249, Test RMSE: 1.5080, Test MAPE: 0.0159
2023-04-12 21:22:46,013 - easytorch-training - INFO - Evaluate best model on test data for horizon 2, Test MAE: 1.0714, Test RMSE: 2.1557, Test MAPE: 0.0215
2023-04-12 21:22:46,023 - easytorch-training - INFO - Evaluate best model on test data for horizon 3, Test MAE: 1.2304, Test RMSE: 2.6389, Test MAPE: 0.0256
2023-04-12 21:22:46,031 - easytorch-training - INFO - Evaluate best model on test data for horizon 4, Test MAE: 1.3444, Test RMSE: 2.9996, Test MAPE: 0.0288
2023-04-12 21:22:46,038 - easytorch-training - INFO - Evaluate best model on test data for horizon 5, Test MAE: 1.4280, Test RMSE: 3.2660, Test MAPE: 0.0313
2023-04-12 21:22:46,046 - easytorch-training - INFO - Evaluate best model on test data for horizon 6, Test MAE: 1.4926, Test RMSE: 3.4677, Test MAPE: 0.0333
2023-04-12 21:22:46,054 - easytorch-training - INFO - Evaluate best model on test data for horizon 7, Test MAE: 1.5432, Test RMSE: 3.6218, Test MAPE: 0.0349
2023-04-12 21:22:46,062 - easytorch-training - INFO - Evaluate best model on test data for horizon 8, Test MAE: 1.5850, Test RMSE: 3.7426, Test MAPE: 0.0363
2023-04-12 21:22:46,070 - easytorch-training - INFO - Evaluate best model on test data for horizon 9, Test MAE: 1.6201, Test RMSE: 3.8362, Test MAPE: 0.0374
2023-04-12 21:22:46,079 - easytorch-training - INFO - Evaluate best model on test data for horizon 10, Test MAE: 1.6513, Test RMSE: 3.9144, Test MAPE: 0.0385
2023-04-12 21:22:46,088 - easytorch-training - INFO - Evaluate best model on test data for horizon 11, Test MAE: 1.6804, Test RMSE: 3.9816, Test MAPE: 0.0393
2023-04-12 21:22:46,096 - easytorch-training - INFO - Evaluate best model on test data for horizon 12, Test MAE: 1.7121, Test RMSE: 4.0462, Test MAPE: 0.0402
2023-04-12 21:22:46,249 - easytorch-training - INFO - Result : [test_time: 218.64 (s), test_MAE: 1.4320, test_RMSE: 3.3536, test_MAPE: 0.0319]
2023-04-12 21:22:47,739 - easytorch-training - INFO - Checkpoint checkpoints/STEP_100/88e31eeaa70eda996a9b04849238f61f/STEP_099.pt saved

Here is a comparison of the two results, STEP's results are from https://arxiv.org/pdf/2206.09113.pdf, and https://github.com/zezhishao/BasicTS.

I would like to ask you to help me figure it out.
Looking forward to your reply.
Best regards

[zezhishao]

That's strange, have you tested other masking ratios? E.g., the ratios in the paper.
Since the code has be significantly refactored, I now need to redo these experiments, which may take some time.

[Jimmy-7664]

Thank you for your reply. I also run with mask ratio=0.75. The results and the performance in PEMS-BAY and METR-LA are not very different from the paper, but I can not reproduce the results of PEMS04 in the paper. My result of PEMS04 is similar with https://github.com/zezhishao/STEP/blob/github/training_logs/STEP_PEMS04.log.

[Jimmy-7664]

@zezhishao Have you tested this problem yet? Looking forward to your reply.

[zezhishao]

Hi @Jimmy-7664 , I am very sorry for the late reply. I have been very busy looking for a job recently.
At present, I haven't finished the test and come to an insightful conclusion.
I tested a few masking ratios, and the results under different masking ratios seem to have no specific rules.
In addition, I have not tested the masking ratios in the paper ,and STEP has been refactored, so I'm not sure if something went wrong.

[zezhishao]

I tested a few more masking ratios again.
From preliminary tests, masking ratio seems to only affect efficiency, and the impact on performance is not significant.
This is not consistent with the results of the ablation study in the paper. I will continue to test the previous version of STEP to see if I can reproduce the results at that time. If it is not consistent, I will immediately revise the results of the paper.
In addition, if you have any further findings or insights, please let me know. Maybe we can figure out why and write a paper together.