Does re-training work as expected?

Competition ADIA Lab
1

Hi experts!

I am wondering whether the re-training works as expected. And I think there might be some problems.

I found out that no matter what re-training frequency I used, I always got the exactly the same score.
Then I tried to use frequency = 1 and check the log of first 5 re-training and examine the number of training date and number of unique training dates. I found out that all the first 5 re-training trained on 269 dates data, but I think each re-training should have 1 more date of data to train?

You can check run 754 for the log.

Thanks so much for the help!

2

I can see that you have different scores?
image

And using a train frequency of 1 will result in training at every date.

Currently retraining, give you all of the dates, maybe you would be interested in a “flag” to only be train on new dates?

Thanks for the Run ID :slight_smile:

3

Thanks for the reply!

Hmm, I have different scores because I use different models. You can see I have lots of 3.037 — I tested one model with different frequency, and all of them have score 3.037.

I tried to use frequency = 1 and check the log of first 5 re-training and check the number of row in the training data and number of unique training dates. I found out that on run 2, 3, 4, 5, they were trained on data from 269 dates, but I think each re-training should have 1 more date of data to train? [so run 2 should have data from 269 days to train, run 3 should have data from 270 days to train, run 4 should have date from 271 days to train etc.]

Checking the log for run 754, we can see that on run 2, 3, 4, all of them were trained on 742670 rows from 269 dates, which I think it’s unexpected.

Log below just for reference:

looping moon=270 train=True (2/31)
3:18:01 PM
user-code
ping(…) took 0.0025 seconds
3:18:01 PM
user-code
read(/tmp/tmpcgfs_kmm/x_train.parquet, …) took 1.8818 seconds
3:18:01 PM
user-code
read(/tmp/tmpcgfs_kmm/y_train.parquet, …) took 0.3990 seconds
3:18:01 PM
user-code
read(/tmp/tmpcgfs_kmm/x_test.parquet, …) took 0.3602 seconds
3:18:01 PM
user-code
training number of features: 461
3:18:01 PM
user-code
training number of example: 742670
3:18:01 PM
user-code
training number of unique dates: 269

looping moon=271 train=True (3/31)
3:18:09 PM
user-code
ping(…) took 0.0025 seconds
3:18:09 PM
user-code
read(/tmp/tmpo1k5aa7_/x_train.parquet, …) took 1.6404 seconds
3:18:09 PM
user-code
read(/tmp/tmpo1k5aa7_/y_train.parquet, …) took 0.4547 seconds
3:18:09 PM
user-code
read(/tmp/tmpo1k5aa7_/x_test.parquet, …) took 0.3580 seconds
3:18:09 PM
user-code
training number of features: 461
3:18:09 PM
user-code
training number of example: 742670
3:18:09 PM
user-code
training number of unique dates: 269

3:18:09 PM
runner
looping moon=272 train=True (4/31)
3:18:16 PM
user-code
ping(…) took 0.0025 seconds
3:18:16 PM
user-code
read(/tmp/tmp9pg87yfp/x_train.parquet, …) took 1.8230 seconds
3:18:16 PM
user-code
read(/tmp/tmp9pg87yfp/y_train.parquet, …) took 0.4516 seconds
3:18:16 PM
user-code
read(/tmp/tmp9pg87yfp/x_test.parquet, …) took 0.3547 seconds
3:18:16 PM
user-code
training number of features: 461
3:18:16 PM
user-code
training number of example: 742670
3:18:16 PM
user-code
training number of unique dates: 269

1 Like
4

Hi @enzo,

Could you take a look here? :slight_smile:

5

Its finally fixed!

Thanks for bringing this issues to us and sorry for the delay.