Submission runs getting stuck after a certain time during inference

Competition ADIA Lab
1

EDIT: It just seems like an issue with the logs not updating after certain period, as soon as the inference finished, it updated the logs showing the determinism check starting.

Hello,

I recently joined this competition, but I am having an issue with the submission system. After a few minutes of the inference function running, which can be seen in the logs, the logs suddenly stop updating/the run seems to get stuck. The runs in which this can be observed are #47336(which i terminated) and #47337.

Could you please provide some assistance with this, or perhaps help me figure out where the problem might be ? Local run works just fine and does not get stuck.

Any help is much appreciated. Best regards.

2

The logs only show the first 1000 lines, and last 500 lines (which are shown in bulk at the end as we do not know in advance how much there will be).

As you only seems to be using a tqdm for 2 steps, I suggest you instead put it in an upper loop that would track the overall progress of the run.

-for dataset in X_test:
+for dataset in tqdm(X_test):
     ...

-    for func in tqdm(functions):
+    for func in functions:
        ...

    yield ...
Results of Timed Out Runs
3

Alright, thanks for the tip, and thanks for the fast response!

1 Like
4

(post deleted by author)