Hi, my submissions are not finished even after 10 hours and I am forced to terminate them. Even though only a lightweight model is used in the code. Is there a way for us to monitor the progress of the code execution? Also, have my weekly or monthly limit for using the execution hardware finished?
There is two possible thing here:
- The logs are limited to the first 1000 lines and last 500 lines.
If your code emit a lot of warnings, this could slow it down considerably. Tqdm is also a time killer if used for very small loops, we recommend to only use it for iterating over the datasets. - In the cloud environment, you are run on 10,000 datasets, not just 100.