I am experiencing the following error when submitting through the cli:
return self.sandbox(
^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/crunch/runner/cloud.py”, line 592, in sandbox
return utils.read(self.prediction_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/crunch/utils.py”, line 135, in read
return pandas.read_parquet(path, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pandas/io/parquet.py”, line 667, in read_parquet
return impl.read(
^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pandas/io/parquet.py”, line 274, in read
pa_table = self.api.parquet.read_table(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pyarrow/parquet/core.py”, line 1793, in read_table
dataset = ParquetDataset(
^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.11/dist-packages/pyarrow/parquet/core.py”, line 1360, in init
[fragment], schema=schema or fragment.physical_schema,
^^^^^^^^^^^^^^^^^^^^^^^^
File “pyarrow/_dataset.pyx”, line 1431, in pyarrow._dataset.Fragment.physical_schema.get
File “pyarrow/error.pxi”, line 155, in pyarrow.lib.pyarrow_internal_check_status
File “pyarrow/error.pxi”, line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Could not open Parquet input source ‘’: Parquet file size is 0 bytes
Hi @enzo , thank you for your time… Now, I switched to CPU and it started working… However, the model is very slow on CPU and it will take days for the inference… (e.g., ~10h per sample x 8).
I wasn’t expecting that the GPU would be the issue… On a side note - when I am submitting the job, the program says that it is not detecting GPU needs even though the code requires it, so I am enforcing the use of the GPU… could this be related?
I am not sure. Something is causing the program to silently exit without error…
I tried to catch exit calls, but it only works if it is done in Python.
Hi, I found that when I submit through the website, the program is not crashing on GPU, but through the crunch cli it is. One of the differences I noticed is the requirements.txt file.
When I submit through crunch cli it enforces the package versions even though I do not specify them:
By the way @enzo , I think the --no-pip-freeze option should be the default 1) Rewriting submission files without the user’s explicit knowledge is bad practice and makes it harder for the user to debug if they are not aware of this. Also, it happens on the fly, so the original file remains as it is. 2) This behavior is different from the website submission process, where it is not enforced.
But I also understand why you might prefer this. However, people also use conda env and maybe during the submission process, they are not inside the one they used during development… But maybe you have better arguments to support it…
We had problems in the past where libraries would update in the middle of competitions and some user codes would break. To improve the situation, we froze some of the requirements (like pandas and numpy), but that was never a stable solution.
Although it helps, having a way to know the original package versions that a user was using locally to run their code would ensure that their environment would be reproducible.
We’re not going to make the --no-pip-freeze option the default, but it should be better communicated, I agree.
Since all the processing is done on the server (and not locally), we can only show a message after a submission has been accepted on the server. “Wasting” a submission if you do not like the processed requirements.txt.
This is also a big reason for the whitelist: since Jupyter users could not submit a requirements.txt, we needed a way to know which package name corresponds to which pypi project (sklearn → scikit-learn).
Internally we keep a boolean to indicate to the system if a library should always be frozen to the latest version (up to the submission creation date).
We cannot do this for all packages as not all latest versions are compatible with each other.
Because we understand that debugging is hard, we always offer more submissions to help. We do not want users to feel stuck if the problem is beyond their control.