Can we somehow use local python package structure for importing code from allowed libraries?

Competition ADIA Lab
1

Hi guys!
Can you clarify is there a way to decompose python code into logical modules (i.e. transformations.py, feature_engineering.py, etc.) stored as local python package (with __init__.py within ) within the same directory as my submission dir (i.e. technical-fedir)?
Please note that all of these modules are built on top of whitelisted packages (i.e. pandas, scikit-learn), my submission passes local tests in a notebook, however, the it does not pass internal validation on submission page here, as there are “forbidden” libraries found (i.e. preprocessing).

So, the question is, are there other ways beyond just copy-pasting all the code back again from modules into a “long, ugly” jupyter notebook :grinning: just to make a valid submission?
i.e. to copy such .py files to a “resources” directory on a server first?

And also a small side question - is there a way to specify exact library versions so they remain the same in requirements.txt ? I wonder whether Crunch uses latest available versions at the time of prediction, which (technically) may cause incompatibilities in the future (i.e. API changes in pandas) leading our models, trained using older package versions, to crush during 2nd stage of a competition?
or, in opposite, what library versions to use locally to make it consistent with the server ones?

Many thanks in advance for your hints and answers!
Fred

2

I am not sure to fully understand.

The “Submit via Notebook” will analyze libraries to be able to build a simple requirements.txt since you would have no way of submitting one with your notebook.

You can have as much code as your want, this includes additional files, directories or local modules.
But you must submit via the CLI, there should be no code analysis if you do so.

You can provide any version of whitelisted library in the requirements.txt.
BUT you should not include your local modules in this file, its only for pip libraries.

For faster execution, latest version of some commonly used libraries are already installed (latest version) but should be able to be reinstalled with a custom version if you want.

As long as you don’t touch too much to pandas and pyarrow (which are used by the runner), everything should be fine.

3

Thank you very much for your answer, @enzo !
Following your answer (and using examples found here about CLI usage) I was able to achieve my goals :+1: