Dylink import is extremely slow.

[choksi81]

While running the nodemanager unit tests, I found that each of the unit test was taking about 1 minute to run (which is extremely slow). After timing the unit tests, I found that most of the time was spent in importing the library files. Below is a breakdown of the time for a sample unit test ut_nm_addfiletovessel.repy.

monzum@TestUbuntu:~/exdisk/work/repy_v2_client$ time python repy.py restrictions.test dylink.repy ut_nm_addfiletovessel.repy
Time for import = 50.1873540878
Time for updatetime = 1.09592890739
Time for nmclient_createhandle = 1.03426098824
Time for nmclient_get_handle = 0.000111103057861
Time for nmclient_listaccessiblevessels = 1.02441596985
Time for addfiletovessel = 1.62042403221
Time for retrievefilefromvessel = 1.64809894562
Terminated

real    0m57.769s
user    0m5.568s
sys 0m2.060s

The file was importing the 3 files: time.repy, nmclient.repy and rsa.repy.

[choksi81]

I broke down the time for imports a little further. Note that this does not include the time for every single import, but the ones that are used for just time.repy (as that was taking the longest).
Time to import 'time_interface.repy' = 1.76680493355 # Does not import any other file.
Time to import 'randome.repy' = 4.61108779907
Time to import 'ntp_time.repy' = 4.76079797745
Time to import 'time_interface.repy' = 0.000697135925293
Time to import 'listops.repy' = 1.72201108932 # Does not import any other file.
Time to import 'centralizedadvertise.repy' = 8.36211299896
Time to import 'centralizedadvertise_v2.repy' = 1.89989495277
Time to import 'parallelize.repy' = 4.22195696831
Time to import 'udpcentralizedadvertise.repy' = 2.3095729351
Time to import 'advertise.repy' = 2.46260595322
Time to import 'random.repy' = 0.000741004943848
Time to import 'sockettimeout.repy' = 0.000627040863037
Time to import 'tcp_time.repy' = 0.157870054245
Time to import 'time.repy' = 0.314615011215
Time to import 'nmclient.repy' = 17.1780560017
Time to import 'rsa.repy' = 0.00134801864624
From what I can tell, if I am importing a library for the first time, it takes roughly ~1.7 second (depending on file size). This can add up as time.repy, nmclient.repy and advertise.repy depend on so many other libraries.

[choksi81]

Since imports go through dylink.repy now, they are subject to the usual resource restrictions just like any other Repy program. If I correctly resolved the (sub-)imports, then ut_nm_addfiletovessel.repy imports 20 libraries totaling 180kB. With each openfile call consuming 4kB of fileread, plus the overall file sizes, the usual 10kBps file read restriction makes these imports take approximately 25 seconds.
Also, the random library is very slow --- the random resource is usually restricted to 100 Bps, but randombytes() (which is used internally by the library) returns 1kiB of random data, so you will wait 10+ seconds for it. This hit is noticeable especially on the first call to any of the random_ functions, before there is data in the library's entropy cache.

[choksi81]

I think the system is working as designed. We need to up the values in the restrictions files if they are too low.
From what I can see, upping the random rate should not be an issue. As for file I/O, we can up this as well, but we may get fewer VMs per system then...
At a minimum, the tests can (and should) have higher settings.