SPARSE_STATUS_ALLOC_FAILED

[alexbovet]

Thanks a lot for this great library. This is extremely useful.

I have a problem happening with large sparse matrices and gram_matrix_mkl returning `2 (SPARSE_STATUS_ALLOC_FAILED)'.

I understand that there is a limit in the available memory to allocate, but this happens well before I reach the limit of my system.

The simplest example is:

from scipy.sparse import rand
rom sparse_dot_mkl import gram_matrix_mkl

p = rand(1,100000,0.7).tocsr()

gram_matrix_mkl(p)

which returns:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-4-456ad292feac> in <module>
----> 1 gram_matrix_mkl(p)

~/-/lib/python3.8/site-packages/sparse_dot_mkl/sparse_dot.py in gram_matrix_mkl(matrix, transpose, cast, dense, debug, reorder_output)
    105         dprint(get_version_string())
    106 
--> 107     return _gm(matrix, transpose=transpose, cast=cast, dense=dense, reorder_output=reorder_output)
    108 
    109 

~/-/lib/python3.8/site-packages/sparse_dot_mkl/_gram_matrix.py in _gram_matrix(matrix, transpose, cast, dense, reorder_output, dprint)
    184         return _gram_matrix_sparse_to_dense(matrix, aat=transpose)
    185     else:
--> 186         return _gram_matrix_sparse(matrix, aat=transpose, reorder_output=reorder_output)
    187 
    188 

~/-/lib/python3.8/site-packages/sparse_dot_mkl/_gram_matrix.py in _gram_matrix_sparse(matrix_a, aat, reorder_output)
     37     # Check return
     38     if ret_val != 0:
---> 39         raise ValueError("mkl_sparse_syrk returned {v} ({e})".format(v=ret_val, e=RETURN_CODES[ret_val]))
     40 
     41     if reorder_output:

ValueError: mkl_sparse_syrk returned 2 (SPARSE_STATUS_ALLOC_FAILED)

But, for example, I can do

import numpy as np
O = np.outer(p.data,p.data)

without any problem. Here, O.nbytes/1024/1024/1024 = 36.50 GB and I have 1.5TB of ram installed.

I know that this could be a problem of intel MKL rather than your library, but I could not find anything on the intel MKL forums or documentation.

I would appreciate If you know if this is expected or if there is a way to fix it.

I am using python3.8, sparse_dot_mkl 0.5.3 and Intel MKL 2020.2.254.

uname -a:
Linux 5.4.0-42-generic #46-Ubuntu SMP Fri Jul 10 00:24:02 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[alexbovet]

Ok, I just realized that this is the same issue than #6 .

if I pass export MKL_INTERFACE_LAYER=ILP64 the problem is solved. :-)

[asistradition]

Appreciate the report anyway; there's an explicit error message about the solution if either input matrix is too big, but I forgot the case where only the product is too big.

I added a clearer error message for this in 7d0d225 and I'll put it on pypi at some point.