Python error when there are too many cells? Script never finishes

[ejagoda14]

I have 4 samples I need to run through CITE-seq count. Everything worked fine for my first two samples that had 4,982 and 35,651 cells respectively. However, my second two samples have 51,458 and 49,904 cells and I suddenly started getting an error in my output (see below). Moreover, the error doesn't actually stop the job but after a few more lines of saying that it processed some more reads it stalls out - it doesn't quit, the job still seems to be running but no more output is ever generated.

I assume it must be an issue with having more cells than my other samples so I tried increasing the cores but even with 256 cores I get the same issue.

Loading whitelist
Counting number of reads
Started mapping
Processing 92,834,317 reads
CITE-seq-Count is running with 256 cores.
Mapping done for process 2332. Processed 362,634 reads
Mapping done for process 2333. Processed 362,634 reads
Mapping done for process 2335. Processed 362,634 reads
Mapping done for process 2338. Processed 362,634 reads
Mapping done for process 2339. Processed 362,634 reads
Mapping done for process 2340. Processed 362,634 reads
Mapping done for process 2341. Processed 362,634 reads
Mapping done for process 2342. Processed 362,634 reads
Mapping done for process 2343. Processed 362,634 reads
Mapping done for process 2344. Processed 362,634 reads
Mapping done for process 2345. Processed 362,634 reads
Mapping done for process 2346. Processed 362,634 reads
Mapping done for process 2347. Processed 362,634 reads
Mapping done for process 2348. Processed 362,634 reads
Mapping done for process 2349. Processed 362,634 reads
Mapping done for process 2350. Processed 362,634 reads
Mapping done for process 2351. Processed 362,634 reads
Mapping done for process 2352. Processed 362,634 reads
Mapping done for process 2353. Processed 362,634 reads
Mapping done for process 2354. Processed 362,634 reads
Mapping done for process 2355. Processed 362,634 reads
Mapping done for process 2356. Processed 362,634 reads
Mapping done for process 2357. Processed 362,634 reads
Mapping done for process 2358. Processed 362,634 reads
Mapping done for process 2359. Processed 362,634 reads
Mapping done for process 2360. Processed 362,634 reads
Mapping done for process 2361. Processed 362,634 reads
Mapping done for process 2362. Processed 362,634 reads
Mapping done for process 2363. Processed 362,634 reads
Mapping done for process 2364. Processed 362,634 reads
Mapping done for process 2365. Processed 362,634 reads
Mapping done for process 2366. Processed 362,634 reads
Mapping done for process 2367. Processed 362,634 reads
Mapping done for process 2368. Processed 362,634 reads
Mapping done for process 2369. Processed 362,634 reads
Mapping done for process 2370. Processed 362,634 reads
Mapping done for process 2372. Processed 362,634 reads
Mapping done for process 2375. Processed 362,634 reads
Mapping done for process 2409. Processed 362,634 reads
Exception in thread Thread-3:
Traceback (most recent call last):
File "/home/groups/engreitz/Software/anaconda3/envs/EngreitzLab/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/home/groups/engreitz/Software/anaconda3/envs/EngreitzLab/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/home/users/ejagoda/.local/lib/python3.6/site-packages/multiprocess/pool.py", line 479, in _handle_results
cache[job]._set(i, obj)
File "/home/users/ejagoda/.local/lib/python3.6/site-packages/multiprocess/pool.py", line 651, in _set
self._error_callback(self._value)
TypeError: '_io.TextIOWrapper' object is not callable

Mapping done for process 2571. Processed 362,634 reads
Mapping done for process 2573. Processed 362,634 reads

[zqzneptune]

We are experiencing the same issue. The number of cells to be processed is ~10K.

[Hoohm]

I would try and run with fewer cores, maybe something like 4 to 8.

50k cells should be fine. Not sure what's going on there. Can you try the https://github.com/Hoohm/CITE-seq-Count/tree/feature/cells_argument branch?

It's a new implementation. Sadly it's worse in terms of memory, but it might survive your bug.