unresponsive (from error?)

[ihanson]

aleph.sagemath.org is now unresponsive, and here is the last bit of the stdout log:

InvalidDocument: BSON document too large (21829064 bytes) - the connected server supports BSON document sizes up to 16777216 bytes.
ERROR:root:Exception in I/O handler for fd <zmq.core.socket.Socket object at 0xaa9a70>
Traceback (most recent call last):
  File "/padic/scratch/jason/sage-4.7.1-sage.math.washington.edu-x86_64-Linux/local/lib/python2.6/site-packages/zmq/eventloop/ioloop.py", line 330, in start
    self._handlers[fd](fd, events)
  File "/padic/scratch/jason/sage-4.7.1-sage.math.washington.edu-x86_64-Linux/local/lib/python2.6/site-packages/zmq/eventloop/zmqstream.py", line 391, in _handle_events
    self._handle_recv()
  File "/padic/scratch/jason/sage-4.7.1-sage.math.washington.edu-x86_64-Linux/local/lib/python2.6/site-packages/zmq/eventloop/zmqstream.py", line 424, in _handle_recv
    self._run_callback(callback, msg)
  File "/padic/scratch/jason/sage-4.7.1-sage.math.washington.edu-x86_64-Linux/local/lib/python2.6/site-packages/zmq/eventloop/zmqstream.py", line 365, in _run_callback
    callback(*args, **kwargs)
  File "trusted_db.py", line 95, in <lambda>
    stream.on_recv(lambda msgs:callback(db,key,pipe,fs_auth_dict if isFS else db_auth_dict,rep,msgs,isFS), copy=False)
  File "trusted_db.py", line 159, in callback
    db.add_messages([c[0] for c in content])
  File "/padic/scratch/jason/simple-python-db-compute/db_mongo.py", line 112, in add_messages
    self.database.messages.insert(messages)
  File "/padic/scratch/jason/sage-4.7.1-sage.math.washington.edu-x86_64-Linux/local/lib/python2.6/site-packages/pymongo/collection.py", line 310, in insert
    continue_on_error, self.__uuid_subtype), safe)
  File "/padic/scratch/jason/sage-4.7.1-sage.math.washington.edu-x86_64-Linux/local/lib/python2.6/site-packages/pymongo/connection.py", line 807, in _send_message
    (request_id, data) = self.__check_bson_size(message)
  File "/padic/scratch/jason/sage-4.7.1-sage.math.washington.edu-x86_64-Linux/local/lib/python2.6/site-packages/pymongo/connection.py", line 784, in __check_bson_size
    (max_doc_size, self.__max_bson_size))
InvalidDocument: BSON document too large (21829064 bytes) - the connected server supports BSON document sizes up to 16777216 bytes.
Traceback (most recent call last):
  File "device_process.py", line 793, in <module>
    keys=keys, resource_limits=resource_limits)
  File "device_process.py", line 343, in device
    for X in db.get_input_messages(device=device_id, limit=-1):
  File "/padic/scratch/jason/simple-python-db-compute/db_zmq.py", line 35, in f
    output=self.socket.recv_pyobj()
  File "socket.pyx", line 801, in zmq.core.socket.Socket.recv_pyobj (zmq/core/socket.c:7113)
cPickle.UnpicklingError: invalid load key, '{'.

[kramer314]

I'm not sure about the pickling error, but for the first error what version of MongoDB is aleph.sagemath running? A brief google search on the traceback message suggests that pre-1.8 versions have this issue.

[jasongrout]

It's running mongodb-linux-x86_64-1.8.1

[kramer314]

Hmm, then it seems like someone's input just generated a massive output message, and the DB complained about it being larger than 16MB (hence the error). Implementing gh-139 could be a good long-term solution, but the quickest solution seems to be to just catch the error when things are added to the database and drop the message. Or, do we want to find a way to let the user know that their output was too large / was rejected?

[jasongrout]

This just halted all of the workers again. As a quick fix, let's just drop the computation (or if there's an easy way, insert a message back to the user saying there was an error).

[jasongrout]

Nice, and I agree with Alex's comment. And I'm really curious: how are you attaching code to an existing issue? You can do that from the API; are you using the API?

[ihanson]

Yes, I’ve been using the Github API.

[jasongrout]

Do you happen to have a shell script or python script you could share with us? I know it's easy enough to write my own, but I thought maybe you might have something already written.

[ihanson]

Here: https://gist.github.com/2156799

[jasongrout]

Oops, there is a tab on this line. That needs to be changed to spaces.

[jasongrout]

I guess that's what I get for adding a few small referee commits and merging---now that it didn't contain your tab commit, it appears that I didn't merge your branch. But I made the same tab fix and a few other fixes that relate to printing enormous messages in the logs.