cluster wait-for-service hangs indefinitely on unschedulable node

[seanjensengrey]

I have more nodes in my cluster than resources available so that my last node will never get spawned. This can occur for a variety of reasons.

In this scenario,

riak-mesos cluster --cluster weatherstation wait-for-service --debug

will spin on requests to the second node (riak-weatherstation-10) that happens to be in the list returned by

HTTP URL: http://riak.mesos:31232/api/v1/clusters/weatherstation/nodes
HTTP Response Text: {"nodes":["riak-weatherstation-3","riak-weatherstation-10","riak-weatherstation-7","riak-weatherstation-9","riak-weatherstation-6","riak-weatherstation-8","riak-weatherstation-1","riak-weatherstation-2"]}

HTTP URL: http://10.1.14.248:31232/api/v1/clusters/weatherstation/nodes/riak-weatherstation-10
HTTP Method: GET
HTTP Body: None
HTTP Status: 200
HTTP Response Text: {"riak-weatherstation-10":{"key":"riak-weatherstation-10","status":"requested","location":{"node_name":"","hostname":"","http_port":"undefined","pb_port":"undefined","disterl_port":"undefined","agent_id_value":""},"container_path":"","persistence_id":""}}
INFO:dcos.http:Sending HTTP ['get'] to ['http://10.1.14.248:31232/api/v1/clusters/weatherstation/nodes/riak-weatherstation-10']: {'Accept': 'application/json'}
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 10.1.14.248
DEBUG:requests.packages.urllib3.connectionpool:"GET /api/v1/clusters/weatherstation/nodes/riak-weatherstation-10 HTTP/1.1" 200 255
INFO:dcos.http:Received HTTP response [200]: {'Date': 'Wed, 20 Jul 2016 15:45:33 GMT', 'Content-Type': 'application/json', 'Server': 'MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)', 'Content-Length': '255'}