[SSL: CERTIFICATE_VERIFY_FAILED] for self-signed certs in Python 2.7.9+

[lukpueh]

Clearinghouse and Custominstallerbuilder use python's xmlrpclib to communicate with each other. xmlrpclib in turn is based on python's httplib which was changed in Python 2.7.9+ to raise an exception during handshake when issuing a request via HTTPS and the server uses a self-signed certificate or the CommonName of the certificate does not match the requested host. (c.f. PEP 474 for further background.)

While this behavior is actually preferred in a production environment, it is a nuisance in a testing setup. Possible remedies are:

• sign your certificates, e.g. with let's encrypt

• Add an unverified ssl context to requests in debug mode, e.g. :

  # https://github.com/SeattleTestbed/clearinghouse/blob/master/website/html/views.py#L1110
  if settings.DEBUG:
      xmlrpclib.ServerProxy(settings.SEATTLECLEARINGHOUSE_INSTALLER_BUILDER_XMLRPC, context=ssl._create_unverified_context())
  else:
      xmlrpclib.ServerProxy(settings.SEATTLECLEARINGHOUSE_INSTALLER_BUILDER_XMLRPC)

[vladimir-v-diaz]

Add an unverified ssl context to requests in debug mode, e.g. :

If you intend for this code to work in versions of Python < 2.7.9, please note that the context argument will not be recognized. It was added in 2.7.9.

[lukpueh]

@vladimir-v-diaz Ok, good to know, thanks for the comment. What do you suggest how to best tackle this issue? I guess, we might consider to stop supporting older Python versions at some point.

[vladimir-v-diaz]

Ideally, I'd recommend you drop support for Python < 2.7.9. In practice, that might not be possible. Seattle will probably support earlier versions of Python for the foreseeable future -- node operators are unlikely to upgrade outdated versions of Python that are installed?

The quick fix:
I think this change (certificate verification by default) was not backported to Python<2.7.9, so you can use conditional statements.

if python_version < 2.7.9:
  # do what you do now

else: 
   # use the `context` argument with `xmlrpclib.ServerProxy`

It might be better to actually test SSL connections... so use a certificate. It is easy with Python, not sure if it's possible in Seattle.

[lukpueh]

Btw. the except clause, where I encountered this problem is not very verbose neither to the user nor to the developer.

[aaaaalbert]

@vladimir-v-diaz, thanks for the heads-up. Indeed, we have to support older Python versions for the time being. This is a bit of pain in various places, and also one of the reasons we cannot expose HTTPS / SSL inside Repy sandboxes cleanly (although this would be a great piece of functionality!)

@lukpueh, the except clause you mention, and every other construct of that sort which blanket-excepts but doesn't log the repr of the excepion it caught should be fixed. Creating a separate issue for this would be in order.

[aaaaalbert]

Note: SeattleTestbed/clearinghouse#172 documents the silent blanket except issue.

[aaaaalbert]

Thinking through my "sandbox" comment again .... there's actually no point using anything but the up-to-date Python 2 version for the clearinghouse. Backwards compatibility problems exist only for Seattle installs on nodes in the wild.

Thus, @lukpueh's proposed patch (relying on the settings.DEBUG flag to be set) is the way to go, assuming we can make sure that operators can't shoot themselves in the foot when going from testing to production. How about displaying a big red warning sign all over the web pages served in debug mode?

[aaaaalbert]

@vladimir-v-diaz @lukpueh Thanks guys!

[vladimir-v-diaz]

I think create_unverified_context() was added in Python 2.7.9, so not all versions of Python 2.7 support it. You might want to explicitly say that Python 2.7.9 and greater is required in the Clearinghouse docs (it only says version 2.7).

[vladimir-v-diaz]

Sorry, I edited my previous reply to use the correct Python version of 2.7.9, instead of 2.5.9...

[aaaaalbert]

Addressed in SeattleTestbed/docs@3dbba8b by mentioning that the latest version of Python 2.7 should be used.