Add a common base config for saas deployments #30

Rotonen · 2019-03-14T15:18:03Z

Harvest: OneGov GEVER - Softwarepflege (Refactoring/Bug)

Added a new common buildout for our SaaS deployments
- OGDS is always in PostgreSQL for SaaS deployments
- Solr is always enabled for SaaS deployments
- Bumblebee is always enabled for SaaS deployments
- All SaaS deployments are production-v2 buildouts
The number of ZEO clients is configured per policy

This is for 4teamwork/opengever.core#5409

deiferni

a draft of one policy would help to see if this works
we add one more level to buildout extends, so i would recommend to take extra care everything is still included

Rotonen · 2019-03-20T15:37:55Z

That means actually setting up a full deployment with it as it relies on there being a zope user and it also uses chgrp with GNU specific flags not present on a BSD userland installation.

Do I proceed with that?

deiferni · 2019-03-20T16:41:40Z

as already discussed, stating a bit more precisely: it could help to see how an existing policy looks when changed to this new schema. also it could (once this is merged) help to quickly figure out if the bug-free buildout extends level is exceeded.

deiferni · 2019-06-04T12:49:34Z

what is going on here? looks to be in undefined state. we should either ✅ or 🛑.

Rotonen · 2019-06-04T12:51:53Z

I've yet to figure out how to actually test it locally.

phgross · 2019-06-11T16:38:28Z

@Rotonen please show the diff of an existing saas-buildout which uses the new saas-deployment.cfg. At the same time you can also check if everything works by executing and comparing "bin/buildout annotate" for both variants, the current one and the new one with saas-deployment.cfg.

For example https://github.com/4teamwork/opengever.edk/blob/master/deployment-ipet-05-edk.onegovgever.ch.cfg

Rotonen · 2019-06-11T16:46:01Z

@phgross you mean to set up a full VM for the verification? Locally the buildout will trip up on not having the correct users and filesystem permissions set up as per the new deployment concept.

phgross · 2019-06-11T16:55:01Z

@Rotonen no!!! 😅
Variante 1

git clone opengever.edk ... 
ln -s deployment-xy.cfg buildout.cfg
bin/buildout

Variante 2

git clone opengever.edk ... 
ln -s deployment-xy.cfg buildout.cfg
vim buildout.cfg --> use the new `saas-deployment.cfg`
bin/buildout

Compare the parts directory, et voilà, should take 15min work and 5-10 minute ☕️ time 😉

Rotonen · 2019-06-11T17:21:56Z

As I've now discovered https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/production-local-dev.cfg, my local troubles disappeared.

Rotonen · 2019-06-11T17:49:41Z

What also threw me off is the implicit requirement for there to be a group by the name of the buildout:os-user combining with the --silent flags on the chgrp invocations, which makes the error output chgrp: illegal option -- -.

For reference, here is how to add a matching group for yourself locally on macOS:
https://apple.stackexchange.com/questions/307173/creating-a-group-via-users-groups-in-command-line

Rotonen · 2019-06-11T17:51:19Z

Comparing buildout annotate results:

[buildout]
extends =
    https://raw.githubusercontent.com/4teamwork/gever-buildouts/master/standard-deployment-v2.cfg
    https://raw.githubusercontent.com/4teamwork/gever-buildouts/master/ogds-postgres.cfg
    https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/zeoclients/2.cfg
    https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/bumblebee.cfg
    versions.cfg
    https://raw.githubusercontent.com/4teamwork/gever-buildouts/master/solr.cfg
    https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/production-v2.cfg

vs.

[buildout]
extends =
    https://raw.githubusercontent.com/4teamwork/gever-buildouts/jo-saas-deployment/saas-deployment.cfg
    https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/zeoclients/2.cfg
    versions.cfg

$ diff -u annotate-old.txt annotate-new.txt
--- annotate-old.txt    2019-06-11 19:46:12.000000000 +0200
+++ annotate-new.txt    2019-06-11 20:05:19.000000000 +0200
@@ -6288,9 +6288,9 @@
 slacker
 maintenance
 precompile
-instance2
 solr
 setpermissions
+instance2
 os-environment
     https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/production.cfg
 +=  https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/chameleon.cfg
@@ -6300,9 +6300,9 @@
 +=  https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/maintenance-server.cfg
 -=  https://raw.githubusercontent.com/4teamwork/gever-buildouts/master/standard-deployment.cfg
 +=  https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/precompile.cfg
-+=  https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/zeoclients/2.cfg
 +=  https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/solr-v2.cfg
 +=  https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/production-v2.cfg
++=  https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/zeoclients/2.cfg
 +=  https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/production-local-dev.cfg
 parts-directory= parts
     DEFAULT_VALUE
@@ -19001,12 +19001,12 @@
 90 instance0 (${buildout:supervisor-client-process-opts} autostart=false) ${buildout:bin-directory}/instance0 [console] true ${buildout:os-user}
 20 instance1 (${buildout:supervisor-client-process-opts}) ${buildout:bin-directory}/instance1 [console] true ${buildout:os-user}
 90 maintenance (autostart=false) ${buildout:bin-directory}/maintenance true ${buildout:os-user}
-20 instance2 (${buildout:supervisor-client-process-opts}) ${buildout:bin-directory}/instance2 [console] true ${buildout:os-user}
 10 solr (startsecs=5 stopasgroup=true) ${buildout:bin-directory}/solr [console] true ${buildout:os-user}
+20 instance2 (${buildout:supervisor-client-process-opts}) ${buildout:bin-directory}/instance2 [console] true ${buildout:os-user}
     https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/production.cfg
 +=  https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/maintenance-server.cfg
-+=  https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/zeoclients/2.cfg
 +=  https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/solr-v2.cfg
++=  https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/zeoclients/2.cfg
 recipe= collective.recipe.supervisor
     https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/production.cfg
 user= supervisor

Rotonen · 2019-06-12T12:44:37Z

With fresh eyes, two additions:

+= https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/chameleon.cfg
+= https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/maintenance-server.cfg

Are these unexpected?

Rotonen · 2019-06-12T15:51:15Z

As those come via https://raw.githubusercontent.com/4teamwork/gever-buildouts/master/standard-deployment-v2.cfg, my testing has definitely hit an extension depth issue.

I was extending from a local buildout.cfg as I've removed the ruby gems stuff out of the way for the ease of testing.

Rotonen · 2019-06-12T18:53:18Z

Yeah, I keep getting various results. I guess this is dead in the water.

I guess one or more of these:
https://github.com/buildout/buildout/labels/plus-minus-issues

Rotonen · 2019-06-12T20:35:39Z

So, with not having my own buildout.cfg in between and adding the local dev extend to the repo provided buildout config:

$ diff -u annotate-old.txt annotate-new.txt
--- annotate-old.txt    2019-06-12 22:19:31.000000000 +0200
+++ annotate-new.txt    2019-06-12 22:29:21.000000000 +0200
@@ -6287,9 +6287,9 @@
 slacker
 maintenance
 precompile
-instance2
 solr
 setpermissions
+instance2
 os-environment
     https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/production.cfg
 +=  https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/chameleon.cfg
@@ -6299,9 +6299,9 @@
 +=  https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/maintenance-server.cfg
 -=  https://raw.githubusercontent.com/4teamwork/gever-buildouts/master/standard-deployment.cfg
 +=  https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/precompile.cfg
-+=  https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/zeoclients/2.cfg
 +=  https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/solr-v2.cfg
 +=  https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/production-v2.cfg
++=  https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/zeoclients/2.cfg
 +=  https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/production-local-dev.cfg
 -=  buildout.cfg
 parts-directory= parts
@@ -19000,12 +19000,12 @@
 90 instance0 (${buildout:supervisor-client-process-opts} autostart=false) ${buildout:bin-directory}/instance0 [console] true ${buildout:os-user}
 20 instance1 (${buildout:supervisor-client-process-opts}) ${buildout:bin-directory}/instance1 [console] true ${buildout:os-user}
 90 maintenance (autostart=false) ${buildout:bin-directory}/maintenance true ${buildout:os-user}
-20 instance2 (${buildout:supervisor-client-process-opts}) ${buildout:bin-directory}/instance2 [console] true ${buildout:os-user}
 10 solr (startsecs=5 stopasgroup=true) ${buildout:bin-directory}/solr [console] true ${buildout:os-user}
+20 instance2 (${buildout:supervisor-client-process-opts}) ${buildout:bin-directory}/instance2 [console] true ${buildout:os-user}
     https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/production.cfg
 +=  https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/maintenance-server.cfg
-+=  https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/zeoclients/2.cfg
 +=  https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/solr-v2.cfg
++=  https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/zeoclients/2.cfg
 recipe= collective.recipe.supervisor
     https://raw.githubusercontent.com/4teamwork/ftw-buildouts/master/production.cfg
 user= supervisor

We'd just barely at the max extends count this way. Does not seem like something we'd like to roll with as extending from that policy would then immediately break.

In time, at least Solr, can go to the standard deployment config.

Add a common base config for saas deployments.

587ebd9

Rotonen requested review from buchi and lukasgraf March 14, 2019 15:18

Rotonen mentioned this pull request Mar 14, 2019

Clean up policy templates 4teamwork/opengever.core#5502

Closed

deiferni reviewed Mar 20, 2019

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a common base config for saas deployments #30

Add a common base config for saas deployments #30

Rotonen commented Mar 14, 2019 •

edited

Loading

deiferni left a comment

Rotonen commented Mar 20, 2019

deiferni commented Mar 20, 2019

deiferni commented Jun 4, 2019

Rotonen commented Jun 4, 2019

phgross commented Jun 11, 2019 •

edited

Loading

Rotonen commented Jun 11, 2019

phgross commented Jun 11, 2019

Rotonen commented Jun 11, 2019

Rotonen commented Jun 11, 2019

Rotonen commented Jun 11, 2019 •

edited

Loading

Rotonen commented Jun 12, 2019

Rotonen commented Jun 12, 2019 •

edited

Loading

Rotonen commented Jun 12, 2019

Rotonen commented Jun 12, 2019

Add a common base config for saas deployments #30

Are you sure you want to change the base?

Add a common base config for saas deployments #30

Conversation

Rotonen commented Mar 14, 2019 • edited Loading

deiferni left a comment

Choose a reason for hiding this comment

Rotonen commented Mar 20, 2019

deiferni commented Mar 20, 2019

deiferni commented Jun 4, 2019

Rotonen commented Jun 4, 2019

phgross commented Jun 11, 2019 • edited Loading

Rotonen commented Jun 11, 2019

phgross commented Jun 11, 2019

Rotonen commented Jun 11, 2019

Rotonen commented Jun 11, 2019

Rotonen commented Jun 11, 2019 • edited Loading

Rotonen commented Jun 12, 2019

Rotonen commented Jun 12, 2019 • edited Loading

Rotonen commented Jun 12, 2019

Rotonen commented Jun 12, 2019

Rotonen commented Mar 14, 2019 •

edited

Loading

phgross commented Jun 11, 2019 •

edited

Loading

Rotonen commented Jun 11, 2019 •

edited

Loading

Rotonen commented Jun 12, 2019 •

edited

Loading