🧪 Solr AWS observability & autoscaling experiments

This repo holds a playground for Solr Cloud observability & autoscaling experiments with AWS and Terraform. It's idea is to quickly spin up a Zookeeper/Solr ensemble and evaluate SolrCloud autoscaling triggers in conjunction with AWS autoscaling group events.

The Solr Autoscaling Framework is very complicated and seems pretty overengineered. That's why I use this playground to test Solr Autoscaling policies.

🚨 The autoscaling framework in its current form is deprecated and will be removed in Solr 9.0.

Project goal

Use this project as a blueprint to:

• scale Solr autoscaling groups at speed (~90s up and running)
• utilize Cloud-Init and SystemD to properly launch and terminate Solr instances.
• See Prometheus metrics and alarms in action
• See Jaeger distributed tracing in action
• Experiment with Solr autoscaling settings

Dependencies

Before you start, provide your AWS access and secret key to Terraform either static in a secrets.auto.tfvars file or in environment variables.

cat << EOF > secrets.auto.tfvars
aws_access_key = "YOUR_AWS_ACCESS_KEY"
aws_secret_key = "YOUR_AWS_SECRET_KEY"
aws_region     = "eu-west-1"
EOF

Up and running

💰 You are about to create resources in AWS that actually cost money. Just be aware of that when scaling your cluster beyond infinity ...

# create VPC and basic security groups
cd tf-workspaces/vpc && tf init && tf apply

# create monitoring instances
cd ../monitoring && tf init && tf apply

# create Zookeeper and Solr cluster
cd ../solr && tf init && tf apply

# retreive Solr urls
cd ../aws-solr-instances && tf init && tf apply -auto-approve && cd ../..

# create film sample collection
./solr-create-collection.sh

Experimenting

./solr-configure-autoscaling.sh

Tear down

cd terraform-workspaces/aws-solr-instances && tf destroy -auto-approve
cd ../aws-solr && tf destroy -auto-approve
cd ../aws-vpc && tf destroy -auto-approve

Local experimenting

$ curl -sfLo jaeger-core.jar https://repo1.maven.org/maven2/io/jaegertracing/jaeger-core/1.8.0/jaeger-core-1.8.0.jar
$ curl -sfLo jaeger-thrift.jar https://repo1.maven.org/maven2/io/jaegertracing/jaeger-thrift/1.8.0/jaeger-thrift-1.8.0.jar
$ curl -sfLo libthrift.jar https://repo1.maven.org/maven2/org/apache/thrift/libthrift/0.16.0/libthrift-0.16.0.jar
$ curl -sfLo solr-jaegertracer-configurator.jar https://repo1.maven.org/maven2/org/apache/solr/solr-jaegertracer-configurator/8.11.1/solr-jaegertracer-configurator-8.11.1.jar

$ curl "http://localhost:8983/solr/admin/collections?action=CLUSTERPROP&name=samplePercentage&val=100"
$ ./src/main/solr/scripts/solr-create-collection.sh