feature: Agent mode support #198

rooty0 · 2023-08-19T01:15:47Z

This PR adds support to run prometheus in agent mode.

Fixes: #187

github-actions · 2023-08-19T01:17:48Z

Docs Build 📝

Thank you for contribution!✨

This PR has been merged and the docs are now incorporated into main:
https://prometheus-community.github.io/ansible/branch/main

roles/prometheus/templates/prometheus.service.j2

rooty0 · 2023-08-19T01:19:44Z

roles/prometheus/templates/prometheus.yml.j2

@@ -17,8 +17,10 @@ remote_read:
  {{ prometheus_remote_read | to_nice_yaml(indent=2,sort_keys=False) | indent(2, False) }}
 {% endif %}

+{% if prometheus_rules_enable and not prometheus_agent_mode %}


rule_files are not allowed in agent mode
Aug 17 20:25:51 master2-int prometheus[50413]: ts=2023-08-17T20:25:51.526Z caller=main.go:479 level=error msg="Error loading config (--config.file=/etc/prometheus/prometheus.yml)" file=/etc/prometheus/prometheus.yml err="field rule_files is not allowed in agent mode"

You probably want to skip copying the rules when using agent mode and not just configure the service to ignore the rules.

ansible/roles/prometheus/tasks/configure.yml

Lines 1 to 25 in 93ed291

---

- name: Alerting rules file

ansible.builtin.template:

src: "alert.rules.j2"

dest: "{{ prometheus_config_dir }}/rules/ansible_managed.rules"

owner: root

group: prometheus

mode: 0640

validate: "{{ _prometheus_binary_install_dir }}/promtool check rules %s"

when:

- prometheus_alert_rules != []

notify:

- reload prometheus

- name: Copy custom alerting rule files

ansible.builtin.copy:

src: "{{ item }}"

dest: "{{ prometheus_config_dir }}/rules/"

owner: root

group: prometheus

mode: 0640

validate: "{{ _prometheus_binary_install_dir }}/promtool check rules %s"

with_fileglob: "{{ prometheus_alert_rules_files }}"

notify:

- reload prometheus

Also I feel that the addition of the prometheus_rules_enable variable is unnecessary.
I'm unsure if we need to be able to disable the rules in setups other than agent mode.
But if we want to offer that option a better approach would be to just leave the prometheus_alert_rules and prometheus_alert_rules_files variables empty/unconfigured and disable the rules based on that.

rooty0 · 2023-08-19T01:21:45Z

roles/prometheus/templates/prometheus.service.j2

@@ -54,6 +58,7 @@ RestrictSUIDSGID=true

 {% if (ansible_facts.packages.systemd | first).version is version('231', '>=') %}
 ReadWritePaths={{ prometheus_db_dir }}
+WorkingDirectory={{ prometheus_db_dir }}


when --storage.tsdb.path is removed, prometheus is trying to write some tsdb related file to / aka root of a filesystem, and since by default it's running from prometheus user we're getting an error:
Aug 17 20:26:44 master2-int prometheus[50516]: ts=2023-08-17T20:26:44.773Z caller=main.go:1155 level=error err="opening storage failed: lock DB directory: mkdir /data-agent: read-only file system"
So defining the WorkingDirectory fixes this issue

You need to set --storage.agent.path.

thanks, replace with --storage.agent.path={{ prometheus_db_dir }}

gardar

Can you please add a molecule scenario for testing the agent mode?

rooty0 · 2023-08-21T19:29:36Z

Can you please add a molecule scenario for testing the agent mode?

@gardar so just to be sure, currently there're 3 targets: alternative, default and latest. Do you want me to create one more target something like agentmode ?

gardar · 2023-08-21T19:35:11Z

Can you please add a molecule scenario for testing the agent mode?

@gardar so just to be sure, currently there're 3 targets: alternative, default and latest. Do you want me to create one more target something like agentmode ?

Yes exactly!

SuperQ · 2023-08-23T05:37:26Z

We recently refactored our test infra. I would recommend rebasing your fork on latest main to pick up the latest changes.

roles/prometheus/molecule/agentmode/converge.yml

roles/prometheus/molecule/agentmode/molecule.yml

Signed-off-by: Stan Rudenko <stan@truera.com>

rooty0 · 2023-08-23T18:49:30Z

@SuperQ thanks!

@gardar looks good now

gardar · 2023-08-24T14:45:25Z

roles/prometheus/molecule/agentmode/tests/test_default.py

Please rename to test_agentmode.py

gardar · 2023-08-24T15:16:07Z

roles/prometheus/templates/prometheus.yml.j2

@@ -17,8 +17,10 @@ remote_read:
  {{ prometheus_remote_read | to_nice_yaml(indent=2,sort_keys=False) | indent(2, False) }}
 {% endif %}

+{% if prometheus_rules_enable and not prometheus_agent_mode %}


You probably want to skip copying the rules when using agent mode and not just configure the service to ignore the rules.

ansible/roles/prometheus/tasks/configure.yml

Lines 1 to 25 in 93ed291

---

- name: Alerting rules file

ansible.builtin.template:

src: "alert.rules.j2"

dest: "{{ prometheus_config_dir }}/rules/ansible_managed.rules"

owner: root

group: prometheus

mode: 0640

validate: "{{ _prometheus_binary_install_dir }}/promtool check rules %s"

when:

- prometheus_alert_rules != []

notify:

- reload prometheus

- name: Copy custom alerting rule files

ansible.builtin.copy:

src: "{{ item }}"

dest: "{{ prometheus_config_dir }}/rules/"

owner: root

group: prometheus

mode: 0640

validate: "{{ _prometheus_binary_install_dir }}/promtool check rules %s"

with_fileglob: "{{ prometheus_alert_rules_files }}"

notify:

- reload prometheus

Also I feel that the addition of the prometheus_rules_enable variable is unnecessary.
I'm unsure if we need to be able to disable the rules in setups other than agent mode.
But if we want to offer that option a better approach would be to just leave the prometheus_alert_rules and prometheus_alert_rules_files variables empty/unconfigured and disable the rules based on that.

Signed-off-by: Stan Rudenko <stan@truera.com>

rooty0 · 2023-08-24T17:26:02Z

@gardar fixed

github-actions bot added enhancement New feature or request roles/prometheus labels Aug 19, 2023

rooty0 commented Aug 19, 2023

View reviewed changes

roles/prometheus/templates/prometheus.service.j2 Show resolved Hide resolved

rooty0 commented Aug 19, 2023

View reviewed changes

rooty0 force-pushed the feat/agent-mode-support branch from be1e7bd to 50539a9 Compare August 19, 2023 01:27

github-actions bot added enhancement New feature or request and removed enhancement New feature or request labels Aug 19, 2023

rooty0 force-pushed the feat/agent-mode-support branch from 989196c to cf566ba Compare August 19, 2023 07:16

github-actions bot added enhancement New feature or request and removed enhancement New feature or request labels Aug 19, 2023

rooty0 requested a review from SuperQ August 20, 2023 16:45

SuperQ approved these changes Aug 21, 2023

View reviewed changes

SuperQ requested a review from gardar August 21, 2023 05:40

gardar reviewed Aug 21, 2023

View reviewed changes

github-actions bot added enhancement New feature or request roles/alertmanager roles/blackbox_exporter roles/chrony_exporter roles/mysqld_exporter roles/node_exporter roles/pushgateway roles/smokeping_prober roles/snmp_exporter and removed enhancement New feature or request labels Aug 22, 2023

SuperQ removed roles/snmp_exporter roles/mysqld_exporter roles/smokeping_prober roles/pushgateway roles/chrony_exporter labels Aug 23, 2023

SuperQ reviewed Aug 23, 2023

View reviewed changes

roles/prometheus/molecule/agentmode/converge.yml Outdated Show resolved Hide resolved

SuperQ reviewed Aug 23, 2023

View reviewed changes

roles/prometheus/molecule/agentmode/molecule.yml Show resolved Hide resolved

rooty0 added 6 commits August 23, 2023 10:20

agent mode support

119bc9a

Signed-off-by: Stan Rudenko <stan@truera.com>

flags are different for agent mode

50a4c1d

Signed-off-by: Stan Rudenko <stan@truera.com>

molecule support

83eee4c

Signed-off-by: Stan Rudenko <stan@truera.com>

template fix

2fde4c9

Signed-off-by: Stan Rudenko <stan@truera.com>

PEP8 fix

f657060

Signed-off-by: Stan Rudenko <stan@truera.com>

test rebase and fixes

8f6880c

Signed-off-by: Stan Rudenko <stan@truera.com>

rooty0 force-pushed the feat/agent-mode-support branch from 78015b5 to 8f6880c Compare August 23, 2023 17:35

github-actions bot added enhancement New feature or request and removed enhancement New feature or request labels Aug 23, 2023

SuperQ approved these changes Aug 23, 2023

View reviewed changes

SuperQ requested a review from gardar August 23, 2023 18:59

gardar requested changes Aug 24, 2023

View reviewed changes

removing prometheus_rules_enable, minor fixes

f637d5f

Signed-off-by: Stan Rudenko <stan@truera.com>

github-actions bot added enhancement New feature or request roles/alertmanager and removed enhancement New feature or request labels Aug 24, 2023

SuperQ approved these changes Aug 24, 2023

View reviewed changes

SuperQ requested a review from gardar August 24, 2023 17:29

gardar approved these changes Aug 24, 2023

View reviewed changes

gardar merged commit ec82955 into prometheus-community:main Aug 24, 2023
46 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature: Agent mode support #198

feature: Agent mode support #198

rooty0 commented Aug 19, 2023

github-actions bot commented Aug 19, 2023 •

edited

Loading

rooty0 Aug 19, 2023

gardar Aug 24, 2023

rooty0 Aug 19, 2023 •

edited

Loading

SuperQ Aug 19, 2023

rooty0 Aug 19, 2023

gardar left a comment

rooty0 commented Aug 21, 2023

gardar commented Aug 21, 2023

SuperQ commented Aug 23, 2023

rooty0 commented Aug 23, 2023

gardar Aug 24, 2023

gardar Aug 24, 2023

rooty0 commented Aug 24, 2023

	---
	- name: Alerting rules file
	ansible.builtin.template:
	src: "alert.rules.j2"
	dest: "{{ prometheus_config_dir }}/rules/ansible_managed.rules"
	owner: root
	group: prometheus
	mode: 0640
	validate: "{{ _prometheus_binary_install_dir }}/promtool check rules %s"
	when:
	- prometheus_alert_rules != []
	notify:
	- reload prometheus

	- name: Copy custom alerting rule files
	ansible.builtin.copy:
	src: "{{ item }}"
	dest: "{{ prometheus_config_dir }}/rules/"
	owner: root
	group: prometheus
	mode: 0640
	validate: "{{ _prometheus_binary_install_dir }}/promtool check rules %s"
	with_fileglob: "{{ prometheus_alert_rules_files }}"
	notify:
	- reload prometheus

feature: Agent mode support #198

feature: Agent mode support #198

Conversation

rooty0 commented Aug 19, 2023

github-actions bot commented Aug 19, 2023 • edited Loading

Docs Build 📝

rooty0 Aug 19, 2023

Choose a reason for hiding this comment

gardar Aug 24, 2023

Choose a reason for hiding this comment

rooty0 Aug 19, 2023 • edited Loading

Choose a reason for hiding this comment

SuperQ Aug 19, 2023

Choose a reason for hiding this comment

rooty0 Aug 19, 2023

Choose a reason for hiding this comment

gardar left a comment

Choose a reason for hiding this comment

rooty0 commented Aug 21, 2023

gardar commented Aug 21, 2023

SuperQ commented Aug 23, 2023

rooty0 commented Aug 23, 2023

gardar Aug 24, 2023

Choose a reason for hiding this comment

gardar Aug 24, 2023

Choose a reason for hiding this comment

rooty0 commented Aug 24, 2023

github-actions bot commented Aug 19, 2023 •

edited

Loading

rooty0 Aug 19, 2023 •

edited

Loading