Add support for generating metrics

[calebhailey]

No description provided.

[jspaleta]

Straw Proposal

1. Add new boolean flag to optionally emit metrics output
2. Internally hold array of process objects that match input search configuration
3. Construct per process metric names from available process functions: https://pkg.go.dev/github.com/shirou/gopsutil/v3@v3.22.2/process#Process
4. tag each per process metric with search_string, full_cmdline, process_name, and 'pid'
5. Add process_count metric tagged by search_string that can be used in threshold alerting
6. Add other count metrics tagged by search string summed over matching processes used in threshold alerting.

[calebhailey]

References:

• Telegraf processes input plugin
• Prometheus' process-exporter (community developed)

Relevant queries from the Sumo Logic process dashboard(s):

metric=procstat field=cpu_usage host.name={{host.name}}  process.executable.name={{process.executable.name}}  | max by host.name, process.executable.name | topk (10, max) 
metric=procstat field=memory_usage host.name={{host.name}}  process.executable.name={{process.executable.name}}  | max by host.name, process.executable.name | topk (10, max) 
metric=procstat field=cpu_usage host.name={{host.name}}  process.executable.name={{process.executable.name}} | avg by host.name, process.executable.name | filter avg > 65 |  topk(10, avg) 
metric=procstat field=memory_usage host.name={{host.name}}  process.executable.name={{process.executable.name}} | avg by host.name, process.executable.name | filter avg > 65 |  topk(10, avg) 
metric=procstat field=created_at host.name={{host.name}} process.executable.name={{process.executable.name}} | min by process.executable.name, host.name | bottomk(10, max) 
metric=procstat field=created_at host.name={{host.name}} process.executable.name={{process.executable.name}} user=* | count by user | topk(10, latest)
metric=procstat field=num_fds host.name={{host.name}}  process.executable.name={{process.executable.name}} | avg by host.name, process.executable.name | filter latest > 1 |  topk(10, latest) 
metric=procstat (field=read_count or field=write_count) AND host.name={{host.name}}  process.executable.name={{process.executable.name}} | sum by host.name, process.executable.name | filter sum > 65 |  topk(10, sum) 
metric=procstat field=num_threads host.name={{host.name}} process.executable.name={{process.executable.name}} | max by process.executable.name, host.name | topk(10, max) 

metric=procstat field=cpu_usage host.name={{host.name}}  process.executable.name={{process.executable.name}} | avg by host.name, process.executable.name | outlier 
metric=procstat field=read_bytes host.name={{host.name}}  process.executable.name={{process.executable.name}} | avg by host.name, process.executable.name | outlier 
metric=procstat field=write_bytes host.name={{host.name}}  process.executable.name={{process.executable.name}} | avg by host.name, process.executable.name | outlier 
metric=procstat field=num_fds  host.name={{host.name}}  process.executable.name={{process.executable.name}} | avg by host.name, process.executable.name | outlier 
metric=procstat field=major_faults host.name={{host.name}}  process.executable.name={{process.executable.name}} | avg by host.name, process.executable.name | outlier 
metric=procstat field=involuntary_context_switches  host.name={{host.name}}  process.executable.name={{process.executable.name}} | avg by host.name, process.executable.name | outlier 
metric=procstat field=signals_pending  host.name={{host.name}}  process.executable.name={{process.executable.name}} | avg by host.name, process.executable.name | outlier 
metric=procstat field=memory_usage host.name={{host.name}} process.executable.name={{process.executable.name}} | avg by host.name, process.executable.name | outlier
metric=procstat field=cpu_usage host.name={{host.name}}  process.executable.name={{process.executable.name}} | avg by field, host.name, user, process.executable.name  
metric=procstat field=memory_usage host.name={{host.name}}  process.executable.name={{process.executable.name}} | avg by field,host.name, user, process.executable.name  
metric=procstat field=num_fds host.name={{host.name}}  process.executable.name={{process.executable.name}} | avg by field, host.name, user, process.executable.name  
metric=procstat field=num_threads host.name={{host.name}}  process.executable.name={{process.executable.name}} | avg by field, host.name, user, process.executable.name
metric=procstat field=read_bytes host.name={{host.name}}  process.executable.name={{process.executable.name}} | avg by field, host.name, user, process.executable.name
metric=procstat field=write_bytes host.name={{host.name}}  process.executable.name={{process.executable.name}} | avg by field, host.name, user, process.executable.name  
metric=procstat field=memory_rss host.name={{host.name}}  process.executable.name={{process.executable.name}} | avg by host.name, process.executable.name  | outlier 

host.name={{host.name}}  metric=processes field=total | sum by host.name | outlier
metric=processes host.name={{host.name}}   field=total_threads | sum by host.name | outlier
host.name={{host.name}}  metric=processes  field=dead   
host.name={{host.name}}    metric=processes field=zombies    
host.name={{host.name}}  metric=processes  AND (field=running OR field=stopped OR field=sleeping OR field=blocked OR field=dead OR field=wait OR field=idle OR field=paging) | avg by field
host.name={{host.name}}   metric=processes field=total | avg by field

[jspaleta]

So... since this uses a field named field.. should i make this a sumologic-compat flag?

[jspaleta]

Also for reference:
https://github.com/influxdata/telegraf/tree/master/plugins/inputs/procstat

[jspaleta]

Development branch pushed... here's initial implementation output:

$ ./sensu-processes-check -s '[{"search_string":"sensu", "full_cmdline" : true }]'
# OK       | 1 >= 1 (found >= required) evaluated true for "sensu"
# Status - OK
# HELP procstat SumoLogic Compatibility
# TYPE procstat gauge
procstat{field="cpu_usage",host.name="carbon",process.executable.name="sensu-backend",search_string="sensu"} 0.7501450595132637 1646347704969

This should be compatible with the first SumoLogic metric query:

metric=procstat field=cpu_usage host.name={{host.name}}  process.executable.name={{process.executable.name}}  | max by host.name, process.executable.name | topk (10, max) 

[jspaleta]

All SumoLogic metric queries referencing procstat metric family should now work with feature branch in linked pull request.

Note: some metrics require privileged access. IThese metrics will be silently ignored if the executable cannot access them.

[jspaleta]

Okay SumoLogic metric queries referencing processes metric family should now work as well
Here is the output matching a single process.

$ ./sensu-processes-check -s '[{"search_string":"sensu", "full_cmdline" : true }]' 
# OK       | 1 >= 1 (found >= required) evaluated true for "sensu"
# Status - OK
# HELP processes SumoLogic Dashboard Compatible Cumulative Process Metrics
# TYPE processes gauge
processes{field="total",host.name="carbon",units="count"} 1 1646355921928
processes{field="total_threads",host.name="carbon",units="count"} 14 1646355921928
processes{field="parked",host.name="carbon",units="count"} 0 1646355921928
processes{field="wait",host.name="carbon",units="count"} 0 1646355921928
processes{field="zombies",host.name="carbon",units="count"} 0 1646355921928
processes{field="running",host.name="carbon",units="count"} 0 1646355921928
processes{field="sleeping",host.name="carbon",units="count"} 1 1646355921928
processes{field="unknown",host.name="carbon",units="count"} 0 1646355921928
processes{field="blocked",host.name="carbon",units="count"} 0 1646355921928
processes{field="dead",host.name="carbon",units="count"} 0 1646355921928
processes{field="stopped",host.name="carbon",units="count"} 0 1646355921928
processes{field="idle",host.name="carbon",units="count"} 0 1646355921928
processes{field="unexpected",host.name="carbon",units="count"} 0 1646355921928
# HELP procstat SumoLogic Dashboard Compatible Per Process Metrics
# TYPE procstat gauge
procstat{field="cpu_usage",host.name="carbon",process.executable.name="sensu-backend",search_string="sensu",units="percent"} 0.8524630207020176 1646355921928
procstat{field="memory_usage",host.name="carbon",process.executable.name="sensu-backend",search_string="sensu",units="percent"} 0.7666301131248474 1646355921928
procstat{field="memory_rss",host.name="carbon",process.executable.name="sensu-backend",search_string="sensu",units="bytes"} 1.25980672e+08 1646355921928
procstat{field="memory_vms",host.name="carbon",process.executable.name="sensu-backend",search_string="sensu",units="bytes"} 6.205018112e+09 1646355921928
procstat{field="memory_swap",host.name="carbon",process.executable.name="sensu-backend",search_string="sensu",units="bytes"} 0 1646355921928
procstat{field="memory_data",host.name="carbon",process.executable.name="sensu-backend",search_string="sensu",units="bytes"} 0 1646355921928
procstat{field="memory_stack",host.name="carbon",process.executable.name="sensu-backend",search_string="sensu",units="bytes"} 0 1646355921928
procstat{field="memory_locked",host.name="carbon",process.executable.name="sensu-backend",search_string="sensu",units="bytes"} 0 1646355921928
procstat{field="created_at",host.name="carbon",process.executable.name="sensu-backend",search_string="sensu",units="nanoseconds"} 1.646275163e+18 1646355921928
procstat{field="num_fds",host.name="carbon",process.executable.name="sensu-backend",search_string="sensu",units="count"} 0 1646355921928
procstat{field="num_threads",host.name="carbon",process.executable.name="sensu-backend",search_string="sensu",units="count"} 1.4e+07 1646355921928
procstat{field="major_faults",host.name="carbon",process.executable.name="sensu-backend",search_string="sensu",units="count"} 1042 1646355921928
procstat{field="minor_faults",host.name="carbon",process.executable.name="sensu-backend",search_string="sensu",units="count"} 37969 1646355921928
procstat{field="child_major_faults",host.name="carbon",process.executable.name="sensu-backend",search_string="sensu",units="count"} 0 1646355921928
procstat{field="child_minor_faults",host.name="carbon",process.executable.name="sensu-backend",search_string="sensu",units="count"} 314065 1646355921928
procstat{field="involuntary_context_switches",host.name="carbon",process.executable.name="sensu-backend",search_string="sensu",units="count"} 9416 1646355921928
procstat{field="voluntary_context_switches",host.name="carbon",process.executable.name="sensu-backend",search_string="sensu",units="count"} 864533 1646355921928

[asachs01]

@jspaleta @calebhailey noting that the request in this internal ref may be related to this. However, the need appears to be such that they want to alert on if a named process is using more than X% CPU. While the metrics above would go part way to solving the request in the internal ref, we'd still need to add a status component, if that's even a possibility.