This Pack enables a variety of Functions when using Cribl Stream to receive data from syslog senders. The pack is specifically to be used in combination with a Cribl Syslog source; syslog-formatted data arriving via other means are not supported by the pack.
Pack benefits:
- Provides a pre-processing Pipeline for handling Cribl Stream's Syslog Source.
- Reduces volume by removing redundant information, such as the human-readable timestamp (typical reduction is 20-30% overall).
- Normalizes timezones when senders do not include timezone information.
- Adds lookup-based enrichment to set additional metadata for a given sender (e.g., index, sourcetype, and timezone).
This Pack also includes sample logs for evaluating which features to enable or disable for specific customer deployments.
Use this Pack's Pipeline to ingest general-purpose syslog senders' data. The Pack's pre-processing Pipeline is for a sender using the Syslog protocol when sending to Cribl Stream. It is not for data in a syslog-compliant format delivered via another method - Splunk forwarder, Elastic Beats agent, etc.
Pre-processing Pipelines (including this one) are particularly suited to general actions that you always want to take for an input Source. Generally, do not use this pre-processing Pipeline for specialized datasets. For specialized datasets, you should specifically tailor a Pipeline for that dataset.
For example, to process all syslog data from all syslog senders on port 514, use a pre-processing Pipeline to normalize the processing of severity and facility information along with basic volume reduction.
When processing a specific dataset, use the pre-processing Pipeline in combination with the other Packs, Routes, or Pipelines specific to that dataset. For example, when receiving Palo Alto firewall data sent via syslog, this pack does the pre-processing and the Palo Alto Pack provides additional processing.
Most Cribl Stream Packs provide a collection of Routes and Pipelines used after the pre-processing phase. This Pack differs in that it also provides a pre-processing Pipeline. Deploying this Pack occurs across three main phases.
The Pack modifies the data being delivered by Cribl Stream, and it supports multiple options for doing so. Prior to putting it into production, it is important to review those options and selectively enable desired features or disable undesired features. Each Function and option is explained within the Pipeline's comments and so those details are omitted here.
Familiarize yourself with the Pack's processing details:
- Open the Pack's
CriblSyslogPreProcessingPipeline. - Select a saved sample and review the IN and OUT versions.
- Read the Pipeline's comments and enable or disable functions as necessary for the deployment environment.
- Optionally, use Sample Data > Capture New with a filter of
__inputId.startsWith('syslog:')to capture data from a deployed production environment with Syslog Sources.
When enabled, the Pack uses a Lookup file to map hostnames or host IPs to metadata including sourcetype, source, index, or timezone. The procedures below provide two ways of editing the Lookup file.
(Recommended for small number of edits or additions, as with testing.)
- Click Knowledge in the Pack's submenu and click the included Lookup
SyslogLookup.csv. (This Lookup file's name is pre-configured in the Lookup Functions within this Pack.) - Change Edit Mode to
Textto edit the metadata column names. Append any additional metadata fields to the first row. - To understand how the rows of the Lookup file work, review the comments in the first lines of the Lookup file.
- Resume editing by changing Edit Mode to
Table. Any added metadata columns will display. - The saved samples included in the Pack reference hosts on lines 3-7. You can remove these lines for deployments where external information is not allowed. Note that the Pipeline's preview mode will no longer show the Lookup results when looking at those samples.
- Click Add Row and provide information for a host. Repeat as needed.
- Click Save.
- Return to the Pipeline to test Lookup Functions against the edited file.
- Click Commit & Deploy when testing is complete.
(Recommended for bulk updates.)
- Click Knowledge in the Pack's submenu and click the included Lookup
SyslogLookup.csv. (This Lookup file's name is pre-configured in the Lookup Functions within this Pack.) - Change Edit Mode to
Textto view the file in.csvformat. - Copy the entire contents and paste to a new file named
SyslogLookup.csv. - Edit the file in your favorite spreadsheet tool, adding or editing columns as desired.
- Upload your edited
.csvfile. Click Knowledge in the Pack's submenu, clickSyslogLookup.csv, click Reupload, navigate to your.csvfile, and then click Open. - Return to the Pipeline to test Lookup Functions against the edited file.
- Click Commit & Deploy when testing is complete. For future bulk edits, repeat steps 4, 5, and 7.
To this point, you have reviewed and prepared the Pack for use. In this step, we enable it by tying it to one or more Syslog Sources.
To set the Pack's Pipeline to use your Sources:
- In the Worker Group's Manage Sources page, select a Syslog Source where you desire pre-processing - probably, most of them.
- In the Pre-Processing tab, select the
[Pack] cribl-syslog-input (syslog pre-processing)Pipeline. - Repeat steps 1 and 2 for additional Syslog Sources as needed.
- Click Commit & Deploy when testing is complete.
Upgrading certian Cribl Packs using the same Pack ID can have unintended consequences. See Upgrading an Existing Pack for details.
Because this Pack has user-modified items (Pipelines, Functions, and Lookups), Cribl recommends that you install future versions with a unique Pack ID by appending the version number to the Pack ID during import. This allows side-by-side comparisons as you configure the updated version.
Once the update is imported, configure the newly installed Pack:
- Review the Pipeline's Functions, enabling and disabling functions as needed.
- Edit the Lookup's
.csvby clicking Reupload or by copying the.csvfrom the previous version and pasting in the new version. - Update the Syslog Sources Pre-Proccessing tab's setting to use the newer Pack version.
- Click Commit & Deploy when testing is complete.
- Re-grouped timezone detection from Message field for easier on/off management
- Auto detection of ISO 8601 time stamps, to avoid unnecessary timezone calculations
- Handling of timestamps from within the message field now support timezone lookups.
- Updates in comments to replace "LogStream" with "Stream"
- Verified support for Cribl Stream 4.0
- Updated README for added clarity during installation and upgrades.
- Added a
C.Lookupoption for returning timezones.
- Changed catch-all Route (used when Source is not Syslog) to use a passthru Pipeline and default Destination.
- Resolved an issue where severity or facility were preserved unintentionally when the value is 0.
- Added an option to perform a lookup using an Eval Function instead of a Lookup Function.
- Minor improvements to the order of processing for missing metadata fields.
- Improved comments to indicate which settings are disabled by default.
- Added metadata for the
packs.cribl.iosuite. - Added sample files for Ubiquiti routers.
- Updated minimum version of Cribl Stream to 3.4.0.
- Increased volume reduction when an event contains multiple timestamps by removing the second timestamp.
- Improved commenting throughout.
- Added log samples from networking gear in an older syslog format.
- Improved metadata lookup attempts - including hardcoded values if all other approaches fail.
The initial release of the Cribl Pack for Syslog Input.
To contribute to the Pack, please connect with Michael Donnelly on Cribl Community Slack. You can suggest new features or offer to collaborate.
This Pack uses the following license: Apache 2.0.