Splunk Enterprise — Reports/Alerts

Table of Contents

  1. Preface
  2. Splunk Scheduled Reports
  3. Splunk Report Manager
  4. Splunk Alerts
  5. Splunk Creating Alerts
  6. Splunk Alert Actions
  7. Splunk Alert Manager
  8. Debrief

https://discord.com/invite/7g9PrxVcc4

1. Preface

I only made this blog in order to provide common notes and information to anyone interested in using Splunk. It is also great as a reference. Please visit Splunk for the official learning courses

Part 1

Part 2

Part 3

Splunk Enterprise — Q&A — Fields

Splunk Enterprise — Reports/Alerts

2. Splunk Scheduled Reports

A scheduled report is a report that runs on a scheduled interval

everytime a scheduled report is run it can automatically be sent by email or trigger additional actions

they can also be used to power dashboard panels to prevent Splunk from running multiple concurrent ad hoc searches

say every Friday you need to report on the previous week’s product sales numbers in order to monitor the progress, rather than run this report manually every time, use a scheduled report

start with the search we want the report to be based on, select save as menu, select report, name it, add time range picker if needed nut usually scheduled reports do not need to include the picker, then select schedule after creating the report from the additional settings links, the edit schedule window now opens, select “Schedule Report” and the frequency to run the report, then select the time range the report will cover

Splunk allows us to set schedule priority for scheduled searches to help us manage scheduled reports so that they put less strain on our deployment

schedule priority is only available to admin users, and allows us to choose between default, higher, and highest

giving a scheduled report a greater priority will make sure that they take priority over other searches being run on the system at that time

the schedule window will allow to set a time frame in which to run our report, so if we have other reports scheduled to run at the same time it can provide window in which to run a report

this provides some time flexibility as to when the report is actually executed

if the system is busy our report will be delayed as long as it falls within our acceptable window

you would include a schedule window only if the report doesn’t have to start at a specific time and you’re okay with the delay

setting the window to “Auto” will allow to Splunk to automatically determine the best time window in which to run the report

we can choose what actions to trigger when a scheduled report is run

there are options to logging an event to a Splunk receiver endpoint outputting results to a lookup sending results to a telemetry endpoint running a script sending an email and posting to an external URL

Splunk also lets you install or build custom alert actions depending on your user role

admin users will have the option to manage and add prebuilt alert actions by selecting “Manage Actions”

to receive the action report via email, select “send email”, this is configured by entering one or more addresses in the “To” field separated by a comma

we set the priority and can customize the subject and message sent in the email by using tokens, here we are supplying the name of the report but there are many tokens that can be used

the options to include things in the email to include links to the report and the results search string run and documents to attach and the option to send the email in HTML and/or plain text format then clicking save will create a scheduled report

3. Splunk Report Manager

managing scheduled reports can be done from the “Searches Reports and Alerts” link in the settings dropdown menu it provides a quick overview of the report and when it is scheduled to run next

clicking on the name of the report will get quick access to change the search string and time range

the edit action next to the name of our report allows to edit the search, permissions, schedule, acceleration and summary indexing for the report, can also disable, clone, embed, move, or delete the report

access report management from the reports tab in the Search and Reporting app, clicking on the name of the report in this window will display the results of our scheduled report, can use the edit button to change settings from this location

users with a power role have the option to display reports for themselves or other users of the app, in this case Search and Reporting

Displaying a report for all apps requires an admin role and selecting App allows us to grant read and write access to all users in our organization, or users based on their role

to make reports available to personnel that do not have access to the Splunk instance, use the embed option in edit then enable embedding for the report then the report will be viewable by anyone that has access or anyone with access to the web page will be able to see the report it is inserted on

“Enable Embedding” gives the code for an iFrame that we can use on any html page, once it is enabled attributes will no longer be able to be edited

An Embedded report will not show data until the scheduled search is run

there is an option to add a scheduled report to a dashboard

4. Splunk Alerts

Splunk alerts are based on searches that run on a scheduled interval or in real time and notify you when the results of a search meets defined conditions and triggered when search is completed

alerts can list in interface, log events, output to lookup, send to a telemetry endpoint, trigger scripts, send emails, use a webhook, run a custom alert

5. Splunk Creating Alerts

alerts are created by first defining a search

create a search for errors then Save As Alert to create an alert, title, set alert permissions (private only you can access, edit, and view the alert) share in app (the results will be displayed for all users of the app), alert type scheduled or real time(scheduled alert allows you to set a schedule and time range for the search to be run)(real time alert type will run the search continuously in the background)

index=web sourcetype=access_combine status=5*

by default everyone has read access and power users have write access to the alert

as soon as alert conditions are satisfied an action is triggered

since real time alerts run continuously they can place more overhead on system performance, but since we want to know immediately when a trigger alert is met it is the better option

to set the trigger condition that will cause the alert to fire, set alerts to trigger per result after a defined number of results, after a defined number of hosts are returned, after a defined number of sources are returned, or with custom conditions you define using the Splunk search language

a bump in the network or a server update could cause a 500 error, but to confirm an actual problem before sending an alert select “Number of Results” trigger condition and set to be alerted if an error is returned more than once and select is greater than from the dropdown menu and enter 1 for the number of results then select a time ranged for the events to happen in before triggering the alert

set the alert to trigger if more than once error happens within 60 minutes

to throttle the number of alerts and how often the alert is executed select Trigger once so that it will only be fired once

for Trigger For each result it will be fired every time our conditions are met

Throttle check box to suppress based off field values or time intervals

6. Splunk Alert Actions

Alert actions allow you to respond to triggered alerts under Trigger actions “+ Add Actions”

when trigger conditions are met there are a choice of actions to take when an alert is triggered and the options will be different depending on the user role

Triggered alerts allows to set the severity of the alert that allows us to sort and filter alerts in the triggered alert list

if you have proper capabilities you can also use a Log Event action, Log events are sent to your Splunk deployment for indexing

be sure to enter the event information that will be written to the log, the source, sourcetype, and host for the alert and the target index for the log event

the “Output results to lookup” option will create or update a .csv lookup table, then can choose to append or replace the data in the lookup table

The “Output results to telemetry endpoint” allows to set a name, input field, select the data type, select categories, and opt -in

The “Run a script” action will trigger a shell or bash file stored on your instance (this action is officially deprecated and suggest using a custom alert action instead)

Send email action lets you set email and select Priority, Subject, Message, what elements to include, and to send in HTML and/or plain text

Use Splunk tokens to add dynamic data to the email

Webhooks allows you to define custom callbacks on a particular web resource, for instance you can set up a webhook to make an alert message pop up in a chat room or create a ticket in a support app

Splunk also allows you to install or build custom alerts actions, admins can find prebuilt alert actions by clicking on the “Manage Alert Actions” link

If there is not an action that fits your needs, you can build your own

7. Splunk Alert Manager

View anytime a Trigger fired by going to “Triggered Alerts” in the Activity menu where you can view the results and edit the search that defines the alert

you can manage alerts the same way you did scheduled reports by accessing the setting and “Searches, reports, and alerts” links

Alerts can also be edited from the Alerts menu in the Search and Reporting app, on the alerts page is the alert’s title, available actions, the owner of the alert, the alert’s app, who the alert is shared with, and whether the alert is enabled or disabled

Additional information about an alert can be viewed by expanding it, and we have direct links to edit permissions, settings for the alert type, trigger conditions, and alert actions, provided we have edit permissions for the selected alert

you can open the alert in search, or use the Edit menu to edit the alert, edit permissions, disable, clone, or delete it

8. Debrief

I hoped this helped answer some general starter questions for anyone just learning Splunk. I really enjoyed doing and this and will be making more notes in the future.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
D.M.

D.M.

26 Followers

Experienced Cyber Security/Intelligence Analyst with a demonstrated history of working in the US Military and IT industry.