https://discord.com/invite/7g9PrxVcc4
1. Preface
I only made this blog in order to provide common notes and information to anyone interested in using Splunk. It is also great as a reference. Please visit Splunk for the official learning courses
2. Splunk Scheduled Reports
A scheduled report is a report that runs on a scheduled interval
everytime a scheduled report is run it can automatically be sent by email or trigger additional actions
they can also be used to power dashboard panels to prevent Splunk from running multiple concurrent ad hoc searches
say every Friday you need to report on the previous week’s product sales numbers in order to monitor the progress, rather than run this report manually every time, use a scheduled report
start with the search we want the report to be based on, select save as menu, select report, name it, add time range picker if needed nut usually scheduled reports do not need to include the picker, then select schedule after creating the report from the additional settings links, the edit schedule window now opens, select “Schedule Report” and the frequency to run the report, then select the time range the report will cover
Splunk allows us to set schedule priority for scheduled searches to help us manage scheduled reports so that they put less strain on our deployment
schedule priority is only available to admin users, and allows us to choose between default, higher, and highest
giving a scheduled report a greater priority will make sure that they take priority over other searches being run on the system at that time
the schedule window will allow to set a time frame in which to run our report, so if we have other reports scheduled to run at the same time it can provide window in which to run a report
this provides some time flexibility as to when the report is actually executed
if the system is busy our report will be delayed as long as it falls within our acceptable window
you would include a schedule window only if the report doesn’t have to start at a specific time and you’re okay with the delay
setting the window to “Auto” will allow to Splunk to automatically determine the best time window in which to run the report
we can choose what actions to trigger when a scheduled report is run
there are options to logging an event to a Splunk receiver endpoint outputting results to a lookup sending results to a telemetry endpoint running a script sending an email and posting to an external URL
Splunk also lets you install or build custom alert actions depending on your user role
admin users will have the option to manage and add prebuilt alert actions by selecting “Manage Actions”
to receive the action report via email, select “send email”, this is configured by entering one or more addresses in the “To” field separated by a comma
we set the priority and can customize the subject and message sent in the email by using tokens, here we are supplying the name of the report but there are many tokens that can be used
the options to include things in the email to include links to the report and the results search string run and documents to attach and the option to send the email in HTML and/or plain text format then clicking save will create a scheduled report
3. Splunk Report Manager
managing scheduled reports can be done from the “Searches Reports and Alerts” link in the settings dropdown menu it provides a quick overview of the report and when it is scheduled to run next
clicking on the name of the report will get quick access to change the search string and time range
the edit action next to the name of our report allows to edit the search, permissions, schedule, acceleration and summary indexing for the report, can also disable, clone, embed, move, or delete the report
access report management from the reports tab in the Search and Reporting app, clicking on the name of the report in this window will display the results of our scheduled report, can use the edit button to change settings from this location
users with a power role have the option to display reports for themselves or other users of the app, in this case Search and Reporting
Displaying a report for all apps requires an admin role and selecting App allows us to grant read and write access to all users in our organization, or users based on their role
to make reports available to personnel that do not have access to the Splunk instance, use the embed option in edit then enable embedding for the report then the report will be viewable by anyone that has access or anyone with access to the web page will be able to see the report it is inserted on
“Enable Embedding” gives the code for an iFrame that we can use on any html page, once it is enabled attributes will no longer be able to be edited
An Embedded report will not show data until the scheduled search is run
there is an option to add a scheduled report to a dashboard
4. Splunk Alerts
Splunk alerts are based on searches that run on a scheduled interval or in real time and notify you when the results of a search meets defined conditions and triggered when search is completed
alerts can list in interface, log events, output to lookup, send to a telemetry endpoint, trigger scripts, send emails, use a webhook, run a custom alert
5. Splunk Creating Alerts
alerts are created by first defining a search
create a search for errors then Save As Alert to create an alert, title, set alert permissions (private only you can access, edit, and view the alert) share in app (the results will be displayed for all users of the app), alert type scheduled or real time(scheduled alert allows you to set a schedule and time range for the search to be run)(real time alert type will run the search continuously in the background)
index=web sourcetype=access_combine status=5*
by default everyone has read access and power users have write access to the alert
as soon as alert conditions are satisfied an action is triggered
since real time alerts run continuously they can place more overhead on system performance, but since we want to know immediately when a trigger alert is met it is the better option
to set the trigger condition that will cause the alert to fire, set alerts to trigger per result after a defined number of results, after a defined number of hosts are returned, after a defined number of sources are returned, or with custom conditions you define using the Splunk search language
a bump in the network or a server update could cause a 500 error, but to confirm an actual problem before sending an alert select “Number of Results” trigger condition and set to be alerted if an error is returned more than once and select is greater than from the dropdown menu and enter 1 for the number of results then select a time ranged for the events to happen in before triggering the alert
set the alert to trigger if more than once error happens within 60 minutes
to throttle the number of alerts and how often the alert is executed select Trigger once so that it will only be fired once
for Trigger For each result it will be fired every time our conditions are met
Throttle check box to suppress based off field values or time intervals
6. Splunk Alert Actions
Alert actions allow you to respond to triggered alerts under Trigger actions “+ Add Actions”
when trigger conditions are met there are a choice of actions to take when an alert is triggered and the options will be different depending on the user role
Triggered alerts allows to set the severity of the alert that allows us to sort and filter alerts in the triggered alert list
if you have proper capabilities you can also use a Log Event action, Log events are sent to your Splunk deployment for indexing
be sure to enter the event information that will be written to the log, the source, sourcetype, and host for the alert and the target index for the log event
the “Output results to lookup” option will create or update a .csv lookup table, then can choose to append or replace the data in the lookup table
The “Output results to telemetry endpoint” allows to set a name, input field, select the data type, select categories, and opt -in
The “Run a script” action will trigger a shell or bash file stored on your instance (this action is officially deprecated and suggest using a custom alert action instead)
Send email action lets you set email and select Priority, Subject, Message, what elements to include, and to send in HTML and/or plain text
Use Splunk tokens to add dynamic data to the email
Webhooks allows you to define custom callbacks on a particular web resource, for instance you can set up a webhook to make an alert message pop up in a chat room or create a ticket in a support app
Splunk also allows you to install or build custom alerts actions, admins can find prebuilt alert actions by clicking on the “Manage Alert Actions” link
If there is not an action that fits your needs, you can build your own
7. Splunk Alert Manager
View anytime a Trigger fired by going to “Triggered Alerts” in the Activity menu where you can view the results and edit the search that defines the alert
you can manage alerts the same way you did scheduled reports by accessing the setting and “Searches, reports, and alerts” links
Alerts can also be edited from the Alerts menu in the Search and Reporting app, on the alerts page is the alert’s title, available actions, the owner of the alert, the alert’s app, who the alert is shared with, and whether the alert is enabled or disabled
Additional information about an alert can be viewed by expanding it, and we have direct links to edit permissions, settings for the alert type, trigger conditions, and alert actions, provided we have edit permissions for the selected alert
you can open the alert in search, or use the Edit menu to edit the alert, edit permissions, disable, clone, or delete it
8. Debrief
I hoped this helped answer some general starter questions for anyone just learning Splunk. I really enjoyed doing and this and will be making more notes in the future.
