Let's analyse WSO2 IS Events with Splunk
Events can be used to analyse data more efficiently and effectively than doing log analysis. WSO2 IS has the capability to publish events in many forms like HTTP, log, wso2event, etc. Splunk HEC(HTTP Event Collector) is one of the effective ways to send data to Splunk for analysis. In this article, we are going to see how we can publish Http events from IS to Splunk enterprises.
If you are wondering about Splunk, In layman’s terms, we can say it is one of the leading products in data analysis. However, it is much more than that. Please check https://www.splunk.com/ to learn more about it.
WSO2 has different publishers to publish different data, here we are going to see how we can send authentication related to data to Splunk. For testing purposes, we have developed a sample app in Splunk. Once you have done the steps as following, you should be able to use that sample dashboard to analyze your authentication data.
Please follow the steps carefully and let me know if you have any doubts
Configure HEC in Splunk
In this part, we will try to get a token in Splunk. If you are already familiar with this setup go to configure IS section. But make sure you have overwritten the source as “is_test” to use the sample application.
- Download and install the Splunk enterprise. (if you have Splunk already skip step 1,2)
2. Start Splunk enterprise
https://docs.splunk.com/Documentation/Splunk/7.2.3/SearchTutorial/StartSplunk
3. Log in to Splunk from http://localhost:8000. (The username password will be requested at the first startup. follow the below article to learn about Splunk free license)
4. From the home page, click Settings in the navigation bar and there under DATA select Data inputs (Settings → DATA → Data inputs)
5. Click on the ‘HTTP Event Collector’
Now you should be able to see this screen
6. If this is your first time creating a token, enable the HEC tokens from the global settings
For the moment lets keep the default settings as it is and use the “HTTP” connect
7. Now let’s add a new token
In the Name field, enter a name for the token. Then override the Source name as “is_test”(This is to use the app we created directly without any modification but you can give any names you wanted).
(Optional) In the Description field, enter a description for the input.
(Optional) In the Output Group field, select an existing forwarder output group.
Please follow
8. To check Splunk dashboard configured as expected send this curl request and observe the changes
curl --location --request POST 'http://localhost:8088/services/collector' \
--header 'Content-Type: application/json' \
--header 'Authorization: Splunk <created_token>' \
--data-raw '{
"sourcetype": "_json",
"event": {
"metaData": {
"tenantId": -1234
},
"payloadData": {
"contextId": "52c79d9a-5147-4ad9-9d47-dfdc3e1b73f3",
"eventId": "05a94580-6878-46aa-91c8-74787d0784eb",
"eventType": "overall",
"authenticationSuccess": true,
"username": "John",
"localUserName": "Snow",
"userStoreDomain": "PRIMARY",
"tenantDomain": "carbon.super",
"remoteIp": "72.229.28.185",
"region": "NOT_AVAILABLE",
"inboundAuthType": "oidc",
"serviceProvider": "postman",
"rememberMeEnabled": false,
"forceAuthEnabled": false,
"passiveAuthEnabled": false,
"rolesCommaSeparated": "NOT_AVAILABLE",
"authenticationStep": "1",
"identityProvider": "LOCAL",
"authStepSuccess": true,
"stepAuthenticator": "BasicAuthenticator",
"isFirstLogin": true,
"identityProviderType": "LOCAL",
"_timestamp": 1571154432757
}
}
}'Configure IS publisher
- Enable data publishing by following
In short, you should following config in your deployment.toml
2. Go to
{IS_HOME}/repository/deployment/server/eventpublishers/IsAnalytics-Publisher-wso2event-AuthenticationData.xml 3. Modify the IsAnalytics-Publisher-wso2event-AuthenticationData.xml file to publish data to Splunk over http.
Use the token we created earlier. Change the mapping type as well.
4. Now you have configured IS and Splunk successfully. You should be able to receive the IS authentication events in Splunk. In the Splunk search them using
source = “is_test”Try out the Sample app
In this section, we are going to install our sample app in Splunk to analyze authentication data.
- Download the Demo.spl file from the location
https://drive.google.com/open?id=1F8hosFZ1UaMkjjLE5gKIFi4n355xszOG
2. Go the Splunk home page and click the settings
3. Select Install app from file and provide the file location for the downloaded app.
Once installed you should be able to see the “Demo” app listed under the Apps.
4. From the Splunk home page select the Demo application.
Data will get updated every 30 sec. If you have configured the token source differently, just change the base search of this dashboard.
In the upcoming articles, we will look deeper into this dashboard. We will discuss how we can add or modify the above dashboard.
Like authentication data, we can publish other data like sessions, users, tokens, etc to Splunk as well. Tweak the relevant publishers to send the data from IS to Splunk.
Resouces
- Splunk dashboard source code https://github.com/inthirakumaaran/LoginAnalyticsDashboard
- https://is.docs.wso2.com/en/5.10.0/learn/prerequisites-to-publish-statistics/
- HTTP event publishers in WSO2 IS https://docs.wso2.com/display/DAS300/HTTP+Event+Publisher
- Update dashboard https://inthiraj1994.medium.com/is-login-analytics-using-splunk-part-1-532e69cc8a7b
- Publishing Login events over HTTP or Log files in WSO2 IS: https://inthiraj1994.medium.com/publishing-login-events-over-http-or-log-files-in-wso2-is-dacd854cec93
