A curated collection of diagnostic and administrative tools for Splunk environments, maintained by Splunk Support for customer use. These scripts help troubleshoot common issues, validate configurations, and perform routine maintenance tasks.
- Splunk Support Scripts
Clone and navigate to the repository:
git clone https://github.com/splunk/support-scripts.git
cd support-scriptsBash script example:
cd test-peers
./testpeers.shPython script example:
cd kvcertverify
$SPLUNK_HOME/bin/python kvcertverify.py --helpRequires Splunk Enterprise access. See individual tool READMEs for detailed usage.
KV Store Certificate Verifier - Verify KV Store certificates before MongoDB upgrade
- Problem: Need to verify KV Store certificates before upgrading MongoDB from 4/4.2 to 7
- Solution: Validates SSL settings, certificate chains, and compatibility requirements
Splunk Config Checker - Generic configuration validator
- Problem: Need to validate Splunk configurations across multiple conf files
- Solution: JSON-based rule validator with custom severity levels and detailed reporting
Splunk User Permissions Checker - Audit user permissions and conflicts
- Problem: Need to audit user permissions and identify conflicts
- Solution: Queries REST API to show role assignments, capabilities, index access, and permission conflicts
Find Duplicate Inputs - Detect overlapping monitor inputs
- Problem: Duplicate data ingestion from overlapping monitor inputs
- Solution: Scans inputs.conf files to identify duplicate or overlapping monitor paths
Lookup Generator - Generate large test lookup files
- Problem: Need large test lookup files for performance testing
- Solution: Generates CSV files with randomized data up to specified size (default 14MB)
Test Peers - Verify distributed search peer connectivity
- Problem: Need to verify distributed search peer connectivity
- Solution: Auto-discovers peers and tests TCP connectivity on port 8089
Note: For advanced stack collection tools, see Splunk UF Diagnosability Tools
Kernel Stacks - D-State Monitor - Monitor D-state processes and collect kernel stacks (Linux only)
- Problem: Need to diagnose processes stuck in uninterruptible sleep (D-state) waiting for I/O or resources
- Solution: Continuously monitors and captures kernel stacks specifically for D-state processes to identify I/O bottlenecks
Kernel Stacks Splunk Threads - Monitor Splunk threads and collect kernel stacks (Linux only)
- Problem: Need to diagnose Splunk thread exhaustion and capture system state during incidents
- Solution: Monitors splunkd thread count and automatically collects kernel stack traces when threshold exceeded
Splunkd Pstacks Threads Monitor - Monitor main splunkd threads and collect pstacks (Linux only)
- Problem: Need to automatically capture user-space stack traces when splunkd thread count exceeds 500
- Solution: Monitors main splunkd process and collects pstack dumps to diagnose lock contention and thread exhaustion
Process-Runner Pstacks Threads Monitor - Monitor process-runner threads and collect pstacks (Linux only)
- Problem: Need to diagnose thread issues specific to Splunk process-runner (scripted inputs, custom commands)
- Solution: Monitors process-runner thread count and collects pstack dumps to identify issues with external process execution
New Log Event - Generate Windows event log entries (Windows only)
- Problem: Need to test Windows event log ingestion with specific event sizes
- Solution: PowerShell script to generate custom-sized event log entries
Most scripts require one or more of:
- Splunk Enterprise installation
- Splunk's Python interpreter (
$SPLUNK_HOME/bin/python) - Bash shell (Linux/macOS)
- PowerShell 3.0+ (Windows-specific tools)
Refer to individual tool documentation for specific requirements.
Team members can contribute new scripts by following these guidelines:
- Create a dedicated folder using kebab-case naming (e.g.,
my-new-tool/) - Include the script(s) with appropriate execute permissions for shell scripts
- Create a README.md with the following sections:
- Purpose (what problem it solves)
- Requirements
- Usage (with clear examples)
- Parameters/options
- Example output (if helpful)
- Notes (compatibility, limitations, etc.)
- Update top-level README.md to add your tool to the appropriate category
# Tool Name
Brief description of what the tool does.
## Purpose
Explain the problem this tool solves and use cases.
## Requirements
- List dependencies
- Note OS compatibility
- Specify version requirements
## Usage
\`\`\`bash
./script.sh [options]
\`\`\`
### Options
- `-o, --option` - Description
### Examples
\`\`\`bash
# Example 1
./script.sh --example
# Example 2
./script.sh --another-example
\`\`\`
## Notes
- Important compatibility notes
- Known limitations
- Security considerations (if applicable)- Scripts should include error handling
- Shell scripts must have shebang lines (
#!/bin/bash) - Cross-platform compatibility when possible (test on Linux/macOS)
- Clear comments for complex logic
- Security best practices (avoid hardcoded credentials, validate inputs)
- Scripts cannot change customer environment apart from creating logging files. See AI usage guidelines
- When using python, if at all possible, use the python packaged with Splunk for consistencies sake
Send scripts to the repository maintainer for review. Include:
- Script file(s)
- README documentation
- Brief description of testing performed
- Tyler Ezell (tezell@splunk.com) - Initial scripts and maintenance
- Robert Phillips (rphillips@splunk.com) - Performance diagnostics and stack collection tools
- Niclas Andersson (nandersson@splunk.com)
- Rob Hilgefort (rhilgefort@splunk.com) - Repository setup, review, and stewardship
Updated:
README.md- Added AI usage guidelines link to contributing sectionsplunk_config_checker- Added rule-based validation system with configurable rulesnew-log-event/README.md- Added disclaimer for test data generation and Security log limitation notelookup-generator/README.md- Added disclaimer for test data generation
Removed:
debugging-splunk- Removed in favor of upstream repo at https://github.com/splunk/uf-diagnosability
Updated:
README.md- Added table of contents, getting started section, and changelog
Added:
debugging-splunk- Advanced pstack collection tool for main splunkd with multiple collection modeskernel-stacks- D-state process monitor for diagnosing uninterruptible sleep issuessplunkd-pstacks-threads- Automatic pstack collection when main splunkd thread count exceeds 500process-runner-pstacks-threads- Automatic pstack collection when process-runner thread count exceeds 500
Included:
kvcertverify- KV Store certificate verifier for MongoDB upgradessplunk_config_checker- Generic configuration validatorsplunk-user-permissions- User permissions and conflicts auditorfind-duplicate-inputs- Duplicate monitor input detectorlookup-generator- Large test lookup file generatortest-peers- Distributed search peer connectivity testerkernel-stacks-splunk-threads- Kernel stack collector triggered by Splunk thread thresholdnew-log-event- Windows event log entry generator