Skip to main content
Question

Splunk props.conf for log streaming from AWS

  • January 15, 2026
  • 0 replies
  • 8 views

I am struggling with the props.conf setup to pull web transaction logs via S3 bucket. I have set up per Stream Logs to Splunk - Netskope Knowledge Portal .

This would replace my setup for using REST API v2 for transaction logs eventually.  I currently have logs going in parallel into a temp index for validation before cut over.

I am using queries below to validate time stamp detection and proper data parsing. If you have this setup to your satisfaction, Can I request you to share your props.conf?

 

Validation queries: 
#Check if _time matches x-cs-timestamp

earliest=-10m
index=nsweb_temp sourcetype="netskope:web:test"
| eval payload_time = tonumber(coalesce('x-cs-timestamp','x_cs_timestamp'))
| eval splunk_time  = _time
| eval diff_seconds = splunk_time - payload_time
| eval status = if(diff_seconds==0, "PERFECT", "OFFSET")
| eval diff_abs_s = abs(diff_seconds)
| eval diff_hms   = tostring(diff_abs_s, "duration")
| table _time payload_time splunk_time diff_seconds diff_hms status
| sort - _time
| head 20



#Check if data parsing is correct

earliest=-10m
index=_internal sourcetype=splunkd 
(log_level=ERROR OR log_level=WARN)
"netskope:web:test"
| table _time, component, message
| sort -_time

#One of the many combinations of settings I have tried in props.conf


[netskope:web:test]
SHOULD_LINEMERGE = false
TRUNCATE = 8388608
CHARSET = UTF-8

# (Optional) Default line breaking is usually fine for JSONL
LINE_BREAKER = ([\r\n]+)

# Target the epoch in the JSON directly in _raw
TIME_PREFIX = "x-cs-timestamp":\s*"
TIME_FORMAT = %s
MAX_TIMESTAMP_LOOKAHEAD = 10
TZ = UTC

# These are optional for timestamping, but fine to keep
INDEXED_EXTRACTIONS = json
KV_MODE = json
AUTO_KV_JSON = true

# Keep the window strict
MAX_DAYS_AGO = 365
MAX_DAYS_HENCE = 1



splunk btool output does confirm that my props.conf settings are in use on the HF.

I am either getting time of ingestion stamped as the event time indicating that my props.conf is not taking effect  OR data parsing errors indicating that splunk is picking other fields (such as certificate expiry etc) and then rejecting them as they are invalid for ingestion time.

I have tried tons of combinations using CHatGPT / Gemini / Copilot, but they are sending me in circles

I would appreciate any pointers you might have. thanks!