Skip to content

Commit ca01441

Browse files
viragtripathiviragtripathi
committed
Splunk (HEC) example
1 parent 48ed69e commit ca01441

12 files changed

+650
-0
lines changed

‎examples/splunk/README.md

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
## Setting up Splunk Enterprise (Source)
2+
3+
Please see an example under [Demo](demo/setup_splunk.sh).
4+
5+
## Setting up Redis Enterprise Databases (Target)
6+
7+
Before using Redis Connect to capture the changes committed on Vertica database into Redis Enterprise Databases, first create a database for the metadata management and metrics provided by Redis Connect by creating a database with [RedisTimeSeries](https://redis.com/modules/redis-timeseries/) module enabled, see [Create Redis Enterprise Database](https://docs.redis.com/latest/rs/administering/creating-databases/#creating-a-new-redis-database) for reference. Then, create (or use an existing) another Redis Enterprise database (Target) to store the changes coming from PostgreSQL. Additionally, you can enable [RediSearch 2.0](https://redis.com/blog/introducing-redisearch-2-0/) module on the target database to enable secondary index with full-text search capabilities on the existing hashes where PostgreSQL changed events are being written at then [create an index, and start querying](https://oss.redis.com/redisearch/Commands/) the document in hashes.
8+
9+
| ℹ️ |
10+
|:-------------------------------------------------|
11+
| Docker demo: Follow the [Docker demo](demo) |
12+
| K8s Setup: Follow the [k8s-docs](../../k8s-docs) |

‎examples/splunk/demo/README.md

+167
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
# Demo Outline
2+
:white_check_mark: Setup and start Vertica database (Source)<br>
3+
:white_check_mark: Setup and start Redis Enterprise database (Target)<br>
4+
:white_check_mark: Setup and start Redis Connect<br>
5+
:white_check_mark: Perform Initial load with Redis Connect<br>
6+
7+
# Prerequisites
8+
Docker compatible [*nix OS](https://en.wikipedia.org/wiki/Unix-like) and [Docker](https://docs.docker.com/get-docker) installed.
9+
<br>Please have 8 vCPU*, 8GB RAM and 50GB storage for this demo to function properly. Adjust the resources based on your requirements. For HA, at least have 2 Redis Connect Connector instances deployed on separate hosts.</br>
10+
<br>Execute the following commands (copy & paste) to download and setup Redis Connect and demo scripts.
11+
i.e.</br>
12+
13+
```bash
14+
wget -c https://github.com/redis-field-engineering/redis-connect-dist/archive/main.zip && \
15+
mkdir -p redis-connect/demo && \
16+
mkdir -p redis-connect/k8s-docs && \
17+
unzip main.zip "redis-connect-dist-main/examples/splunk/*" -d redis-connect && \
18+
cp -R redis-connect/redis-connect-dist-main/examples/splunk/demo/* redis-connect/demo && \
19+
cp -R redis-connect/redis-connect-dist-main/examples/splunk/k8s-docs/* redis-connect/k8s-docs && \
20+
rm -rf main.zip redis-connect/redis-connect-dist-main && \
21+
cd redis-connect && \
22+
chmod a+x demo/*.sh && \
23+
cd demo
24+
```
25+
26+
Expected output:
27+
```bash
28+
demo$ ls
29+
README.md config extlib setup_re.sh setup_splunk.sh
30+
```
31+
32+
## Setup Splunk Enterprise (Source)
33+
<br>Execute [setup_splunk.sh](setup_splunk.sh)</br>
34+
```bash
35+
demo$ ./setup_splunk.sh latest 5433
36+
```
37+
38+
## Setup Redis Enterprise cluster, databases and RedisInsight in docker (Target)
39+
<br>Execute [setup_re.sh](setup_re.sh)</br>
40+
```bash
41+
demo$ ./setup_re.sh
42+
```
43+
**NOTE**
44+
45+
The above script will create a 1-node Redis Enterprise cluster in a docker container, [Create a target database with RediSearch module](https://docs.redislabs.com/latest/modules/add-module-to-database/), [Create a job management and metrics database with RedisTimeSeries module](https://docs.redislabs.com/latest/modules/add-module-to-database/), [Create a RediSearch index for emp Hash](https://redislabs.com/blog/getting-started-with-redisearch-2-0/), [Start a docker instance of grafana with Redis Data Source](https://redisgrafana.github.io/) and [Start an instance of RedisInsight](https://docs.redislabs.com/latest/ri/installing/install-docker/).
46+
47+
## Start Redis Connect
48+
49+
<details><summary>Review options by running Redis Connect docker container </summary>
50+
<p>
51+
52+
```bash
53+
demo$ docker run \
54+
-it --rm --privileged=true \
55+
--name redis-connect-$(hostname) \
56+
-v $(pwd)/config:/opt/redislabs/redis-connect/config \
57+
-v $(pwd)/config/samples/credentials:/opt/redislabs/redis-connect/config/samples/credentials \
58+
--net host \
59+
redislabs/redis-connect:0.9.5.5-rc1
60+
```
61+
62+
</p>
63+
</details>
64+
65+
<details><summary>Expected output:</summary>
66+
<p>
67+
68+
```bash
69+
-------------------------------
70+
Redis Connect startup script.
71+
*******************************
72+
Please ensure that these environment variables are correctly mapped before executing start and cli options. They can also be found in /opt/redislabs/redis-connect/bin/redisconnect.conf
73+
Example environment variables and volume mapping for docker based deployments
74+
-e REDISCONNECT_JOB_MANAGER_CONFIG_PATH=/opt/redislabs/redis-connect/config/jobmanager.properties [OPTIONAL]
75+
-e REDISCONNECT_LOGBACK_CONFIG=/opt/redislabs/redis-connect/config/logback.xml [OPTIONAL]
76+
-e REDISCONNECT_JAVA_OPTIONS=-Xms1g -Xmx2g [OPTIONAL]
77+
-e REDISCONNECT_EXTLIB_DIR=/opt/redislabs/redis-connect/extlib [OPTIONAL]
78+
-v <HOST_PATH_TO_JOB_MANAGER_PROPERTIES>:/opt/redislabs/redis-connect/config
79+
-v <HOST_PATH_TO_CREDENTIALS>:/opt/redislabs/redis-connect/config/samples/credentials
80+
-v <HOST_PATH_TO_EXTLIB>:/opt/redislabs/redis-connect/extlib [OPTIONAL]
81+
-p 8282:8282
82+
83+
Usage: [-h|cli|start]
84+
options:
85+
-h: Print this help message and exit.
86+
-v: Print version.
87+
cli: init Redis Connect CLI
88+
start: init Redis Connect Instance (Cluster Member)
89+
-------------------------------
90+
```
91+
92+
</p>
93+
</details>
94+
95+
<details><summary><b>Start Redis Connect Instance</b></summary>
96+
<p>
97+
98+
```bash
99+
docker run \
100+
-it --rm --privileged=true \
101+
--name redis-connect-$(hostname) \
102+
-v $(pwd)/config:/opt/redislabs/redis-connect/config \
103+
-v $(pwd)/extlib:/opt/redislabs/redis-connect/extlib \
104+
--net host \
105+
redislabs/redis-connect:0.9.5.5-rc1 start
106+
```
107+
108+
</p>
109+
</details>
110+
111+
<details><summary>Expected output:</summary>
112+
<p>
113+
114+
```bash
115+
-------------------------------
116+
Starting redis-connect v0.9.5.5 Instance using JAVA 11.0.15 on virag-cdc started by root in /opt/redislabs/redis-connect/bin
117+
Loading redis-connect Instance configuration from /opt/redislabs/redis-connect/config/jobmanager.properties
118+
Instance classpath /opt/redislabs/redis-connect/lib/*:/opt/redislabs/redis-connect/extlib/*
119+
23:32:54.102 [main] INFO redis-connect-manager - ----------------------------------------------------------------------------------------------------------------------------
120+
/####### /## /## /###### /##
121+
| ##__ ## | ## |__/ /##__ ## | ##
122+
| ## \ ## /###### /####### /## /####### | ## \__/ /###### /####### /####### /###### /####### /######
123+
| #######/ /##__ ## /##__ ##| ## /##_____/ | ## /##__ ##| ##__ ##| ##__ ## /##__ ## /##_____/|_ ##_/
124+
| ##__ ##| ########| ## | ##| ##| ###### | ## | ## \ ##| ## \ ##| ## \ ##| ########| ## | ##
125+
| ## \ ##| ##_____/| ## | ##| ## \____ ## | ## ##| ## | ##| ## | ##| ## | ##| ##_____/| ## | ## /##
126+
| ## | ##| #######| #######| ## /#######/ | ######/| ######/| ## | ##| ## | ##| #######| ####### | ####/
127+
|__/ |__/ \_______/ \_______/|__/|_______/ \______/ \______/ |__/ |__/|__/ |__/ \_______/ \_______/ \___/
128+
Powered by Redis Enterprise
129+
23:32:59.108 [main] INFO redis-connect-manager - ----------------------------------------------------------------------------------------------------------------------------
130+
23:33:01.015 [main] INFO redis-connect-manager - Instance: 30@virag-cdc successfully established Redis connection with ClientId: JobManager ConnectionId: JobManager
131+
23:33:01.043 [main] INFO redis-connect-manager - Instance: 30@virag-cdc successfully established Redis connection with ClientId: JobManager ConnectionId: JobReaper
132+
23:33:01.063 [main] INFO redis-connect-manager - Instance: 30@virag-cdc successfully established Redis connection with ClientId: JobManager ConnectionId: JobClaimer
133+
23:33:01.085 [main] INFO redis-connect-manager - Instance: 30@virag-cdc successfully established Redis connection with ClientId: JobManager ConnectionId: JobOrchestrator
134+
23:33:01.112 [main] INFO redis-connect-manager - Instance: 30@virag-cdc successfully established Redis connection with ClientId: JobManager ConnectionId: HeartbeatManager
135+
23:33:01.138 [main] INFO redis-connect-manager - Instance: 30@virag-cdc successfully established Redis connection with ClientId: JobManager ConnectionId: MetricsReporter
136+
23:33:01.170 [main] INFO redis-connect-manager - Instance: 30@virag-cdc successfully established Redis connection with ClientId: JobManager ConnectionId: CredentialsRotationEventListener
137+
23:33:01.210 [main] INFO redis-connect-manager - Instance: 30@virag-cdc successfully established Redis connection with ClientId: JobManager ConnectionId: ChangeEventQueue
138+
23:33:01.297 [main] INFO redis-connect-manager - Instance: 30@virag-cdc skipped creating Job Claim Assignment Consumer Group since it already exists
139+
23:33:01.303 [main] INFO redis-connect-manager - Instance: 30@virag-cdc successfully started JobManager service
140+
23:33:01.304 [main] INFO redis-connect-manager - Instance: 30@virag-cdc successfully started JobReaper service
141+
23:33:01.307 [main] INFO redis-connect-manager - Instance: 30@virag-cdc successfully started JobClaimer service
142+
23:33:01.308 [main] INFO redis-connect-manager - Instance: 30@virag-cdc Metrics are not enabled so MetricsReporter threadpool will not be instantiated
143+
23:33:06.841 [main] INFO redis-connect-manager - Instance: 30@virag-cdc started Redis Connect REST API listening on ["http-nio-8282"]
144+
23:33:06.841 [main] INFO redis-connect-manager - ----------------------------------------------------------------------------------------------------------------------------
145+
23:33:06.841 [main] INFO redis-connect-manager -
146+
23:33:06.842 [main] INFO redis-connect-manager - Started Redis Connect Instance
147+
23:33:06.842 [main] INFO redis-connect-manager -
148+
23:33:06.842 [main] INFO redis-connect-manager - ----------------------------------------------------------------------------------------------------------------------------
149+
23:33:11.303 [JobManagerThreadpool-2] INFO redis-connect-manager - Instance: 30@virag-cdc was successfully elected Redis Connect cluster leader
150+
```
151+
152+
</p>
153+
</details>
154+
155+
**Open browser to access Swagger UI -** [http://localhost:8282/swagger-ui/index.html]()
156+
<br>_For quick start, use '**cdc_job**' as **jobName**_
157+
<br><br><img src="/images/quick-start/Redis Connect Swagger Front Page.jpg" style="float: right;" width = 700px height = 425px/>
158+
159+
**Create Job Configuration** - `/connect/api/vi/job/config/{jobName}`
160+
<br>_For quick start, use the sample `cdc-job.json` configuration:_ <a href="/examples/splunk/demo/config/samples/payloads/cdc-job.json">Splunk</a>
161+
<br><br><img src="/images/quick-start/Redis Connect Save Job Config.png" style="float: right;" width = 700px height = 375px/>
162+
<br>
163+
164+
**Or Use `curl` to create the `cdc-job` configuration** <br>
165+
`demo$ curl -v -X POST "http://localhost:8282/connect/api/v1/job/config/cdc-job" -H "accept: */*" -H "Content-Type: multipart/form-data" -F "file=@config/samples/payloads/cdc-job.json;type=application/json"`
166+
167+
-------------------------------

‎examples/splunk/demo/config/jobmanager.properties

+38
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
##################################################################################################
2+
##### Properties to start a Redis Connect cluster. Default values are commented out. #####
3+
##################################################################################################
4+
5+
##### Cluster properties
6+
#cluster.name=default
7+
#cluster.leader.heartbeat.lease.renewal.interval=5000
8+
#cluster.election.attempt.interval=5000
9+
#cluster.timeseries.metrics.enabled=false
10+
11+
##### Job Manager Services properties
12+
#job.manager.services.enabled=true
13+
#job.manager.services.threadpool.size=2
14+
#job.reap.attempt.interval=7000
15+
#job.claim.service.enabled=true
16+
#job.claim.attempt.interval=5000
17+
#job.claim.batch.size.per.attempt=4
18+
#job.claim.max.capacity=4
19+
#job.claim.heartbeat.lease.renewal.interval=10000
20+
21+
##### REST properties
22+
#rest.api.enabled=true
23+
#rest.api.port=8282
24+
25+
##### Job Manager Database properties
26+
redis.connection.url=redis://127.0.0.1:14001
27+
#redis.connection.sslEnabled=false
28+
#truststore=<path_to_JKS>
29+
#keystore=<path_to_PKCS>
30+
#redis.connection.insecure=false
31+
#redis.connection.timeout.duration=5
32+
#redis.connection.auto.reconnect=true
33+
#redis.connection.suspend.reconnect.on.protocol.failure=true
34+
35+
##### Credentials properties
36+
credentials.file.path=/opt/redislabs/redis-connect/config/samples/credentials
37+
#credentials.rotation.event.listener.enabled=false
38+
#credentials.rotation.event.listener.interval=60000

‎examples/splunk/demo/config/logback-cli.xml

+113
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
<configuration debug="false" scan="true" scanPeriod="15 seconds">
2+
3+
<property name="LOG_REDIS_CONNECT_PATH" value="logs/redis-connect"/>
4+
<property name="LOG_REDIS_CONNECT_MANAGER_PATH" value="logs/redis-connect-manager.log"/>
5+
<property name="LOG_REDIS_CONNECT_HEARTBEAT_PATH" value="logs/redis-connect-heartbeat.log"/>
6+
7+
<appender name="REDIS_CONNECT_HEARTBEAT" class="ch.qos.logback.core.rolling.RollingFileAppender">
8+
<file>${LOG_REDIS_CONNECT_HEARTBEAT_PATH}</file>
9+
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
10+
<fileNamePattern>logs/archived/redis-connect-heartbeat.%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
11+
<!-- each archived file, size max 10MB -->
12+
<maxFileSize>10MB</maxFileSize>
13+
<!-- total size of all archive files, if total size > 20GB, it will delete old archived file -->
14+
<totalSizeCap>20GB</totalSizeCap>
15+
<!-- 60 days to keep -->
16+
<maxHistory>60</maxHistory>
17+
</rollingPolicy>
18+
<encoder>
19+
<pattern>%d %p %c{1.} [%t] %m%n</pattern>
20+
</encoder>
21+
</appender>
22+
<appender name="REDIS_CONNECT_MANAGER" class="ch.qos.logback.core.rolling.RollingFileAppender">
23+
<file>${LOG_REDIS_CONNECT_MANAGER_PATH}</file>
24+
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
25+
<fileNamePattern>logs/archived/redis-connect-manager.%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
26+
<!-- each archived file, size max 10MB -->
27+
<maxFileSize>10MB</maxFileSize>
28+
<!-- total size of all archive files, if total size > 20GB, it will delete old archived file -->
29+
<totalSizeCap>20GB</totalSizeCap>
30+
<!-- 60 days to keep -->
31+
<maxHistory>60</maxHistory>
32+
</rollingPolicy>
33+
<encoder>
34+
<pattern>%d %p %c{1.} [%t] %m%n</pattern>
35+
</encoder>
36+
</appender>
37+
<appender name="REDIS_CONNECT" class="ch.qos.logback.core.rolling.RollingFileAppender">
38+
<file>${LOG_REDIS_CONNECT_PATH}-${PID}.log</file>
39+
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
40+
<fileNamePattern>logs/archived/redis-connect-${PID}.%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
41+
<!-- each archived file, size max 10MB -->
42+
<maxFileSize>10MB</maxFileSize>
43+
<!-- total size of all archive files, if total size > 20GB, it will delete old archived file -->
44+
<totalSizeCap>20GB</totalSizeCap>
45+
<!-- 60 days to keep -->
46+
<maxHistory>60</maxHistory>
47+
</rollingPolicy>
48+
<encoder>
49+
<pattern>%d %p %c{1.} [%t] %m%n</pattern>
50+
</encoder>
51+
</appender>
52+
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
53+
<encoder>
54+
<pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
55+
</encoder>
56+
</appender>
57+
58+
<logger name="redis-connect-heartbeat" level="OFF" additivity="false">
59+
<appender-ref ref="REDIS_CONNECT_HEARTBEAT"/>
60+
<appender-ref ref="CONSOLE" />
61+
</logger>
62+
<logger name="redis-connect-manager" level="OFF" additivity="false">
63+
<appender-ref ref="REDIS_CONNECT_MANAGER"/>
64+
<appender-ref ref="CONSOLE" />
65+
</logger>
66+
<logger name="redis-connect" level="OFF" additivity="false">
67+
<appender-ref ref="REDIS_CONNECT"/>
68+
<appender-ref ref="CONSOLE" />
69+
</logger>
70+
<logger name="io.netty" level="OFF" additivity="false">
71+
<appender-ref ref="REDIS_CONNECT"/>
72+
<appender-ref ref="CONSOLE" />
73+
</logger>
74+
<logger name="io.lettuce" level="OFF" additivity="false">
75+
<appender-ref ref="REDIS_CONNECT"/>
76+
<appender-ref ref="CONSOLE" />
77+
</logger>
78+
<logger name="com.zaxxer" level="OFF" additivity="false">
79+
<appender-ref ref="REDIS_CONNECT"/>
80+
<appender-ref ref="CONSOLE"/>
81+
</logger>
82+
<logger name="io.debezium" level="OFF" additivity="false">
83+
<appender-ref ref="REDIS_CONNECT"/>
84+
<appender-ref ref="CONSOLE"/>
85+
</logger>
86+
<logger name="org.apache.kafka" level="OFF" additivity="false">
87+
<appender-ref ref="REDIS_CONNECT"/>
88+
<appender-ref ref="CONSOLE"/>
89+
</logger>
90+
<logger name="org.springframework" level="OFF" additivity="false">
91+
<appender-ref ref="REDIS_CONNECT"/>
92+
<appender-ref ref="CONSOLE"/>
93+
</logger>
94+
<logger name="springfox" level="OFF" additivity="false">
95+
<appender-ref ref="REDIS_CONNECT"/>
96+
<appender-ref ref="CONSOLE"/>
97+
</logger>
98+
<logger name="org" level="OFF" additivity="false">
99+
<appender-ref ref="REDIS_CONNECT"/>
100+
<appender-ref ref="CONSOLE"/>
101+
</logger>
102+
<logger name="Validator" level="OFF" additivity="false"/>
103+
<logger name="_org.springframework.web.servlet.HandlerMapping" level="OFF" additivity="false"/>
104+
<logger name="com.redis.connect.cluster.RedisConnectSpringBootApplication" level="OFF" additivity="false"/>
105+
<logger name="reactor.util" level="OFF" additivity="false"/>
106+
107+
<root>
108+
<appender-ref ref="REDIS_CONNECT"/>
109+
<appender-ref ref="REDIS_CONNECT_MANAGER"/>
110+
<appender-ref ref="REDIS_CONNECT_HEARTBEAT"/>
111+
</root>
112+
113+
</configuration>

0 commit comments

Comments
 (0)