Skip to content

Commit 4236c43

Browse files
Calvin979yuluo-yxtomsun28Aias00
authored
[feature] integrate with Apache Arrow (apache#2864)
Signed-off-by: tomsun28 <[email protected]> Signed-off-by: shown <[email protected]> Co-authored-by: shown <[email protected]> Co-authored-by: tomsun28 <[email protected]> Co-authored-by: aias00 <[email protected]>
1 parent 6a64752 commit 4236c43

File tree

159 files changed

+2705
-6072
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

159 files changed

+2705
-6072
lines changed

‎.all-contributorsrc

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2135,6 +2135,33 @@
21352135
"contributions": [
21362136
"code"
21372137
]
2138+
},
2139+
{
2140+
"login": "Rancho-7",
2141+
"name": "Nick Guo",
2142+
"avatar_url": "https://avatars.githubusercontent.com/u/59016860?v=4",
2143+
"profile": "https://github.com/Rancho-7",
2144+
"contributions": [
2145+
"doc"
2146+
]
2147+
},
2148+
{
2149+
"login": "doveLin0818",
2150+
"name": "doveLin",
2151+
"avatar_url": "https://avatars.githubusercontent.com/u/190927907?v=4",
2152+
"profile": "https://github.com/doveLin0818",
2153+
"contributions": [
2154+
"code"
2155+
]
2156+
},
2157+
{
2158+
"login": "yunfan24",
2159+
"name": "yunfan24",
2160+
"avatar_url": "https://avatars.githubusercontent.com/u/91836599?v=4",
2161+
"profile": "https://zzrl.cc/",
2162+
"contributions": [
2163+
"code"
2164+
]
21382165
}
21392166
],
21402167
"contributorsPerLine": 7,

‎README.md

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -160,7 +160,7 @@ Detailed config refer to [Install HertzBeat via Package](https://hertzbeat.apach
160160
##### 3：Start via source code
161161

162162
1. Local source code debugging needs to start the back-end project `manager` and the front-end project `web-app`.
163-
2. Backend：need `maven3+`, `java17`, `lombok`, start the `manager` service.
163+
2. Backend：need `maven3+`, `java17`, `lombok`, add VM options in IDE: ` --add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED `, then start the `manager` service.
164164
3. Web：need `nodejs npm angular-cli` environment, Run `ng serve --open` in `web-app` directory after backend startup.
165165
4. Access `http://localhost:4200` to start, default account: `admin/hertzbeat`
166166

@@ -483,6 +483,9 @@ Thanks to these wonderful people, welcome to join us:
483483
<td align="center" valign="top" width="14.28%"><a href="https://github.com/starryCoder"><img src="https://avatars.githubusercontent.com/u/46510059?v=4?s=100" width="100px;" alt="starryCoder"/><br /><sub><b>starryCoder</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=starryCoder" title="Code">💻</a></td>
484484
<td align="center" valign="top" width="14.28%"><a href="https://github.com/hasimmollah"><img src="https://avatars.githubusercontent.com/u/32538599?v=4?s=100" width="100px;" alt="hasimmollah"/><br /><sub><b>hasimmollah</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=hasimmollah" title="Code">💻</a></td>
485485
<td align="center" valign="top" width="14.28%"><a href="https://github.com/ayu-v0"><img src="https://avatars.githubusercontent.com/u/127600988?v=4?s=100" width="100px;" alt="Ayu"/><br /><sub><b>Ayu</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=ayu-v0" title="Code">💻</a></td>
486+
<td align="center" valign="top" width="14.28%"><a href="https://github.com/Rancho-7"><img src="https://avatars.githubusercontent.com/u/59016860?v=4?s=100" width="100px;" alt="Nick Guo"/><br /><sub><b>Nick Guo</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=Rancho-7" title="Documentation">📖</a></td>
487+
<td align="center" valign="top" width="14.28%"><a href="https://github.com/doveLin0818"><img src="https://avatars.githubusercontent.com/u/190927907?v=4?s=100" width="100px;" alt="doveLin"/><br /><sub><b>doveLin</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=doveLin0818" title="Code">💻</a></td>
488+
<td align="center" valign="top" width="14.28%"><a href="https://zzrl.cc/"><img src="https://avatars.githubusercontent.com/u/91836599?v=4?s=100" width="100px;" alt="yunfan24"/><br /><sub><b>yunfan24</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=yunfan24" title="Code">💻</a></td>
486489
</tr>
487490
</tbody>
488491
</table>

‎README_CN.md

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -159,7 +159,7 @@
159159
##### 方式三：本地代码启动
160160

161161
1. 此为前后端分离项目，本地代码调试需要分别启动后端工程 `manager` 和前端工程 `web-app`
162-
2. 后端：需要 `maven3+`, `java17` 和 `lombok` 环境，修改 `YML` 配置信息并启动 `manager` 服务
162+
2. 后端：需要 `maven3+`, `java17` 和 `lombok` 环境，修改 `YML` 配置信息，添加JVM参数`--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED`后启动 `manager` 服务即可。
163163
3. 前端：需要 `nodejs npm angular-cli`环境，待本地后端启动后，在 `web-app` 目录下启动 `ng serve --open`
164164
4. 浏览器访问 `http://localhost:4200` 即可开始，默认账号密码 `admin/hertzbeat`
165165

@@ -482,6 +482,9 @@ Thanks these wonderful people, welcome to join us:
482482
<td align="center" valign="top" width="14.28%"><a href="https://github.com/starryCoder"><img src="https://avatars.githubusercontent.com/u/46510059?v=4?s=100" width="100px;" alt="starryCoder"/><br /><sub><b>starryCoder</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=starryCoder" title="Code">💻</a></td>
483483
<td align="center" valign="top" width="14.28%"><a href="https://github.com/hasimmollah"><img src="https://avatars.githubusercontent.com/u/32538599?v=4?s=100" width="100px;" alt="hasimmollah"/><br /><sub><b>hasimmollah</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=hasimmollah" title="Code">💻</a></td>
484484
<td align="center" valign="top" width="14.28%"><a href="https://github.com/ayu-v0"><img src="https://avatars.githubusercontent.com/u/127600988?v=4?s=100" width="100px;" alt="Ayu"/><br /><sub><b>Ayu</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=ayu-v0" title="Code">💻</a></td>
485+
<td align="center" valign="top" width="14.28%"><a href="https://github.com/Rancho-7"><img src="https://avatars.githubusercontent.com/u/59016860?v=4?s=100" width="100px;" alt="Nick Guo"/><br /><sub><b>Nick Guo</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=Rancho-7" title="Documentation">📖</a></td>
486+
<td align="center" valign="top" width="14.28%"><a href="https://github.com/doveLin0818"><img src="https://avatars.githubusercontent.com/u/190927907?v=4?s=100" width="100px;" alt="doveLin"/><br /><sub><b>doveLin</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=doveLin0818" title="Code">💻</a></td>
487+
<td align="center" valign="top" width="14.28%"><a href="https://zzrl.cc/"><img src="https://avatars.githubusercontent.com/u/91836599?v=4?s=100" width="100px;" alt="yunfan24"/><br /><sub><b>yunfan24</b></sub></a><br /><a href="https://github.com/apache/hertzbeat/commits?author=yunfan24" title="Code">💻</a></td>
485488
</tr>
486489
</tbody>
487490
</table>

‎hertzbeat-alerter/pom.xml

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,14 @@
7373
<version>${easy-poi.version}</version>
7474
<scope>compile</scope>
7575
</dependency>
76+
<dependency>
77+
<groupId>org.apache.arrow</groupId>
78+
<artifactId>arrow-vector</artifactId>
79+
</dependency>
80+
<dependency>
81+
<groupId>org.apache.arrow</groupId>
82+
<artifactId>arrow-memory-netty</artifactId>
83+
</dependency>
7684
</dependencies>
7785

7886
</project>

‎hertzbeat-alerter/src/main/java/org/apache/hertzbeat/alert/calculate/CalculateAlarm.java

Lines changed: 85 additions & 76 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,10 @@
2626
import static org.apache.hertzbeat.common.constants.CommonConstants.TAG_MONITOR_APP;
2727
import static org.apache.hertzbeat.common.constants.CommonConstants.TAG_MONITOR_ID;
2828
import static org.apache.hertzbeat.common.constants.CommonConstants.TAG_MONITOR_NAME;
29+
30+
import com.google.common.collect.Maps;
2931
import jakarta.persistence.criteria.Predicate;
32+
3033
import java.util.ArrayList;
3134
import java.util.HashMap;
3235
import java.util.List;
@@ -45,8 +48,10 @@
4548
import org.apache.hertzbeat.alert.service.AlertService;
4649
import org.apache.hertzbeat.alert.util.AlertTemplateUtil;
4750
import org.apache.hertzbeat.common.constants.CommonConstants;
51+
import org.apache.hertzbeat.common.constants.MetricDataConstants;
4852
import org.apache.hertzbeat.common.entity.alerter.Alert;
4953
import org.apache.hertzbeat.common.entity.alerter.AlertDefine;
54+
import org.apache.hertzbeat.common.entity.arrow.RowWrapper;
5055
import org.apache.hertzbeat.common.entity.manager.Monitor;
5156
import org.apache.hertzbeat.common.entity.manager.TagItem;
5257
import org.apache.hertzbeat.common.entity.message.CollectRep;
@@ -124,6 +129,7 @@ private void startCalculate() {
124129
if (metricsData != null) {
125130
calculate(metricsData);
126131
}
132+
dataQueue.sendMetricsDataToStorage(metricsData);
127133
} catch (InterruptedException ignored) {
128134
Thread.currentThread().interrupt();
129135
} catch (Exception e) {
@@ -155,94 +161,97 @@ private void calculate(CollectRep.MetricsData metricsData) {
155161
if (defineMap.isEmpty()) {
156162
return;
157163
}
158-
List<CollectRep.Field> fields = metricsData.getFieldsList();
159-
Map<String, Object> fieldValueMap = new HashMap<>(8);
160-
int valueRowCount = metricsData.getValuesCount();
161-
for (Map.Entry<String, List<AlertDefine>> entry : defineMap.entrySet()) {
162-
List<AlertDefine> defines = entry.getValue();
163-
for (AlertDefine define : defines) {
164-
final String expr = define.getExpr();
165-
if (StringUtils.isBlank(expr)) {
166-
continue;
167-
}
168-
if (expr.contains(SYSTEM_VALUE_ROW_COUNT) && metricsData.getValuesCount() == 0) {
169-
fieldValueMap.put(SYSTEM_VALUE_ROW_COUNT, valueRowCount);
170-
try {
171-
boolean match = execAlertExpression(fieldValueMap, expr);
172-
try {
173-
if (match) {
174-
// If the threshold rule matches, the number of times the threshold has been triggered is determined and an alarm is triggered
175-
afterThresholdRuleMatch(currentTimeMilli, monitorId, app, metrics, "", fieldValueMap, define);
176-
// if the threshold is triggered, ignore other data rows
177-
continue;
178-
} else {
179-
String alarmKey = String.valueOf(monitorId) + define.getId();
180-
triggeredAlertMap.remove(alarmKey);
181-
if (define.isRecoverNotice()) {
182-
handleRecoveredAlert(currentTimeMilli, define, expr, alarmKey);
183-
}
184-
}
185-
} catch (Exception e) {
186-
log.error(e.getMessage(), e);
187-
}
188-
} catch (Exception ignored) {}
189-
}
190-
for (CollectRep.ValueRow valueRow : metricsData.getValuesList()) {
164+
Map<String, Object> fieldValueMap = Maps.newHashMapWithExpectedSize(8);
191165

192-
if (CollectionUtils.isEmpty(valueRow.getColumnsList())) {
166+
try {
167+
for (Map.Entry<String, List<AlertDefine>> entry : defineMap.entrySet()) {
168+
List<AlertDefine> defines = entry.getValue();
169+
for (AlertDefine define : defines) {
170+
final String expr = define.getExpr();
171+
if (StringUtils.isBlank(expr)) {
193172
continue;
194173
}
195-
fieldValueMap.clear();
196-
fieldValueMap.put(SYSTEM_VALUE_ROW_COUNT, valueRowCount);
197-
StringBuilder tagBuilder = new StringBuilder();
198-
for (int index = 0; index < valueRow.getColumnsList().size(); index++) {
199-
String valueStr = valueRow.getColumns(index);
200-
if (CommonConstants.NULL_VALUE.equals(valueStr)) {
201-
continue;
202-
}
203-
204-
final CollectRep.Field field = fields.get(index);
205-
final int fieldType = field.getType();
206-
207-
if (fieldType == CommonConstants.TYPE_NUMBER) {
208-
final Double doubleValue;
209-
if ((doubleValue = CommonUtil.parseStrDouble(valueStr)) != null) {
210-
fieldValueMap.put(field.getName(), doubleValue);
174+
175+
if (expr.contains(SYSTEM_VALUE_ROW_COUNT) && metricsData.getValues().size() == 0L) {
176+
fieldValueMap.put(SYSTEM_VALUE_ROW_COUNT, 0);
177+
try {
178+
boolean match = execAlertExpression(fieldValueMap, expr);
179+
try {
180+
if (match) {
181+
// If the threshold rule matches, the number of times the threshold has been triggered is determined and an alarm is triggered
182+
afterThresholdRuleMatch(currentTimeMilli, monitorId, app, metrics, "", fieldValueMap, define);
183+
// if the threshold is triggered, ignore other data rows
184+
continue;
185+
} else {
186+
String alarmKey = String.valueOf(monitorId) + define.getId();
187+
triggeredAlertMap.remove(alarmKey);
188+
if (define.isRecoverNotice()) {
189+
handleRecoveredAlert(currentTimeMilli, define, expr, alarmKey);
190+
}
191+
}
192+
} catch (Exception e) {
193+
log.error(e.getMessage(), e);
211194
}
212-
} else if (fieldType == CommonConstants.TYPE_TIME) {
213-
final Integer integerValue;
214-
if ((integerValue = CommonUtil.parseStrInteger(valueStr)) != null) {
215-
fieldValueMap.put(field.getName(), integerValue);
195+
} catch (Exception ignored) {}
196+
}
197+
198+
RowWrapper rowWrapper = metricsData.readRow();
199+
while (rowWrapper.hasNextRow()) {
200+
rowWrapper = rowWrapper.nextRow();
201+
StringBuilder tagBuilder = new StringBuilder();
202+
fieldValueMap.clear();
203+
fieldValueMap.put(SYSTEM_VALUE_ROW_COUNT, metricsData.getValues().size());
204+
rowWrapper.cellStream().forEach(cell -> {
205+
String valueStr = cell.getValue();
206+
if (CommonConstants.NULL_VALUE.equals(valueStr)) {
207+
return;
216208
}
217-
} else {
218-
if (StringUtils.isNotEmpty(valueStr)) {
219-
fieldValueMap.put(field.getName(), valueStr);
209+
210+
final String fieldName = cell.getField().getName();
211+
final int fieldType = cell.getMetadataAsInteger(MetricDataConstants.TYPE);
212+
213+
if (fieldType == CommonConstants.TYPE_NUMBER) {
214+
final Double doubleValue;
215+
if ((doubleValue = CommonUtil.parseStrDouble(valueStr)) != null) {
216+
fieldValueMap.put(fieldName, doubleValue);
217+
}
218+
} else if (fieldType == CommonConstants.TYPE_TIME) {
219+
final Integer integerValue;
220+
if ((integerValue = CommonUtil.parseStrInteger(valueStr)) != null) {
221+
fieldValueMap.put(fieldName, integerValue);
222+
}
223+
} else {
224+
if (StringUtils.isNotEmpty(valueStr)) {
225+
fieldValueMap.put(fieldName, valueStr);
226+
}
220227
}
221-
}
222228

223-
if (field.getLabel()) {
224-
tagBuilder.append("-").append(valueStr);
225-
}
226-
}
227-
try {
228-
boolean match = execAlertExpression(fieldValueMap, expr);
229+
if (cell.getMetadataAsBoolean(MetricDataConstants.LABEL)) {
230+
tagBuilder.append("-").append(valueStr);
231+
}
232+
});
229233
try {
230-
if (match) {
231-
// If the threshold rule matches, the number of times the threshold has been triggered is determined and an alarm is triggered
232-
afterThresholdRuleMatch(currentTimeMilli, monitorId, app, metrics, tagBuilder.toString(), fieldValueMap, define);
233-
} else {
234-
String alarmKey = String.valueOf(monitorId) + define.getId() + tagBuilder;
235-
triggeredAlertMap.remove(alarmKey);
236-
if (define.isRecoverNotice()) {
237-
handleRecoveredAlert(currentTimeMilli, define, expr, alarmKey);
234+
boolean match = execAlertExpression(fieldValueMap, expr);
235+
try {
236+
if (match) {
237+
// If the threshold rule matches, the number of times the threshold has been triggered is determined and an alarm is triggered
238+
afterThresholdRuleMatch(currentTimeMilli, monitorId, app, metrics, tagBuilder.toString(), fieldValueMap, define);
239+
} else {
240+
String alarmKey = String.valueOf(monitorId) + define.getId() + tagBuilder;
241+
triggeredAlertMap.remove(alarmKey);
242+
if (define.isRecoverNotice()) {
243+
handleRecoveredAlert(currentTimeMilli, define, expr, alarmKey);
244+
}
238245
}
246+
} catch (Exception e) {
247+
log.error(e.getMessage(), e);
239248
}
240-
} catch (Exception e) {
241-
log.error(e.getMessage(), e);
242-
}
243-
} catch (Exception ignored) {}
249+
} catch (Exception ignored) {}
250+
}
244251
}
245252
}
253+
} catch (Exception e) {
254+
log.error(e.getMessage(), e);
246255
}
247256
}
248257

0 commit comments

Comments
 (0)