You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -9,3 +9,11 @@ data, without the need to set up an ingestion chain (which could be Azure IoTHub
9
9
## Maximizing Performance
10
10
11
11
To maximize performance multiple instances of the Data Generator must be run in parallel. One way to achieve this is using kubernetes how to do this is documented [here](KUBERNETES.md).
Copy file name to clipboardExpand all lines: query_timer/README.md
+31-15
Original file line number
Diff line number
Diff line change
@@ -12,11 +12,14 @@ The Query Timer is a tool to run queries against different databases and determi
12
12
13
13
#### Pip install
14
14
15
-
The Query Timer is part of the tsdb-data-generator package and can be installed using `pip install tsdb-data-generator`.
15
+
The Query Timer is part of the tsdb-data-generator package and can be installed using `pip install tsdg`.
16
16
17
-
By calling `tsqt -h` the possible configurations are listed. For further details see [Query Timer Configuration](#query-timer-configuration). All configurations can be done with either command line arguments or environment variables but when both are set then command line arguments will be used.
17
+
By calling `tsqt -h` the possible configurations are listed. For further details see
18
+
[Query Timer Configuration](#query-timer-configuration). All configurations can be done with either command line
19
+
arguments or environment variables but when both are set then command line arguments will be used.
18
20
19
-
When calling `tsqt` with the desired arguments the Query Timer outputs live updated statistics on the query execution. This includes:
21
+
When calling `tsqt` with the desired arguments the Query Timer outputs live updated statistics on the query execution.
22
+
This includes:
20
23
21
24
+ concurrency: how many threads are running, defined by [CONCURRENCY](#concurrency)
22
25
+ iterations: how many queries will be done in each thread, defined by [ITERATIONS](#iterations)
@@ -48,7 +51,8 @@ Currently 7 Databases are
48
51
49
52
##### Client Library
50
53
51
-
For CrateDB the [crate](https://pypi.org/project/crate/) library is used. To connect to CrateDB the following environment variables must be set:
54
+
For CrateDB the [crate](https://pypi.org/project/crate/) library is used. To connect to CrateDB the following
55
+
environment variables must be set:
52
56
53
57
+[HOST](#host): hostname including port e.g. `localhost:4200`
54
58
+[USERNAME](#username): CrateDB username.
@@ -58,7 +62,8 @@ For CrateDB the [crate](https://pypi.org/project/crate/) library is used. To con
58
62
59
63
##### Client Library
60
64
61
-
For InfluxDB the [influx-client](https://pypi.org/project/influxdb-client/) library is used as the Data Generator only supports InfluxDB V2. To connect to InfluxDB the following environment variables must be set:
65
+
For InfluxDB the [influx-client](https://pypi.org/project/influxdb-client/) library is used as the Data Generator only
66
+
supports InfluxDB V2. To connect to InfluxDB the following environment variables must be set:
62
67
63
68
+[HOST](#host): hostname
64
69
+[TOKEN](#token): InfluxDB Read/Write token
@@ -86,7 +91,8 @@ To connect with TimescaleDB the following environment variables must be set:
86
91
87
92
##### Client Library
88
93
89
-
For MongoDB the [MongoClient](https://mongodb.github.io/node-mongodb-native/api-generated/mongoclient.html) library is used.
94
+
For MongoDB the [MongoClient](https://mongodb.github.io/node-mongodb-native/api-generated/mongoclient.html) library is
95
+
used.
90
96
91
97
To connect with MongoDB the following environment variables must be set:
92
98
@@ -97,7 +103,8 @@ To connect with MongoDB the following environment variables must be set:
97
103
98
104
##### Specifics
99
105
100
-
Because `pymongo` does not support queries as string, Support for MongoDB is turned of in the binary. To still use the Query Timer with Mongo DB have a look at the [Using MongoDB](#using-mongodb) section of this documentation.
106
+
Because `pymongo` does not support queries as string, Support for MongoDB is turned of in the binary. To still use the
107
+
Query Timer with Mongo DB have a look at the [Using MongoDB](#using-mongodb) section of this documentation.
101
108
102
109
#### PostgreSQL
103
110
@@ -128,7 +135,8 @@ To connect with AWS Timestream the following environment variables must be set:
128
135
129
136
##### Specifics
130
137
131
-
+ Tests have shown that queries often fail due to server errors. To accommodate this an automatic retry is implemented that tries to execute the query a second time. If it fails again the query is marked as failure.
138
+
+ Tests have shown that queries often fail due to server errors. To accommodate this an automatic retry is implemented
139
+
that tries to execute the query a second time. If it fails again the query is marked as failure.
132
140
133
141
#### Microsoft SQL Server
134
142
@@ -146,21 +154,24 @@ To connect with Microsoft SQL Server the following environment variables must be
146
154
147
155
### Using MongoDB
148
156
149
-
To use the Query Timer with MongoDB the code of the Query Timer needs to be changed. Therefore checkout this [repository](https://www.github.com/crate/ts-data-generator).
157
+
To use the Query Timer with MongoDB the code of the Query Timer needs to be changed. Therefore checkout this
158
+
[repository](https://www.github.com/crate/tsdg).
150
159
151
160
+ In [this](query_timer/__main__.py) file uncomment the import statement of the `MongoDBWriter`
152
161
+ Also uncomment the instantiation of the `db_writer` in the `get_db_writer` function
153
162
+ Comment the `ValueError` in the line above
154
163
155
164
This should let you start the Query Timer using `DATABASE` set to MongoDB.
156
165
157
-
To add the query you want to measure add a variable containing your query to the script and pass this variable to `db_writer.execute_query()` in the `start_query_run` function, instead of `config.query`.
166
+
To add the query you want to measure add a variable containing your query to the script and pass this variable to
167
+
`db_writer.execute_query()` in the `start_query_run` function, instead of `config.query`.
158
168
159
169
Now the Query Timer is able to measure query execution times for MongoDB.
160
170
161
171
## Query Timer Configuration
162
172
163
-
The Query Timer is mostly configured by setting Environment Variables (or command line arguments start with `-h` for more information). This chapter lists all available Environment Variables and explains their use in the Query Time.
173
+
The Query Timer is mostly configured by setting Environment Variables (or command line arguments start with `-h` for
174
+
more information). This chapter lists all available Environment Variables and explains their use in the Query Time.
164
175
165
176
### Environment variables configuring the behaviour of the Query Time
166
177
@@ -288,7 +299,8 @@ Default: empty string
288
299
used with TimescaleDB, MongoDB, AWS Timestream, Postgresql, MSSQL.
289
300
290
301
**TimescaleDB, Postgresql, MSSQL:**
291
-
The value of `DB_NAME` is used when connecting to TimescaleDB. This database must already exist in your TimescaleDB instance and must have already been initialized with `CREATE EXTENSION IF NOT EXISTS timescaledb CASCADE;`.
302
+
The value of `DB_NAME` is used when connecting to TimescaleDB. This database must already exist in your TimescaleDB
303
+
instance and must have already been initialized with `CREATE EXTENSION IF NOT EXISTS timescaledb CASCADE;`.
292
304
293
305
**MongoDB:**
294
306
The value of `DB_NAME` is used as the database parameter of MongoDB.
@@ -362,11 +374,14 @@ Default: empty string
362
374
363
375
## Alternative Query Timers
364
376
365
-
As the Query Timer is just a by-product of the Data Generator there are other alternatives that offer more features and ways to time queries. The main advantage of the Query Timer is that it supports all Databases that are also supported by the Data Generator and is easy and fast to use.
377
+
As the Query Timer is just a by-product of the Data Generator there are other alternatives that offer more features and
378
+
ways to time queries. The main advantage of the Query Timer is that it supports all Databases that are also supported by
379
+
the Data Generator and is easy and fast to use.
366
380
367
381
### cr8
368
382
369
-
[cr8](https://github.com/mfussenegger/cr8) is a highly sophisticated tool that offers the possibility to measure query execution times for CrateDB and other Databases using the postgres protocol.
383
+
[cr8](https://github.com/mfussenegger/cr8) is a highly sophisticated tool that offers the possibility to measure query
384
+
execution times for CrateDB and other Databases using the postgres protocol.
370
385
371
386
Pros:
372
387
@@ -380,7 +395,8 @@ Cons:
380
395
381
396
### JMeter
382
397
383
-
[Jmeter](https://jmeter.apache.org/) is a well known and great tool that offers the possibility to measure query execution times for Databases using JDBC.
398
+
[Jmeter](https://jmeter.apache.org/) is a well known and great tool that offers the possibility to measure query
0 commit comments