Finish the GCP pipeline #166
Conversation
PR Review ChecklistDo not edit the content of this comment. The PR reviewer should simply update this comment by ticking each review item below, as they get completed. Trivial Change
Code
Architecture
|
dmitrii-ubskii
force-pushed
the
gcp
branch
5 times, most recently
from
53efc84 to
59c3b5a
Compare
| (curl -o "$DISTRIBUTION_TARGZ" "$DISTRIBUTION_URL_SNAPSHOT" && tar -xf "$DISTRIBUTION_TARGZ") || | ||
| (curl -o "$DISTRIBUTION_TARGZ" "$DISTRIBUTION_URL_RELEASE" && tar -xf "$DISTRIBUTION_TARGZ") |
There was a problem hiding this comment.
Try downloading the version given from the snapshot url, retry from the release url on failure.
tool/typedb3/install-driver.sh
Outdated
| pip install typedb-driver=="$DRIVER_VERSION" --index-url https://repo.typedb.com/public/public-snapshot/python/simple/ No newline at end of file | ||
| pip install typedb-driver=="$DRIVER_VERSION" --extra-index-url https://repo.typedb.com/public/public-snapshot/python/simple/ |
There was a problem hiding this comment.
Releases are on PyPI, so we don't want to actually override the index.
| sudo mongod --replSet rs0 --bind_ip localhost --config ./tool/mongodb/mongod.conf & | ||
| sudo mongod --replSet rs0 --bind_ip localhost --config ./tool/mongodb/mongod.conf >&/dev/null & |
There was a problem hiding this comment.
mongod keeps the SSH connection open by keeping the stdour/stderr open, so the script cannot progress unless we do this.
|
|
||
| KEEP_SERVER= | ||
| while getopts ":d:w:c:s:t:k" opt; do |
There was a problem hiding this comment.
getopts can only handle single letter flags. There's also getopt (note no s) which is a lot more powerful.
However, since getopts is a shell builtin and getopt is a side binary which behaves differently on different platforms, I'm sticking with this simlpe one.
There was a problem hiding this comment.
I've merged the individual stages (setup, load, execute) into this one script. Each one effectively called a single command, and it was way too much indirection to keep track of.
| BENCH_ID=b-$USER-db$DB_SHORT-$SERVER_VERSION_SHORT-$DRIVER_VERSION_SHORT-$MACHINE_TYPE_SHORT-$DISK_SIZE-sf$SCALE_FACTOR-w$WAREHOUSES-c$CLIENTS-dur$DURATION-r$RUN_NUM | ||
| BENCH_ID=$USER-$DB_SHORT-$SERVER_VERSION_SHORT-$DRIVER_VERSION_SHORT-$MACHINE_TYPE_SHORT-$DISK_SIZE-sf$SCALE_FACTOR-w$WAREHOUSES-c$CLIENTS-dur$DURATION-$ID |
There was a problem hiding this comment.
GCP has a limit on the machine name, so I had to make a few cuts.
| # four digit random number with zero padding | ||
| ID=0000$RANDOM | ||
| ID=${ID:(-4)} |
There was a problem hiding this comment.
This is so that we can run multiples of the same configuration at once.
What is the goal of this PR?
We add configuration for Neo4J and Postgres and finish the end-to-end pipeline script that is to run a benchmark in GCP.
What are the changes implemented in this PR?