Skip to content

Commit 95bedad

Browse files
ohsayanohsayan
committed
Add all BlueQL docs
1 parent 02549e9 commit 95bedad

21 files changed

+5002
-3156
lines changed

‎docs/1.index.md

+20-11
Original file line numberDiff line numberDiff line change
@@ -5,17 +5,26 @@ sidebar_label: Home
55
slug: /
66
---
77

8-
Welcome to Skytable's docs! You will find information about how you can get started with Skytable, installation options, configuration and clients.
9-
10-
## Users
11-
12-
We have an easy-to-follow guide for [Getting Started](getting-started). Once you've got everything up and running, you can take a look at the available actions [here](all-actions) and [configuration](config).
13-
Once you've learned the basics, start using a [client driver](clients)! When you're ready to deploy, check
14-
out the [deployment notes](deployment-notes).
15-
16-
## Developers
17-
18-
You can find information on how to build your own clients [here](protocol/skyhash). The primary idea is to implement the Skyhash Protocol.
8+
Welcome to Skytable's docs! Skytable is a free and open-source modern NoSQL database that builds on the foundations of performance, scalability, powerful querying and a robust type system. Skytable can be deployed as just a single binary file with no special system dependencies and only relies on the operating system's `libc` implementation.
9+
10+
## Getting started
11+
12+
Skytable has it's own query language, BlueQL<sup>TM</sup> which provides everything across DDL, DCL and DML queries and exists to be a very powerful and secure alternative to SQL. If you're coming from SQL, you should feel just at home — BlueQL has a few small but important operating differences from SQL but has very similar syntax.
13+
14+
We recommend you to follow the guide in this sequence (but feel free to skip any sections):
15+
- [**Installation**](installation) and [**using the CLI**](using-the-repl): Get everything installed on your local system
16+
- [**System overview**](system-overview): A brief introduction to Skytable, including an overview of the data model, query systems and storage
17+
- **BlueQL**:
18+
- [**Introduction**](blueql/overview): Serves as a basic introduction to the query language and an (currently incomplete) informal specification with information on keywords, syntax and stuctures.
19+
- [**DDL**](blueql/ddl): Data definition with BlueQL
20+
- [**DML**](blueql/dml): Data manipulation with BlueQL
21+
- [**DCL**](blueql/dcl): Data control with BlueQL
22+
- [**Configuration**](system/configuration): Information to help you configure Skytable with custom settings such as custom ports, hosts, TLS, and etc.
23+
- [**Administration**](system/administration): Information on access control, user and other administration features
24+
- [**A guide to the new Skytable**](migration): For old our returning Skytable users who are coming from older versions
25+
- [**Benchmarking**](benchmarking): A guide for load testing Skytable
26+
- [**Deployment**](deployment): An useful guide for deploying
27+
- [**Limits**](limits): An useful guide on system limits
1928

2029
## Contributing
2130

‎docs/10.administration.md

+66
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
---
2+
id: system+administration
3+
title: Administration
4+
slug: /system/administration
5+
---
6+
7+
Skytable's access control is very simple:
8+
- there is only one `root` account
9+
- there are zero or more standard users
10+
11+
## Types of users
12+
13+
### Root users
14+
15+
As noted earlier there can be only one `root` account and this is primarily for security concerns. We however do plan to support
16+
creating multiple users (the implementation isn't hard but security is key).
17+
18+
#### Exclusive rights
19+
20+
Root users have exclusive rights to use plus everything that standard users have access to:
21+
- `CREATE`
22+
- `ALTER`
23+
- `DROP`
24+
- `SYSCTL CREATE`
25+
- `SYSCTL DROP`
26+
27+
#### Resetting the root password
28+
29+
We strongly recommend that you keep the root password in someplace safe, but if you happen to lose it — just like many other
30+
databases, you will first need to stop the server to reset it. Once you have stopped the server, you will need to modify the root
31+
password that you set in your [configuration source](configuration) (such as CLI/ENV/configuration file). You will then
32+
need to restart the server. The server will issue a warning in the logs that the root password has changed but since that is
33+
what you intended, you can safely ignore it.
34+
35+
### Standard users
36+
37+
You can have any number of standard users. Standard users can essentially manipulate data but can't modify the objects that store them.
38+
39+
#### Rights
40+
41+
Standard users can access the following query types:
42+
- `SYSCTL REPORT STATUS`
43+
- `INSERT`
44+
- `SELECT`
45+
- `UPDATE`
46+
- `DELETE`
47+
- `INSPECT`
48+
49+
### Global management
50+
51+
The single DDL query that lets you do a "sneak peek" into the status of the entire system is the `INSPECT GLOBAL` query. It
52+
returns a JSON string like this:
53+
```json
54+
{
55+
"spaces:"["space1", "space2"],
56+
"users":["root", "staging_server"],
57+
"settings:{},
58+
}
59+
```
60+
61+
- `spaces`: lists all the spaces in the system
62+
- `users`: lists all users
63+
:::info Access control note
64+
This is only returned if you are the `root` user. Standard users cannot see the other users in the system
65+
:::
66+
- `settings`: returns system settings (currently an empty dictionary is returned)

‎docs/11.migrating.md

+26
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
---
2+
id: migration
3+
title: Migrating from an older release
4+
---
5+
6+
Firstly, thank you patron! You're a part of an exclusive *club* which has seen Skytable take shape from *just another key-value*
7+
store into a powerful database and we couldn't be more honored to be a choice in your stack. This document is meant for people
8+
coming from version prior to Skytable Octave (0.8).
9+
10+
Let's get you up to speed with the new semantics:
11+
12+
- **Keyspaces are now spaces**
13+
- **Tables are now models** and can store multiple fields
14+
- **Nested lists are now supported**
15+
- **All the actions are gone!**
16+
- **There is no `default:default` keyspace and model**: You will need to create your own `space` and `model`
17+
18+
## From actions to BlueQL
19+
20+
- `SET x y` becomes: `INSERT INTO mymodel(x, ?)`
21+
- `GET x` becomes `SELECT * FROM mymodel WHERE <primary_key> = x`
22+
- `UPDATE x y` becomes `UPDATE mymodel SET field_name = ? WHERE <primary_key> = x`
23+
- `DEL X` becomes `DELETE FROM mymodel WHERE <primary_key> = x`
24+
25+
Essentially, you no longer have the key/value semantics but a more SQL-like interface for querying with BlueQL. You can continue
26+
to use lists (create, update) but individual element access is currently limited. See [this page for more information](limits).

‎docs/12.benchmarking.md

+35
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
---
2+
id: benchmarking
3+
title: Benchmarking
4+
---
5+
6+
Due to Skytable having in-house implementations of almost everything, starting from the protocol, the storage engine and query
7+
language — we have our own custom load testing tool called `sky-bench` that is distributed with the bundle.
8+
9+
## Setting up for benchmarking
10+
11+
**Quick notes**:
12+
- The benchmark tool will create:
13+
- a space called `bench`
14+
- a model called `bench`
15+
- **Be sure that these objects don't already exist!** But don't worry, if they do — the benchmark tool will error. You won't lose any data
16+
- **Once the benchmark is complete, the `bench` space is removed!**
17+
- **Do not use the `bench` space during the benchmark**: You might lose data
18+
19+
**The benchmark tool will:**
20+
- Run a total of 4,000,000 queries (don't worry, they run pretty fast!):
21+
- Run 1,000,000 `INSERT`s
22+
- Run 1,000,000 `SELECT`s
23+
- Run 1,000,000 `UPDATE`s
24+
- Run 1,000,000 `DELETE`s
25+
- The model used has the declaration `(un: string, pw: uint8)`
26+
- The `SELECT` will select and return all fields
27+
- The `UPDATE` will increment the value of `pw` like this `pw += 1`
28+
- The `DELETE` removes the entire collection
29+
- The default primary key size is 7 bytes. All generated keys will be padded with zeros from to 0 to 999,999 like this: `0000000` or `0999999`
30+
31+
## Off to the races
32+
33+
1. Start up the database server
34+
2. Run `./sky-bench --password <your root password>`. We need your `root` password because only the root account can create, alter and drop models and the benchmark tool needs to run these queries
35+
3. Wait for it to happen. You may not believe your eyes, so we recommend that you keep your eyes hydrated 🔥🚀✨💣

‎docs/13.deployment.md

+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
---
2+
id: deployment
3+
title: Deployment
4+
---
5+
6+
Here are some recommendations for deployment:
7+
1. **Make sure you have enough memory and storage!** The server will start returning errors when your server runs out of resources, as you'd expect.
8+
2. **When deploying on docker**:
9+
- Try to map the volume to a local path. We've had unwarranted data losses when we accidentally ended up running a `docker system prune`
10+
- Make sure you have your networking configured right. For example, if you don't map your ports correctly, you'll not be able to access the database outside docker (without running `docker inspect` to find the IP of the container)
11+
- Keep a separate folder for all your docker containers:
12+
- Create a directory in your home directory like: `$HOME/docker-data`
13+
- Then for every Docker image (whether it is Skytable or any other container) create a subfolder in that directory and map
14+
the subfolder as a volume for persistence
15+
3. **Check your firewall**: You want to be very sure about this! If you're starting your database on a different server (as you should; ideally you keep your database server separate from the host running your application server) then make sure that the ports are open and allowed
16+
4. **Set up virtual memory and monitoring**: To avoid exhausting resources, set up monitoring on your node and enable virtual memory to temporarily avoid OOMs

‎docs/14.limits.md

+40
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
---
2+
id: limits
3+
title: Limits
4+
---
5+
6+
We've made every effort to provide a robust querying interface, but there are some **temporary limitations** that we think you
7+
should know about. We aim to remove the limitations over the next few releases which we expect should happen fairly quickly.
8+
9+
Skytable's limitations primarily come from a bunch of concerns:
10+
- **Performance and scalability**: Most of our design decisions are influenced by concerns about performance. For example, it's very hard to efficiently scale multi-column indexes.
11+
- **Reliability**: how reliable is the execution of the task? If it's like walking on eggshells, then we're not going to implement it (for example, unreliable distributed locking)
12+
- **Security**: If it can't be run securely, then it's off our list
13+
14+
## Temporary limitations
15+
16+
- **Collections in fields are *partially frozen at the moment***:
17+
- You can add data to collections as you desire during an `INSERT`
18+
- However, a `SELECT` will return the *entire* collection and not an individual element
19+
- Similarly, an `UPDATE` can append to a non-nested list, but cannot append to a nested list
20+
- For example if your model has a field defined as `mylist: list {type: string}` you can add more elements to it in an `UPDATE` query like this: `UPDATE mymodel SET mylist += 'new element'`
21+
- However, if you have a field defined as `mylist: { list { type: list{type: string}} }` then you will not be able to append another list to it.
22+
- On the same note, you cannot `DELETE` an individual element right now
23+
- **Models cannot be `volatile` yet**. If you've used Skytable before, you'd know that you could previously create `volatile`
24+
models which were used as *caching tables* as in they didn't persist data across restarts. The `volatile` feature has been
25+
temporarily removed because we're working on integrating it with the new storage engine.
26+
- **`SELECT` doesn't allow multi-row fetch**. This is a severe limitation that will be removed very soon. This means that a `SELECT` will match one, and only one query
27+
28+
:::tip Work in progress
29+
Individual element access in nested collections and multi-row `SELECT`s are on their way to stability. If you need these features
30+
(which we consider to be a very plausible use-case) hang in with us and you should have it by 0.8.1.
31+
32+
**We're on it!**
33+
:::
34+
35+
## Hard limitations
36+
37+
Following Skytable's design philosophy that closely encompasses NoSQL systems, the following hard limitations are set:
38+
- **Only one index**: Right now, the only index that you can use is the primary index (primary_key -> row). This is due to concerns about performance and scale
39+
- **No mass updates**: We consider mass updates, such as setting `counter += 1` to every row in a model with multi-million rows
40+
to be very slow and bad for performance. Hence, we do not allow mass updates at this time.

‎docs/2.getting-started.md

-74
This file was deleted.

‎docs/2.installation.md

+67
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
---
2+
id: installation
3+
title: Installation
4+
---
5+
6+
Getting started with Skytable involves choosing a mode of installation, downloading any required files and then starting up the database. You can choose to either use:
7+
8+
- [**Native binaries (recommended)**](#native-binaries): This is what is generally recommended for the best performance. You will need to download a bundle and then start the server binary; no expert knowledge required
9+
- [**A Docker image**](#docker-image): We generally recommend using a Docker image for experimenting with Skytable on your local system during development and you want to keep your local system *clean*. If you want to use a Docker image for deployment, you're always free to do so!
10+
> **Note:** You might experience slightly degraded performance from the storage engine due to Docker engine's abstractions.
11+
12+
13+
## Native binaries
14+
15+
:::info
16+
Your operating system might sometimes not let you run binaries directly. On Unix based systems, you'll need to run: `chmod +x skyd skysh sky-bench`.
17+
18+
And on Windows systems you might need to right-click on the binaries and click on "unblock"
19+
:::
20+
21+
To use native binaries you need to download a bundle which is simply a ZIP file with all the necessary binaries that you'll ever need to develop on and deploy Skytable.
22+
23+
1. **First download the latest bundle** for your platform. You can find [download links on the releases page](https://github.com/skytable/skytable/releases).
24+
2. **Unzip the ZIP file**. You'll find the following binaries in the extracted archive:
25+
- `skyd`: This is the database server binary which when started runs as a daemon, serving requests
26+
- `skysh`: This is the Skytable shell and it provides a very helpful interactive REPL database client
27+
- `sky-bench`: This is the benchmarking tool that you can use to load test Skytable
28+
3. **Start up the server**. You need to choose a `root` password for the `root` account which will have complete control over the database.
29+
```bash
30+
./skyd --auth-root-password=<your root password> --auth-plugin=pwd
31+
```
32+
**Replace with your own secure password!**
33+
34+
Explanation:
35+
- `--auth-root-password`: sets the root password
36+
- `--auth-plugin`: sets the authentication plugin to `pwd` which is a simple password based authentication system
37+
38+
The server starts up at `localhost:2003` and is ready to run queries.
39+
40+
## Docker image
41+
42+
:::info You must have docker set up!
43+
- Use [this great guide from Docker](https://docs.docker.com/engine/install/) to install and get started
44+
- To be able to run `docker run` and related commands, you may need administrative privileges
45+
:::
46+
47+
1. **Download the bundle**: To be able to run queries you need to download the bundle as described above
48+
2. **Create the data directory**: To ensure that our database is persistent and all our data doesn't vanish as soon as the container is terminated, we'll map the data directory to an actual directory on our local system.
49+
> **Note:** Create a folder called `skytable` in a convenient location. We recommend having a directory in `$HOME/docker-containers` where you can store the Skytable container's data and any other containers that you might use. It's a great way to keep things organized.
50+
3. **Start the container**:
51+
```shell
52+
docker run -d --name skydb \
53+
-v /home/docker-containers/skytable:/var/skytable \
54+
-p 2003:2003 \
55+
-e SKYDB_AUTH_ROOT_PASSWORD=<your root password> \
56+
-e SKYDB_AUTH_PLUGIN=pwd skytable/skytable:latest
57+
```
58+
**Replace with your own secure password!**
59+
60+
Explanation:
61+
- This starts a container with name `skydb`
62+
- It maps the folder (as discussed earlier) `/home/docker-containers/skytable` from your local file system to `/var/skytable` (in the container's file system)
63+
- Maps port `2003` on the host to the containers port `2003` so that you can use the command-line client `skysh` without having to inspect the container's IP address
64+
- Set's some basic configuration:
65+
- `SKYDB_AUTH_ROOT_PASSWORD`: sets the root password. you can't use Skytable without a `root` account
66+
- `SKYDB_AUTH_PLUGIN`: this sets the authentication plugin to `pwd` which is the simplest authentication plugin
67+

0 commit comments

Comments
 (0)