Skip to content

feat: standard database functions everywhere #1750

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 23 commits into from
May 6, 2025
Merged
Changes from 14 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
227 changes: 168 additions & 59 deletions guides/databases.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,9 @@ impl-variants: true

<div class="impl node">

### Migrating to New Database Services? {.node}
### Migrating to the `@cap-js/` Database Services? {.node}

With CDS 8, the new database services for SQLite, PostgreSQL, and SAP HANA are now generally available. It's highly recommended to migrate. You can find instructions in the [migration guide](databases-sqlite#migration). Although the guide is written in the context of the new SQLite Service, the same hints apply to PostgreSQL and SAP HANA.
With CDS 8, the [`@cap-js`](https://github.com/cap-js/cds-dbs) database services for SQLite, PostgreSQL, and SAP HANA are generally available. It's highly recommended to migrate. You can find instructions in the [migration guide](databases-sqlite#migration). Although the guide is written in the context of the SQLite Service, the same hints apply to PostgreSQL and SAP HANA.

### Adding Database Packages {.node}

Expand Down Expand Up @@ -358,63 +358,6 @@ The operator mappings are available for runtime queries only, but not in CDS fil
:::


### Functions Mappings for Runtime Queries {.node}

A specified set of standard functions is supported in a **database-agnostic**, hence portable way, and translated to database-specific variants or polyfills.
Note that these functions are only supported within runtime queries, but not in CDS files.
This set of functions are by large the same as specified in OData:

* `concat(x,y,...)` — concatenates the given strings or numbers
* `trim(x)` — removes leading and trailing whitespaces
* `contains(x,y)` — checks whether `y` is contained in `x`, may be fuzzy
* `startswith(x,y)` — checks whether `y` starts with `x`
* `endswith(x,y)` — checks whether `y` ends with `x`
* `matchespattern(x,y)` — checks whether `x` matches regex `y`
* `substring(x,i,n?)` <sup>1</sup> —
Extracts a substring from `x` starting at index `i` (0-based) with optional length `n`.
* **`i`**: Positive starts at `i`, negative starts `i` before the end.
* **`n`**: Positive extracts `n` items; omitted extracts to the end; negative is invalid.
* `indexof(x,y)` <sup>1</sup> — returns the index of the first occurrence of `y` in `x`
* `length(x)` — returns the length of string `x`
* `tolower(x)` — returns all-lowercased `x`
* `toupper(x)` — returns all-uppercased `x`
* `ceiling(x)` — rounds the input numeric parameter up to the nearest numeric value
* `floor(x)` — rounds the input numeric parameter down to the nearest numeric value
* `round(x)` — rounds the input numeric parameter to the nearest numeric value.
The mid-point between two integers is rounded away from zero, i.e. 0.5 is rounded to 1 and ‑0.5 is rounded to -1.
* `year(x)` `month(x)`, `day(x)`, `hour(x)`, `minute(x)`, `second(x)` —
returns parts of a datetime for a given `cds.DateTime` / `cds.Date` / `cds.Time`
* `time(x)`, `date(x)` - returns a string representing the `time` / `date` for a given `cds.DateTime` / `cds.Date` / `cds.Time`
* `fractionalseconds(x)` - returns a a `Decimal` representing the fractions of a second for a given `cds.Timestamp`
* `maxdatetime()` - returns the latest possible point in time: `'9999-12-31T23:59:59.999Z'`
* `mindatetime()` — returns the earliest possible point in time: `'0001-01-01T00:00:00.000Z'`
* `totalseconds(x)` — returns the duration of the value in total seconds, including fractional seconds. The [OData spec](https://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part2-url-conventions.html#sec_totalseconds) defines the input as EDM.Duration: `P12DT23H59M59.999999999999S`
* `now()` — returns the current datetime
* `min(x)` `max(x)` `sum(x)` `average(x)` `count(x)`, `countdistinct(x)` — aggregate functions
* `search(xs,y)` — checks whether `y` is contained in any of `xs`, may be fuzzy → [see Searching Data](../guides/providing-services#searching-data)
* `session_context(v)` — with standard variable names → [see Session Variables](#session-variables)
> <sup>1</sup> These functions work zero-based. E.g., `substring('abcdef', 1, 3)` returns 'bcd'

> You have to write these functions exactly as given; all-uppercase usages aren't supported.

In addition to the standard functions, which all `@cap-js` database services support, `@cap-js/sqlite` and `@cap-js/postgres` also support these common SAP HANA functions, to further increase the scope for portable testing:

* `years_between` — Computes the number of years between two specified dates.
* `months_between` — Computes the number of months between two specified dates.
* `days_between` — Computes the number of days between two specified dates.
* `seconds_between` — Computes the number of seconds between two specified dates.
* `nano100_between` — Computes the time difference between two dates to the precision of 0.1 microseconds.

The database service implementation translates these to the best-possible native SQL functions, thus enhancing the extent of **portable** queries.
With open source and the new database service architecture, we also have methods in place to enhance this list by custom implementation.

> For the SAP HANA functions, both usages are allowed: all-lowercase as given above, as well as all-uppercase.

::: warning Runtime Only
The function mappings are available for runtime queries only, but not in CDS files.
:::


### Session Variables {.node}

The API shown below, which includes the function `session_context()` and specific pseudo variable names, is supported by **all** new database services, that is, *SQLite*, *PostgreSQL* and *SAP HANA*.
Expand Down Expand Up @@ -940,7 +883,173 @@ Instead, they protect the integrity of your data in the database layer against p
→ Use [`@assert.target`](providing-services#assert-target) for corresponding input validations.
:::

## Standard Database Functions
{ #functions-mappings-for-runtime-queries }

A specified set of standard functions - inspired by [OData](https://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part2-url-conventions.html#sec_StringandCollectionFunctions) and [SAP HANA](https://help.sap.com/docs/hana-cloud-database/sap-hana-cloud-sap-hana-database-sql-reference-guide/alphabetical-list-of-functions?locale=en-US) - is supported in a **database-agnostic**, hence portable way, and translated to the best-possible native SQL functions or polyfills.

Those functions are automatically mapped during runtime (Node.js) already today.
To switch on the same mappings for your CDL files, you have to set:

<Config>cds.cdsc.standardDatabaseFunctions = true</Config>

::: tip
this will be the default starting with `@sap/cds-compiler >= 9` and hence is only needed for earlier versions.
:::

<!--
TODO: remove the above with cds9
-->

### OData standard functions

The `@sap/cds-compiler` and all CAP Node.js database services come with out of the box support for common OData functions.

::: warning Case Sensitivity
The OData function mappings are case-sensitive and must be written as in the list below.
:::

e.g.

```cds
entity V as select from Books {
startswith(title, 'Raven') as lowerCase, // mapped to native SQL equivalent
startsWith(title, 'Raven') as camelCase, // passed as-is
}
```

`❯ cds compile -2 sql --dialect hana`

```sql
CREATE VIEW V AS SELECT
(CASE WHEN locate(title, 'Raven') = 1 THEN TRUE ELSE FALSE END) AS lowerCase,
-- the below will most likely fail on SAP HANA
startsWith(title, 'Raven') AS camelCase
FROM Books;
```

💡 If you have your own User-Defined Functions (UDFs) with the same name, you can still use them,
by deviating from the casing given below.

#### String Functions

- `concat(x, y, ...)`
Concatenates the given strings or numbers.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes we support the concat operator


- `trim(x)`
Removes leading and trailing whitespaces.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • not available in CAP Java
  • returns String

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to OData, this should be whitespace characters, according to Unicode rules.

I think these are:

  • u2002
  • \u3000
  • \r
  • \u0085
  • \u200A
  • \u2005
  • \u2000
  • \u3000
  • \u2029
  • \u000B
  • \u3000
  • \u2008
  • \u2003
  • \u205F
  • \u3000
  • \u1680
  • \u0009
  • \u0020
  • \u2006
  • \u2001
  • \u202F
  • \u00A0
  • \u000C
  • \u2009
  • \u3000
  • \u2004
  • \u3000
  • \u3000
  • \u2028
  • \n
  • \u2007
  • \u3000

@patricebender - do you trim these characters or just whatever the DB considers to be a whitespace? I think the latter would be fine for most practical purposes. (Although we recently had an issue with a Chinese WS character (!)). At least we need to document which interpretation of whitespace character we apply.


- `contains(x, y)`
Checks whether `y` is contained in `x` (case-sensitive).

- `startswith(x, y)`
Checks whether `y` starts with `x` (case-sensitive).

- `endswith(x, y)`
Checks whether `y` ends with `x` (case-sensitive).

- `matchespattern(x, y)`
Checks whether `x` matches the regular expression `y`.

- `indexof(x, y)` <sup>1</sup>
Returns the index of the first occurrence of `y` in `x` (case-sensitive).

- `substring(x, i, n?)` <sup>1</sup>
Extracts a substring from `x` starting at index `i` (0-based) with an optional length `n`.
- `i`:
- Positive: starts at index `i`
- Negative: starts `i` positions before the end
- `n`:
- Positive: extracts `n` characters
- Omitted: extracts until the end of the string
- Negative: invalid

- `length(x)`
Returns the length of the string `x`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • not available in CAP Java
  • returns Int64 ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

currently Int32


- `tolower(x)`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • CAP Java: toLower
  • returns String

Converts all characters in `x` to lowercase.

- `toupper(x)`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • CAP Java: toUpper
  • returns String

Converts all characters in `x` to uppercase.

<sup>1</sup> These functions work zero-based. E.g., `substring('abcdef', 1, 3)` returns 'bcd'

#### Numeric Functions

- `ceiling(x)`
Rounds the numeric parameter up to the nearest integer.

- `floor(x)`
Rounds the numeric parameter down to the nearest integer.

- `round(x)`
Rounds the numeric parameter to the nearest integer.
Copy link
Contributor

@agoerler agoerler Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • CAP Java: WIP: https://github.wdf.sap.corp/cds-java/cds4j/pull/3590
  • What's the return type? Type of x? Int32? Int64?
  • SQL also know round with two parameters where the second parameter which indicates the number of digits to round to. There are issues on SQLite with this second parameter. I assume there is no plan to support the method with two parameters.
  • On Postgres we observed that Postgres and Double elements rounds to the nearest even integer

https://www.postgresql.org/docs/16/functions-math.html

Rounds to nearest integer. For numeric, ties are broken by rounding away from zero. For double precision, the tie-breaking behavior is platform dependent, but “round to nearest even” is the most common rule.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Java:

  • return type: Type of x.
  • available now

The midpoint between two integers is rounded away from zero (e.g., `0.5` → `1` and `-0.5` → `-1`).

::: warning `round` function with more than one argument
Note that most databases support `round` functions with multiple arguments,
the second parameter being the precision. SAP HANA even has a third argument which for the rounding mode.
If you provide more than one argument, the `round` function may behave differently depending on the database.
:::

#### Date and Time Functions

- `year(x)`, `month(x)`, `day(x)`, `hour(x)`, `minute(x)`, `second(x)`
Extracts and returns specific date / time parts as integer value from a given `cds.DateTime`, `cds.Date`, or `cds.Time`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • not available in CAP Java
  • in CAP Java, we treat DateTime as a Timestamp with seconds precision. Hence we can't extract date/time components unless we would make the assumption that the extraction is in UTC
  • What's the return type?
  • Zero-based or one-based?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the return type?

an Int32

Zero-based or one-based?

could you elaborate? It always returns the requested part of the date as it is written in the given cds.DateTime, cds.Date, or cds.Time


- `time(x)`, `date(x)`
Extracts and returns a time or date from a given `cds.DateTime`, `cds.Date`, or `cds.Time`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • not available in CAP Java
  • in CAP Java, we treat DateTime as a Timestamp with seconds precision. Hence we can't extract date/time components unless we would make the assumption that the extraction is in UTC
  • I assume the return types are Time and Date, resp..


- `fractionalseconds(x)`
Returns a `Decimal` representing the fractional seconds for a given `cds.Timestamp`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

n/a in CAP Java


- `maxdatetime()`
Returns the latest possible point in time: `'9999-12-31T23:59:59.999Z'`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • n/a in CAP Java
  • DateTime has no fractional seconds!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in OData, this functions returns an DateTimeOffset which is defined to contain the fractional part: https://docs.oasis-open.org/odata/odata-csdl-xml/v4.01/csprd02/odata-csdl-xml-v4.01-csprd02.html#sec_DateTimeOffset


- `mindatetime()`
Returns the earliest possible point in time: `'0001-01-01T00:00:00.000Z'`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • n/a in CAP Java
  • I would assume this returns no fractional seconds as DateTime has no fractional seconds. See.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in OData, this functions returns an DateTimeOffset which is defined to contain the fractional part: https://docs.oasis-open.org/odata/odata-csdl-xml/v4.01/csprd02/odata-csdl-xml-v4.01-csprd02.html#sec_DateTimeOffset


#### Aggregate Functions

- `min(x)`, `max(x)`, `sum(x)`, `average(x)`, `count(x)`, `countdistinct(x)`
Standard aggregate functions used to calculate minimum, maximum, sum, average, count, and distinct count of values.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Standard aggregate functions used to calculate minimum, maximum, sum, average, count, and distinct count of values.
Standard aggregate functions used to calculate minimum, maximum, sum, average, count, and count of distinct values.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

returns

  • min(x), max(x): type of x
  • average(x): Double?
  • sum(x) : type of x? Or next larger numeric type?
  • count(x), coundistinct(x): Int64

It would be useful to have a type cds.Number!



### SAP HANA Functions

In addition to the OData standard functions, the cds-compiler and all CAP Node.js database services come with
out of the box support for some common SAP HANA functions, to further increase the scope for portable testing:

::: warning Upper- and Lowercase are supported
For the SAP HANA functions, both usages are allowed: all-lowercase as given above, as well as all-uppercase.
:::

- `years_between`
Computes the number of years between two specified dates. ([link](https://help.sap.com/docs/hana-cloud-database/sap-hana-cloud-sap-hana-database-sql-reference-guide/years-between-function-datetime?locale=en-US))
- `months_between`
Computes the number of months between two specified dates. ([link](https://help.sap.com/docs/hana-cloud-database/sap-hana-cloud-sap-hana-database-sql-reference-guide/months-between-function-datetime?locale=en-US))
- `days_between`
Computes the number of days between two specified dates. ([link](https://help.sap.com/docs/hana-cloud-database/sap-hana-cloud-sap-hana-database-sql-reference-guide/days-between-function-datetime?locale=en-US))
- `seconds_between`
Computes the number of seconds between two specified dates. ([link](https://help.sap.com/docs/hana-cloud-database/sap-hana-cloud-sap-hana-database-sql-reference-guide/seconds-between-function-datetime?locale=en-US))
- `nano100_between`
Computes the time difference between two dates to the precision of 0.1 microseconds. ([link](https://help.sap.com/docs/hana-cloud-database/sap-hana-cloud-sap-hana-database-sql-reference-guide/nano100-between-function-datetime?locale=en-US))

### Special Runtime Functions

In addition to the OData and SAP HANA standard functions, the **CAP runtimes** provides special functions that are only available for runtime queries:

- `search(x, y)`
Checks whether `y` is contained in any element of `x` (fuzzy matching may apply).
See [Searching Data](../guides/providing-services#searching-data) for more details.

- `session_context(<var>)`
Utilizes standard variable names to maintain session context.
Refer to [Session Variables](#session-variables) for additional information.

- `now()`
Returns the current timestamp.

## Using Native Features { #native-db-functions}

Expand Down