Skip to content

updating standard analyzer docs #9747

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

updating standard analyzer docs #9747

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
79982b0
updating standard analyzer docs
AntonEliatra Apr 28, 2025
72c65e1
Update _analyzers/supported-analyzers/standard.md
kolchfa-aws Apr 28, 2025
cfe0817
addressing the PR comments
AntonEliatra Jun 26, 2025
a62523c
replacing add Data Type with Data type
AntonEliatra Jun 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
93 changes: 59 additions & 34 deletions _analyzers/supported-analyzers/standard.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,17 +7,20 @@

# Standard analyzer

The `standard` analyzer is the default analyzer used when no other analyzer is specified. It is designed to provide a basic and efficient approach to generic text processing.
The `standard` analyzer is the built-in default analyzer used for general-purpose full-text search in OpenSearch. It is designed to provide consistent, language-agnostic text processing by efficiently breaking down text into searchable terms.

This analyzer consists of the following tokenizers and token filters:
The `standard` analyzer performs the following operations:

- `standard` tokenizer: Removes most punctuation and splits text on spaces and other common delimiters.
- `lowercase` token filter: Converts all tokens to lowercase, ensuring case-insensitive matching.
- `stop` token filter: Removes common stopwords, such as "the", "is", and "and", from the tokenized output.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did we miss adding detail about stop words or is it part of tokenization or lowercasing itself.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sandeshkr419 but the stop words are not removed in standard analyzer, is that what you mean?

- **Tokenization**: It uses the [`standard`]({{site.url}}{{site.baseurl}}/analyzers/tokenizers/standard/) tokenizer, which splits text into words based on Unicode text segmentation rules, handling spaces, punctuation, and common delimiters.
- **Lowercasing**: It applies the [`lowercase`]({{site.url}}{{site.baseurl}}/analyzers/token-filters/lowercase/) token filter to convert all tokens to lowercase, ensuring consistent matching regardless of input case.

Check failure on line 15 in _analyzers/supported-analyzers/standard.md

View workflow job for this annotation

GitHub Actions / style-job 

[vale] reported by reviewdog 🐶
[OpenSearch.Spelling] Error: Lowercasing. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.

Raw Output:
{"message": "[OpenSearch.Spelling] Error: Lowercasing. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_analyzers/supported-analyzers/standard.md", "range": {"start": {"line": 15, "column": 5}}}, "severity": "ERROR"}

## Example
This combination makes the `standard` analyzer ideal for indexing a wide range of natural language content without needing language-specific customizations.

Use the following command to create an index named `my_standard_index` with a `standard` analyzer:
---

## Example: Creating an index with the standard analyzer

You can assign the `standard` analyzer to a text field when creating an index:

```json
PUT /my_standard_index
Expand All @@ -26,41 +29,44 @@
"properties": {
"my_field": {
"type": "text",
"analyzer": "standard"
"analyzer": "standard"
}
}
}
}
```
{% include copy-curl.html %}

---

## Parameters

You can configure a `standard` analyzer with the following parameters.
The `standard` analyzer supports the following optional parameters:

Parameter | Required/Optional | Data type | Description
:--- | :--- | :--- | :---
`max_token_length` | Optional | Integer | Sets the maximum length of the produced token. If this length is exceeded, the token is split into multiple tokens at the length configured in `max_token_length`. Default is `255`.
`stopwords` | Optional | String or list of strings | A string specifying a predefined list of stopwords (such as `_english_`) or an array specifying a custom list of stopwords. Default is `_none_`.
`stopwords_path` | Optional | String | The path (absolute or relative to the config directory) to the file containing a list of stop words.
| Parameter | Data type | Default | Description |
|:----------|:-----|:--------|:------------|
| `max_token_length` | Integer | `255` | Sets the maximum length of a token before it is split. |
| `stopwords` | String or list of strings | None | A list of stopwords or a predefined stopword set like `_english_` to remove during analysis. |
| `stopwords_path` | String | None | Path to a file containing stopwords to be used during analysis. |
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you verify once if both stopwords and stopwords_path be used in a single request? If not, let's put a small note or clarification?


Use only one of the parameters `stopwords` or `stopwords_path`. If both are used, no error is returned but only `stopwords` parameter is applied.
{: .note}

## Configuring a custom analyzer
## Example: Analyzer with parameters

Use the following command to configure an index with a custom analyzer that is equivalent to the `standard` analyzer:
The following example creates index `products` and configures `max_token_length` and `stopwords`:

```json
PUT /my_custom_index
PUT /animals
{
"settings": {
"analysis": {
"analyzer": {
"my_custom_analyzer": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"lowercase",
"stop"
"my_manual_stopwords_analyzer": {
"type": "standard",
"max_token_length": 10,
"stopwords": [
"the", "is", "and", "but", "an", "a", "it"
]
}
}
Expand All @@ -70,28 +76,47 @@
```
{% include copy-curl.html %}

## Generated tokens

Use the following request to examine the tokens generated using the analyzer:
Use the following `_analyze` API to see how the `my_manual_stopwords_analyzer` processes text:

```json
POST /my_custom_index/_analyze
POST /animals/_analyze
{
"analyzer": "my_custom_analyzer",
"text": "The slow turtle swims away"
"analyzer": "my_manual_stopwords_analyzer",
"text": "The Turtle is Large but it is Slow"
}
```
{% include copy-curl.html %}

The response contains the generated tokens:
The returned tokens are:

- separated based on spacing
- lowercased
- stopwords removed

```json
{
"tokens": [
{"token": "slow","start_offset": 4,"end_offset": 8,"type": "<ALPHANUM>","position": 1},
{"token": "turtle","start_offset": 9,"end_offset": 15,"type": "<ALPHANUM>","position": 2},
{"token": "swims","start_offset": 16,"end_offset": 21,"type": "<ALPHANUM>","position": 3},
{"token": "away","start_offset": 22,"end_offset": 26,"type": "<ALPHANUM>","position": 4}
{
"token": "turtle",
"start_offset": 4,
"end_offset": 10,
"type": "<ALPHANUM>",
"position": 1
},
{
"token": "large",
"start_offset": 14,
"end_offset": 19,
"type": "<ALPHANUM>",
"position": 3
},
{
"token": "slow",
"start_offset": 30,
"end_offset": 34,
"type": "<ALPHANUM>",
"position": 7
}
]
}
```
4 changes: 2 additions & 2 deletions _api-reference/index-apis/alias.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ POST _aliases

All parameters are optional.

Parameter | Data Type | Description
Parameter | Data type | Description
:--- | :--- | :---
cluster_manager_timeout | Time | The amount of time to wait for a response from the cluster manager node. Default is `30s`.
timeout | Time | The amount of time to wait for a response from the cluster. Default is `30s`.
Expand All @@ -34,7 +34,7 @@ timeout | Time | The amount of time to wait for a response from the cluster. Def

In your request body, you need to specify what action to take, the alias name, and the index you want to associate with the alias. Other fields are optional.

Field | Data Type | Description | Required
Field | Data type | Description | Required
:--- | :--- | :--- | :---
actions | Array | Set of actions you want to perform on the index. Valid options are: `add`, `remove`, and `remove_index`. You must have at least one action in the array. | Yes
add | N/A | Adds an alias to the specified index. | No
Expand Down
6 changes: 3 additions & 3 deletions _im-plugin/index-transforms/transforms-apis.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,15 +28,15 @@ PUT _plugins/_transform/<transform_id>

### Path parameters

Parameter | Data Type | Description
Parameter | Data type | Description
:--- | :--- | :---
transform_id | String | Transform ID |

### Request body fields

You can specify the following options in the HTTP request body:

Option | Data Type | Description | Required
Option | Data type | Description | Required
:--- | :--- | :--- | :---
enabled | Boolean | If true, the transform job is enabled at creation. | No
continuous | Boolean | Specifies whether the transform job should be continuous. Continuous jobs execute every time they are scheduled according to the `schedule` field and run based off of newly transformed buckets as well as any new data added to source indexes. Non-continuous jobs execute only once. Default is `false`. | No
Expand Down Expand Up @@ -184,7 +184,7 @@ Parameter | Description | Required

You can update the following fields.

Option | Data Type | Description
Option | Data type | Description
:--- | :--- | :---
schedule | Object | The schedule for the transform job. Contains the fields `interval.start_time`, `interval.period`, and `interval.unit`.
start_time | Integer | The Unix epoch start time of the transform job.
Expand Down
2 changes: 1 addition & 1 deletion _search-plugins/sql/datatypes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
layout: default
title: Data Types
title: Data types
parent: SQL and PPL
nav_order: 7
---
Expand Down
2 changes: 1 addition & 1 deletion _search-plugins/sql/ppl/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,5 +54,5 @@ Developers can find information in the following resources:
- [Piped Processing Language](https://github.com/opensearch-project/piped-processing-language) specification
- [OpenSearch PPL Reference Manual](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/index.rst)
- [Observability](https://github.com/opensearch-project/dashboards-observability/) using [PPL-based visualizations](https://github.com/opensearch-project/dashboards-observability#event-analytics)
- PPL [Data Types](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/general/datatypes.rst)
- PPL [Data types](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/general/datatypes.rst)
- [Cross-cluster search](https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/cross_cluster_search.rst#using-cross-cluster-search-in-ppl) in PPL
6 changes: 3 additions & 3 deletions _search-plugins/sql/sql-ppl-api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,14 +15,14 @@ Sends an SQL/PPL query to the SQL plugin. You can pass the format for the respon

### Query parameters

Parameter | Data Type | Description
Parameter | Data type | Description
:--- | :--- | :---
[format]({{site.url}}{{site.baseurl}}/search-plugins/sql/response-formats/) | String | The format for the response. The `_sql` endpoint supports `jdbc`, `csv`, `raw`, and `json` formats. The `_ppl` endpoint supports `jdbc`, `csv`, and `raw` formats. Default is `jdbc`.
sanitize | Boolean | Specifies whether to escape special characters in the results. See [Response formats]({{site.url}}{{site.baseurl}}/search-plugins/sql/response-formats/) for more information. Default is `true`.

### Request body fields

Field | Data Type | Description
Field | Data type | Description
:--- | :--- | :---
query | String | The query to be executed. Required.
[filter](#filtering-results) | JSON object | The filter for the results. Optional.
Expand Down Expand Up @@ -151,7 +151,7 @@ The response contains the schema and the results:

### Response body fields

Field | Data Type | Description
Field | Data type | Description
:--- | :--- | :---
schema | Array | Specifies the field names and types for all fields.
data_rows | 2D array | An array of results. Each result represents one matching row (document).
Expand Down
Loading