updating standard analyzer docs #9747
Conversation
Signed-off-by: Anton Rubin <[email protected]>
AntonEliatra
requested review from
kolchfa-aws,
Naarcha-AWS,
AMoo-Miki,
natebower,
dlvenable and
epugh
as code owners
|
Thank you for submitting your PR. The PR states are In progress (or Draft) -> Tech review -> Doc review -> Editorial review -> Merged. Before you submit your PR for doc review, make sure the content is technically accurate. If you need help finding a tech reviewer, tag a maintainer. When you're ready for doc review, tag the assignee of this PR. The doc reviewer may push edits to the PR directly or leave comments and editorial suggestions for you to address (let us know in a comment if you have a preference). The doc reviewer will arrange for an editorial review. |
Signed-off-by: kolchfa-aws <[email protected]>
|
@udabhas Could you please review this PR? Thanks! |
kolchfa-aws
added
3 - Tech review
| `max_token_length` | Optional | Integer | Sets the maximum length of the produced token. If this length is exceeded, the token is split into multiple tokens at the length configured in `max_token_length`. Default is `255`. | ||
| `stopwords` | Optional | String or list of strings | A string specifying a predefined list of stopwords (such as `_english_`) or an array specifying a custom list of stopwords. Default is `_none_`. | ||
| `stopwords_path` | Optional | String | The path (absolute or relative to the config directory) to the file containing a list of stop words. | ||
| The `standard` analyzer supports the following parameters: |
There was a problem hiding this comment.
Since all the parameters are optional, can we reword it as supports the following optional parameters?
| | Parameter | Type | Default | Description | | ||
| |:----------|:-----|:--------|:------------| | ||
| | `max_token_length` | Integer | `255` | Sets the maximum length of a token before it is split. | | ||
| | `stopwords` | List or String | None | A list of stopwords or a predefined stopword set like `_english_` to remove during analysis. | |
There was a problem hiding this comment.
String or list of strings was better description
| |:----------|:-----|:--------|:------------| | ||
| | `max_token_length` | Integer | `255` | Sets the maximum length of a token before it is split. | | ||
| | `stopwords` | List or String | None | A list of stopwords or a predefined stopword set like `_english_` to remove during analysis. | | ||
| | `stopwords_path` | String | None | Path to a file containing stopwords to be used during analysis. | |
There was a problem hiding this comment.
Can you verify once if both stopwords and stopwords_path be used in a single request? If not, let's put a small note or clarification?
|
|
||
| Use the following command to configure an index with a custom analyzer that is equivalent to the `standard` analyzer: | ||
| The following example crated index `products` and configures `max_token_length` and `stopwords`: |
| } | ||
| ``` | ||
| {% include copy-curl.html %} | ||
|
|
||
| The response contains the generated tokens: | ||
| The returned token are separated based on spacing, lowercased and stopwords are removed: |
There was a problem hiding this comment.
rephrasing:
The returned token are 1/ separated based on spacing, 2/ lowercased and 3/ stopwords removed:
want to emphasize on e actions which happened.
|
|
||
| - `standard` tokenizer: Removes most punctuation and splits text on spaces and other common delimiters. | ||
| - `lowercase` token filter: Converts all tokens to lowercase, ensuring case-insensitive matching. | ||
| - `stop` token filter: Removes common stopwords, such as "the", "is", and "and", from the tokenized output. |
There was a problem hiding this comment.
Did we miss adding detail about stop words or is it part of tokenization or lowercasing itself.
There was a problem hiding this comment.
@sandeshkr419 but the stop words are not removed in standard analyzer, is that what you mean?
|
|
||
| | Parameter | Type | Default | Description | |
There was a problem hiding this comment.
Most of the documentation pages use Data type instead of Type. Eg: https://github.com/opensearch-project/documentation-website/pull/9479/files
Let's stick to a single nomenclature across documentations - either type or data type - either is fine, as long as it is consistent.
There was a problem hiding this comment.
@sandeshkr419 thats updated now across the repo, All "Data Type" changed to "Data type"
Signed-off-by: Anton Rubin <[email protected]>
Signed-off-by: Anton Rubin <[email protected]>
Description
updating standard analyzer docs
Version
all
Checklist
For more information on following Developer Certificate of Origin and signing off your commits, please check here.