feat: Add "latest" and "related" search.

[jimsynz]

Here's my first stab at adding latest and related search as per #2013

I'm keen for early feedback, thus this draft PR.

I have a few questions:

1. Is there a standard function in the JS (I couldn't find one) to generate a link to hexdocs for a specific package name, version and ref? I feel iffy about just having string interpolation in the code.
2. It looks like it might be difficult (but not impossible) to correctly generate the metadata (and thus the excerpts) from the response payload. Each hit contains a list of highlight snippets however it treats underscores as token separators so searching for to_ast is returns results for to ast.
3. It seems to return the same hits over and over again. Each returned hit has a different document ID so I think it's actually an index problem not a search problem. I can dedup them by ref but I'd rather that the search index returned better results.

I haven't yet changed any of the markup as I wanted to get this step right before I go adding things like package names/versions to results, etc.

[josevalim]

Is there a standard function in the JS (I couldn't find one) to generate a link to hexdocs for a specific package name, version and ref? I feel iffy about just having string interpolation in the code.

There isn't because this is the first time we do something like this. However, since it comes from hexdocs, I believe it is fair to assume a naming scheme.

Each hit contains a list of highlight snippets however it treats underscores as token separators so searching for to_ast is returns results for to ast.

@ruslandoga, is this something we can change in the indexer?

It seems to return the same hits over and over again. Each returned hit has a different document ID so I think it's actually an index problem not a search problem. I can dedup them by ref but I'd rather that the search index returned better results.

The reason it returns the same results over and over again is because we break the documentation of a single function into multiple entries, one per h2 to be more precise. The benefit is that we can provide more precise links too. For now, I don't think we dedup them for regular results right? So I would not dedup them here, but we can dedup them later if we want to (we can even render it like Google results, where we show the main entry and below we refer to specific sections in the result).

---

This is fantastic progress. :)

[garazdawi]

FYI, in the algolia for https://erlang.org/doc/search we disable _ as punctuation just as proposed here for the same reasons.

[ruslandoga]

👋

I couldn't find any options to make highlighting ignore the token separators. I'll try asking Typesense people if it's supported or planned to be supported.

For now, I think we might need to roll our own highlighter, possibly similar to the one used in autocomplete:

ex_doc/assets/js/autocomplete/suggestions.js

Lines 289 to 311 in 51cd422

	/**
	* Returns an HTML string highlighting the individual tokens from the query string.
	*/
	function highlightMatches (text, query) {
	// Sort terms length, so that the longest are highlighted first.
	const terms = tokenize(query).sort((term1, term2) => term2.length - term1.length)
	return highlightTerms(text, terms)
	}
	function highlightTerms (text, terms) {
	if (terms.length === 0) return text
	const [firstTerm, ...otherTerms] = terms
	const match = text.match(new RegExp(`(.*)(${escapeRegexModifiers(firstTerm)})(.*)`, 'i'))
	if (match) {
	const [, before, matching, after] = match
	// Note: this has exponential complexity, but we expect just a few terms, so that's fine.
	return highlightTerms(before, terms) + '<em>' + escapeHtmlEntities(matching) + '</em>' + highlightTerms(after, terms)
	} else {
	return highlightTerms(text, otherTerms)
	}
	}

---

Removing _ from the current token_separators somewhat breaks non-prefix search. For example, searching upload stops returning allow_upload. Typesense does have a way around it with infix query option but I couldn't make it work well. We can try it again :)

[ruslandoga]

Just a couple nitpicks :)

Can we have a race condition here? When the previous request returns after the current request and updates the items to stale results. I think it's possible with multiple HTTP/1.1 connections, but not sure about multiple streams on the same HTTP/2 connection, are they guaranteed to be ordered? Or maybe JS runtime resolves it in some way?

Also, do we need to debounce on remote search or check for response.ok and results.length > 0?

For some reason I decided to do these things in ruslandoga#1 but I don't remember if I actually had these problems or was just playing it safe ...

[jimsynz]

Thanks. I'm sure you're right. As you can probably tell it's been almost a decade since I wrote any JavaScript so I'm still getting the hang of the new idioms.

[jimsynz]

So looking at the code more carefully, it appears that the search function is only called on page load, so should only be run once in the page's lifecycle.

[jimsynz]

Additionally, the search result handlebars template takes care of whether any results were actually returned.

[jimsynz]

Okay so I've pushed some updates:

1. Uses the shorter package filter syntax in the API query.
2. Repurpose the existing highlighter code to highlight search terms in the API search results.

[jimsynz]

I'll be moving on to adding the search type dropdown in the search bar next.

[jimsynz]

Okay, so I've added the search type to the UI, although you can probably tell that CSS is not my strong suit. It enables and disables the "latest" and "related" search based on the contents of searchNodes and versionNodes. It also completely disables the select box when the browser is not online.

[josevalim]

Thank you @jimsynz. I have been a bit busy this week but I will try to add review and add polish to this feature this week.

[jimsynz]

Thanks @josevalim 🙌

[josevalim]

Is this considered a filter by typesense?

[jimsynz]

This came from the original implementation of local search. The filters passed off to the typesense API are correctly quoted due to the use of the browser's URLSearchParams API so unless I'm misunderstanding the question I'm 99% sure it's fine.

[josevalim]

Thank you @jimsynz, I have dropped some comments. About the online vs offline, maybe instead of checking it upfront, we should try to run the search and, if the search fails for any reason, we fallback to regular search with an error message? I believe there is a toast we could use.

[jimsynz]

Thanks for the review @josevalim - I'll get onto it.

[jimsynz]

I'm dreadfully sorry that it has taken this long for me to get back to this PR. I have rebased it and addressed most of the feedback. I haven't yet changed the offline-detection as suggested - but I wanted to get the PR in a good shape for when @zachdaniel runs out of patience with me and takes over 😝