Fulltext search is not working

[haristariqmage4]

Preconditions

Magento Version : 2.4.0

ElasticSuite Version : 2.1.0

Environment : Developer

Third party modules :
Amasty_Base
Amasty_CronScheduleList
Amasty_Customform
Amasty_InvisibleCaptcha
Amasty_RequestQuote
Amasty_QuoteAttributesManagement
Amasty_RequestAQuoteProSubscriptionPackage
Amasty_QuoteAttributes
Amazon_Core
Amazon_Login
Amazon_Payment
Clarion_CustomerAttribute
Codazon_AjaxCartPro
Codazon_AjaxLayeredNav
Codazon_AjaxLayeredNavPro
Codazon_Core
Codazon_GoogleAmpManager
Codazon_ImproveBundle
Codazon_Lookbookpro
Codazon_MegaMenu
Codazon_OneStepCheckout
Codazon_ProductFilter
Codazon_ThemeOptions
Codazon_QuickShop
Codazon_ShippingCostCalculator
Codazon_Shopbybrandpro
Codazon_Slideshow
Codazon_ProductLabel
Codazon_Utility
Dotdigitalgroup_Email
Dotdigitalgroup_ChatHarrigo_EverCrumbs
Klarna_Core
Klarna_Ordermanagement
Klarna_Onsitemessaging
Klarna_Kp
Klaviyo_Reclaim
MageMe_HidePrice
MageWorx_SearchSuiteAutocomplete
Magefan_Community
Magefan_Blog
Magefan_WysiwygAdvanced
Magemonkeys_CategoryFilter
Magemonkeys_CompanyName
Magemonkeys_Customerinfo
Magemonkeys_FeaturedProduct
Magemonkeys_HideMyOrders
Magemonkeys_Ordermail
Magemonkeys_Product
Magemonkeys_Quote
Magemonkeys_RemoveQuoteCartPrice
Magemonkeys_RepresentativeAttr
Magemonkeys_RestrictCategory
Magemonkeys_WelcomeEmailCc
Mageplaza_Core
Mageplaza_BannerSlider
Mageplaza_BackendReindex
Mageplaza_MassProductActions
Mageplaza_Smtp
Magestat_SplitOrder
OlegKoval_RegenerateUrlRewrites
PayPal_Braintree
PayPal_BraintreeGraphQl
RapideWeb_ProductListTable
Smile_ElasticsuiteCore
Smile_ElasticsuiteCatalog
Smile_ElasticsuiteCatalogGraphQl
Smile_ElasticsuiteCatalogRule
Smile_ElasticsuiteCatalogOptimizer
Smile_ElasticsuiteTracker
Smile_ElasticsuiteThesaurus
Smile_ElasticsuiteSwatches
Smile_ElasticsuiteIndices
Smile_ElasticsuiteAnalytics
Smile_ElasticsuiteVirtualCategory
Temando_ShippingRemover
Ulmod_Ordernotes
Vertex_Tax
Vertex_AddressValidation
WebShopApps_MatrixRate
Yotpo_Yotpo
Zero1_Patches

How do we make results for "dextrose 5% water" show the same as results for "d5w"? Since one is multiple words and the other is technically just one?
How can we make sure that items like Sharps Container 26 1/4 ° 20 w * 14 3/4 D Inch 19 BD Gallon are not included in the search results for 'D5W'?

Expected result

More narrow product search that will only allow for exact terms to be fetched
Searches like 'D5W' should not have results that include hits for 'D' '5' 'W'

Actual result

[rbayet]

Hello @haristariqmage4,

This is probably due to the "word_delimiter" of the "standard" (text) analyzer which will transform your product names before indexing it.
This "word_delimiter" component DO split words like "D5W" when switching from a letter to a digit and vice versa, so you are correct assuming that we do search for "D", "5" and "W" when searching for "D5W".
The issue is then that you have other product names with those isolated letters (for example coming from a product name string like "3/5 H X 10 7/10 W X 6 D").
You can check what's happening on the analyzer side of things from the admin interface in the Elasticsuite > System > Analysis.

If you can't have "simpler" product names, I would recommend trying to change the configuration of the "word_delimiter" token filter by disabling "split_on_numerics" in the elasticsuite_analysis.xml (through a composer patch or a re-definition of the XML in a custom module)

Then for the original issue, a thesaurus entry for associating "D5W" or "d5w" to "Dextrose 5% Water" should to the trick.

Regards,

[haristariqmage4]

There is only System > Indices. Please define more.

[rbayet]

Hello @haristariqmage4,

Indeed, I've just saw

Magento Version : 2.4.0

ElasticSuite Version : 2.1.0 (I guess it's 2.10.0)

That's ... old :)
Indeed, you will not have that screen which has been introduced in 2.10.13, so in Magento 2.4.1 and above only.
You can install cerebro locally and reproduce what that screen does from cerebro's "analysis" screen :

Or use directly the _analyze endpoint of your Elasticsearch in CLI

/var/www/html $ curl -H "Content-Type: application/json" -XPOST http://opensearch:9200/magento2_fr_fr_catalog_product/_analyze?pretty -d '{"analyzer":"standard","text":"d5w"}'
{
  "tokens" : [
    {
      "token" : "d5w",
      "start_offset" : 0,
      "end_offset" : 3,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "d",
      "start_offset" : 0,
      "end_offset" : 1,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "d5w",
      "start_offset" : 0,
      "end_offset" : 3,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "5",
      "start_offset" : 1,
      "end_offset" : 2,
      "type" : "<ALPHANUM>",
      "position" : 1
    },
    {
      "token" : "w",
      "start_offset" : 2,
      "end_offset" : 3,
      "type" : "<ALPHANUM>",
      "position" : 2
    }
  ]
}

(Replace http://opensearch:9200/magento2_fr_fr_catalog_product/ by http://[your_elasticsearch_server_address_or_hostname]/[your_catalog_product_index_name])

Regards,

[haristariqmage4]

Hi @rbayet ,
Hereis my analysis, now what should i do?

[rbayet]

Hello @haristariqmage4,

So now that your thesaurus is in place, you have two options (that could be combined, actually)

1. reducing the score penalty for products matching a synonym
2. altering the "word_delimiter" token filter in the way I described

1. reducing the score penalty for products matching a synonym
When searching for "d5w" you will now also search for "dextrose 5% water" but by default the products matching only "dextrose 5% water" will suffer a score penalty with a tenth of their expected score.

You can change that by reducing (up to 1, ie "no penalty") the setting available at Elasticsuite > Search Relevance > Thesaurus Configuration > Synonyms Configuration > Synonyms Weight Divider

The products matching individually "D", "5" and "W" will still be present in the search results list but maybe at a lower place for you to be satisfied.

2. altering the "word_delimiter" token filter in the way I described
If you're not satisfied, or as an alternative, you can redefine or finetune the word_delimiter token filter which is responsible for splitting "D5W" into "D", "5" and "W".

You probably only need to change the "split_on_numerics" from "true" to "false".

You can do that either with a composer patch on that distribution file OR create a custom module in app/code with a local elasticsuite_analysis.xml which will contain just the re-defined word_delimiter token filter.
In both cases, this will require clearing the Magento cache and performing a full reindex.

Please be aware that this approach could have adverse side effects, for instance preventing finding products with a "L48B" in their name or their SKU by searching for "L 48 B" for instance.

Regards,

[haristariqmage4]

@rbayet
Should i generate_word_parts -> false too in elasticsuite_analysis.

[haristariqmage4]

Also what can be the solution to avoid:

Please be aware that this approach could have adverse side effects, for instance preventing finding products with a "L48B" in their name or their SKU by searching for "L 48 B" for instance.

this side effects

[haristariqmage4]

@rbayet ?