Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tutorial: generate embedding for arrays of object #2477

Merged
merged 1 commit into from
May 28, 2024
Merged

tutorial: generate embedding for arrays of object #2477

merged 1 commit into from
May 28, 2024

Conversation

ylwu-amzn
Copy link
Collaborator

Description

Add tutorial: generate embedding for arrays of object

Issues Resolved

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@ylwu-amzn
tutorial: generate embedding for arrays of object
8f985bd 
Signed-off-by: Yaliang Wu <ylwu@amazon.com>
@ylwu-amzn ylwu-amzn temporarily deployed to ml-commons-cicd-env May 28, 2024 16:58 — with GitHub Actions Inactive
@ylwu-amzn ylwu-amzn temporarily deployed to ml-commons-cicd-env May 28, 2024 16:58 — with GitHub Actions Inactive
@ylwu-amzn ylwu-amzn temporarily deployed to ml-commons-cicd-env May 28, 2024 16:58 — with GitHub Actions Inactive
@ylwu-amzn ylwu-amzn temporarily deployed to ml-commons-cicd-env May 28, 2024 16:58 — with GitHub Actions Inactive
@ylwu-amzn ylwu-amzn temporarily deployed to ml-commons-cicd-env May 28, 2024 16:58 — with GitHub Actions Inactive
@ylwu-amzn ylwu-amzn temporarily deployed to ml-commons-cicd-env May 28, 2024 16:58 — with GitHub Actions Inactive
@ylwu-amzn ylwu-amzn merged commit 0722df1 into opensearch-project:main May 28, 2024
11 checks passed
opensearch-trigger-bot bot pushed a commit that referenced this pull request May 28, 2024
@ylwu-amzn @github-actions
tutorial: generate embedding for arrays of object (#2477)
b135e91 
Signed-off-by: Yaliang Wu <ylwu@amazon.com>
(cherry picked from commit 0722df1)
@opensearch-trigger-bot opensearch-trigger-bot bot mentioned this pull request May 28, 2024
Merged
dhrubo-os
dhrubo-os reviewed May 28, 2024
View reviewed changes
docs/tutorials/semantic_search/generate_embeddings_for_arrays_of_object.md
Create sub-pipeline to generate embedding for one item in the array.

This pipeline contains 3 processors
- set processor: The `text_embedding` processor is unable to identify "_ingest._value.title". You need to copy "_ingest._value.title" to a temporary field for text_embedding to process it.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how customers will know what will be the name of the temporary field? if the original field name is xyz then embedding field will be xyz_embedding?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could be arbitrary name, just make sure it's not same with any field which is being used in document.

docs/tutorials/semantic_search/generate_embeddings_for_arrays_of_object.md
}
```

Create pipeline with foreach processor:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May be we can add a link for foreach processor to explain what does this processor do?

Copy link
Collaborator Author

@ylwu-amzn ylwu-amzn May 28, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, OpenSearch doesn't have foreach processor doc, read https://opensearch.org/docs/latest/ingest-pipelines/processors/index-processors/. There is one issue tracking the gap opensearch-project/documentation-website#4647

docs/tutorials/semantic_search/generate_embeddings_for_arrays_of_object.md
"description": "This is first book"
},
{
"title": "second book",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how did you get this title here?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, should remove this, will fix in next PR soon

ylwu-amzn added a commit that referenced this pull request May 28, 2024
@opensearch-trigger-bot @ylwu-amzn
tutorial: generate embedding for arrays of object (#2477) (#2478)
a339153 
Signed-off-by: Yaliang Wu <ylwu@amazon.com>
(cherry picked from commit 0722df1)

Co-authored-by: Yaliang Wu <ylwu@amazon.com>
@ochrist-eis
Copy link

Thanks a lot for providing this tutorial! 👍

@ylwu-amzn ylwu-amzn temporarily deployed to ml-commons-cicd-env May 28, 2024 17:53 — with GitHub Actions Inactive
@ylwu-amzn ylwu-amzn temporarily deployed to ml-commons-cicd-env May 28, 2024 17:53 — with GitHub Actions Inactive
@ylwu-amzn ylwu-amzn temporarily deployed to ml-commons-cicd-env May 28, 2024 17:53 — with GitHub Actions Inactive
opensearch-trigger-bot bot pushed a commit that referenced this pull request Sep 30, 2024
@ylwu-amzn @github-actions
tutorial: generate embedding for arrays of object (#2477)
b61d500 
Signed-off-by: Yaliang Wu <ylwu@amazon.com>
(cherry picked from commit 0722df1)
@opensearch-trigger-bot opensearch-trigger-bot bot mentioned this pull request Sep 30, 2024
Merged
dhrubo-os pushed a commit that referenced this pull request Sep 30, 2024
@opensearch-trigger-bot @ylwu-amzn
tutorial: generate embedding for arrays of object (#2477) (#3007)
0a74930 
Signed-off-by: Yaliang Wu <ylwu@amazon.com>
(cherry picked from commit 0722df1)

Co-authored-by: Yaliang Wu <ylwu@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dhrubo-os dhrubo-os dhrubo-os left review comments

@Zhangxunmt Zhangxunmt Zhangxunmt approved these changes

@jngz-es jngz-es jngz-es approved these changes

@b4sjoo b4sjoo Awaiting requested review from b4sjoo b4sjoo is a code owner

@model-collapse model-collapse Awaiting requested review from model-collapse model-collapse is a code owner

@rbhavna rbhavna Awaiting requested review from rbhavna rbhavna is a code owner

@zane-neo zane-neo Awaiting requested review from zane-neo zane-neo is a code owner

@austintlee austintlee Awaiting requested review from austintlee austintlee is a code owner

@HenryL27 HenryL27 Awaiting requested review from HenryL27 HenryL27 is a code owner

@samuel-oci samuel-oci Awaiting requested review from samuel-oci samuel-oci is a code owner

@xinyual xinyual Awaiting requested review from xinyual xinyual is a code owner

Assignees
No one assigned
Labels
backport feature/multi_tenancy backport 2.x
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@ylwu-amzn @ochrist-eis @Zhangxunmt @jngz-es @dhrubo-os