If I use moganet to train a model, and then use the model for feature extraction and image retrieval, how should I apply this framework?

[watertianyi]

Is there an existing issue for this?

• I have searched the existing issues.

Is your feature request related to a problem? Please describe.

If I use moganet to train a model, and then use the model for feature extraction and image retrieval, how should I apply this framework?

Describe the solution you'd like.

If I use moganet to train a model, and then use the model for feature extraction and image retrieval, how should I apply this framework?

Describe an alternate solution.

If I use moganet to train a model, and then use the model for feature extraction and image retrieval, how should I apply this framework?

Anything else? (Additional Context)

If I use moganet to train a model, and then use the model for feature extraction and image retrieval, how should I apply this framework?

[junjiejiangjjj]

hi @goldwater668 , towhee integrates image models through timm. If your model can be called through timm, use the following code:

from towhee import pipe, ops, DataCollection

p = (
    pipe.input('path')
        .map('path', 'img', ops.image_decode())
        .map('img', 'vec', ops.image_embedding.timm(model_name='model_name', checkpoint_path="your local weights"))
        .output('img', 'vec')
)

DataCollection(p('towhee.jpeg')).show()

If not, you need to implement an operator yourself, https://towhee.readthedocs.io/en/latest/operator/usage.html#custom-operators

[watertianyi]

@junjiejiangjjj
https://github.com/Westlake-AI/MogaNet
This is the warehouse of the model. I saw the following call. It should support timm. If you have time, you can help and see if it can be called directly.

from timm.models import create_model, apply_test_time_pool, load_checkpoint, is_model, list_models
from timm.data import create_dataset, create_loader, resolve_data_config, RealLabelsImagenet
from timm.utils import accuracy, AverageMeter, natural_key, setup_default_logging, set_jit_legacy
checkpoint = torch.load(checkpoint_path)
model = create_model(
        "moganet_xtiny",
        pretrained=False,
        num_classes=2,
        in_chans=3,
        global_pool=None,
        scriptable=False,
        mem_index=[10],
        K=1024)
model.load_state_dict(checkpoint["state_dict"], strict=True)
model = model.cuda()
model.eval()

transform = create_transform(input_size=(3, 224, 224),mean=(0.485, 0.456, 0.406),std=(0.229, 0.224, 0.225),crop_pct=0.9)
for root, subdirs, files in os.walk(folder, topdown=False, followlinks=True):
    rel_path = os.path.relpath(root, folder) if (root != folder) else ''
    label = os.path.basename(rel_path) if True else rel_path.replace(os.path.sep, '_')
    for f in files:
        base, ext = os.path.splitext(f)
        if ext.lower() in ['.png', '.jpg', '.jpeg']:
            image_path = os.path.join(root,f)
            img=Image.open(image_path)
            img = img.convert("RGB")
            input_image = transform(img).unsqueeze(0)
            input_image = Variable(input_image).to(DEVICE)
            out = model(input_image)
            img_num+=1
            classes = ('NG', 'OK')
            _, pred = torch.max(out.data, 1)

[junjiejiangjjj]

It seems that you need to run it in the directory of this project.

git clone https://github.com/Westlake-AI/MogaNet 
cd MogaNet 

import models
from towhee import pipe, ops, DataCollection

p = (
    pipe.input('path')
        .map('path', 'img', ops.image_decode())
        .map('img', 'vec', ops.image_embedding.timm(model_name='moganet_xtiny', checkpoint_path="your local weights"))
        .output('img', 'vec')
)

DataCollection(p('towhee.jpeg')).show()

[junjiejiangjjj]

hi @goldwater668 , if you change the channels, you also need to change it when initializing the model.

[junjiejiangjjj]

This is determined by the model. This model has these specifications

[junjiejiangjjj]

Towhee accelerates model inference through Triton Server , but only supports a subset of models. For your own models, you can consider converting them to ONNX to accelerate inference.

[watertianyi]

@junjiejiangjjj
Where is the similarity threshold for retrieval set?Then how much memory does the inserted data occupy and where can I check it?

[watertianyi]

@junjiejiangjjj
Where can I check the similarity threshold? How to set the similarity measurement threshold? What should I do if I want to extract feature points from surf in opencv and store them in the database?

[junjiejiangjjj]

I don't quite understand your question. Could you look at these cases to see if there are any similar scenarios to yours? https://github.com/towhee-io/examples

[watertianyi]

@junjiejiangjjj
The first thing is where to set the top5 threshold in accuracy? Can the top5 similarities be printed out?

The second thing is whether I use traditional algorithms for feature extraction and then vector retrieval, such as surf feature point extraction in opencv

[junjiejiangjjj]

@junjiejiangjjj The first thing is where to set the top5 threshold in accuracy? Can the top5 similarities be printed out?

The second thing is whether I use traditional algorithms for feature extraction and then vector retrieval, such as surf feature point extraction in opencv

1. When searching, just set the top 5

# Search pipeline
p_search_pre = (
        p_embed.map('vec', ('search_res'), ops.ann_search.milvus_client(
                    host=HOST, port=PORT, limit=TOPK,
                    collection_name=COLLECTION_NAME))
               .map('search_res', 'pred', lambda x: [str(Path(y[0]).resolve()) for y in x])
#                .output('img_path', 'pred')
)
p_search = p_search_pre.output('img_path', 'pred')

2. Milvus stores this information through scalars

You can refer to this example
https://github.com/towhee-io/examples/blob/main/image/reverse_image_search/1_build_image_search_engine.ipynb

[watertianyi]

@junjiejiangjjj
INDEX_TYPE = 'HNSW'
METRIC_TYPE = 'IP'
pymilvus.exceptions.MilvusException: <MilvusException: (code=65535, message=fail to search on QueryNode 10: worker(10) query failed: metric type not match: invalid parameter[expected=IP][actual=L2])>

Is it only limited to the calculation method of similarity measure as L2, and other methods such as inner product (IP)
Doesn’t even cosine similarity (COSINE) work?