Allow training a model on multiple annotations #8071

philippotto · 2024-09-11T15:05:46Z

URL of deployed dev instance (used for testing):

https://multiannotraining.webknossos.xyz
I already created a few annotations (with differing configurations) and made the default user a super-admin
details about the annotations:
- https://multiannotraining.webknossos.xyz/annotations/66e2e04b010000df00b3f6fa
  - no-tracing segmentation layer: "segmentation"
- https://multiannotraining.webknossos.xyz/annotations/66e2dfe6010000ab00b3f6f7
  - volume layer with fallback: "segmentation"
- https://multiannotraining.webknossos.xyz/annotations/66e2e06d010000ad00b3f6ff
  - volume layer with fallback: "custom_segmentation"
- https://multiannotraining.webknossos.xyz/annotations/66e2e08c010000ad00b3f702
  - volume layer with fallback: custom_segmentation
  - volume layer without fallback: custom_empty
    - finest mag is mag4
- https://multiannotraining.webknossos.xyz/annotations/66e2e0d6010000ac00b3f706
  - no-tracing segmentation layer: "segmentation"
  - volume layer without fallback: custom_empty

Steps to test:

go to admin > ai models
click on the "train new model" button
paste this CSV

66e2e04b010000df00b3f6fa
https://multiannotraining.webknossos.xyz/annotations/66e2dfe6010000ab00b3f6f7
https://multiannotraining.webknossos.xyz/annotations/66e2e06d010000ad00b3f6ff
https://multiannotraining.webknossos.xyz/annotations/66e2e08c010000ad00b3f702
https://multiannotraining.webknossos.xyz/annotations/66e2e0d6010000ac00b3f706

click "add"
use the form
submit it (it will fail because there is no worker configured on the dev instance). however, you can use the network tab to inspect the sent payload.

also open an annotation and use the train-model dialog from there to ensure that it still works
special care needs to be taken for the volume layer name, because we usually use the tracing id as the name and then have a human-readable name for the UI. however, the worker always wants the human readable one and never the tracing id.

Issues:

fixes Start train_model with multiple annotations #8005

Updated changelog
Updated documentation if applicable
Removed dev-only changes like prints and application.conf edits

…ti-anno-training

philippotto · 2024-09-12T12:05:45Z

frontend/javascripts/oxalis/default_state.ts

@@ -240,7 +240,7 @@ const defaultState: OxalisState = {
    showVersionRestore: false,
    showDownloadModal: false,
    showPythonClientModal: false,
-    aIJobModalState: "invisible",
+    aIJobModalState: "neuron_inferral",


Suggested change

aIJobModalState: "neuron_inferral",

aIJobModalState: "invisible",

daniel-wer

Thank you for implementing this, I'm looking forward to use the feature! Very thorough implementation 👍

Notes from testing:

Either the placeholder or the validation for the csv modal is wrong. The placeholder indicates that the color and segmentation layer names can be supplied, but the validation complains that only the url/id should be given.
The csv and the training modal are closed when clicking onto the back drop. The second one (and arguably the first one as well) should not do that, because it makes it to easy to lose entered information.
Whether the modal is triggered from within an annotation or from the models list, allows to select different layers (i.e., the fallback layer):

frontend/javascripts/admin/voxelytics/ai_model_list_view.tsx

frontend/javascripts/oxalis/view/jobs/train_ai_model.tsx

normanrz · 2024-09-18T09:03:09Z

@MichaelBuessemeyer Could you please carry this over the finish line?

…ti-anno-training

- includes extending bbox validation

…ti-anno-training

MichaelBuessemeyer · 2024-09-24T12:27:52Z

Ok here we go :)
I hope I implemented everything you mentioned or commented it. Could you please do some retesting & re-reviewing of the changes @daniel-wer 🙏?

Either the placeholder or the validation for the csv modal is wrong. The placeholder indicates that the color and segmentation layer names can be supplied, but the validation complains that only the url/id should be given.

I removed the placeholder as the current code does not support directly supplying the color and segmentation layer to use for an annotation. If this is a helpful feature I'd prefer to do this as a follow-up :)

Whether the modal is triggered from within an annotation or from the models list, allows to select different layers (i.e., the fallback layer):

I don't understand what the issue is with the option to be able to select layers. An annotation might have multiple volume layers and thus the segmentation data that should be used is most of the time ambiguous. Same goes for color layers.

daniel-wer · 2024-09-24T13:00:01Z

I don't understand what the issue is with the option to be able to select layers. An annotation might have multiple volume layers and thus the segmentation data that should be used is most of the time ambiguous. Same goes for color layers.

What I wanted to express is that the layer options offered to the user are different ones, depending on where the modal is opened. If you look at the screenshots, one time there are two options offered, and the other time there are three options, although I specified the same annotation. This should not be the case.

daniel-wer

Thank you for taking over this PR!

Please also see my previous comment regarding the testing feedback. I can no longer test this on the dev instance, because the AI modal cannot be opened from within an annotation (Philipp patched the code to open the modal automatically, but you already removed that). Please test this locally or let me know if you don't know what I mean :)

daniel-wer · 2024-09-24T13:08:40Z

frontend/javascripts/oxalis/view/jobs/train_ai_model.tsx

-function areBoundingBoxesValid(userBoundingBoxes: UserBoundingBox[] | undefined): {
-  valid: boolean;
-  reason: string | null;
+function areAllAnnotationsInvalid<T extends HybridTracing | APIAnnotation>(


The All seems to be incorrect and can probably simply be dropped

daniel-wer · 2024-09-24T13:10:16Z

frontend/javascripts/oxalis/view/jobs/train_ai_model.tsx

            .map(
              (layer) =>
                getTracingForAnnotationType(annotation, layer) as Promise<ServerVolumeTracing>,
            ),
        );
+        // TODO: make bboxs a member again


ups, forgot to remove this. I already implemented it this way.

daniel-wer · 2024-09-24T13:10:25Z

frontend/javascripts/oxalis/view/jobs/train_ai_model.tsx

+          }
+        }
+
+        console.log(volumeTracings); // TODOM remove


thanks, sorry I sometimes tend to skip reading comments 🙈

frontend/javascripts/oxalis/view/jobs/train_ai_model.tsx

daniel-wer · 2024-09-24T13:20:10Z

frontend/javascripts/oxalis/view/jobs/train_ai_model.tsx

+        const volumeTracings = volumeServerTracings.map((tracing) =>
+          serverVolumeToClientVolumeTracing(tracing),
+        );
+        let userBoundingBoxes = volumeTracings[0]?.userBoundingBoxes;
+        if (!userBoundingBoxes) {
+          const skeletonLayer = annotation.annotationLayers.find(
+            (layer) => layer.typ === "Skeleton",
+          );
+          if (skeletonLayer) {
+            const skeletonTracing = await getTracingForAnnotationType(annotation, skeletonLayer);
+            userBoundingBoxes = convertUserBoundingBoxesFromServerToFrontend(
+              skeletonTracing.userBoundingBoxes,
+            );
+          } else {


Could you add a comment here that user bounding boxes are always saved in all annotation layers, but if there is no volume annotation layer, the skeleton layer is checked. At least that's how I interpret it, but I was a bit confused at first :)

MichaelBuessemeyer · 2024-09-24T14:54:41Z

frontend/javascripts/oxalis/view/jobs/train_ai_model.tsx


 const { TextArea } = Input;
 const FormItem = Form.Item;

+export type AnnotationInfoForAIJob<GenericAnnotation> = {


Please note that I also added this comment. It hopefully explains why all these additional properties (volumeTracings, userBoundingBoxes, volumeTracingResolutions) are needed. Please tell me your opinion on whether you understand what I want to express and maybe on wording improvements 🙏

MichaelBuessemeyer · 2024-09-24T14:55:29Z

What I wanted to express is that the layer options offered to the user are different ones, depending on where the modal is opened. If you look at the screenshots, one time there are two options offered, and the other time there are three options, although I specified the same annotation. This should not be the case.

Oh good finding 🎉

I think I fixed it now by simply filtering out volume data layers of the dataset that work as a fallback layer for one of the volume tracing layers.

The rest of your awesome feedback 🦶 🦶 🔙 should also be applied. 👯

Co-authored-by: Daniel <daniel.werner@scalableminds.com>

- add comment about why AnnotationInfoForAIJob is so complex - filter out dataset segmentation layers that are used as fallback layers of other volume tracing layers of an annotation

…ssos into multi-anno-training

daniel-wer

Awesome, thank you for polishing this. I'm looking forward to using this :)

frontend/javascripts/oxalis/view/jobs/train_ai_model.tsx

daniel-wer · 2024-09-24T19:18:20Z

As a note, I've created #8097 for two smaller followup issues that came to my mind, but I didn't want to further bloat and delay this PR :) After merging this, we should test it in production, extend the mentioned issue with any other shortcomings we notice, and then implement the followups 🚀

Co-authored-by: Daniel <daniel.werner@scalableminds.com>

philippotto added 7 commits September 9, 2024 17:01

implement train-model for multiple annotations (WIP)

b6aefe3

further refactoring of train model form

618cd38

make onFinish work for multi-annotation training

8a8a0cc

Merge branch 'master' of github.com:scalableminds/webknossos into mul…

89d1c78

…ti-anno-training

clean up

7e4e3ea

fix annotationId and csv validation

a927a7e

fix training model from annotation view

9167b0e

philippotto self-assigned this Sep 11, 2024

philippotto and others added 2 commits September 12, 2024 13:59

Merge branch 'master' into multi-anno-training

79f859d

clean up

732a22f

philippotto changed the title ~~[WIP] Allow training a model on multiple annotations~~ Allow training a model on multiple annotations Sep 12, 2024

philippotto commented Sep 12, 2024

View reviewed changes

philippotto marked this pull request as ready for review September 12, 2024 12:59

philippotto requested a review from daniel-wer September 12, 2024 13:00

daniel-wer reviewed Sep 16, 2024

View reviewed changes

normanrz requested a review from MichaelBuessemeyer September 18, 2024 09:02

MichaelBuessemeyer self-assigned this Sep 18, 2024

MichaelBuessemeyer removed their request for review September 18, 2024 09:31

Michael Büßemeyer added 7 commits September 23, 2024 15:49

Merge branch 'master' of github.com:scalableminds/webknossos into mul…

b405d11

…ti-anno-training

WIP: Apply feedback

6eccb7e

- includes extending bbox validation

make bbox checking work while not refetching tracings

4926361

some cleanup

77e0439

apply missing pr feedback

f3b4c16

Merge branch 'master' of github.com:scalableminds/webknossos into mul…

f748d6b

…ti-anno-training

improve ai modal csv input parsing

66ecfef

daniel-wer reviewed Sep 24, 2024

View reviewed changes

MichaelBuessemeyer reviewed Sep 24, 2024

View reviewed changes

MichaelBuessemeyer and others added 3 commits September 24, 2024 17:02

Update frontend/javascripts/oxalis/view/jobs/train_ai_model.tsx

e2546b9

Co-authored-by: Daniel <daniel.werner@scalableminds.com>

apply feedback

acc2b2d

- add comment about why AnnotationInfoForAIJob is so complex - filter out dataset segmentation layers that are used as fallback layers of other volume tracing layers of an annotation

Merge branch 'multi-anno-training' of github.com:scalableminds/webkno…

463de3c

…ssos into multi-anno-training

daniel-wer approved these changes Sep 24, 2024

View reviewed changes

frontend/javascripts/oxalis/view/jobs/train_ai_model.tsx Outdated Show resolved Hide resolved

MichaelBuessemeyer and others added 3 commits September 25, 2024 07:06

Update frontend/javascripts/oxalis/view/jobs/train_ai_model.tsx

75aa6b1

Co-authored-by: Daniel <daniel.werner@scalableminds.com>

Merge branch 'master' into multi-anno-training

6e378a3

Merge branch 'master' into multi-anno-training

1045a03

MichaelBuessemeyer enabled auto-merge (squash) September 25, 2024 07:05

MichaelBuessemeyer merged commit 4d2261d into master Sep 25, 2024
2 checks passed

MichaelBuessemeyer deleted the multi-anno-training branch September 25, 2024 07:18

fm3 mentioned this pull request Oct 15, 2024

Followup for multi-annotation trainings #8097

Open

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow training a model on multiple annotations #8071

Allow training a model on multiple annotations #8071

philippotto commented Sep 11, 2024 •

edited

Loading

philippotto Sep 12, 2024

daniel-wer left a comment •

edited

Loading

normanrz commented Sep 18, 2024

MichaelBuessemeyer commented Sep 24, 2024

daniel-wer commented Sep 24, 2024

daniel-wer left a comment

daniel-wer Sep 24, 2024

daniel-wer Sep 24, 2024

MichaelBuessemeyer Sep 24, 2024

daniel-wer Sep 24, 2024

MichaelBuessemeyer Sep 24, 2024

daniel-wer Sep 24, 2024

MichaelBuessemeyer Sep 24, 2024

MichaelBuessemeyer commented Sep 24, 2024 •

edited

Loading

daniel-wer left a comment

daniel-wer commented Sep 24, 2024

	aIJobModalState: "neuron_inferral",
	aIJobModalState: "invisible",

Allow training a model on multiple annotations #8071

Allow training a model on multiple annotations #8071

Conversation

philippotto commented Sep 11, 2024 • edited Loading

URL of deployed dev instance (used for testing):

Steps to test:

Issues:

Choose a reason for hiding this comment

daniel-wer left a comment • edited Loading

Choose a reason for hiding this comment

normanrz commented Sep 18, 2024

MichaelBuessemeyer commented Sep 24, 2024

daniel-wer commented Sep 24, 2024

daniel-wer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MichaelBuessemeyer commented Sep 24, 2024 • edited Loading

daniel-wer left a comment

Choose a reason for hiding this comment

daniel-wer commented Sep 24, 2024

philippotto commented Sep 11, 2024 •

edited

Loading

daniel-wer left a comment •

edited

Loading

MichaelBuessemeyer commented Sep 24, 2024 •

edited

Loading