Some doubts about prompting format

[gilroff]

How will these prompts be processed?
I know the difference between AND_SALT, AND_PERP and TOPK. But Im not sure the order and the context in which they are processed here:
A woman AND_TOPK [ goddess of ornaments ] AND_PERP [ dancing ] in heaven.
Is prompting like this even make sense? lol
Pls gimme your tips or advice :)

[ljleb]

Hi, conceptually there are 2 types of prompts:

• regular prompts
• auxiliary prompts

I'll start by defining what happens without nesting the prompts into each other, and only then will I cover how nesting is implemented. This will make things easier to understand I believe.

Single nesting level

Regular prompts are defined as:

• the first prompt before any keyword separator
• any regular AND [ x ] prompt

These prompts are processed and combined together before anything else.

Auxiliary prompts are defined with respect to the regular prompts. They are always processed after the regular prompts because they need to know the regular prompts to work. This is the only common feature of these prompts, otherwise the auxiliary prompts can be made to do whatever we need them to do. A side effect of this constraint is that different auxiliary prompt keywords do not see each other. Each auxiliary keywords AND_PERP, AND_SALT, etc. never see the prompts of other auxiliary keywords. Each keyword only ever sees the prompts using the same keyword, and the shared regular prompts.

Now, we can define each keyword however we want, with the only constraint described above that they have to be processed after the regular prompts:

• ... AND_PERP [ x ] ...: Orthogonalize x with the regular prompts. There is no particular interaction between any two AND_PERP prompts.
• ... AND_SALT [ x ] ... AND_SALT [ y ] ...: Take the regular prompts and the salient prompts x and y, and put them in a list. The first element of the list is the regular prompts combined together, the second element is the first salient prompt x, the third is the salient prompt y, etc. Then, the strongest pixel of any prompt at any particular pixel coordinate is picked, so as to create a combined prompt with the strongest features of each prompt.
• ... AND_TOPK [ x ] ...: Keep only the top 5% strongest features of the topk prompt x. There is no interaction between each topk prompt, and they do not use the regular prompts.

Again, only after all auxiliary keywords have been processed, do they finally get added to the main prompt with regular addition, resulting in a final single combined prompt.

Nested prompts

First, an example:

a woman
AND_TOPK [
    goddess of ornaments
    AND_SALT in heaven
]
AND_PERP [
    dancing
]

The leaf prompts are processed first to turn them into single combined prompts. In this case, this is the non-nested prompt groups a woman, goddess of ornaments AND_SALT in heaven and dancing. Only once these prompts have been individually processed without talking to each other, are they combined together using the keywords AND_TOPK and AND_PERP.

Hope this helps! Please let me know if anything was unclear or if you have other questions.

[ljleb]

As a side note, for the particular prompt you are using: I'm not completely sure what the parser does in this case, as the prompt has extra text at the end.

Each prompt separated by a keyword should try to describe as much as possible the entire image. They split the prompt into chunks that are passed individually to the text encoder, so there is no sharing of concepts or arrangement distribution.

For example, this is not a good prompt:

a formidable
AND_SALT [ flying castle ]
AND_PERP [ with fans as propeller ]

Because the text encoder only sees a formidable and flying castle individually before being passed to the model individually, where you might have wanted instead for a formidable flying castle to be passed together. Also, the last prompt with fans as propeller would benefit starting with a castle, i.e. a castle with fans as propeller. This is because, even though this prompt only contributes to a fraction of the final image (say, 1/3), the unet still generates noises of the size of the final image. Some regions will be more important than others, but the region of the generated noise for the prompt is still the entire image.

If you want to target a specific region of the image, I strongly advise using something like regional prompter with this extension. I'm not sure if the 2 extensions are fully compatible at the moment, let me know if there are any issues when combining these tools.

[gilroff]

I understand better now. this method is most useful to prompt for things that SD is reluctant to combine. I tried to add fx and stuff like that and it does a good job. I still have some doubts about how the prompts are combined. when, u say the noise generated is combined to regular prompt, whats the noise like? is it like an image or is it like information of the concept in an 'AND_XX' prompt? what actually happens in simple terms if that's possible?
Btw thanks so much for the detailed explanations :)

[ljleb]

When the webui generates an image from text, it starts with a fully noisy image. At each step of the denoising process, the model generates the noise to remove from the noisy image based on the given prompt at that step, so that python can take it out of the current noisy image to make it less noisy and more like a normal image. The size of these generated noises is exactly the size of the initial noise.

When I say "noise" above, I refer to this delta of the current image at each step.

Each prompt separated by a keyword will have exactly one noise associated to it, which you can think of as letting the model interpret what to do with the current noisy image and prompt.

In other words, the noises are the differences used to get to the next consecutive step of the generation process.

The extension will take all the noises generated from the prompt at a particular step, add the current noisy image to each of them, subtract the negative denoised image, combine the resulting differences using the rules I described above, and then apply CFG so that there is a single final denoised image remaining.

Let me know if that is still not clear enough.

[gilroff]

Yes I understand much better now. thanks a lot :)

[ljleb]

So I recently learned that all of this happens in denoised space. No noise is ever directly merged together: only partially formed images are. I updated the readme to remove the word "noise" in accordance. Apologies for slightly misleading you on this point!