Can't replicate image generation after update

[krones9000]

Hi all,

I recently updated my version of Forge which I'd been avoiding due to previously breaking installations.

I was on: f0.0.17v1.8.0rc-latest-276-g29be1da7

and have updated to: f2.0.1v1.10.1-previous-483-g33b5b9d2

Everything worked fine and I can now run Flux within Forge which was the purpose of the update.

HOWEVER, I can no longer replicate images using the same image metadata. Origin of the test images is this post just for clarity lol.

This is the image made using the old Forge install:

The image metadata is as follows:

score_9,score_8_up,score_7_up,1girl, (MARCILLEDONATO:1.2), BLONDE HAIR, SINGLE BRAID ,covered in green spiders, witch ,Billie Eilish, fantasy witch, witch, (half shot:1.4), BREAK witches hovel, fantasy house, potions background,spiders, (green spiders:1.2), swarm,
lora:darkstyle:1
lora:Marcille_PonyXL:1

Negative prompt: score_6, score_5, score_4, rating_explicit, rating_questionable, source_pony, realistic, photo, 3D, render, 3d, photorealistic, (bats:1.3)

Steps: 40,

Sampler: Euler a,

CFG scale: 7,

Seed: 666666721,

Size: 832x1216,

Model hash: 44d3daf083, Model: 7thAnimeXLPonyA_v10,

VAE hash: 735e4c3a44, VAE: sdxl_vae.safetensors,

Clip skip: 2,

RNG: NV, "*I had a comment regarding this but looks to be deleted, my response: Ahh, that was to do with seed generation (how the "random" number generator works).

It's got to do with how it creates random seeds, NOT anything do to with how the image generates, especially given that I'm specifying a specific seed.*"

Lora hashes: "darkstyle: 8046fb0ecc46, Marcille_PonyXL: d4b765526e19",

Version: f0.0.17v1.8.0rc-latest-276-g29be1da7

And this is an image made using the same spec, imported across into Txt2Img from the png info tab in the new Forge install:

The image metadata is as follows:

score_9,score_8_up,score_7_up,1girl, (MARCILLEDONATO:1.2), BLONDE HAIR, SINGLE BRAID ,covered in green spiders, witch ,Billie Eilish, fantasy witch, witch, (half shot:1.4), BREAK witches hovel, fantasy house, potions background,spiders, (green spiders:1.2), swarm,
lora:darkstyle:1
lora:Marcille_PonyXL:1

Negative prompt: score_6, score_5, score_4, rating_explicit, rating_questionable, source_pony, realistic, photo, 3D, render, 3d, photorealistic, (bats:1.3)

Steps: 40,

Sampler: Euler a,

Schedule type: Automatic,

CFG scale: 7,

Seed: 666666721,

Size: 832x1216,

Model hash: a881029f19, Model: 7thAnimeXLPonyA_v10,

RNG: NV,

Lora hashes: "darkstyle: 8046fb0ecc46, Marcille_PonyXL: d4b765526e19",

Version: f2.0.1v1.10.1-previous-483-g33b5b9d2,

Module 1: sdxl_vae

Highlighted are the bits that I can see are different.

I also don't seem to be able to specify ClipSkip 2 within the "XL" radio selection in the new top right of this interface though it appears that's applied automatically.

I've also set the UI to "All" mode and set everything as before and comes out wrong.

I've also done a grid using all the different "Schedulers" as those are now new options that weren't there before but none of them produce the right output image.

---

Here is an image of my generation UI in the old forge post-completion showing the image generated and the obvious settings:

And here is one in the new one:

Everything is being created with the same models and LoRAs being loaded from the same folders. All launch arguments are identical: --cuda-stream --pin-shared-memory --cuda-malloc and I've run both versions with and without.

---

Am I just being totally stupid?

[krones9000]

Just to add a note of interest as highlighted by a reddit user:

The last two images show new generations I did in both installs just for this post to demonstrate I can still recreate it in the old install. Both models are drawing from the same model source folder with just the 7thAnime model in it.

But each install calculates a different model hash for the exact same model.

I'm not sure if that's significant or why that would occur.

[lllyasviel]

I will take a look this next time I am on forge dev
what will you get if you use the info in original newest a1111?

[krones9000]

I will take a look this next time I am on forge dev what will you get if you use the info in original newest a1111?

So this might have added more confusion but:

I did a fresh install of A1111 and I get a more similar output to the original but still different.

You'll note that the hash for the model in A1111 is the same as the hash in the OLD Forge install...

I also noticed a couple of other differences.

A1111: "Torch 2.1.2+cu121"

NEW Forge: "Torch: 2.3.1+cu121"

OLD Forge: "Torch: 2.1.2+cu121"

Worth noting that A1111 and the OLD Forge install both use the same Torch but don't produce the same output. So it can't JUST be down to Torch if that's even related.

I wondered if this could be related to this post:

https://www.reddit.com/r/StableDiffusion/comments/1f6t09l/different_versions_of_pytorch_produce_different/

---

Though to be clear, this doesn't resolve the model hash issue.

I think there are now 2 questions:

1. Why is a different image being produced in all three instances.
2. Why is the model hash different in latest Forge?

[krones9000]

A couple more tests in A1111 with different schedulers:

SGM Uniform gets close but still not identical to the original.

[lllyasviel]

I will take a look next week.

if possible please give download links to everything you used in making this image

[krones9000]

I will take a look next week.

if possible please give download links to everything you used in making this image

All below. Thanks for your help.

I'm sure you're aware, but for anyone else reading this, NSFW caveats for Civitai, access at your own discretion:

7th anime XL-Pony A Checkpoint: civitai.com/models/395554?modelVersionId=441236

Marcille_PonyXL.safetensors LoRA: civitai.com/models/316750?modelVersionId=515982

DarkstyleXL LoRA: civitai.com/models/107375/pony-xl-and-15-neondark-backgrounds

I had renamed this to darkstyle hence it will be <lora:darkstyleXL:1> in your prompts. I just did a quick test, just to be 1000% sure renaming couldn't be causing it (obviously it shouldn't but just to be sure), and re-downloaded and did the generation with the new <lora:darkstyleXL:1> instead of <lora:darkstyle:1>. Exact same output as before.

[Juqowel]

Can you do a compare test with dpm++ sde?

[krones9000]

Can you do a compare test with dpm++ sde?

I don't think that should work because it's diverging further from how the piece was originally generated. I've done it either way replacing euler_a with 'dpm++ sde' (and with each scheduler). It did not produce the original output:

To return to the two questions I think are now in play:

1. Why is a different image being produced in all three instances.
2. Why is the model hash different in latest Forge?

Regarding 1.: I've now got a much clearer understanding of the process of creating the images and that actually there's quite a lot of things that can vary system-to-system and install-to-install that might be introducing the variance. That's not necessarily to say that 1 is "solved" (see below) but is more explainable as possibly normal, as far as I understand now. So I shouldn't necessarily expect the same image to be produced by the same input, especially if things like Torch versions have change.

Regarding 2.: this seems weird to me still. The model hash should be the same. Especially given that testing it using A1111 confirms that the model does have the correct hash. Does the new Forge version hash the model before or after it loads it into VRAM and could that be causing this? I understand hashing, but not enough about how and when Forge does it to really grasp why you'd get a different hash. Unless the hashing process itself has changed. If the hashing process remains the same, but when/how this occurs is now different, then perhaps that helps to explain some of the variation in the image output as well, even if some smaller variation is still to be expected either way?

edit:

So I guess possibly it is just variation between the generation processes and perhaps the hashing thing is related to this: #1647 ?

[Juqowel]

I read everything again more carefully.

RNG: NV, "*I had a comment regarding this but looks to be deleted, my response: Ahh, that was to do with seed generation (how the "random" number generator works).

It's got to do with how it creates random seeds, NOT anything do to with how the image generates, especially given that I'm specifying a specific seed.*"

No. It's important for how the image generates.

But anyway I can't reproduce your image at the new forge... Maybe some little thing in settings differs.

And this with CPU RNG:

[krones9000]

No. It's important for how the image generates.

Interesting, thank you. I'd clearly misunderstood it to be about seed progression rather than generation well. As you say though, testing either still seems to yield a different outcome (same on my end).

I think we're establishing it's basically within the norms of expected variation. I tested a couple of other models against their provided example images and found similar results (nearly identical images but with minor variations).

[KiraSlith]

Seconding this issue, with a fix. Saving settings with CLIP above 1 in the top causes it to get stuck as 1 no matter how much you rattle the slider from then onwards.
Fix: Set it back to 1, go to settings, click "Apply Settings" (It says nothing was changed but it's lying, I was just as confused by that), then go back to txt2img and increase CLIP to whatever you want it to be, from then on it will respect your CLIP as long as you don't save with it above 1 again.

[krones9000]

Seconding this issue, with a fix. Saving settings with CLIP above 1 in the top causes it to get stuck as 1 no matter how much you rattle the slider from then onwards. Fix: Set it back to 1, go to settings, click "Apply Settings" (It says nothing was changed but it's lying, I was just as confused by that), then go back to txt2img and increase CLIP to whatever you want it to be, from then on it will respect your CLIP as long as you don't save with it above 1 again.

I was unable to verify if your description is accurate or not but I followed the steps either way.

So I'm just add that, after following your steps regarding CLIP, the image output remains as shown in #1650 (comment) using the new Forge install. I've already said that I think the image divergence is probably within acceptable norms now based on what I've learnt. But still useful to confirm that, if there's an issue with CLIP as you described, it does not appear to be causing or even contributing to the divergences in my image output.