New Quant: flux1-dev-bnb-nf4-v2.safetensors

[lllyasviel]

See also: https://huggingface.co/lllyasviel/flux1-dev-bnb-nf4

https://huggingface.co/lllyasviel/flux1-dev-bnb-nf4/blob/main/flux1-dev-bnb-nf4-v2.safetensors

---

Copied from readme:

Always use V2 by default.

V2 is quantized in a better way to turn off the second stage of double quant.

V2 is 0.5 GB larger than the previous version, since the chunk 64 norm is now stored in full precision float32, making it much more precise than the previous version. Also, since V2 does not have second compression stage, it now has less computation overhead for on-the-fly decompression, making the inference a bit faster.

The only drawback of V2 is being 0.5 GB larger.

[rakib91221]

please add a support flux 6gb model.. thanks

[yamfun]

I was trying out the difference of Queue/Async/CPU/shared to my speed and then this post appeared and so I downloaded and git pulled again to try some speed of v2 too. But perhaps the options and models difference will matter to 8gb but not 12gb as all the speed seem similar.
NV 4070 12gb, 32gb, Intel i5-12400f, GPU weights 11257, cmd args: --disable-xformers (to avoid None Type error)

flux1-dev-bnb-nf4:
batch count 1
Auto / Queue / CPU - speed: 1.68s / taken: 40s
Auto / Queue / Shared - speed: 1.68s / taken: 40s
Auto / Async / Shared - speed: 1.73s / taken: 41s
Auto / Async / CPU - speed: 1.72s / taken: 40s

batch count 4
Auto / Queue / CPU : taken: 2 min. 25.1 sec.
Auto / Async / CPU : taken 2 min. 23.5 sec.

flux1-dev-bnb-nf4-v2:
batch count 4
Auto / Queue / CPU : taken: 2 min. 26.1 sec.
Auto / Async / Shared : taken: 2 min. 28.9 sec.

batch count 2 batch size 2
Auto / Queue / CPU : taken: 2 min. 22.6 sec.

if anyone's 4070 gives faster it/s, please share your secret to me

[RamonGuthrie]

Does this model now work with Loras?

[kroketagamer]

Not

[newxhy]

这个模型国内下载真的好慢，fuck GFW!

[newxhy]

国内度盘地址：https://pan.baidu.com/s/1BoDl5Bauwz2pSg0hpimxnA?pwd=rmqm

[gmf1982]

其他模型我都是用迅雷直接在huggingface下载的，速度有十几M/秒，这个模型不知为什么不能直接下载，你们可以用国内镜像下载，速度飞快！
https://hf-mirror.com/lllyasviel/flux1-dev-bnb-nf4/resolve/main/flux1-dev-bnb-nf4-v2.safetensors

[cmcjas]

I've noticed a noticeable increased in VRAM usage with v2, for example, generating a relatively high res image 1792x1008, v1 was using around 7.4GB VRAM on my 4060, while v2 gave me OOM error, and the VRAM usage went up to like 8.8GB VRAM usage causing some slow down in inference speed.

[cmcjas]

After carried out some testing at lower resolutions to get the VRAM usage under control, in my case, it seems v2 performs on par with v1, maybe a couple of seconds faster. However, on average, v2 used up to 500-600mb more VRAM in comparison to v1.

[a3nima]

How about image quality? I dont know how "more precision" translates to the output image? Can you compare?

[cmcjas]

How about image quality? I dont know how "more precision" translates to the output image? Can you compare?

Basically the same between the two. Composition remains unchanged at least in my case, although I found that v2 does a slightly better job at capturing small finer details, otherwise it's hard to tell the difference between the two. At this point, I'm not sure if it's worth it to go with v2 due to its increased VRAM demand

[LoneVit]

Does "Distilled CFG Scale" slider works for anyone? I changing it (with fixed seed and CFG Scale=1), but generated image always stay the same. Both checkpoints flux1-dev-bnb-nf4 and flux1-dev-bnb-nf4-v2.

version: f2.0.1v1.10.1-previous-260-gaadc0f04  •  python: 3.10.6  •  torch: 2.3.1+cu121  •  xformers: N/A  •  gradio: 4.40.0

[iKurama]

No, with CFG 1 & Distilled cfg anything between 0 and 120, the image looks the same, when using the same seed.
This happens on the regular flux dev version too. It also happens when using a lora, so there's no placebo with that :P

[belenkoksas]

@lllyasviel for the President 2024!

my 3080 with new v2 works ~2s faster, for rezults need more comparison but looks like better also

[RoninReilly]

How long does it take to generate 1 image? I have rtx 3080 +32 ram generation on standard settings, it takes ~ 35 seconds

[ZeroCool22]

Without clothes now, please.

[kusobabaa29]

My comparison between NF4 V1 ,V2 and FP8. ( Euler + Beta, Seed - 114, 1024 x 1024, CFG - 4)
NF4 and FP8 might not be very suitable for comparison because in complex scenes, the composition itself can change.

Nikon Z8. The image shows two boxers in a boxing ring engaged in a fight. The boxer on the left is wearing white shorts and red boxing gloves and is throwing a punch at the other boxer who is wearing black shorts and white gloves. The field shows the ropes of the ring and a referee in the middle of the fight. Above the ring there is a red neon sign with a heart-shaped design. The image appears to have been taken during a boxing match.

[cmcjas]

Thanks for the comparison. I didn't know the composition would change that much with FP8, too bad FP8 is not usable on my laptop due to lack of vram/ram. Personally, I think for those with lower specs machines, it's best to stick with NF4 V1, as it's less demanding than V2 in terms ram/vram usage, while the two are practically the same in quality.

[mirek190]

nf4-v2 comparing to nf4-v1 seems has better details and object have more real sense. Look on their foots , ears, etc

But fp8 seems much better ... I know co position has been changed for no reason but just looks better in details.

[HMRMike]

But fp8 seems much better ... I know co position has been changed for no reason but just looks better in details.

So far in my checks it looks like fp8 is much closer in composition to fp16. So considering fp16 as a "ground truth", it's nf4 that deviates the most.

[mirek190]

Have to test if is better than nf4v1 where is loosing a lot details comparing to fp8.

[FLYleoYBQ]

第二张是最新的NF4_V2生成的图，它对手指的理解反而变差了。
有趣的是V1版本的细节虽然比不上V2，但是"狡猾"的V1似乎更加善于简化。O(∩_∩)O

[FLYleoYBQ]

当我换了其他采样器，手的问题又解决了

[Woukim]

之後您改用哪種取樣器？

[FLYleoYBQ]

最后经过大量测试后，我发现保持默认采样器，然后把Guidance设置为3.7的效果最好。默认它是3.5

[Myoko]

最后经过大量测试后，我发现保持默认采样器，然后把Guidance设置为3.7的效果最好。默认它是3.5

其他的采样器 + Guidance 3.7 有测试过吗？ 默认Euler 效果最好？

[Njasa2k]

@lllyasviel is there a quant like this for schnell?

[Yuliucheng]

https://huggingface.co/silveroxides/flux1-nf4-weights/tree/main

[RamonGuthrie]

No that Schnell version is not nf4 v2

[gi0baro]

jfyi, on simple prompts it seems nf4v2 is somewhat more balanced; when compared to fp8 the composition is still very different.
Also, no noticeable change in generation speed on a 3090ti.

[ashuoAI]

Waiting for more tests from the gods

[HMRMike]

fp16/8/nf4V2/nf4 series on RTX3090, shared/async swap.
fp16 used ae, clip_l, t5 fp8. the rest (all from Illyasviel links) used whatever was baked in.

Can't comment on memory usage in this case (because of async?), and maybe some leak kept increasing usage (which eventually reached full VRAM for fp16 and fp8 but simply slowed to a crawl instead of errorring out). Best case for fp16 was 18GB VRAM, worst was OOM error, sometimes straight on the first generation after launching the UI, I can't reproduce this behavior reliably, feels almost random.

Best case speed:
fp16: 1.8s/it
fp8: 1.37s/it
nf4v2: 1.35s/it
nf4: 1.35s/it

fp8 stayed closest to fp16 in general composition, nf4 variants significantly different but extremely similar to each other. Although some objects from fp16 appear in nf4 but not in fp8 (billboard obviously, also some tree branches). Only fp16 didn't fail the text (it did fail with 20 steps, that's the only reason for using 30) but this is not the rule, in other prompts fp16 failed and fp8 didn't. nf4v2 is the only one that produced damaged letters, even though fp8 added nonsense- it was readable at least.

Very fancy detailed photorealistic forest, mountains in background, with large sign in ground that says "this is a text test", five old people standing for a group photo
Steps: 30, Sampler: Euler, Schedule type: Simple, CFG scale: 1, Distilled CFG Scale: 3.5, Seed: 12345, Size: 1024x1024

Short prompt:
cat ice cream
in this case interestingly it's fp8 that deviated a lot, and nf4 was closer to fp16

[danilomaiaweb]

I was testing a merged version of schnell+dev and was getting good results with only 4 steps and an average of 18s to generate each image.

Now I'm testing this new version dev-bnb-nf4-v2 and I only get reasonably good images with 10 steps, which takes up to 50s to generate the image.

A significant increase in generation time.

How many steps are you using this new version to get a reasonable image?

My setup is:
Windows 11 Pro
AMD64 Rayzen 5 3500X
RTX 3050 8GB
32GB RAM
Latest Forge version f2.0.1v1.10.1-previous-252-g00f1cd36
Python 3.10.6
Torch 2.3.1+cu121
Cuda 12.1
Cudnn 8907
Diffusers 0.29.2
Transformers 4.44.0

Please leave your comments
How many steps are you using with dev-bnb-nf4-v2

Thank in advance
Best regards
Brazil

[Njasa2k]

where did you get the schnell-dev merger with nf4 quantization?

[gohan2091]

Comparison between different Distilled CFGs using the nf4 (v1) model: https://www.dropbox.com/scl/fi/ewdrxrggz8u01ld681emh/CFG-Flux.jpg?rlkey=qppaj6mfsn2s3usfzktm0nck8&st=bg0hgs2e&dl=0

[lonngxiang]

how to use with diffusrrs？

[aggregate15]

Can someone please explain what all of these different versions of Flux models are ? For this NF4 alone, there's so many other versions like GGUF4, BNB4, Schnell_Dev, BNB_NF4, GGUF4_K, GGUF4_K_S, etc with them and the more I search the more there are. All with different file sizes. What is the difference between all of these in term of speed and Quality and which ones are best optimized for speed and which ones for Quality ? Is there some article or page that explains these in simple terms ?

[aggregate15]

Thank you so much, both, for the great explanation. Really appreciated and much more clear now. Just one more question please, when you say quality, what exactly does that mean ? Better prompt adherence ? Or better details ? I'm really looking for a model that gives good prompt adherence, since quality can be pretty much amended by loras. Any recommendation on this part ?

[lllyasviel]

Precision: fp16 >> Q8 > Q4
Precision For Q8: Q8_K (not available) >Q8_1 (not available) > Q8_0 >> fp8
Precision For Q4: Q4K_S >> Q4_1 > Q4_0
Precision NF4: between Q4_1 and Q4_0, may be slightly better or worse since they are in different metric system
Speed (if not offload, e.g., 80GB VRAM H100) from fast to slow: fp16 ≈ NF4 > fp8 >> Q8 > Q4_0 >> Q4_1 > Q4K_S > others
Speed (if offload, e.g., 8GB VRAM) from fast to slow: NF4 > Q4_0 > Q4_1 ≈ fp8 > Q4K_S > Q8_0 > Q8_1 > others ≈ fp16

[HMRMike]

Thank you so much, both, for the great explanation. Really appreciated and much more clear now. Just one more question please, when you say quality, what exactly does that mean ? Better prompt adherence ? Or better details ? I'm really looking for a model that gives good prompt adherence, since quality can be pretty much amended by loras. Any recommendation on this part ?

Quality will be judged by detail and lack of weird deformities in my opinion.
Overall the choice would be the highest bits your VRAM can tolerate, unless time is critical for you then go for fp8 and nf4. Also Flux is quite good at lower resolution, so quickly generating tests at even 512x512 then upscaling the best, is a decent option.

But prompt adherence will also be affected by the training and text encoders, and original Flux just doesn't "know" many concepts and styles which would be problematic legally or too NSFW or just not popular enough. That's where LORAs come in but for more flexibility we need trained checkpoints and other solutions. For example Nepotism FUX uses fp8 and includes their own CLIP so it responds differently, better, to prompts that suit this model's purpose. Some just merge a ton of LORAs into a checkpoint to suit their needs (but this might cause weird visual issues). It's all very new and something pops up every day so whatever I recommend now might be vastly inferior tomorrow.

[lunar-studio]

@HMRMike Which Q8 are you using? I tend to use Dev-fp8.

[HMRMike]

For original Flux, I mean the Q8 from here:
https://huggingface.co/city96/FLUX.1-dev-gguf/tree/main

But lately many custom models turn to fp8 as "standard", some decide otherwise, it's a bit of a wild mess right now.

[Kovhel1]

Hello! Im very green! I try to start using flux1-dev-bnb-nf4 with python, but I cantt understand. Its really difficult for me.
I try different ways, fo example, with FluxTransformer2DModel.from_single_file. I get error. I also try to use FluxPipeline.from_pretrained, but I don`know, how to write config files.
I know about ae.safetensors, clip_l.safetensors and t5xxl_fp8_e4m3fn.safetensors
I try to do smth like this and have a lot of mistakes

#scheduler = FlowMatchEulerDiscreteScheduler.from_pretrained(bfl_repo, subfolder="scheduler", revision=revision)
text_encoder = CLIPTextModel.from_pretrained("./flux/clip", torch_dtype=dtype)
tokenizer = CLIPTokenizer.from_pretrained("./flux/clip", torch_dtype=dtype)
text_encoder_2 = T5EncoderModel.from_pretrained('./flux/encoder', subfolder="text_encoder_2", torch_dtype=dtype, revision=revision)
tokenizer_2 = T5TokenizerFast.from_pretrained('./flux/encoder', subfolder="tokenizer_2", torch_dtype=dtype, revision=revision)
vae = AutoencoderKL.from_pretrained('./flux/vae', subfolder="vae", torch_dtype=dtype, revision=revision)
transformer = FluxTransformer2DModel.from_pretrained('./flux/transformer', subfolder="transformer", torch_dtype=dtype, revision=revision)


pipe = FluxPipeline(
    scheduler=scheduler,
    text_encoder=text_encoder,
    tokenizer=tokenizer,
    text_encoder_2=text_encoder_2,
    tokenizer_2=tokenizer_2,
    vae=vae,
    transformer=transformer,
)

Can someone help me?