[QUESTION] Performance of SKImage via SKCodec and SKSurface vs. SKBitmap

[AndersMad]

Q1) I use SKCodec to read image (need EncodedOrigin), then create and SKImage from that using SKImage.FromPixelCopy(codec.info, codec.Pixels). Question: Is there a faster way of doing this? As I read it as a double up on memory there even though data is immutable. E.g. if SKCodec could expose the data using e.g. ReadOnlySpan<>.

Q2) Also i use SKSurface for drawing (mostly scaling and filters) instead of SKBitmap as that is newer/better - true? Or is that only true when GPU is available (I am doing all on server/asp.net).

[mattleibow]

So, for Q1, the reason the codec needs to be copied is because that is more of a decode operation. I think I might need to deprecate the Pixels property as that is sort of a lie... It is actually decoding the image right then and there. For each call. A total waste.

The better way would be to either call Pixels once and pin the array, or actually not use it. The issue is that you have no control over how the pixels are decoded. It may come in a format that is not particularly useful. Or, it may be in a state that on every draw of the image, it has to be converted.

Finally, SKImage.FromPixelCopy makes a fresh copy of that data.

A good way would be to just use GetPixels to a new image:

// load
using var codec = SKCodec.Create(path);
var sourceInfo = codec.Info;

// prepare new image
var destInfo = sourceInfo.WithColorType(SKImageInfo.PlatformColorType);
using var image = SKImage.Create(destInfo);
using var pixmap = image.PeekPixels();
var pixelAddress= pixmap.GetPixels();

// read
var result = codec.GetPixels(destInfo, pixelAddress);

Full code:

using var codec = SKCodec.Create(pathToImage);

// get things
var origin = codec.EncodedOrigin;
Console.WriteLine("Origin: " + origin);
var sourceInfo = codec.Info;
Console.WriteLine(
	$"W: {sourceInfo.Width}, " +
	$"H: {sourceInfo.Height}, " +
	$"CT: {sourceInfo.ColorType}, " +
	$"AT: {sourceInfo.AlphaType}, " +
	$"CS.G{sourceInfo.ColorSpace.NamedGamma}, " +
	$"CS.T{sourceInfo.ColorSpace.Type}");

// make sure the color type is the best for drawing
var destInfo = sourceInfo.WithColorType(SKImageInfo.PlatformColorType);
Console.WriteLine(
	$"W: {destInfo.Width}, " +
	$"H: {destInfo.Height}, " +
	$"CT: {destInfo.ColorType}, " +
	$"AT: {destInfo.AlphaType}, " +
	$"CS.G{destInfo.ColorSpace.NamedGamma}, " +
	$"CS.T{destInfo.ColorSpace.Type}");

// create the container image
using var image = SKImage.Create(destInfo);

// get the pixmap so e can get the pointer to the memory location
using var pixmap = image.PeekPixels();
var pixelAddress= pixmap.GetPixels();

// get pixels
var result = codec.GetPixels(destInfo, pixelAddress);
Console.WriteLine($"Decode result: {result}");

[mattleibow]

For Q2, yes, SKImage or SKSurface is the new way to do things.

SKImage is used for the representation of an image that can be moved from CPU to/from GPU as a unit. Or loaded from a file.

SKSurface is used for drawing to.

But, all this doesn't really matter. A surface can wrap an image. Both can wrap a chunk of random memory. But the typical use is images provide pixels to draw, surfaces provide a place to draw on. For raster, this is all the same, but for GPU, this matters because you can't really draw on a GPU image.

[AndersMad]

thanks for the detailed answer!..

Q1: in the var destInfo = sourceInfo.WithColorType(SKImageInfo.PlatformColorType); should it have this too?

if (destInfo.AlphaType == SKAlphaType.Unpremul)
	destInfo.AlphaType = SKAlphaType.Premul;
destInfo.ColorSpace = (SKColorSpace)null;

Q2: when I get the time will try measure the diff between:

using var outputRaster = new SKBitmap(info);
using var canvas = new SKCanvas(outputRaster));
...
outputRaster.Encode(...);

vs:

using var surface = SKSurface.Create(info);
using var canvas = surface.Canvas);
...
using var outputRaster = surface.Snapshot();
outputRaster.Encode(...);

[AndersMad]

maybe this 3. way is the fastest/best way (mem wise)?

using var outputRaster = SKImage.Create(info);
using var pixmap = outputRaster.PeekPixels();
using var surface = SKSurface.Create(info, pixmap.GetPixels());
using var canvas = surface.Canvas;
...
outputRaster.Encode(...);

[mattleibow]

Q1: in the var destInfo = sourceInfo.WithColorType(SKImageInfo.PlatformColorType); should it have this too?

Probably the alpha type, yes.
The colorspace, you might want to keep it. Might be related to color things like this: #931 (comment)

maybe this 3. way is the fastest/best way (mem wise)?

Hmmm. That will work for raster, yes. Then you can draw directly on the surface. But, you need to make sure you use the correct color/alpha types for best results. The surface is more picky about the combos. But, the defaults usually work.

But, I believe all the implementations of Encode go through the same SKPixmap.Encode path. So, I would say the actual perf would be the object used. I think a SKSurface is a bit more expensive that drawing directly to a bitmap with a canvas.

But, the bitmap can be used in place of an image if it is marked SKBitmap.SetImmutable(). This tells the drawing code to make an image when drawing, but don't copy.

But... again. This is 100% a perf test requirement. could/should/might means nothing. Test some options, and then let us all know.

[AndersMad]

I tried two ways now - the diff seems to be negligible - however I'm baffled about the reduced memory consumption using the image (sharpening) filter:

v1.68.2 canvas via bitmap:
	draw scaled bitmap:
		CPU: 17,555s / avg: 0,176s
		MEM: avg: 383 / min: 254 / max: 525
	.. with sharpening filter:
		CPU: 22,993s / avg: 0,230s
		MEM: avg: 256 / min: 178 / max: 415

v1.68.2 canvas via image peek pixels and surface:
	draw scaled bitmap:
		CPU: 16,932s / avg: 0,169s
		MEM: avg: 390 / min: 254 / max: 485
	.. with sharpening filter:
		CPU: 22,564s / avg: 0,226s
		MEM: avg: 254 / min: 194 / max: 449
		
v1.68.2 canvas from surface with snapshot:
	draw scaled bitmap:
		CPU: 17,070s / avg: 0,171s
		MEM: avg: 386 / min: 254 / max: 476
	.. with sharpening filter:
		CPU: 22,205s / avg: 0,222s
		MEM: avg: 259 / min: 186 / max: 440

I did not test the alpha as I could not get it to work with LinqPad5

[mattleibow]

Good to know the difference is negligible. It is a bit weird when drawing with the filter. What does the test source look like?

[AndersMad]

ran this fragment in LINQPad 5:

if (sharpen) {
	using (var paint = new SKPaint { FilterQuality = SKFilterQuality.High, IsAntialias = false, }) {
		var kernel = new[] { 0, -.1f, 0, -.1f, 1.4f, -.1f, 0, -.1f, 0, };

		using (var cropRect = new SKImageFilter.CropRect(codec.Info.Rect, SKCropRectFlags.HasAll)) {
			paint.ImageFilter = SKImageFilter.CreateMatrixConvolution(
					new SKSizeI(3, 3), kernel, 1f, 0f, new SKPointI(1, 1),
					SKMatrixConvolutionTileMode.Clamp, false, cropRect: cropRect);
			canvas.DrawImage(bmp, SKRect.Create(0, 0, w, h), SKRect.Create(0, 0, newWidth, newHeight), paint);
		}
	}
}
else {
	canvas.DrawImage(bmp, SKRect.Create(0, 0, w, h), SKRect.Create(0, 0, newWidth, newHeight));
}