I have a <canvas> that I'm updating every 100 ms with bitmap image data coming from a HTTP request:

var ctx = canvas.getContext("2d");

setInterval(() => {
    fetch('/get_image_data').then(r => r.arrayBuffer()).then(arr => {
        var byteArray = new Uint8ClampedArray(arr);
        var imgData = new ImageData(byteArray, 500, 500);
        ctx.putImageData(imgData, 0, 0);
    });
}, 100);

This works when /get_image_data gives RGBA data. In my case, since alpha is always 100%, I don't send A channel through the network. Question:

• how to efficiently do this when the request delivers RGB binary data?
• and also when the request delivers grayscale binary data?

(Can we avoid a for loop which might be slow in Javascript for megabytes of data 10 times per second?)

Example in the grayscale => RGBA case: each input value ..., a, ... should be replaced by ..., a, a, a, 255, ... in the output array.

Here is a pure JS solution: ~10 ms for a 1000x1000px grayscale => RGBA array conversion.

CodePudding user response：

Converting an ArrayBuffer from RGB to RGBA is conceptually straightforward: just splice in an opaque alpha channel byte (255) after every RGB triplet. (And grayscale to RGBA is just as simple: for every gray byte: copy it 3 times, then insert a 255.)

The (slightly) more challenging part of this problem is offloading the work to another thread with wasm or a worker.

Because you expressed familiarity with JavaScript, I'll provide an example of how it can be done in a worker using a couple of utility modules, and the code I'll show will use TypeScript syntax.

On the types used in the example: they are very weak (lots of anys) — they're present just to provide structural clarity about the data structures involved in the example. In strongly-typed worker application code, the types would need to be re-written for the specifics of the application in each environment (worker and host) because all types involved in message passing are just contractual anyway.

Task-oriented worker code

The problem in your question is task-oriented (for each specific sequence of binary RGB data, you want its RGBA counterpart). Inconveniently in this case, the Worker API is message-oriented rather than task-oriented — meaning that we are only provided with an interface for listening for and reacting to every single message regardless of its cause or context — there's no built-in way to associate a specific pair of messages to-and-from a worker. So, the first step is to create a task-oriented abstraction on top of that API:

task-worker.ts:

export type Task<Type extends string = string, Value = any> = {
  type: Type;
  value: Value;
};

export type TaskMessageData<T extends Task = Task> = T & { id: string };

export type TaskMessageEvent<T extends Task = Task> =
  MessageEvent<TaskMessageData<T>>;

export type TransferOptions = Pick<StructuredSerializeOptions, 'transfer'>;

export class TaskWorker {
  worker: Worker;

  constructor (moduleSpecifier: string, options?: Omit<WorkerOptions, 'type'>) {
    this.worker = new Worker(moduleSpecifier, {...options ?? {}, type: 'module'});

    this.worker.addEventListener('message', (
      {data: {id, value}}: TaskMessageEvent,
    ) => void this.worker.dispatchEvent(new CustomEvent(id, {detail: value})));
  }

  process <Result = any, T extends Task = Task>(
    {transfer, type, value}: T & TransferOptions,
  ): Promise<Result> {
    return new Promise<Result>(resolve => {
      const id = globalThis.crypto.randomUUID();

      this.worker.addEventListener(
        id,
        (ev) => resolve((ev as unknown as CustomEvent<Result>).detail),
        {once: true},
      );

      this.worker.postMessage(
        {id, type, value},
        transfer ? {transfer} : undefined,
      );
    });
  }
}

export type OrPromise<T> = T | Promise<T>;

export type TaskFnResult<T = any> = { value: T } & TransferOptions;

export type TaskFn<Value = any, Result = any> =
  (value: Value) => OrPromise<TaskFnResult<Result>>;

const taskFnMap: Partial<Record<string, TaskFn>> = {};

export function registerTask (type: string, fn: TaskFn): void {
  taskFnMap[type] = fn;
}

export async function handleTaskMessage (
  {data: {id, type, value: taskValue}}: TaskMessageEvent,
): Promise<void> {
  const fn = taskFnMap[type];

  if (typeof fn !== 'function') {
    throw new Error(`No task registered for the type "${type}"`);
  }

  const {transfer, value} = await fn(taskValue);

  globalThis.postMessage(
    {id, value},
    transfer ? {transfer} : undefined,
  );
}

I won't over-explain this code: it's mostly just about picking and moving properties between objects so you can avoid all that boilerplate in your application code. Notably: it also abstracts the necessity of creating unique IDs for every task instance. I will talk about the three exports:

• a class TaskWorker: For use in the host — it is an abstraction over instantiating a worker module and exposes the worker on its worker property. It also has a process method which accepts task information as an object argument and returns a promise of the result of processing the task. The task object argument has three properties:

  • type: the type of task to be performed (more on this below). This is simply a key that points to a task processing function in the worker.
  • value: the payload value that will be acted on by the associated task function
  • transfer: an optional array of transferable objects (I'll bring this up again later)

• a function registerTask: For use in the worker — sets a task function to its associated type name in a dictionary so that the worker can use the function to process a payload when a task of that type is received.

• a function handleTaskMessage: For use in the worker — this is simple, but important: it must be assigned to self.onmessage in your worker module script.

Efficient conversion of RGB (or grayscale) to RGBA

The second utility module has the logic for splicing the alpha bytes into the RGB data, and there's also a function for conversion from grayscale to RGBA:

rgba-conversion.ts:

/**
 * The bytes in the input array buffer must conform to the following pattern:
 *
 * ```
 * [
 *   r, g, b,
 *   r, g, b,
 *   // ...
 * ]
 * ```
 *
 * Note that the byte length of the buffer **MUST** be a multiple of 3
 * (`arrayBuffer.byteLength % 3 === 0`)
 *
 * @param buffer A buffer representing a byte sequence of RGB data elements
 * @returns RGBA buffer
 */
export function rgbaFromRgb (buffer: ArrayBuffer): ArrayBuffer {
  const rgb = new Uint8ClampedArray(buffer);
  const pixelCount = Math.floor(rgb.length / 3);
  const rgba = new Uint8ClampedArray(pixelCount * 4);

  for (let iPixel = 0; iPixel < pixelCount; iPixel  = 1) {
    const iRgb = iPixel * 3;
    const iRgba = iPixel * 4;
    // @ts-expect-error
    for (let i = 0; i < 3; i  = 1) rgba[iRgba   i] = rgb[iRgb   i];
    rgba[iRgba   3] = 255;
  }

  return rgba.buffer;
}

/**
 * @param buffer A buffer representing a byte sequence of grayscale elements
 * @returns RGBA buffer
 */
export function rgbaFromGrayscale (buffer: ArrayBuffer): ArrayBuffer {
  const gray = new Uint8ClampedArray(buffer);
  const pixelCount = gray.length;
  const rgba = new Uint8ClampedArray(pixelCount * 4);

  for (let iPixel = 0; iPixel < pixelCount; iPixel  = 1) {
    const iRgba = iPixel * 4;
    // @ts-expect-error
    for (let i = 0; i < 3; i  = 1) rgba[iRgba   i] = gray[iPixel];
    rgba[iRgba   3] = 255;
  }

  return rgba.buffer;
}

I think the iterative math code is self-explanatory here (however — if any of the APIs used here or in other parts of the answer are unfamiliar — MDN has explanatory documentation). I think it's noteworthy to point out that both the input and output values (ArrayBuffer) are transferable objects, which means that they can essentially be moved instead of copied between the host and worker contexts for improved memory and speed efficiency.

Also, thanks @Kaiido for providing information that was used to improve the efficiency of this approach over a technique used in an earlier revision of this answer.

Creating the worker

The actual worker code is pretty minimal because of the abstractions above:

worker.ts:

import {
  rgbaFromGrayscale,
  rgbaFromRgb,
} from './rgba-conversion.js';
import {handleTaskMessage, registerTask} from './task-worker.js';

registerTask('rgb-rgba', (rgbBuffer: ArrayBuffer) => {
  const rgbaBuffer = rgbaFromRgb(rgbBuffer);
  return {value: rgbaBuffer, transfer: [rgbaBuffer]};
});

registerTask('grayscale-rgba', (grayscaleBuffer: ArrayBuffer) => {
  const rgbaBuffer = rgbaFromGrayscale(grayscaleBuffer);
  return {value: rgbaBuffer, transfer: [rgbaBuffer]};
});

self.onmessage = handleTaskMessage;

All that's needed in each task function is to move the buffer result to the value property in the return object and to signal that its underlying memory can be transferred to the host context.

Example application code

I don't think anything will surprise you here: the only boilerplate is mocking fetch to return an example RGB buffer since the referenced server in your question isn't available to this code:

main.ts:

import {TaskWorker} from './task-worker.js';

const tw = new TaskWorker('./worker.js');

const buf = new Uint8ClampedArray([
  /* red */255, 0, 0, /* green */0, 255, 0, /* blue */0, 0, 255,
  /* cyan */0, 255, 255, /* magenta */255, 0, 255, /* yellow */255, 255, 0,
  /* white */255, 255, 255, /* grey */128, 128, 128, /* black */0, 0, 0,
]).buffer;

const fetch = async () => ({arrayBuffer: async () => buf});

async function main () {
  const canvas = document.createElement('canvas');
  canvas.setAttribute('height', '3');
  canvas.setAttribute('width', '3');

  // This is just to sharply upscale the 3x3 px demo data so that
  // it's easier to see the squares:
  canvas.style.setProperty('image-rendering', 'pixelated');
  canvas.style.setProperty('height', '300px');
  canvas.style.setProperty('width', '300px');

  document.body
    .appendChild(document.createElement('div'))
    .appendChild(canvas);

  const context = canvas.getContext('2d', {alpha: false})!;

  const width = 3;

  // This is the part that would happen in your interval-delayed loop:
  const response = await fetch();
  const rgbBuffer = await response.arrayBuffer();

  const rgbaBuffer = await tw.process<ArrayBuffer>({
    type: 'rgb-rgba',
    value: rgbBuffer,
    transfer: [rgbBuffer],
  });

  // And if the fetched resource were grayscale data, the syntax would be
  // essentially the same, except that you'd use the type name associated with
  // the grayscale task that was registered in the worker:

  // const grayscaleBuffer = await response.arrayBuffer();

  // const rgbaBuffer = await tw.process<ArrayBuffer>({
  //   type: 'grayscale-rgba',
  //   value: grayscaleBuffer,
  //   transfer: [grayscaleBuffer],
  // });

  const imageData = new ImageData(new Uint8ClampedArray(rgbaBuffer), width);
  context.putImageData(imageData, 0, 0);
}

main();

Those TypeScript modules just need to be compiled and the main script run as a module script in your HTML.

I can't make performance claims without access to your server data, so I'll leave that to you. If there's anything that I overlooked in explanation (or anything that's still not clear), feel free to ask in a comment.

CodePudding user response：

For completeness, here is a pure JS version.

1000 x 1000 px grayscale array → RGBA array

~ 9 or 10 milliseconds on my machine.

Can we do better with WASM or other techniques?

var width = 1000, height = 1000;
var array = new Uint8Array(width*height).fill().map(() => Math.round(Math.random() * 255))
var ctx = document.getElementById("canvas").getContext("2d");
grayscale_array_to_canvas(array, width, height, ctx);

function grayscale_array_to_canvas(array, width, height, ctx) {
    var startTime = performance.now();
    var rgba = new Uint8ClampedArray(4*width*height);
    for (var i = 0; i < width*height; i  ) {
        rgba[4*i] = array[i];
        rgba[4*i 1] = array[i];
        rgba[4*i 2] = array[i];
        rgba[4*i 3] = 255;
    }    
    console.log(`${performance.now() - startTime} ms`);    
    var imgData = new ImageData(rgba, width, height);
    ctx.putImageData(imgData, 0, 0);
}

<canvas id="canvas"></canvas>