Getting ImageData/HTMLImageElement/HTMLCanvasElement object from image url

Question

I'm trying to develop a program that classifies images using tensorflow and knn-classifier. However, in order to train the program, I need to get images as either ImageData, HTMLImageElement or HTMLCanvasElement.

Hence, I'm looking for a way to get one of these objects simply from the image url (e.g. "https://upload.wikimedia.org/wikipedia/commons/thumb/b/b6/Image_created_with_a_mobile_phone.png/1200px-Image_created_with_a_mobile_phone.png")

It may be worth mentioning that I'm using nodejs for development. I've attached some code below to help explain what I'm trying to achieve.

const tf = require("@tensorflow/tfjs");
const mobilenetModule = require("@tensorflow-models/mobilenet");
const knnClassifier = require("@tensorflow-models/knn-classifier");

const classifer = knnClassifier.create();

async function start() {
  const mobilenet = await mobilenetModule.load();

  const pic0 = // Get image from "https://upload.wikimedia.org/wikipedia/commons/thumb/b/b6/Image_created_with_a_mobile_phone.png/1200px-Image_created_with_a_mobile_phone.png";
  const img0 = tf.browser.fromPixels(pic0);
  const logits0 = mobilenet.infer(img0, true);
  classifier.addExample(logits0, 0);

  const pic1 = // Get image from "https://www.pixsy.com/wp-content/uploads/2021/04/ben-sweet-2LowviVHZ-E-unsplash-1.jpeg";
  const img1 = tf.browser.fromPixels(pic1);
  const logits1 = mobilenet.infer(img1, true);
  classifier.addExample(logits1, 0);

  const check = await axios.get(
    "https://cryptotvnetwork.com/wp-content/uploads/2021/04/4ee1ad2ffbb00866fb7c55c61786e95d.jpg",
    { responseType: "arrayBuffer", }
  );
  const x = tf.browser.fromPixels(check);
  const xlogits = mobilenet.infer(x, true);
  const p = classifier.predictClass(xlogits);
  console.log(p);
}

start();

I apologize in advance if this is a stupid or duplicate question however I haven't been able to find on this during my time searching.

@Andreas, I want to know how to get an ImageData, HTMLImageElement or HTMLCanvasElement object from an image url :) — Bob, Sep 04 '21 at 10:18
What you've tried so far to solve this on your own? Get the image as array buffer and `tf.node.decodeImage`? ... ? — Andreas, Sep 04 '21 at 10:28
I've researched extensively though I believe my methods and approaches may have been inefficient due to my recent exposure to nodejs and js. This mainly includes reading many tutorials on both js and nodejs, searching for similar questions on this forum, and looking into libraries that may be able to aid me in this. While experimenting, I came across a image classification library based on tensorflow, called nsfwjs. While messing with it, I was able to use a library called axios to get images from the internet. However, this did not work with the current libraries I'm working with. — Bob, Sep 04 '21 at 10:32
As for ```tf.node.decodeImage```, I can't seem to find it on the API documentation (https://js.tensorflow.org/api/1.0.0). Could you tell me where you found that and what it does? — Bob, Sep 04 '21 at 10:34
You're using node, so... -> [Node API - `node.decodeImage()`](https://js.tensorflow.org/api_node/3.9.0/#node.decodeImage) ;) — Andreas, Sep 04 '21 at 10:36
Thx for that . Now I just need to find a way get the image encoded in an Uint8Array from just the url — Bob, Sep 04 '21 at 10:46
Could I get a hint as to how to achieve this . I've looked into some tutorials and documentations but haven't found a way to get a Uint8Array from an image url. — Bob, Sep 04 '21 at 11:16
Still haven't managed to find a way to load an image from the web into tensorflow... — Bob, Sep 04 '21 at 21:13
@Andreas, the following is what I've got from your guidance so far. However, it gives me an unsupported image error on the second line. `const resp0 = await axios.get(urls[0], { responseType: "arraybuffer" }); const pic0 = tfn.node.decodeImage(resp0); const img0 = tf.browser.fromPixels(pic0); const logits0 = mobilenet.infer(img0, true); classifier.addExample(logits0, 0);` — Bob, Sep 05 '21 at 00:13
The following is what I've finally landed on: `const pic = new Image();` `pic.src = url;` `img = tf.browser.fromPixels(pic);` However, I get an error saying that the fromPixels function expects `HTMLVideoElement, HTMLImageElement, HTMLCanvasElement, ImageData in browser...`. My understanding is that the Image constructor returns a HTMLImageElement then why is it not getting accepted? — Bob, Sep 05 '21 at 01:36
@Andreas, when I try to run the code, it tells me that `tf.node.decode()` is not a function... — Bob, Sep 05 '21 at 22:59

score 0 · Accepted Answer · answered Sep 05 '21 at 11:19

0

The tensor has to be created once the image is loaded.

async function getImage() {
  var pic0 = new Image();
  pic0.src = 'https://www.google.com/favicon.ico';

  pic0.onload = () => {
    var img0 = tf.browser.fromPixels(pic0);
    return img0;
  }
}

answered Sep 05 '21 at 11:19

Jofbr

455
3
23

Getting this error: `Error: pixels passed to tf.browser.fromPixels() must be either an HTMLVideoElement, HTMLImageElement, HTMLCanvasElement, ImageData in browser, or OffscreenCanvas, ImageData in webworker or {data: Uint32 Array, width: number, height: number}, but was Image` – Bob Sep 05 '21 at 11:27

Getting ImageData/HTMLImageElement/HTMLCanvasElement object from image url

1 Answers1