Frame Resizing and Drawing Confusion

[paulsermon-gemfair]

Not sure if this is just me or a complexity that could do with abstracting out.

I've set up on an android.

My camera is rendering square on screen, and seems to be showing dead center of what the lens shows.

The frame that the frameProcessor is receiving is not a 1/1 aspect ratio as there doesn't seem to be an option to do that. It often receives 1080 x800 or some similar ratio.

This is then resized to 320x320 by the resize plugin, which if I understand correctly, resizes and crops to the center by default.

I'm using effecientdet which seems to return normalized bbox coordinates. My issue is converting those normalized values back to the full frame for rendering the bbox to the screen. I'm taking the normalized value and multiplying by framewidth and frameheight but of course these don't map exactly back to the original aspect ratio. So the box is rendered in the wrong place (obviously).

I'm wondering if I'm missing something obvious here.

`const resized = resize(frame, {
scale: {
width: 320,
height: 320,
},

    pixelFormat: 'rgb',
    dataType: 'uint8',
  })
  const result = actualModel.runSync([resized])

  // const num_detections = result[3]?.[0] ?? 0
  // console.log('Number of Detections: ' + num_detections)

  const detection_scores = result[2]
  const detection_boxes = result[0]

  frame.render();
  const frameWidth = frame.width;
  const frameHeight = frame.height;

  
  let thisDetectedClasses: string[] = []

  for (let i = 0; i < detection_boxes.length; i += 4) {
    const confidence = detection_scores[i / 4]
    if (confidence > 0.6) {
      
      const classId = result[1][i / 4] || 0
      const className = cocoLabels[Number(classId)] || `Class ${classId}`;

      thisDetectedClasses.push(className)
      // console.log(`Detected ${className} with confidence: ${confidence}`)

      const left = frameWidth * detection_boxes[i]
      const top = frameHeight * detection_boxes[i + 1]
      const right = frameWidth * detection_boxes[i + 2]
      const bottom = frameHeight * detection_boxes[i + 3]
      

      const rect = Skia.XYWHRect(left, top, 10, 10);
      const paint = Skia.Paint();
      paint.setColor(Skia.Color('red'));
      paint.setStyle(PaintStyle.Stroke);
      paint.setStrokeWidth(4);
      frame.drawRect(rect, paint);
    }
  }`