Drawing selectable rectangles on an image

JupyterLab
1

Hi

I’m new to using Jupiter Lab and to Python - trying to find a way to perform the following :

I want to display an image (a page from a document) - and draw rectangles on top of it (bounding boxes of different texts). I get the list of bounding boxes from an external server using rest api.
I managed to do that using PIL and request easily.

But now - I want to be able to “select” specific rectangles by clicking on the image - and get the boxes of those rectangles (so the user can select the texts he wants).

What will be the easiest way to accomplish this ?
Any reference to any example will help me get started.

Thanks
Inbal

2

The bokeh BoxEditTool (nicely wrapped up by holoviews) is quite good at this:
https://holoviews.org/reference/streams/bokeh/BoxEdit.html

You can put an image in there, and then get back well-registered coordinates back.

1 Like
3

Thank you for this fast reply
Will give it a try

4

Maybe you’ll find the jupyter_bbox_widget handy for this task. I made it initially for annotating images with bounding boxes but I think it can cover this use case as well.

You create the widget with

from jupyter_bbox_widget import BBoxWidget
widget = BBoxWidget(
    image = 'url or base64-encoded data',
    bboxes = [
        # a list of boxes with pixel coordinates
        {'x': 100, 'y': 100, 'width': 500, 'height':200, 'label':''},
    ],
    view_only = True, # to prevent users from moving/resizing boxes
)

Now this part isn’t really documented :sweat_smile: … But there is a selected_index trait you can observe to know which box the user clicked on:

def on_bbox_selected(change):
    selected_index = change['new']
    # a value of -1 means the user clicked outside of any bboxes
    if selected_index == -1:
        selected_bbox = None
    else:
        selected_bbox = widget.bboxes[selected_index]
    # do whatever with the bbox

widget.observe(on_bbox_selected, names=['selected_index'])

And that’s about all the boilerplate code you need =). There’s a demo notebook on Binder where you could try this out.

3 Likes
5

Thanks so much for this reply - looks really helpful
I will give it a try

6

Hi There @gereleth

Thanks for pointing me to your project - works great.
Since I’m really new to this environment I have 2 questions:

1 - how can a resize the image displayed ?
2 - in the on_box_selected - I would like to know what is the active class (so I can add the selected box with the label to a new array)

Thanks again for assisting me on this task
Inbal

7
  1. You can resize the whole widget by including something like layout={'width': '50%'} as a keyword arg when creating it.
  2. selected_bbox['label'] should give the class label of the clicked box.
8

Thanks for this fast reply

2 - what I actually wanted to get is the class selected at the bottom of the widget - cause I want the user to “pick” the class - and then click the appropriate Box to match the class

1 Like
9

Oh, this hasn’t come up before. I’m afraid the selected class label is not currently exposed to the python side.
I could add that in the next version =). You’d access it as widget.label.

1 Like
10

Thanks so much - it would be great to have that - this will enable the widget to support both scenarios.

11

Ok, if you update the widget to version 0.4.0 then widget.label should have the currently selected class label.
Let’s hope I didn’t break anything :grin:

2 Likes
12

Thank you so much for this addition. I will test it soon