Skip to content

Commit 156ebaf

Browse files
giswqsCopilot
giswqs
and
Copilot
authored
Add support for segmentation with box prompts and SAM 3 (#449)
* Add support for segmentation with box prompts and SAM 3 * Update samgeo/samgeo3.py Co-authored-by: Copilot <[email protected]> * Update samgeo/samgeo3.py Co-authored-by: Copilot <[email protected]> * Update samgeo/samgeo3.py Co-authored-by: Copilot <[email protected]> --------- Co-authored-by: Copilot <[email protected]>
1 parent 24eccbd commit 156ebaf

File tree

3 files changed

+488
-24
lines changed

3 files changed

+488
-24
lines changed

‎docs/examples/sam3_box_prompts.ipynb‎

Lines changed: 359 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,359 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# Segmenting remote sensing imagery with box prompts and SAM 3\n",
8+
"\n",
9+
"[![image](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/opengeos/segment-geospatial/blob/main/docs/examples/sam3_box_prompts.ipynb)\n",
10+
"\n",
11+
"This notebook shows how to generate object masks from box prompts with the Segment Anything Model 3 (SAM 3). \n",
12+
"\n",
13+
"Make sure you use GPU runtime for this notebook. For Google Colab, go to `Runtime` -> `Change runtime type` and select `GPU` as the hardware accelerator.\n"
14+
]
15+
},
16+
{
17+
"cell_type": "markdown",
18+
"metadata": {},
19+
"source": [
20+
"## Install dependencies\n",
21+
"\n",
22+
"Uncomment and run the following cell to install the required dependencies.\n"
23+
]
24+
},
25+
{
26+
"cell_type": "code",
27+
"execution_count": null,
28+
"metadata": {},
29+
"outputs": [],
30+
"source": [
31+
"# %pip install \"segment-geospatial[samgeo3]\""
32+
]
33+
},
34+
{
35+
"cell_type": "markdown",
36+
"metadata": {},
37+
"source": [
38+
"## Import libraries"
39+
]
40+
},
41+
{
42+
"cell_type": "code",
43+
"execution_count": null,
44+
"metadata": {},
45+
"outputs": [],
46+
"source": [
47+
"import leafmap\n",
48+
"from samgeo import SamGeo3, download_file\n",
49+
"from samgeo.common import raster_to_vector, regularize"
50+
]
51+
},
52+
{
53+
"cell_type": "markdown",
54+
"metadata": {},
55+
"source": [
56+
"## Download Sample Data\n",
57+
"\n",
58+
"Let's download a sample satellite image covering Washington State:"
59+
]
60+
},
61+
{
62+
"cell_type": "code",
63+
"execution_count": null,
64+
"metadata": {},
65+
"outputs": [],
66+
"source": [
67+
"image_url = \"https://github.com/opengeos/datasets/releases/download/places/wa_building_image.tif\"\n",
68+
"image = download_file(image_url)"
69+
]
70+
},
71+
{
72+
"cell_type": "code",
73+
"execution_count": null,
74+
"metadata": {},
75+
"outputs": [],
76+
"source": [
77+
"m = leafmap.Map()\n",
78+
"m.add_raster(image, layer_name=\"Satellite image\")\n",
79+
"m"
80+
]
81+
},
82+
{
83+
"cell_type": "markdown",
84+
"metadata": {},
85+
"source": [
86+
"## Initialize SAM 3\n",
87+
"\n",
88+
"To use point and box prompts (SAM1-style interactive segmentation), initialize SAM3 with `enable_inst_interactivity=True`."
89+
]
90+
},
91+
{
92+
"cell_type": "code",
93+
"execution_count": null,
94+
"metadata": {},
95+
"outputs": [],
96+
"source": [
97+
"sam = SamGeo3(backend=\"meta\", enable_inst_interactivity=True)"
98+
]
99+
},
100+
{
101+
"cell_type": "markdown",
102+
"metadata": {},
103+
"source": [
104+
"Specify the image to segment."
105+
]
106+
},
107+
{
108+
"cell_type": "code",
109+
"execution_count": null,
110+
"metadata": {},
111+
"outputs": [],
112+
"source": [
113+
"sam.set_image(image)"
114+
]
115+
},
116+
{
117+
"cell_type": "markdown",
118+
"metadata": {},
119+
"source": [
120+
"## Create bounding boxes\n",
121+
"\n",
122+
"Use the drawing tools to draw some rectangles around the features you want to extract, such as trees, buildings.\n",
123+
"\n",
124+
"If no rectangles are drawn, the default bounding boxes will be used as follows:"
125+
]
126+
},
127+
{
128+
"cell_type": "code",
129+
"execution_count": null,
130+
"metadata": {},
131+
"outputs": [],
132+
"source": [
133+
"if m.user_rois is not None:\n",
134+
" boxes = m.user_rois\n",
135+
"else:\n",
136+
" boxes = [\n",
137+
" [-117.5995, 47.6518, -117.5988, 47.652],\n",
138+
" [-117.5987, 47.6518, -117.5979, 47.652],\n",
139+
" ]"
140+
]
141+
},
142+
{
143+
"cell_type": "markdown",
144+
"metadata": {},
145+
"source": [
146+
"## Segment the image\n",
147+
"\n",
148+
"Use the `generate_masks_by_boxes_inst()` method to segment the image with specified bounding boxes. The `boxes` parameter accepts a list of bounding box coordinates in the format of [[xmin, ymin, xmax, ymax], [xmin, ymin, xmax, ymax], ...].\n"
149+
]
150+
},
151+
{
152+
"cell_type": "code",
153+
"execution_count": null,
154+
"metadata": {},
155+
"outputs": [],
156+
"source": [
157+
"sam.generate_masks_by_boxes_inst(boxes=boxes, box_crs=\"EPSG:4326\")"
158+
]
159+
},
160+
{
161+
"cell_type": "markdown",
162+
"metadata": {},
163+
"source": [
164+
"Save the masks to a file."
165+
]
166+
},
167+
{
168+
"cell_type": "code",
169+
"execution_count": null,
170+
"metadata": {},
171+
"outputs": [],
172+
"source": [
173+
"sam.save_masks(output=\"mask.tif\", dtype=\"uint8\")"
174+
]
175+
},
176+
{
177+
"cell_type": "markdown",
178+
"metadata": {},
179+
"source": [
180+
"## Display the result\n",
181+
"\n",
182+
"Add the segmented image to the map."
183+
]
184+
},
185+
{
186+
"cell_type": "code",
187+
"execution_count": null,
188+
"metadata": {},
189+
"outputs": [],
190+
"source": [
191+
"m.add_raster(\"mask.tif\", cmap=\"viridis\", nodata=0, opacity=0.5, layer_name=\"Mask\")\n",
192+
"m"
193+
]
194+
},
195+
{
196+
"cell_type": "markdown",
197+
"metadata": {},
198+
"source": [
199+
"## Use an existing vector file as box prompts\n",
200+
"\n",
201+
"Alternatively, you can specify a file path to a vector file. Let's download a sample vector file from GitHub.\n"
202+
]
203+
},
204+
{
205+
"cell_type": "code",
206+
"execution_count": null,
207+
"metadata": {},
208+
"outputs": [],
209+
"source": [
210+
"url = \"https://github.com/opengeos/datasets/releases/download/samgeo/building_bboxes.geojson\"\n",
211+
"geojson = \"building_bboxes.geojson\"\n",
212+
"download_file(url, geojson)"
213+
]
214+
},
215+
{
216+
"cell_type": "markdown",
217+
"metadata": {},
218+
"source": [
219+
"Display the vector data on the map."
220+
]
221+
},
222+
{
223+
"cell_type": "code",
224+
"execution_count": null,
225+
"metadata": {},
226+
"outputs": [],
227+
"source": [
228+
"m = leafmap.Map()\n",
229+
"m.add_raster(image, layer_name=\"Image\")\n",
230+
"style = {\n",
231+
" \"color\": \"#ffff00\",\n",
232+
" \"weight\": 2,\n",
233+
" \"fillColor\": \"#7c4185\",\n",
234+
" \"fillOpacity\": 0,\n",
235+
"}\n",
236+
"m.add_vector(geojson, style=style, zoom_to_layer=True, layer_name=\"Bboxes\")\n",
237+
"m"
238+
]
239+
},
240+
{
241+
"cell_type": "markdown",
242+
"metadata": {},
243+
"source": [
244+
"## Segment image with box prompts from vector file\n",
245+
"\n",
246+
"The `generate_masks_by_boxes_inst()` method can directly accept a file path to a vector file (GeoJSON, Shapefile, etc.). It will automatically extract bounding boxes from geometries and filter out any boxes outside the image bounds."
247+
]
248+
},
249+
{
250+
"cell_type": "code",
251+
"execution_count": null,
252+
"metadata": {},
253+
"outputs": [],
254+
"source": [
255+
"output_masks = \"building_masks.tif\""
256+
]
257+
},
258+
{
259+
"cell_type": "code",
260+
"execution_count": null,
261+
"metadata": {},
262+
"outputs": [],
263+
"source": [
264+
"sam.generate_masks_by_boxes_inst(\n",
265+
" boxes=geojson,\n",
266+
" box_crs=\"EPSG:4326\",\n",
267+
" output=output_masks,\n",
268+
" dtype=\"uint16\",\n",
269+
" multimask_output=False,\n",
270+
")"
271+
]
272+
},
273+
{
274+
"cell_type": "markdown",
275+
"metadata": {},
276+
"source": [
277+
"Display the segmented masks on the map."
278+
]
279+
},
280+
{
281+
"cell_type": "code",
282+
"execution_count": null,
283+
"metadata": {},
284+
"outputs": [],
285+
"source": [
286+
"m.add_raster(\n",
287+
" output_masks, cmap=\"jet\", nodata=0, opacity=0.5, layer_name=\"Building masks\"\n",
288+
")\n",
289+
"m"
290+
]
291+
},
292+
{
293+
"cell_type": "markdown",
294+
"metadata": {},
295+
"source": [
296+
"## Convert raster to vector"
297+
]
298+
},
299+
{
300+
"cell_type": "code",
301+
"execution_count": null,
302+
"metadata": {},
303+
"outputs": [],
304+
"source": [
305+
"output_vector = \"building_vector.geojson\"\n",
306+
"raster_to_vector(output_masks, output_vector)"
307+
]
308+
},
309+
{
310+
"cell_type": "markdown",
311+
"metadata": {},
312+
"source": [
313+
"## Regularize building footprints"
314+
]
315+
},
316+
{
317+
"cell_type": "code",
318+
"execution_count": null,
319+
"metadata": {},
320+
"outputs": [],
321+
"source": [
322+
"output_regularized = \"building_regularized.geojson\"\n",
323+
"regularize(output_vector, output_regularized)"
324+
]
325+
},
326+
{
327+
"cell_type": "code",
328+
"execution_count": null,
329+
"metadata": {},
330+
"outputs": [],
331+
"source": [
332+
"m.add_vector(\n",
333+
" output_regularized, style=style, layer_name=\"Building regularized\", info_mode=None\n",
334+
")"
335+
]
336+
}
337+
],
338+
"metadata": {
339+
"kernelspec": {
340+
"display_name": "geo",
341+
"language": "python",
342+
"name": "python3"
343+
},
344+
"language_info": {
345+
"codemirror_mode": {
346+
"name": "ipython",
347+
"version": 3
348+
},
349+
"file_extension": ".py",
350+
"mimetype": "text/x-python",
351+
"name": "python",
352+
"nbconvert_exporter": "python",
353+
"pygments_lexer": "ipython3",
354+
"version": "3.12.2"
355+
}
356+
},
357+
"nbformat": 4,
358+
"nbformat_minor": 2
359+
}

‎mkdocs.yml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,7 @@ nav:
6868
- examples/sam2_point_prompts.ipynb
6969
- examples/sam2_text_prompts.ipynb
7070
- examples/tree_mapping.ipynb
71+
- examples/image_captioning.ipynb
7172
- examples/sam3_image_segmentation.ipynb
7273
- examples/sam3_image_segmentation_jpg.ipynb
7374
- examples/sam3_interactive.ipynb
@@ -77,7 +78,7 @@ nav:
7778
- examples/sam3_object_tracking.ipynb
7879
- examples/sam3_point_prompts.ipynb
7980
- examples/sam3_point_prompts_batch.ipynb
80-
- examples/image_captioning.ipynb
81+
- examples/sam3_box_prompts.ipynb
8182
- Workshops:
8283
- workshops/purdue.ipynb
8384
- workshops/cn_workshop.ipynb

0 commit comments

Comments
 (0)