It’s 2025, and object detection is a solved problem. Or is it?
Object detection is like giving machines eyes. It’s the computer vision task of finding and localizing people, cars, cups, and other objects in images or videos. Unlike plain image classification (which simply states “this is a cat”), object detection draws little boxes around objects.
At JigsawStack, we’ve been developing the best small object detection and grounding model available, built to handle not just detection, but also spatial grounding, segmentation, and even UI element detection for computer use. One model. All the sauce.
In a world where browser agents now see, click, and reason, object detection is the backbone behind not just smart cameras, but UI automation, accessibility tech, and even creative tools. Computer use itself has exploded in the last three years, and under the hood, it’s all powered by the same core: visual grounding for object detection.
Vision-Language Models in Detection
Historically, object detection has been specialized for specific industries, factories, construction sites, and even medical research. But at JigsawStack, we asked: Can we build a single, small, general-purpose model that handles everything from segmenting any real-world object to grounding UI buttons?
Early VLMs were used for classification or captioning, but now developers are adapting them to draw boxes. For example, CLIP-inspired methods (such as GLIP, Grounding DINO, and OWL-ViT) transform language supervision into object localization. These models let you pose queries like “Where is the apple?” and get back bounding boxes or masks. Grounding DINO and OWL-ViT are prime examples: they leverage text prompts to find novel objects at first attempt (zero-shot).
Here’s the state of object detection in 2025:
You can now just type “find the red mug” and open-vocabulary detectors like Grounding DINO 1.5 will box it, even if “red mug” was never in the training set.
Meta’s shiny SAM 2.1 goes beyond boxes by carving out pixel-perfect cut-outs of any object with one click or prompt.
Chatty AIs such as GPT-4o can reply with bounding-box coordinates straight from plain-English questions.
Edge-tuned versions like Grounding DINO 1.5 Edge now run on phones and drones, so object detection works offline in real time.
Devs are chaining detectors to segmenters, think Grounded SAM 2 for one-click “find-and-mask” pipelines in production APIs.
Examples: What can our model do?
- Basic Object Detection: detects all visible objects.
- Prompt-based Object Detection: Find anything you can describe—“find the red mug,” “show me cranes,” or even “utensils which can be used for frying”. Take a look at what our model detects for the prompt “four-legged fren <3“
- Computer Use: understands and localizes user interface elements (e.g., buttons, sliders) for computer use. Following is a screenshot for detections for VScode with a chat input pop-up for local computer use.
- Browser Automation: Automate website navigation by GUI element grounding. With prompting e.g. “search field“, “give me the search button“, e.t.c.
Can JigsawStack see an image differently?
Totally! Because we built the model around phenomenon vision-language alignment instead of just cranking up the model size. Alignment means the pixel-side encoder and the text-side encoder learn to land in the same semantic space, so “oplayer” and the blob of pixels that is a player overlap on the inside. Do that well and you get:
Generalisation on tap – snap to the right region even if they never showed up in training.
Promptable precision – ask for a UI button, a road sign, or a corgi wearing goggles, and the model adaptively tightens its boxes or masks.
One-shot spatial reasoning – verbs like left of, under, right, or next to that pillar come baked in, no extra heads required.
Small model, aligned brains, fresh set of eyes for every prompt. Let’s take a look to see if our model can point to the right objects ;)
Grid: Comparing Gemini-2.5-Pro (top-right), GPT-4.1 (bottom-left), & JigsawStack-ObjectDetection (bottom-right) on Minecraft (GUI-grounding), with prompt “player“
Grid: Comparing Gemini-2.5-Pro (top-right), GPT-4.1 (bottom-left), & JigsawStack-Object Detection (bottom-right) for object_detection with prompt “crane“
Let’s take a look at segmentation ability as well!
Image: JigsawStack-Object Detection for Segmentation, with prompt “helmets“
Image: fal-ai/SAM-evf for Segmentation, with prompt “helmets“
Quick Start with JigsawStack Object Detection
Getting started with our object detection model is incredibly straightforward. Whether you're developing a web or mobile app, our JavaScript and Python SDKs make the process seamless. With just a few lines of code, you can tap into JigsawStack's capabilities to detect, segment, and annotate objects in images, adding a layer of intelligence and interactivity to your application. Ready to dive in? Let's explore how you can set up and make your first call using our SDKs
Image: Golden Gate Bridge in San Francisco
Let’s start by setting up JigsawStack SDK,
npm install jigsawstack (or) pip install jigsawstack
Make your first call with ease through our JS and Python SDKs
import { JigsawStack } from "jigsawstack";
const jigsaw = JigsawStack({ apiKey: "your-api-key" });
const response = await jigsaw.vision.object_detection({
"url": "https://blog.viajemos.com/wp-content/uploads/2024/06/Que-hacer-en-San-Francisco-3-dias-1-1.png",
"annotated_image": true,
})
Similarly, in Python, you can use the object detection service as follows:
from jigsawstack import JigsawStack
jigsaw = JigsawStack(api_key="your-api-key")
response = jigsaw.vision.object_detection({
"url": "https://blog.viajemos.com/wp-content/uploads/2024/06/Que-hacer-en-San-Francisco-3-dias-1-1.png",
"annotated_image": true,
})
And voila!
{
"success": true,
"annotated_image": "https://jigsawstack-temp.b1e91a466694ad4af04df5d05ca12d93.r2.cloudflarestorage.com/temp/62818eaf-0573-45c1-ac55-601302e567ab.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=7b9a19349842b7b1a9e4c2e19f05b232%2F20250624%2Fauto%2Fs3%2Faws4_request&X-Amz-Date=20250624T172243Z&X-Amz-Expires=604800&X-Amz-Signature=8d5bdade297836deada8c33d81dd5d638c89a3a83014ffc18dc834c6939f4938&X-Amz-SignedHeaders=host&x-amz-checksum-mode=ENABLED&x-id=GetObject",
"gui_elements": [],
"objects": [
{
"bounds": {
"top_left": {
"x": 0,
"y": 0
},
"top_right": {
"x": 1900,
"y": 0
},
"bottom_left": {
"x": 0,
"y": 728
},
"bottom_right": {
"x": 1900,
"y": 728
},
"width": 1900,
"height": 728
},
"label": "Sky",
"mask": "https://jigsawstack-temp.b1e91a466694ad4af04df5d05ca12d93.r2.cloudflarestorage.com/temp/37a2ad76-ab17-4ffc-98d9-58aad848d162.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=7b9a19349842b7b1a9e4c2e19f05b232%2F20250624%2Fauto%2Fs3%2Faws4_request&X-Amz-Date=20250624T172244Z&X-Amz-Expires=604800&X-Amz-Signature=d1da3b9fb1cd46cac0d5aa9d8178f970568a016abd6864c5cfa18257f3e4ba32&X-Amz-SignedHeaders=host&x-amz-checksum-mode=ENABLED&x-id=GetObject"
},
{
"bounds": {
"top_left": {
"x": 411,
"y": 143
},
"top_right": {
"x": 1900,
"y": 143
},
"bottom_left": {
"x": 411,
"y": 1066
},
"bottom_right": {
"x": 1900,
"y": 1066
},
"width": 1489,
"height": 923
},
"label": "Golden Gate Bridge",
"mask": "https://jigsawstack-temp.b1e91a466694ad4af04df5d05ca12d93.r2.cloudflarestorage.com/temp/dcc4363c-4239-47ab-a58c-48ad036deb43.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=7b9a19349842b7b1a9e4c2e19f05b232%2F20250624%2Fauto%2Fs3%2Faws4_request&X-Amz-Date=20250624T172244Z&X-Amz-Expires=604800&X-Amz-Signature=1b7587e4b39ec4640895dfffc54fb4ecea8c738ff19f6150c44c38bebaa35bd0&X-Amz-SignedHeaders=host&x-amz-checksum-mode=ENABLED&x-id=GetObject"
},
...
],
"_usage": {
"input_tokens": 36,
"output_tokens": 3185,
"inference_time_tokens": 23758,
"total_tokens": 26979
}
}
Here's the annotated image showcasing our model's ability to detect, segment, label, and annotate.
Image**: Result from JigsawStack-Object Detection
Play With JigsawStack Object Detection
Users can also pass in natural language prompts to ground objects based on the intent & visual context in the scene.
Consider the following example:
from jigsawstack import JigsawStack
jigsaw = JigsawStack(api_key="your-api-key")
response = jigsaw.vision.object_detection({
"prompts": ["Objects that can be used to illuminate the room."],
"url": "https://images.unsplash.com/photo-1607448421773-f74d4b9b5919?q=80&w=1754&auto=format&fit=crop&ixlib=rb-4.1.0&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D",
"annotated_image": true
})
Our model detects:
Image**: Result from JigsawStack-Object Detection for user query “Objects that can be used to illuminate the room.“
Our approach to generalizing object detection and segmentation is transforming the way developers integrate visual intelligence into their applications. With our user-friendly SDKs and powerful vision-language alignment, you can effortlessly enhance your projects with cutting-edge technology. Whether you're working on a web or mobile app, JigsawStack provides the tools you need to bring your ideas to life. And we’re excited to see devs build with our smoll models capable of big impact!
Have questions or want to show off what you’ve built? Join the JigsawStack developer community on Discord and X/Twitter. Let’s build something amazing together!
