Unleash the Power of Computer Vision!

This repository provides a suite of tools designed to tackle your image and video-based computer vision challenges. Whether you're working on object detection, image classification, QR reading, counting items, or other visual tasks, these tools can streamline your development process.

Key Features:

• Image & Video Support.
• Detailed Documentation: Get started quickly and explore advanced features with our documentation: https://landing-ai.github.io/vision-agent-tools/.
• Seamless Integration: These tools are designed to work in conjunction with the powerful Vision Agent.

Ready to Get Started?

For a quick and easy introduction to the core functionalities, head over to the Vision Agent web app: https://va.landing.ai/tool. This is a great starting point to get familiar with the capabilities and potential of the tools before diving deeper into the code.

Let's Build Something Amazing!

We encourage you to explore the tools, leverage the documentation, and contribute to the project.

Installation

Easy way

make install

Advanced usage

You can install by running poetry install --extras "all" to install all tools, or with
poetry install --extras "owlv2 florence2" to install specific tools such as owlv2
and florence2.

Usage

Models

Models in this project are machine learning models that perform specific tasks (like object detection and instance segmentation).

Here's a simple example of how to use the Owlv2 model to detect objects in an image:

from PIL import Image
from vision_agent_tools.models.owlv2 import Owlv2

# load image
image = Image.open("/path/to/my/image.png")
model = Owlv2()

detections = model(image=image, prompts=["cat"])

Tools

Tools are higher-level abstractions that wrap around one or more models to accomplish specific tasks. Each tool is designed to work with different models via a dynamic model registry, allowing users to switch between models.

Here's an example of how to use the TextToObjectDetection tool to detect objects in an image based on text prompts:

from PIL import Image
from vision_agent_tools.tools.text_to_object_detection import TextToObjectDetection

# load image
img_path = "/path/to/my/image.jpg"
image = Image.open(img_path)

# Initialize the text-to-object detection tool with the desired model
detector = TextToObjectDetection(model="owlv2")

# Run the detector with the image and a text prompt
detections = detector(image=image, prompts=["find dogs in the picture"])

In this example:

• TextToObjectDetection tool is initialized with the "florence2" model.
• The tool detects objects based on the text prompt "find dogs in the picture" and returns a list of TextToObjectDetectionOutput containing the detection results.

Contributing

Clone the repo and install it

poetry install
poetry run pre-commit install

Adding new model code

Tools can be added in vision_agent_tools/models. Simply create a new python file with
the model name and add a class with the same name. The class should inherit from
BaseMLModel and implement the __call__ method. Here's a simplified example for adding
Owlv2 from the transformers library:

from Image import Image
from vision_agent_tools.shared_types import BaseMLModel
from transformers import Owlv2Processor, Owlv2ForObjectDetection

class Owlv2(BaseMLModel):
    def __init__(self):
        self.processor = Owlv2Processor.from_pretrained("google/owlv2-base-patch16-ensemble")
        self.model = Owlv2ForObjectDetection.from_pretrained("google/owlv2-base-patch16-ensemble")

    def __call__(self, image: Image.Image, prompt: list[str]):
        inputs = self.processor(image, [prompt], return_tensors="pt")
        outputs = self.model(**inputs)
        target_sizes = torch.Tensor(image.size[::-1])
        results = self.processor.post_process_object_detection(outputs, target_sizes=target_sizes, threshold=0.1)
        output = []
        for box, score, label in zip(resuts[0]["boxes"], results[0]["scores"], results[0]["labels"]):
            output.append({"box": box.tolist()), "score": score.item(), "label": label.item()}
        return output

Registering a model in the model registry

To use a model with your tool, you need to register it