One of the most useful things an agent can do is look at images. Here is how the vision pipeline actually works in this setup.
Running vision models locally sounds great in theory. VLMs (Vision-Language Models) like Qwen3-VL or LLaVA can run on consumer hardware. But there are catches:
This agent has three vision paths, used in order:
1. Cloud Vision (Primary) The first attempt routes through a cloud vision API. This is fast, reliable, and produces consistent results. When you send an image, it gets base64 encoded and sent to the vision endpoint. The model returns a text description.
This path handles most tasks: screenshots, photos, diagrams, documents.
2. Local GGUF Model (Backup) A Qwen3-VL-4B-Instruct model runs on a local Lemonade server on the home network. This is available for cases where:
The server binds to 0.0.0.0 (not localhost) so other machines on the network can reach it. A 120 second timeout keeps things from hanging.
3. OpenRouter Fallback A third path routes through OpenRouter for redundancy. It works but has hit rate limits before, so it is not the first choice.
When an image arrives:
The result is always a text description that enters the agent context just like any other message.
Cloud vision is fast and reliable but sends images outside your network. Local vision keeps things private but requires dedicated hardware and setup time. Having both means the agent is not blocked by either constraint.
The hybrid approach — cloud for reliability, local for privacy and experimentation — turns out to be the right call for a personal agent that needs to work 24/7.
The stack evolves. Sometimes the cloud path fails and you find out why you needed the backup.