Senseplane Provides Perception Capabilities for AI Agents
Senseplane positions itself as eyes and ears for AI agents. The system connects to cameras, microphones, and sensors already present in a setup, enabling any agent — whether Claude, ChatGPT, Cursor, or custom code — to see, hear, and remember events as they unfold. The platform handles perception while leaving decisions to the agent: it reports what it observed, includes confidence estimates, and provides evidence such as snapshots and clips.
Core Features
The platform offers several operational modes accessible through its interface or API:
- Look — provides a snapshot of the current visual environment and supports ad-hoc questions about what is present.
- Watch — configures event-driven alerts such as "notify me when a package is left at the door." Alerts can be delivered via signed webhooks, the Model Context Protocol (MCP), or Telegram.
- Ask — answers temporal queries like "when did the courier arrive?" with the associated event, a snapshot, and a short clip.
- Measure — extracts quantitative answers from ongoing activity, such as "how many meals were recorded today," "how long did she sleep," or "is the environment quieter than usual." A daily summary option is available.
- Replay — accesses any range of recorded video or sensor data within the retention period, which defaults to seven days.
Demo footage includes recordings from vpdonline, the Museum of Municipal Engineering in Kraków, and Diego Torres Silvestre via Wikimedia Commons under CC BY license.
Architecture and Components
The system distributes responsibilities across specialized modules:
| Part | Folder | Language | Role |
|---|---|---|---|
| Media plane | media/ | TypeScript (mediasoup) | Ingest, continuous recording, playback, analysis feed |
| Brain | brain/ | Python | Motion, objects, sounds, episodes, watches, memory |
| API | api/ | TypeScript (Fastify) | Public /v1 REST API, webhooks, auth |
| MCP server | mcp/ | TypeScript | 10 tools for agents |
| SDK | sdk/typescript/ | TypeScript | npm i senseplane |
| Console | console/ | TypeScript (React) | Web UI for sources, playback, watches, chat, access |
| Bridge | bridge/ | Go | RTSP cameras, outbound only |
| Home Assistant | integrations/home-assistant/ | Python | Forwards sensors as readings |
| Spec | spec/ | OpenAPI + JSON Schema | Single source of truth for API, SDK and MCP |
All components run on a single machine from one docker-compose file. Caddy provides HTTPS termination in front. The repository is early software (v0.2); the project notes expect rough edges and welcomes bug reports.
Getting Started with Docker
To run Senseplane locally, create a directory and pull the docker-compose file:
mkdir senseplane && cd senseplane
curl -fsSLO https://raw.githubusercontent.com/senseplane-ai/senseplane/main/docker-compose.yml
# edit the settings block at the top: your domain, email, passwords and (optionally) a model key
chmod 600 docker-compose.yml
docker compose up -d
After the stack starts, open the provided domain and create an account — the first account owns the workspace. Add a phone or laptop camera from the console, then connect your agent to the MCP endpoint at /mcp. The full guide covering DNS setup, firewall rules, backups, and upgrades is available in the documentation titled "Run Senseplane on a VM."
Agent Connectivity
Agents connect to Senseplane through multiple pathways. The Model Context Protocol (MCP) server provides 10 tools that agents can use. A REST API is available at /v1. Webhooks can be configured for event delivery. A TypeScript SDK is published as npm i senseplane and integrates with agent runtimes such as Claude, ChatGPT, and Cursor. The platform reports perception results including the data itself, confidence levels, and supporting evidence, allowing the downstream agent to decide the next action.
Retention and Playback
Recorded events are retained for seven days by default. Within that window, any range of video or sensor data can be replayed. This enables queries such as reviewing activity from a specific hour or day, useful for investigating incidents or extracting metrics over time.
Security and Licensing
The project outlines security considerations in a separate SECURITY.md file. The software uses the Apache 2.0 license; see LICENSE and NOTICE for details. The "Senseplane" name is a trademark; forks must rename the project per the guidance in TRADEMARKS.md.