Agents Over SSH vs Agents on the Box

Deciding between running agents over SSH versus deploying agents on the box represents a fundamental choice in how we structure modern remote development environments. As AI coding assistants become more integrated into the daily flow, the latency, security, and context availability of these two topologies dictate the speed at which a developer can iterate. This guide explores the technical trade-offs of both approaches to help you determine which configuration fits your infrastructure.

Understanding Agents Over SSH

The concept of agents over SSH involves a local client machine that initiates a secure connection to a remote server. In this model, the "intelligence" or the orchestrator typically resides on the local machine. The SSH tunnel acts as a transport layer for commands, file system events, and terminal output.

When you use this topology, the AI agent reads local representations of remote files or executes shell commands through the existing SSH session. This is common in traditional IDE extensions where the editor runs locally but interacts with a remote workspace.

Advantages of the Remote Tunnel

• Centralized Secrets: You keep your API keys and configuration files on your local machine rather than distributing them across multiple remote instances.
• Consistent Environment: The agent uses your local shell configurations and custom aliases that you have perfected over time.
• Lower Resource Consumption: The remote server does not need to allocate CPU or memory to run the LLM orchestration logic, which is beneficial for smaller VPS instances.

Technical Challenges

Latency is the primary drawback. Every time the agent needs to "see" a file or check a process status, it must wait for the round trip over the network. In high latency scenarios, this can make agentic workflows feel sluggish. Additionally, complex file system watches over SSH can be brittle, leading to sync issues between what the agent thinks the code looks like and the actual state on the disk.

The Case for Agents on the Box

Running agents on the box means the AI agent software is installed and executed directly on the remote environment. The agent has native, low latency access to the file system and the system bus.

In this scenario, the developer interacts with a thin client or a web interface that simply displays what the agent is doing on the remote host. The agent is a citizen of the remote OS, allowing it to trigger builds, run tests, and inspect logs with sub millisecond response times.

Performance and Context

Because the agent is local to the code, it can index large repositories much faster. It does not have to pipe megabytes of text over a socket to "understand" the project structure. For teams working on massive monorepos, having the agent live on the box is often the only way to maintain acceptable performance.

Security Implications

Deploying agents on the box requires you to manage credentials on the remote server. If you are using a local-first approach for your primary workspace, you must ensure that your remote environment is as secure as your workstation. If the remote box is compromised, the agent and any keys it holds are also at risk.

Comparative Overview: Agent Topologies

How Deska Handles Agent Environments

Deska provides a different perspective on this choice by offering a workspace that treats both local and remote contexts as first class citizens. It is a free desktop app for Mac, Windows, and Linux that uses an infinite canvas to organize your tools.

Within a single workspace, you can place multiple terminals and coding agents side by side. Because Deska is local-first, it stays on your machine, but its panels can reach out to various environments.

Parallel Execution on the Canvas

The canvas allows you to run tools like Claude Code or Codex CLI as individual panels. You can have one panel running an agent over an SSH terminal while another panel runs a local agent on your machine. This side by side comparison helps developers choose the right tool for the specific task at hand. If you need to monitor these sessions while away from your desk, the mobile app allows you to check on your progress through a secure relay without exposing ports.

Driving the Workspace with AI

Deska includes a feature called Ask Deska. This is a voice and chat assistant that can actually drive the workspace. It can open new panels, run commands, and check your active sessions. This reduces the friction of managing multiple SSH connections because the assistant handles the layout of your workspaces.

Hybrid Strategies for Remote Work

Many developers find that a hybrid approach works best. They might use agents over SSH for quick configuration changes or scripts, but move to agents on the box for heavy refactoring or when the network connection is unstable.

1. Use SSH forwarding to securely share keys from your local machine to the remote agent.
2. Maintain a persistent session using a terminal multiplexer to ensure agents on the box continue working if the connection drops.
3. Utilize a unified workspace to keep track of multiple remote contexts without losing focus.

When using Deska, you can integrate your own API keys via the BYOK model. This ensures that whether you are interacting with agents via a local panel or a remote terminal, you maintain control over your costs and identity. For more details on managing these settings, see the data and storage documentation.

FAQ

How do I use agents over SSH safely?

You should use SSH agent forwarding or specialized secret management tools to avoid hardcoding API keys on the remote server. Ensure your ~/.ssh/config is properly hardened and use key-based authentication for all sessions.

Is latency a dealbreaker for remote AI agents?

It depends on the task. For simple code completions, latency is manageable. For autonomous agents that need to perform hundreds of file reads to plan a refactor, network lag can significantly increase the time to completion and the cost of the operation.

Can I run multiple AI agents side by side?

Yes, using a tool with a flexible UI like a canvas allows you to run different agents simultaneously. This is useful for comparing how different models handle the same codebase or for running a frontend agent and a backend agent in parallel.

Getting Started with Agents

If you are looking to optimize your workflow between local and remote environments, exploring a dedicated workspace is a productive next step. You can organize your editors, terminals, and agents in one place to see which topology serves your needs.

Download Deska to begin building your custom developer workspace today. Whether you prefer the isolation of agents over SSH or the raw speed of agents on the box, having a unified interface makes all the difference in productivity. Pero if you are more comfortable in Spanish, the siguiente section covers these topics for your team.