A security baseline for self-hosted AI memory and API keys

A self-hosted agent can put more of the work environment under your control. It does not automatically keep every part of an AI workflow inside your server. The model may run locally or at an external endpoint, and the agent may store context between conversations. Those are separate security boundaries.

For a practical baseline, take one task: use AIDA by Autafy to triage work email. AIDA runs on your server, supports local execution with Ollama or API keys for external model providers, and integrates with Gmail and Outlook. Before connecting an inbox, decide what information may be processed, where it may travel and who can access the server.

1. Map the data path before connecting email

Write down the systems involved: the mail service, the server running AIDA, the model endpoint, any messaging surface you plan to use, and the places where server data is backed up. This simple inventory is more useful than the label “self-hosted.” The agent’s server location does not settle where inference happens or how information is carried to a chat client.

With local execution through Ollama, model inference runs locally. With an API key, requests go to the selected external model endpoint. Treat information included in those requests as leaving your server boundary. The provider’s handling depends on its own terms and configuration; check those before using real customer, employee or confidential data. Local inference also does not, by itself, establish that connected email, stored memory, backups or messaging are all local.

Choose the model path based on the data you intend to process, not just convenience. If you are comparing the wider trade-offs, our analysis of self-hosted agents, privacy and operating costs provides useful context.

2. Prepare the server and protect the key

Before setup, limit access to the server to the people who need to administer it. Apply your normal server patching, account and backup practices. A private server is still a shared responsibility: anyone who can read its files, logs or backups may be able to reach sensitive agent data.

If you use an external model, create an API key for this workload where the provider’s options allow it. Use the deployment method documented for AIDA to supply the key. Do not put it in a chat message, source-code repository, shared document or command that may be saved in shell history. Store it in an access-controlled secret store or equivalent protected configuration supported by your deployment environment. Do not assume AIDA includes a particular key vault or storage control; confirm the setup details in its documentation.

Keep a record of which key belongs to which deployment and who can access it. Check whether provider-side limits or restrictions are available, and set them if appropriate. Know how to revoke and replace the key before you need to respond to a leak. Review server logs and backups for accidental copies, and avoid putting secrets in diagnostic material.

3. Connect only the inbox you need

Start with one work account and the smallest practical scope for the task. During the connection process, read the access request rather than assuming it grants only inbox reading. The exact permissions depend on the integration and setup. If the requested access is broader than your use case, pause and find out whether a narrower configuration is available.

Define what “urgent” means before enabling scheduled triage. AIDA supports inbox monitoring on a schedule, along with criteria such as quiet hours and ignore lists. Use those boundaries to reduce unnecessary processing. Begin with low-risk messages, check the summaries and any drafts, and confirm they match your expectations before relying on the workflow for sensitive correspondence. For the email workflow itself, see our guide to setting up self-hosted email triage.

4. Treat persistent memory as stored work data

AIDA offers persistent memory across conversations. That can save repeated explanations, but it also means context may remain beyond the message that supplied it. Treat memory as a data store, not as a private thought that disappears when a chat ends.

Give the agent only the context needed for the task. Avoid deliberately adding passwords, authentication codes, payment details or sensitive personal information. Before using memory with real inbox data, find out how the deployed version handles viewing, changing and removing remembered information, and how that information is included in backups. These details are not established by the fact that memory is persistent; confirm them in the product documentation and your own deployment.

Apply the same access restrictions and backup protections you use for work records. If you cannot identify where memory files or backups live, or who can reach them, do not put sensitive correspondence into the workflow yet.

5. Test the boundary, then review it

Run a small test with non-sensitive email. Confirm which model path is active, what arrives in the chat surface, and whether the result contains information you did not expect to share. Check the server’s access and backup arrangements, then repeat the review when you change the model, integrations or messaging channel.

The operational baseline is straightforward: know where inference happens, protect the API key as a credential, and manage persistent memory like any other work data. Self-hosting gives you control over the server; keeping that control depends on the choices around it.