security · last updated 8 November 2026
What we can and cannot see
Margin measures what your AI spends and whether the work succeeded, without receiving what your models were asked or what they said back. Below is the whole record that arrives for a call.
Every column of the calls table, grouped by what it is for.
- Your labels
- source event_id workflow_id session_id prompt_id operation environment team task_key task_class span_id parent_span_id case_id
- Which model ran
- provider model billing_tier inference_geo speed substrate serving_host harness reasoning_effort
- How much it used
- input_tokens output_tokens cache_read_tokens cache_creation_tokens cache_creation_1h_tokens reasoning_tokens prefix_chars cache_reported
- How it went
- status finish_reason error_type outcome_passed failing_criteria is_retry is_simulated orchestration_pattern branch_width stopping_rule prompt_hash prefix_hash output_class retry_kind
- Cost and time
- id ts received_at cost_usd cost_unpriced cost_claimed_usd latency_ms
Source: margin.models.Call, held to this figure by tests/test_security_page_matches_schema.py.
- Your production prompts and completions never reach us. Not redacted, not encrypted, not retained briefly: there is no field for them in the SDK, the API or the meter’s tables. Two exceptions, both started by you, are below.
- Margin is a meter, not a proxy. Your agents call OpenAI, Anthropic and the rest directly, and the SDK posts a short record after each call. Remove it and your agents keep running unchanged.
- Thirty-nine fields are free text. Most are labels you choose; a few, like the model name, the SDK fills in from your provider’s reply. They are the only way content could reach us, and only if something puts it there.
the record
Everything we store about a call
One row per model call. Fifty-one columns, listed here in full: Figure 1 draws each one, and this is what each group is for.
- Your labels
- Strings you choose, so the numbers can be grouped the way your org is.
- Which model ran
- Enough to price the call and compare routes.
- How much it used
- Counts. Never the tokens themselves.
- How it went
- Whether the work succeeded, and how the agent was shaped.
- Cost and time
- The two numbers the whole product divides by, a flag when a cost could not be priced, and the cost you sent when it was too high for its tokens to count as spend. received_at is set only when Margin's own log forwarder (which runs with database access, not over the ingest API) backdated a row to when it really happened; it keeps the time we received it.
The outcome record is shorter: whether it passed, a quality score and how it was graded, an optional link back to your own system, and a value if you supplied one.
A third record exists only if you run our wait checker: one row per trace of an agent’s actions, holding five numbers (seconds traced, seconds idle, actions, actions measured, actions with an unknown send time) and two labels, trace_id and adapter. The checker runs on your machine and your trace files stay there. trace_id is free text you choose, such as a file name, so do not put content in it. A row carrying any other field is refused.
by construction
What is absent, and why that is structural
No prompt. No completion. No message content. No user identifiers, email addresses or IP addresses. Not because we delete them: nothing collects them. The SDK builds its payload from the usage block your provider returns, which carries counts rather than text.
where content does reach us
Two ways your text reaches us, and what else we hold
A measurement run you start
It clones your repository and runs your own eval harness in a fresh virtual machine on Vercel Sandbox, one machine per run, destroyed when the run ends. A run we pay for uses a model key made for that run alone, capped in dollars and revoked when it ends. A run you pay for uses the OpenRouter key you saved in the console, which we store encrypted and do not cap or restrict: its spending limit and the hosts it may reach are whatever you set on that key. Either way, while it runs, your evals’ prompts and responses go from the sandbox to OpenRouter. We keep only the run’s cost and outcome records. The checkout is deleted when the run ends, but a daily server backup taken during a run can hold a copy for up to 7 days.
What a run leaves behind, and how to delete it
What a run leaves behind: the file paths it read, the agents it found in them, which of those you picked, and its cost and outcome records. No file contents are kept from a repository it cloned, and the code the container executes is deleted with the run. Delete workspace data on the console’s home removes the path list, the agent list, every run for the project, the evaluation criteria drafted from your code, the reviews you recorded against sessions and the tasks derived from your repository history, and forgets the model key you saved. Metered calls and outcomes are deleted on a written request.
The sandbox: four hosts, no secrets
The machine is fresh for each run and holds nothing from the last one, so it cannot read our files or another project’s. Its outbound network is closed except for four hosts: OpenRouter, PyPI and its file host, and the npm registry. Everything else is refused, and we measured that the cloud metadata address is unreachable from inside it. Its environment holds no database address, no server secret and no real model key: the variables a model library reads carry a placeholder, and the sandbox’s network proxy adds the run’s actual key (ours, made for that run, or yours) as a header on requests to OpenRouter, so a run that prints its environment shows no key. Eval code that needs any other host fails to reach it rather than leave the sandbox.
The console’s AI tools
Asking the console a question, having it draft evaluation criteria from your agent’s outputs or source, and having it grade an output all send that text from our server through OpenRouter to one of the nine model hosts named below, and only to an endpoint that keeps nothing, which OpenRouter’s documentation says means it cannot train on it. If none of them can serve a request, it fails rather than going anywhere else. A run on our key is held to those nine hosts by a restriction on the key itself, unless a signed-in member has chosen “any zero-retention host” for the project (below); a run on your own key is held only to what you set on it.
GitHub, sign-in, and a folder you drop
The GitHub App: four permissions, and it never merges
Connecting GitHub installs our GitHub App, margin-governor on the repositories you pick at install time and on no others. It asks for four permissions: repository contents (read and write), pull requests (read and write), repository metadata (read), and organization members (read), which is used only to check that you own an organization before connecting it, never to list or contact its people. Beyond the installation events every app receives, it is built for one webhook event, push, which says that the default branch moved and which files changed. That subscription is configured on the app and is not on yet; GitHub shows its state on the app’s page. When it is on, we use it to re-read the repository’s agents, so the list you see is the code you have; it never starts a run and never spends anything. A run clones with a one-hour, read-only token narrowed to that one repository. Write access is used for one thing: opening a pull request on a branch of its own. Margin never merges; a person on your side does.
Sign-in for your GitHub organization
A signed-in member can let people from the connected organization sign in to the project. It is off until someone turns it on, and we store one row for it: the project, the GitHub installation it applies to, and the time. Moving the project to another installation turns it off. Those people use a separate sign-in link that also asks GitHub to let us read their organization memberships. We use it for one question, whether the person is an active member, and we do not keep their GitHub token.
A folder you drop instead of connecting GitHub
If you drop a folder instead of connecting GitHub, we keep it so a later run or face-off can use it, encrypted with a key that only our API and our run server hold, and stored against your project alone. Files that look like keys or credentials are left out before anything is stored. We delete it when you ask, or after 30 days with no activity in your project’s workspace; database backups can keep the encrypted copy for up to 35 days after that.
the one route
The thirty-nine fields you control
These carry text we store as written, so they are the one route by which your content could reach us. On each call:
source · event_id · workflow_id · session_id · prompt_id · operation · environment · team · task_key · task_class · span_id · parent_span_id · stopping_rule · harness · provider · model · finish_reason · failing_criteria · case_id
On each outcome:
source · event_id · workflow_id · operation · span_id · link · environment · arm · quality_provenance · check_results · judge_model_id · judge_signature · value_source · value_field · eval_commit · input_features
On each tool cost you send, if you send any:
event_id · workflow_id · session_id · name
They exist to be labels, and short ones work best in the console: name a workflow invoice-extraction rather than pasting the invoice. The SDK’s eval adapters fill link with a record id or a row number, never the question an eval asked. The promptfoo and deepeval adapters use the name or description you gave a test case first, so keep those short too.
Each ingest request is also stored whole, one row per attempt, the accepted and the refused alike, which is how a mis-shaped payload stays visible. That copy is not pruned on the thirty-day window the dropped folder gets. It carries those same fields and no prompt or completion.
One more: when a measurement run fails a test, we keep the first 200 characters of its failure message so the switch report can name the reason, and withhold any line that looks like a credential.
sub-processors and hosts
Where it lives
Postgres hosted by Neon in AWS us-east-1, behind an application hosted by Vercel. Encrypted in transit and at rest by both. Measurement runs are prepared on a server run by boat (Dedale AI, Corp.) and execute your code in a per-run sandbox on Vercel, model calls go through OpenRouter, and Sentry receives errors. Those five are our sub-processors. Four are in the United States; boat is a US company whose server is in France, on OVH. Zoho sends your sign-in email and sees only your address and the link.
PostHog receives page views and autocapture events. On the marketing pages autocapture sends the text of the element you click. On every route that renders your data it records only that a click happened and on what kind of element, without the element's text or attributes, so a spend figure on the console does not travel with it. A session recording stays off on those routes too, and no prompt or completion can reach PostHog, because no column could hold one. It is named on /privacy as well.
The model hosts OpenRouter may reach for your data are Microsoft Azure, Google Vertex AI, Amazon Bedrock, DeepInfra, Morph, InferenceNet, Parasail and Cloudflare in the United States, and Mistral AI in France, each on a zero-data-retention endpoint. Morph, InferenceNet, Parasail and Cloudflare were added on 17 October 2026. The Data Processing Addendum names all of them, commits to thirty days notice before any change, and gives you the right to object. That notice covers the named hosts; the “any zero-retention host” choice below is your general authorisation of OpenRouter’s list as it changes, so it carries no per-host notice.
How a measurement run’s hosts are chosen
Two policies decide which of those hosts a measurement run may use. The default is the nine named hosts. A signed-in member can instead choose “any zero-retention host” in the console, which lets a run use any host on OpenRouter’s public zero-data-retention list at the time of the call, so a model the nine do not serve can still be measured. Choosing it is your general authorisation of that list as it changes, so no per-host notice is given. You can switch back to the named hosts at any time; doing so also stops the keys of that project’s runs in progress, which then fail and say why. Under both policies every call goes only to a zero-retention endpoint, and the run key itself enforces zero retention, not a separate no-training setting. The list is read when a run key is issued and refreshed on later issues, so a host removed from it can stay allowed on a key issued before, for at most four hours; OpenRouter’s own zero-retention check still applies on every call. An ingest key cannot change the choice, a list that cannot be read means the run does not start, each call records the host that served it, and each choice is recorded with who made it and when.
A customer who prefers it gets a separate deployment with its own database, so isolation is physical rather than a query filter.
commitments
What we will not do
- Train or fine-tune any model on your data.
- Show your results to another customer. Other customers see model results only combined across at least five customers, and you can opt out by written notice (DPA Section 9).
- Sell or share your data with anyone outside the subprocessors above.
- Route your model traffic, which is what keeps our advice neutral.
These are commitments in the DPA, not preferences. Deletion is thirty days from a written request, and breach notice is without undue delay and inside seventy-two hours.
the grade
If your agent games its tests, is the parity number still true?
An agent can find a shortcut to the score it is graded on, such as editing a test or printing a passing summary. Four guards are built and tested today; each has a test that fails if it stops.
- A passing summary over a failing exit is not a pass. If the test output says everything passed but the process exited non-zero, the two signals disagree and the run is skipped, never counted (
extract_tests_passinoss_outcome.py). An unclear result is skipped the same way. This can drop a real pass, for example when a coverage gate exits non-zero, and a dropped pass can only make parity harder to reach. - A suite that grades with the model under test is left out. Its pass rate moves when the model moves, so it is not evidence about the route. It is dropped from both arms alike, and when every suite is like that there is no verdict (
faceoff_cases.pyandworkspace_faceoff.py). - A grader nobody has read can be checked by you. You open five failing and five passing cases with the transcripts open and say whether you would have passed each. If more than one in ten disagree with your grader, a switch that would have been banked is held (
grader_check.py). It is optional, so an unread grader is labelled unchecked rather than blocked. - The agent cannot name the file the verdict is read from. A stricter grading path runs a separate grader after the agent and reads its verdict from a location outside the agent’s checkout. A task with no such grader is refused rather than read from the agent’s own files (
oss_executor._grade). Face-offs on your tests grade by test output today, so this guard is built but is not the one your face-off uses.
What is not covered
- Your tests run in the agent’s own checkout. An agent that edits them, or arranges both a zero exit and a faked summary, still passes. Closing that needs the agent and the grader to run as separate users or in separate containers, and that work is open.
- Even on the stricter path, the agent runs as the same user as the harness, so it could rewrite the verdict file. Not observed in practice; read from the code.
- Your cases are reused across face-offs, and the confirmation set is small. A face-off keeps about a fifth of your cases out of the choice of candidate, and a switch is held unless it is confirmed on them. That slice is too small to prove a 3% tolerance, so it only rules out a material regression, and a slice too small to answer holds the switch. A suite with no per-case identity has no held-out slice, so its switch is held. A model that has seen your cases can still score higher for a reason that is not capability.
audits and reviews
What procurement can have today
If your company runs a vendor security review before it buys, this is what the review can have today.
- A filled security questionnaire. Yours, or a standard one.
- The data map on this page. Every field we store, the two ways text can reach us, and each sub-processor by name.
- The DPA. The commitments above are in it.
- An on-site review on reasonable notice.
No SOC 2 report today, and no ISO 27001 certificate. If your policy requires one, we are not yet a vendor you can approve, and we would rather you read that here than in the third meeting.
Questions, or a questionnaire to fill: subh@trymargin.io. The legal text: Terms, Privacy, DPA.
Margin never sees your live prompts.
A meter, not a proxy: we cannot read a request, and an ingest outage cannot stop your agents. The exception is a console run of your own evals on our server.
margin-meter is public on PyPI and npm. Read it before you install it: the privacy policy and DPA list every field it records.
The published package trails this build: PyPI is 0.6.23 (repo 0.6.24). A plain pip install does not fetch every adapter shown yet. Setup through /start works on it. Registries read 2026-11-02.
On the wire
- workflow name
- provider
- model
- input tokens
- output tokens
- latency
- pass or fail
- computed cost
Plus a link you choose to attach, or nothing.
Never on the wire
- prompt text
- model output
- tool arguments
- your users' data