Private AI Inference: Who Else Sees Your Prompt?

What is private AI inference?

Private AI inferenceInferenceRunning a trained AI model to produce an answer. Inference is what happens when you type a prompt into ChatGPT and get a response. The model takes your input, computes a best guess, and returns it.Like asking an expert for their opinion. The training was the decades they spent becoming an expert. The inference is the 30 seconds it takes them to answer your specific question.Inference in the glossary → means running an AI model on your promptPromptThe text you give an AI model to tell it what to generate. A prompt can be a simple question, a long instruction, a chunk of context plus a task, or a conversation history the model uses to produce its response.Like a brief you give to a junior designer. A vague brief gets a vague result. A detailed brief with context, constraints, and examples gets something usable. The quality of the output depends heavily on the quality of the brief.Prompt in the glossary → so that nobody except you can read the prompt or the answer while the model works. There are three ways to get there: run the model on your own hardware, run it inside a hardware enclaveEnclaveAn isolated region of CPU or GPU memory protected by hardware. Code and data inside the enclave are inaccessible to the operating system, the hypervisor, or even the machine's physical owner.Like a secure room inside a much larger office building. The building's caretakers have keys to every other room but not this one. What happens inside is invisible to them by design.Enclave in the glossary → the operator can’t look into, or compute on encrypted data. Anything short of that is privacy by policy: someone can read your prompt and promises not to.

Every AI privacy project says it’s private. Most of them are using the word differently.

Venice alone offers three meanings. In Private mode your identity is stripped before the prompt reaches a GPUGPUGraphics Processing Unit. Originally designed to render video game graphics, GPUs turned out to be exceptionally good at the massively parallel math that AI models need. Modern AI training and inference runs almost entirely on GPUs.Like a factory with 10,000 workers doing the same simple task in parallel, versus a CPU which is more like 10 workers each doing different complex tasks. AI training involves doing simple math a million times per second on a million numbers, which is exactly what the GPU factory is designed for.GPU in the glossary →, but the GPU operator still reads the text. In TEETEETrusted Execution Environment. A hardware-secured region of a CPU or GPU where code runs in isolation, so even the machine's operator can't read what's happening inside. TEEs give decentralised AI inference privacy guarantees.Like a bank vault inside a bank. The bank owns the building, staffs the lobby, and runs the security cameras. But what's inside the vault is invisible to everyone, including the bank staff, unless the customer opens it.TEE in the glossary → mode the model runs inside an enclave the operator can’t inspect. In E2EE mode the prompt is encrypted on your device and only decrypted inside a verified enclave. Run a model on your own Mac and there is no operator at all.

The use case sets the level you need. A blog post draft needs far less protection than your company’s financial projections, a patient’s records, or your legal strategy. Where a platform sits on the spectrum tells you what you can safely send it.

The five levels of AI inference privacy

Level 0: No privacy. OpenAI, Anthropic (default)

The provider sees your prompt in plaintext and keeps records of it. How long depends on the product and the settings you pick. OpenAI’s API keeps abuse-monitoring logs for up to 30 days by default, and zero data retention needs OpenAI’s approval, per its data controls guide. Anthropic’s August 2025 consumer terms keep Claude chats for five years if you allow them to be used for training, and 30 days if you don’t.

The leaks have already happened. In April 2023 Samsung staff pasted faulty source code into ChatGPT, and the company restricted generative AI tools on company devices the following month. Business Insider reported in January 2023 that Amazon had warned its staff not to share confidential information with ChatGPT.

With default settings, a centralised provider knows who you are and what you asked, and keeps both for as long as its policy says.

Level 1: Policy-based privacy. Venice (Private mode), Morpheus (standard providers)

Most “private” AI platforms sit here, and it pays to be precise about what that buys you.

Venice in its default Private mode sends your prompt through a proxy that strips your identity before it reaches a GPU. Venice’s privacy page says inference runs on Venice-controlled GPUs or zero-data-retention partner infrastructure, with privacy enforced by contractual commitments. In practice:

• Your identity is hidden. The GPU operator has no way to tie the prompt back to you.
• Your prompt content is visible. The GPU must see plaintext to run the model. Nothing is encrypted during processing.
• Nothing is stored, by policy. Venice says it doesn’t log prompts and its partners are contracted not to. No hardware or cryptography stops an operator copying a prompt while it sits in GPU memory.

To use a construction analogy: it’s like sending a confidential document by courier with the return address torn off. The recipient doesn’t know who sent it, but they open the envelope and read the contents. They promise not to photocopy it, and you trust them on that.

Level 1 is a big step up from Level 0. There’s no persistent record, no identity link, and no log to subpoena. For most work (writing, coding, analysis, creative work) it’s enough.

If you’re pasting a cap table, a patient’s records or a client’s legal strategy into a Level 1 system, understand that a GPU operator could read it during inference. They don’t know it’s yours and they say they won’t keep it, but the text passes through their hardware in plaintext.

With a standard provider, Morpheus is a Level 1 service. Its peer-to-peer routing is encrypted in transit, per Morpheus’s docs, but the provider’s GPU sees the prompt. Route to a TEE-tagged provider and it moves to Level 2, covered below.

Level 2: Hardware-enforced privacy. Venice (TEE/E2EE), NEAR AI Cloud, Phala, Morpheus (TEE providers)

At Level 2 the guarantee moves from policy to silicon.

Trusted Execution EnvironmentsTEETrusted Execution Environment. A hardware-secured region of a CPU or GPU where code runs in isolation, so even the machine's operator can't read what's happening inside. TEEs give decentralised AI inference privacy guarantees.Like a bank vault inside a bank. The bank owns the building, staffs the lobby, and runs the security cameras. But what's inside the vault is invisible to everyone, including the bank staff, unless the customer opens it.TEE in the glossary → (TEEs) are isolated regions built into modern processors and GPUs. Memory inside the enclave is encrypted, so the operator who owns the server can’t read what’s being processed. The enclave also produces an attestationAttestationA cryptographic proof that a piece of code is running on a specific hardware enclave in an unmodified state. Attestation lets remote users verify that a service is genuinely running what it claims to be running.Like a tamper-evident seal on a medicine bottle. The seal itself doesn't make the medicine safe, but it gives you a way to verify that nobody opened the bottle and swapped the contents before you bought it.Attestation in the glossary → report: a signed statement from the chip maker’s hardware that says what code is running and that it hasn’t been tampered with. Today’s stack pairs an Intel TDX or AMD SEV-SNP confidential VM with an NVIDIA GPU in confidential computing mode, which NVIDIA introduced with the H100.

Venice TEE and E2EE modes

Venice launched TEE and E2EE modes on 18 March 2026. Inference in both runs inside enclaves operated by NEAR AI Cloud and Phala Network. In E2EE mode your prompt is encrypted in the browser, stays encrypted through Venice’s proxy, and is only decrypted inside the verified enclave. Each E2EE response carries a verification icon that links to the attestation report.

The trade-offs:

• E2EE switches off features. Web search, memory, file uploads and function calling are unavailable, because each would need your data decrypted outside the enclave. TEE mode keeps them.
• Paid tiers only. Both modes sit behind Venice’s Pro plan and above.
• A subset of models. Only the models Venice tags tee- or e2ee- run this way.

Both modes also work through the APIAPIApplication Programming Interface. A structured way for one piece of software to talk to another. In DeAI, APIs let applications request inference from a model without running the model themselves.Like a waiter in a restaurant. You don't walk into the kitchen and cook your own meal. You tell the waiter what you want, they tell the kitchen, the kitchen cooks it, and the waiter brings it back. The API is the waiter.API in the glossary →. Developers pick models by prefix, and E2EE needs client-side ECDH key exchange (secp256k1, HKDF-SHA256) with AES-256-GCM encryption, per Venice’s developer docs.

Venice wasn’t the first to put hardware-enforced privacy in front of consumers. Apple announced Private Cloud Compute in June 2024 and opened a research environment for inspecting its builds that October. Apple’s version is bound to Apple devices and Apple’s verification tooling. Venice runs in any browser on inference operated by independent providers, and the attestation report is yours to check.

NEAR AI Cloud

NEAR AI Cloud sells private inference directly, and it’s also one of Venice’s two enclave operators. Its docs describe Intel TDX confidential VMs paired with NVIDIA GPU isolation, with each environment producing cryptographic proofs of its integrity and configuration. NEAR publishes open-source verifier code, so developers can check a deployment’s evidence before sending a prompt and check a response’s signature afterwards.

Read the model list carefully. NEAR tags open-weight models such as DeepSeek, Kimi and Qwen as “Private TEE”. The Anthropic, OpenAI and Google models on the same page carry an “Incognito” label instead. Those companies don’t release their weights, so NEAR can’t load them into its own enclaves.

Phala Network

Phala Network runs GPU TEE inference and cloud hosting on NVIDIA H100, H200 and B300 hardware paired with Intel TDX, and holds SOC 2 Type I and HIPAA compliance, per its own site. Its open-source dstack SDKSDKSoftware Development Kit. A collection of code libraries, documentation, and tools that lets developers integrate a service into their applications without writing everything from scratch. SDKs are how projects become easy to build with.Like a plug-and-play kit for building furniture. You don't have to mill your own wood, forge your own screws, or design the joinery from scratch. The kit gives you pre-cut parts and instructions so you can assemble the thing in an afternoon.SDK in the glossary → packages standard Docker containers as confidential VMs.

Bittensor’s Chutes subnet (SN64) is heading the same way. It says it runs a TEE mode on Intel TDX and NVIDIA confidential computing, and says prompts on eligible TEE-backed instances are encrypted on the client and only decrypted inside a verified enclave.

Morpheus with a TEE provider

Morpheus shipped attestation in two phases. v6.0.0 (18 March 2026) attests the provider’s proxy-router: your node checks that the provider is running the published, signed build inside Intel TDX, with chat logging switched off, before opening a session. v7.0.0 (23 April 2026, with SCRT Labs’ SecretVM) extends the chain to the GPU, binding the CPU enclave to NVIDIA’s GPU attestation on every prompt, according to its release notes. v7.2.0 (2 June 2026) added AMD SEV-SNP hosts alongside Intel TDX, though the v7.15.0 release notes say that SEV check failed open until 14 October: an empty measurement was skipped, so any SEV host passed.

So Morpheus reaches Level 2 end to end against a TEE provider. Against a standard provider it stays at Level 1, because Morpheus is a marketplace and runs no GPUs of its own. The privacy level is whatever the provider you route to delivers, and Morpheus’s contribution is letting your node verify it.

What it costs, and where it breaks

The speed penalty is small. A 2024 benchmark of confidential computing on NVIDIA Hopper GPUs found overhead below 7% for most typical LLM queries, falling to nearly zero for larger models and longer sequences. Most of that cost is moving data between CPU and GPU, not the computation itself.

The trust model is the weak point. You’re trusting Intel, AMD and NVIDIA to have built the enclave correctly, and physical attacks keep landing:

• WireTap (September 2025). Researchers pulled SGX attestation keys out of DDR4 servers with under $1,000 of equipment. Phala stopped admitting SGX workers the same day and moved its roadmap entirely onto TDX and NVIDIA confidential computing.
• TEE.fail (October 2025, IEEE S&P 2026). A DDR5 memory interposer, also under $1,000, extracted attestation keys from Intel TDX and broke AMD SEV-SNP. The researchers showed they could “borrow” NVIDIA attestations from GPUs they didn’t own, so a user could be convinced their chat was running inside a GPU TEE while it ran in the clear. Phala’s dstack was one of the systems they demonstrated against.

Both attacks need physical access to the server. That turns “a rogue operator reads your prompt” into “a rogue operator with a soldering iron, a budget and intent forges your proof.” That’s a far higher bar than Level 1, but it isn’t zero, and TEE.fail shows attestation proves the hardware is authentic. It doesn’t on its own prove your prompt ran on that hardware.

Practical guidance: for healthcare, legal work and proprietary data, Level 2 is now the sensible default when you need a frontier-scale open model you can’t run locally. Prefer services whose attestation binds to your specific connection, and keep the most sensitive material for Level 4.

Level 3: Cryptographic privacy. FHE, MPC, ZK

Level 3 is the mathematical guarantee. There’s no trust in hardware vendors or policies, only the cryptography.

Fully Homomorphic Encryption (FHE)

Fully Homomorphic EncryptionFHEFully Homomorphic Encryption. A cryptographic technique that lets you compute on encrypted data without decrypting it. The result is also encrypted, and only the data owner can read it. FHE is the strongest form of computational privacy.Like sending a sealed box of ingredients to a chef, having the chef cook a meal inside the box without ever opening it, and getting back a sealed box with the finished meal. Only you can unseal it and see what's inside.FHE in the glossary → lets a server compute on data that stays encrypted the whole time. Your prompt is encrypted on your machine, the GPU works out the answer without ever decrypting it, and only your key can read the result. Without that key the provider can’t see anything.

It’s slow. Zama’s own Concrete ML demo runs a single attention block from GPT-2, a 2019 model, under FHE in about 14 minutes on a 196-core server. A modern LLM has dozens of those blocks per token, so FHE is nowhere near real-time chat today.

Multi-Party Computation (MPC)

Multi-Party ComputationMPCMulti-Party Computation. A cryptographic technique where multiple parties jointly compute a function over their inputs without any party revealing its input to the others. Useful for shared computation on private data.Like a group of friends calculating the average of their salaries without anyone revealing their actual salary to the others. Each person contributes a piece of the answer, and the pieces combine into the result, but nobody learns anything except the average itself.MPC in the glossary → splits your data into shares held by several parties. Each computes on its share, none sees the whole, and the result is reconstructed at the end.

For LLMs the coordination cost is brutal. The PUMA research system needed around five minutes to generate a single token from LLaMA-7B under MPC in 2023. MPC suits smaller, well-defined jobs: private statistics, key management, and encrypted storage.

Arcium is the most visible MPC network in crypto. It acquired Inpher’s core team and technology, including the Manticore MPC protocol, in November 2024, launched Mainnet Alpha on Solana in February 2026, and has since launched its ARX token. Arcium is a member of NVIDIA’s Inception programme. Nillion uses MPC for its nilDB storage, but its nilAI LLM service runs inside TEEs on NVIDIA confidential computing, which makes Nillion’s AI inference Level 2, not Level 3.

Zero-Knowledge Proofs (ZKPs)

Zero-knowledge proofsZKZero Knowledge. A class of cryptographic proofs that let you prove something is true without revealing any of the underlying information. ZK lets a network verify a transaction without seeing the transaction's contents.Like proving you know the password to a safe by demonstrating you can open it, without ever saying the password out loud. The verifier learns that you know the password and nothing more.ZK in the glossary → prove a computation was done correctly without revealing its inputs. You could prove “this model produced this output from a valid input” without showing the input.

ZK proves a result is correct, and on its own it leaves the data exposed during computation. For inference privacy it works alongside TEE or MPC: one keeps the data hidden during processing, ZK proves the result afterwards.

Level 4: Local and on-premises inference

The most private inference never touches a network. Run an open-weight model on hardware you own and there’s no provider, no policy and no attestation to check. The prompt never leaves the machine.

What that takes in 2026:

• Hardware. Memory is the constraint. Ollama lists Llama 3.3 70B at 43GB in its default 4-bit build, so a Mac Studio with 64GB or more of unified memory, or a workstation with enough GPU memory, runs it. Smaller models run on an ordinary laptop.
• Software. Ollama, llama.cpp, LM Studio and Apple’s MLX are free and run the major open-weight families. Our open-weight model comparison compares the families by size and licence.
• On-premises for teams. Businesses that can’t send data out are buying turnkey inference servers from vendors such as Zanus AI, or building their own racks, and serving models behind their own firewall.

The limits: you’re restricted to open-weight models (no Claude, no GPT), capable hardware is a capital cost, and speed depends on your own specs. Our case for self-hosted AI tests where that trade-off pays on a Mac Studio. For legal documents, medical analysis and proprietary code, local inference is still the gold standard.

Where each project sits

How to check a private inference claim

A privacy claim is only as good as what you can check yourself. We run these six questions against any service that calls itself private:

1. Who holds plaintext at each hop? List every component between your keyboard and the GPU: app, proxy, gateway, provider. Any one that sees plaintext sets your privacy level.
2. Can you verify the attestation yourself? A green tick in the vendor’s own interface is a claim. An attestation report you can check with open-source tooling is evidence.
3. Is the attestation bound to your connection? Look for a TLS certificate fingerprint or a fresh nonce inside the signed report. Without that binding, a valid report can be replayed from a different machine, which is exactly what TEE.fail demonstrated.
4. Does it cover the GPU? A CPU enclave around the router says nothing about the model server unless GPU attestation is chained to it.
5. What sits outside the enclave? Your IP address, account, payment details and timing are metadata, and enclaves don’t hide them.
6. Who can change the code? An enclave running a build that one company can swap overnight is only as trustworthy as that company’s release process. Signed, reproducible builds narrow that.

Here’s the method applied to Morpheus. Our own refresher reads every TEE-tagged model on chain and checks the provider’s Intel TDX quote against the TLS certificate the host presents. At the last read:

• TEE-tagged models on chain: 5
• Provider nodes serving them: 1
• Host quotes our check verified: 1
• Passing GPU attestation, per the provider’s own health endpoint: 2
• Flagged unverified, per the same endpoint: 0 (a model whose backend is down, rather than failing attestation, counts in neither)

That answers questions 2 to 4 for Morpheus with evidence rather than trust. Question 1 depends on how you connect. Run your own node (from v7.9.0, since older routers fail TEE sessions) and the chain is attested from your machine to the GPU. Use the api.mor.org gateway and its own node verifies the provider for you, but the gateway hop itself is private only by policy, per Morpheus’s TEE page, and it knows your identity from sign-up.

What shipped in 2026, and what comes next

The first nine months of 2026 moved hardware-attested inference from pilots into consumer products:

• Venice TEE and E2EE (18 March). Per-response attestation in a consumer chat app, on hardware run by independent operators.
• Morpheus Phase 1 and Phase 2 (18 March and 23 April). Attestation from the consumer’s node to the GPU, with AMD SEV-SNP hosts added on 2 June (a check that failed open until 14 October).
• Arcium Mainnet Alpha (February). Arcium’s MPC network went live on Solana, though MPC is still far too slow for LLM chat.

What’s still open: Morpheus has a single TEE partner, with a second on its roadmap and gateway-side attestation at the research stage, per its TEE page. Its release notes list native RA-TLS, which embeds attestation in the TLS handshake itself, as future work. On the hardware side, the TEE.fail class of physical attacks needs a response from Intel, AMD and NVIDIA, since both Intel and AMD treat physical memory attacks as out of scope for their current designs.

FHE is the long game. We think fully encrypted inference for interactive chat is still years away. Dedicated FHE accelerator hardware is the development that would change that.

A practical decision framework

Pick the level from the use case. The best level on paper is rarely the one you need.

Level 1 is enough when:

• Your prompts are everyday knowledge work (writing, coding, analysis)
• The content wouldn’t cause harm if read without your name attached
• Your threat model is “I don’t want a company logging and training on my data”

Level 2 is required when:

• Your prompts contain personal information about other people (healthcare, legal)
• Regulation expects verifiable protection during processing (HIPAA, GDPR processing obligations)
• Your threat model includes a compromised infrastructure provider
• You need to show an auditor the data was protected during processing

Level 4 is required when:

• Maximum assurance is non-negotiable (trade secrets, privileged legal material)
• You can’t trust any outside infrastructure, including hardware-attested enclaves
• You have the hardware and accept the model and speed limits

Most people overestimate the level they need and underestimate the friction of the higher ones. If you’re not handling data covered by HIPAA, securities rules or legal privilege, Level 1 is probably enough. The bigger problem is Level 0: sending everything to a provider that keeps it.

The infrastructure battle

Three forces are pushing private inference from niche to default.

Regulation. GDPR, the EU AI Act, HIPAA, CCPA and a growing list of US state privacy laws all push toward the same requirement: show that data is protected while it’s processed. “We have a policy” won’t satisfy auditors for long.

Enterprise demand. Every corporate legal team restricting ChatGPT is a potential customer for private inference. Banks and healthcare providers face the same wall Samsung hit in 2023, with client and patient data in place of source code.

Cost. TEE overhead is now in single digits for most LLM work, so privacy no longer carries a large performance tax. Decentralised compute is pushing the base cost down too: Akash, for one, claims savings of up to 85% against AWS, Azure and Google Cloud.

Venice has shown there’s a mass market for Level 1, reporting 3.5 million registered users when it announced its Series A on 1 July 2026. Its TEE and E2EE modes, running on NEAR and Phala hardware, show a consumer app can offer Level 2 without asking users to understand the cryptography. The question is which projects make private inference invisible enough that people choose it without thinking, the way HTTPS became the default for the web.

Frequently asked questions

What is private AI inference?

Running an AI model on your prompt so that nobody except you can read the prompt or the response during processing. It’s delivered by local models, hardware enclaves (TEEs) or encrypted computation. Services that only promise not to log your data offer policy-based privacy, a weaker level.

Is TEE inference really private?

Against the operator of the server, yes, as long as the enclave hardware holds. TEEs encrypt memory so the operator can’t read your prompt, and attestation lets you check the enclave is genuine. The residual risk is physical attacks such as WireTap and TEE.fail, which need hands-on access to the machine.

What’s the difference between private inference and confidential computing?

Confidential computing is the hardware technology: CPU and GPU enclaves that keep data encrypted while in use. Private inference is one application of it, alongside local models and cryptographic methods such as FHE and MPC.

Is Venice AI private?

It depends on the mode. Venice’s default Private mode hides your identity but the GPU operator can read the prompt (Level 1). Its TEE and E2EE modes run inside attested hardware enclaves operated by NEAR AI Cloud and Phala, so the operator can’t read it (Level 2). Our Venice review covers the detail.

Can I run private AI inference at home?

Yes. Open-weight models run locally with free tools such as Ollama, llama.cpp, LM Studio or MLX. Small models run on a laptop. A 70B model in its 4-bit build needs roughly 43GB of memory, so a machine with 64GB or more.