Guide / Checked Nov 2026
How do I test an AI trading agent on paper?
Forward test it, because an agent's calls change from run to run and are hard to backtest. Give each model its own strategy with its own token and a paper bot on Nyria Paper, the same Budget per entry, the same instruments and the same window. Write the pass rule before the first alert, run a plain baseline alongside, and read the Logs tabs daily. Paper shows the setup works. It doesn't show whether the approach makes money.
Official documentation
Run a fair paper test
- 1
Write the pass rule first
Decide what counts before any alert fires: the window in trading days, the instruments, and the number you'll compare. Write it down. A rule written after the fact bends to the result.
- 2
One strategy per model
Create a strategy with AI Agent (MCP) as the alert source for each model you want to compare. Each has its own token, so each model's alerts land in its own Logs tab.
- 3
Same bot settings everywhere
Deploy a paper bot on Nyria Paper under each strategy with the same Budget per entry, allowed instruments and trading window. Nyria Paper is free and takes stocks, options, spot crypto and perpetuals.
- 4
Add a plain baseline
Run one more strategy with a fixed rule you already trust, on the same settings. An agent's result means little without a dull comparison next to it.
- 5
Preview, then let it send
Ask each agent to preview before it sends. Sending the same alert twice in a row sends it once, and bursts are refused. The agent sees that in its reply, not in the Logs tab. An agent that sends many different entries still reaches your bot, so read the Logs tab daily.
- 6
Read the logs and decide
Each day, open the strategy's Logs tab for passed and failed alerts, and the bot's Logs tab for Placed, Skipped and Failed. At the end of the window, apply the rule you wrote. Only then think about a live bot.
Pass rule to copy
A pass rule, written before the test
Window: 20 trading days, the same for every strategy Instruments: SPY and QQQ only, long only Budget per entry: $500 on every paper bot Baseline: one fixed-rule strategy on the same settings Pass: closed-trade P&L above the baseline AND zero failed alerts in the log
Change the numbers to fit your account. 20 days is an example length.
What paper doesn't simulate
- Nyria Paper fills a market order at the ask to buy and the bid to sell. It never slips.
- On Nyria Paper, partial fills, broker rejections, halts, assignment and funding aren't simulated.
- Paper P&L usually reads better than live for those reasons. Don't treat a paper pass as a forecast.
- Alpaca paper fills from real-time quotes and doesn't simulate dividends. Its paper key differs from the live key.
- Nyria can't connect a paper account at Charles Schwab or tastytrade. Test those strategies on Nyria Paper.
Keep the test fair
- Up to 10 paper bots run free, counted across Nyria Paper and brokers' paper accounts.
- Don't change Budget per entry or the window mid-test. Start a new strategy instead.
- The agent reads fills and average price over MCP, not P&L. Judge from the bot page, not from what the agent says about itself.
- Test alert wording in the alert tester before the window opens.
Questions
- Why not backtest the agent?
- Its calls change from run to run, so a backtest can't replay them. Nyria has no built-in backtester; you test a strategy going forward, on a paper account, in real time.
- Can two models share one strategy?
- They can, but their alerts mix in one Logs list and one bot. Separate strategies keep each model's log and P&L apart.
- Does a paper pass mean the agent will make money live?
- No. Paper shows the setup works. Live fills slip, partial fills happen and brokers reject orders. Move one bot to live with a small Budget per entry and compare its log with paper.
- Which model trades well?
- Run the test above. The answer holds only for your rules, your instruments and your window.
Next
- Paste an alert into the alert tester to see how Nyria reads it.
- Step by step for one account: how to automate Webhook to Alpaca.
- Step by step for one account: how to automate Webhook to Schwab.
Guides
Broker sandbox vs paper trading
What Nyria Paper, Tradier's sandbox and Alpaca paper each prove.
Read guideHow do I connect Claude or ChatGPT to my broker?
Claude or ChatGPT sends the alert over MCP, and a Nyria bot places the order at your broker.
Read guideHow do I build an AI trading agent with Claude?
Claude decides, a Nyria bot places. Paper first.
Read guideHow do I move a paper trading bot to live trading?
Go live starts a new live bot with your paper settings. Same webhook URL.
Read guideTry a paper bot first
No card required. Paper is free. Live trading from $15 a month.