Launch TapeA lot of people now build their Solana bot with an AI doing most of the code. That works surprisingly well for the plumbing. Where it goes wrong is the testing: the AI happily writes a backtest, runs it on whatever data is handy (or invents some), and tells you the results look promising.
The fix is not a smarter AI. It is giving it the right data and asking the right questions. Here is how we do it.
Start with real files, not descriptions
If you describe your data in words ("I have trades with prices and times"), the AI fills the gaps with guesses about column names, units and time zones. Sometimes the guesses are right. Often they are quietly wrong, and the backtest still runs.
Give it the actual file, or at least the first few rows plus the schema. Real rows answer questions you did not think to ask: is the price in SOL or USD, is time in seconds or milliseconds, is there one row per trade or one per block.
Takeaway: paste a sample of the data, not a summary of it.
Here are the first 5 rows of trades.jsonl and the schema.
Before writing any code, tell me in one line each:
what one row means, the unit of every price, and the unit of time.Small files first
Chat tools have upload size limits, and long files also eat the AI's working memory. Trade data gets big fast: a single full day of our graduation trades (October 12, 2026) is about 800 MB compressed. No chat window is going to read that.
So work in two steps. First, a small slice (a couple of hours, or one coin) that fits in the chat. Use it to get the code right and check a few trades by hand. Then run the same script on your own machine over the big files.
Takeaway: use the chat to build and check the script; use your computer to run it at full size.
Make it state its assumptions
The single most useful line you can add to any prompt is: "Before you run anything, list every assumption you are making." Typical answers: fills at the last seen price, no fees, coins with missing data skipped.
Each of those is a choice. Some are fine. Some make the result useless. You cannot judge them if you never see them.
Takeaway: an assumption you can see is one you can fix.
Ask for honest fills, by name
AI-written backtests nearly always fill at the signal's price, because that is the simplest code. Tell it not to, in plain words:
- every buy and sell fills at the end of the first block after the signal
- fees and slippage on both sides
- your trade size moves the pool's price
- coins that never reached your entry are counted, not dropped
Then ask it to run the same rules both ways, naive and honest, and print the two results side by side. If they are far apart, the naive one was flattering you. Our post on backtests that lie goes through each of these.
Takeaway: if you do not ask for honest fills, you will not get them.
A prompt skeleton
Here is a short version you can adapt. Fill in the brackets.
I'm testing a Solana memecoin bot's rules on real trade data.
Files: [file names], schema: [paste or attach].
RULES
- Entry: [e.g. buy 15 s after graduation]
- Exit: [e.g. sell at 3 min, or -20% from my entry]
- Size: [e.g. 0.1 SOL a trade]
HOW TO TEST
- Fill every signal at the end of the next block, never at its own price.
- Charge [my real costs] on the buy and on the sell.
- My size moves the pool price (use the pool's reserves).
- Count coins that never reached my entry; don't drop them.
BEFORE CODING: list every assumption you're making.
AFTER: print win rate, median and mean return a trade, total,
the 3 worst trades, and the same rules at signal price with no
costs, for comparison. Then a plain verdict, including how small
the sample is.Our Test your bot page has a longer, complete version that already knows our file formats.
Takeaway: a good prompt reads like a spec. If a stranger could not run your test from it, neither can the AI.
Check one trade by hand
When the results come back, pick one trade, open the raw rows for that coin, and follow it yourself: when the signal fired, which block the buy filled in, what it cost, when it sold. If you cannot match what the script says, something is off.
Takeaway: one trade checked by hand is worth more than a hundred trusted blindly.
The hopeful part
An AI is a fast, tireless helper for this kind of work, as long as you hold it to the same standard you would hold a person. Better rules might exist for your bot. Give the AI real data and honest instructions, and it can help you find out.
If you want real files to start with, the free graduates CSV is two hours of pump.fun graduations on one sheet, small enough to try today, and one full day is $10.