Is an AI-built internal tool ready for production? What our 13 build tests didn't test

Thirteen AI builds of internal tools passed all 358 of their acceptance checks. Every one of them still left the same things untested, starting with production sign-in and hosting. Here is the list from our build notes, with how often each item came up and what to check before your team relies on the tool.

Updated October 17, 2026 · Markdown version

The short answer

Not yet. Passing the spec's acceptance checks shows the tool does what the spec says, on one machine, with test data. It does not show that sign-in, email, scheduled jobs, the database and hosting work in the real world. Those are exactly the parts our test builds did not exercise.

We had Claude, working in Type, build thirteen internal tools from our blueprints, from an approval workflow to a help desk. All thirteen passed every acceptance check in their final browser runs: 358 checks in all. Each blueprint's build notes then list what we did not test. Put side by side, the lists are nearly identical:

Not tested	Builds	What we did instead
Production sign-in	13 of 13	A development "sign in as" switcher
Hosting and a production database	13 of 13	SQLite on one server
Real email delivery	12 of the 12 that send email	Emails written to an outbox table and checked there
A real scheduler	11 of the 12 with a scheduled job	The job's endpoint called directly, often with a development clock
Two people or processes writing at once	7 of 13	Automated tests, or a repeated click, not parallel sessions
A live outside system (Slack, inbound email, calendar apps, payroll, accounting)	5 of 13	Signed test requests, documented example payloads, or file contents
Other browsers and real phones	All ran in Chromium only; 5 call it out	Chromium, including a 390-pixel-wide window
HTTPS, secure cookies and rate limits behind a real proxy	4 of 13 call it out	Not tested

Source: the "What we did not test" section of each build-tested blueprint, as published on October 17, 2026. The Approval Workflow has a scheduled job, but its notes don't mention the scheduler either way, so it isn't counted in that row.

The pattern is not a quirk of these thirteen tools. A test build is checked on a laptop-sized setup because that is what makes checks repeatable. Everything that depends on real accounts, real time and real infrastructure is left for deployment, and that is where a team has to look before it relies on the tool.

What to check before your team relies on it

These are our recommendations from the gaps above. They are not results from our build tests.

Every one of our builds had a development control to sign in as any seed user. That is essential for testing and dangerous in production.

Confirm the "sign in as" switcher, any ?as= parameter and any development routes are switched off when the app runs in production, and try them on the live URL to prove it.
Use your company's identity provider (Google Workspace or Microsoft) and restrict sign-in to your domain. Try signing in with a personal account and confirm it is refused.
Check what happens when someone leaves: removing them from your directory should stop them signing in.

2. The database and hosting

All thirteen builds ran on SQLite on one server. On most hosting platforms, especially serverless ones, the files on disk don't persist and several copies of the app may run at once.

Move to a managed Postgres, or your platform's database, using the same schema.
Turn on automatic backups and restore one once, so you know it works.
Decide how schema changes will be applied when you change the tool later.

3. Email that actually arrives

Twelve of our builds send email, and none of them delivered a real one. They wrote each message to an outbox table, which is how the checks could read them.

Connect an email provider, verify your sending domain (SPF and DKIM) and send a real message to an outside address.
Check that a failed send is retried and then shown as failed, rather than lost.
If the tool receives email, like the help desk, send a real reply from a real mailbox and confirm it threads onto the right record. Inbound webhooks should require a secret.

4. Scheduled jobs on a real schedule

Twelve builds have a job that runs on a schedule: daily digests, reminders, auto-closing tickets. In eleven, the notes say the job was run by calling its endpoint directly, often with a development clock to fake the date.

Set the job up on your platform's scheduler and watch it run at least once on its own.
Make sure it requires a secret header and refuses requests without it.
Check the time zone: a job that should run at 08:00 local time moves by an hour twice a year if the scheduler runs in UTC.
Make the job safe to run twice. Schedulers retry, and two runs shouldn't send two reminders.

5. Two people at once

Seven builds list simultaneous writes as untested: two managers approving the same request, two people moving the same candidate, overlapping job runs.

Open the same record in two browser windows as two people and make both changes. One should win cleanly and the other should see a clear message, not a silent overwrite.
Where the spec says something can happen only once, such as a quote accepted or a payment period closed, make sure the database enforces it, not just the screen.

6. Outside systems

Five builds connect to something outside the app (Slack, an email inbox, calendar apps, payroll or accounting files) and none was tested against the real thing.

Connect the real workspace or account and run the one action the tool depends on most, end to end.
Check what happens when the outside system is down or rejects the request.

7. Phones, browsers, HTTPS and limits

Open the tool on the phones and browsers your team actually uses, not just one desktop browser.
Confirm the live site is HTTPS only and its session cookies are marked secure.
If a public form has a rate limit, check it still works behind your host's proxy. A limit keyed on the wrong IP address either blocks everyone or no one.

8. Who looks after it

A bought SaaS product comes with someone who fixes it. A built one comes with you. Before the team depends on it, decide who updates it, where the spec and code live, and how someone exports the data if you ever move to a paid product. Our build-or-buy guide covers when that trade-off is worth it.

Why the build notes say this out loud

Each blueprint publishes what its build test did not cover because "passed every check" is easy to over-read. A build that passes is evidence that the spec is complete enough to build from and that the rules work as written. Production readiness is a separate test, and it depends on your accounts, your host and your team. The How we test page describes the method, and What AI app builders get wrong covers the 126 gaps the same builds found in the specs themselves.