# AX Check: fly.io Checked 2026-09-17. Fly.io's docs, quickstart, and pricing are all clear upfront. Quickstart gives concrete install and deploy commands, and the pricing page lists exact USD rates per second/GB-month with no login required. ## Onboarding needs a login ## Coding sessions Of three independent sessions, one completed and produced a detailed pricing breakdown covering machine size, region, and always-on versus scale-to-zero costs. The other two stopped before finishing, as the task called for launching and deploying an app through a local CLI setup rather than something reviewable in-session. ### DeepSeek V4 Pro [View public run](https://agents.withgauge.com/p/runs/49c28a6e-3f57-433e-91e7-4993474a7355) · [Read transcript](https://www.ax-check.com/fly.io/sessions/deepseek.json) #### End-to-end onboarding - **Onboarding needs a login**: The agent never obtained a Fly.io API token or account. It searched environment variables and found none (only unrelated PI/gateway keys), then confirmed the platform requires authentication by getting a 401 from the Machines API and an UNAUTHORIZED error from the GraphQL API. No credential-issuance step (like `fly auth signup`) was available in this non-interactive sandbox, so the agent pivoted entirely to reading public docs and public GitHub source instead of performing any authenticated operation. Event 24: ```text PI_GATEWAY_API_KEY= --- pi env --- PI_CODING_AGENT= PI_REASONING_LEVEL= PI_SESSION_FILE= PI_PROVIDER= PI_MODEL= PI_SESSION_ID= PI_GATEWAY_API_KEY= done ``` Event 81: ```text === https://api.machines.dev/v1/apps/demo-xyz/machines === 401 === https://api.machines.dev/v1/apps/demo-xyz/machines/abc === 404 === https://api.machines.dev/v1/apps === 404 --- graphql post no auth --- {"data":null,"errors":[{"message":"You must be authenticated to view this.","locations":[{"line":1,"column":3}],"path":["currentUser"],"extensions":{"code":"UNAUTHORIZED"}}]} ``` Event 16: ```text --- home configs --- --- net check --- 200 --- machines api --- 404 done ``` #### Hallucinated URLs - **Guessed markdown/API doc URL variants that 404'd**: The agent tried several speculative URL patterns to fetch a markdown or JSON version of the Machines API docs (appending .md, /index.md, ?format=md) that are not documented anywhere in the transcript. All three returned Fly's generic 404 page, consistent with invented paths rather than a broken documented link. Event 75: ```text === https://fly.io/docs/machines/api.md === Error 404 · Fly ``` Event 101: ```text === https://raw.githubusercontent.com/superfly/flyctl/main/internal/command/apps/create.go === 404: Not Found === https://raw.githubusercontent.com/fly-apps/flyctl/main/internal/command/apps/create.go === 404: Not Found ``` #### Blockers - **No Fly.io credentials available in sandbox, blocking any authenticated call**: This is a test-environment/credential limitation, not a product defect: the sandbox had no flyctl binary, no ~/.fly config, and no FLY_API_TOKEN in the environment. Fly.io correctly rejected unauthenticated calls to both its GraphQL and Machines REST APIs, which is expected product behavior for a platform requiring signup. Because there is no self-service token-issuance flow reachable without human interaction (e.g. `fly auth login` requires a browser/human), the agent could not move past documentation review. Event 7: ```text total 16 drwxr-xr-x 3 root root 4096 Sep 16 22:29 . drwxr-xr-x 8 root root 4096 Sep 16 22:29 .git -rw-r--r-- 1 root root 408 Sep 16 22:29 .gitignore --- done ``` Event 16: ```text --- home configs --- --- net check --- 200 --- machines api --- 404 done ``` Event 81: ```text --- graphql post no auth --- {"data":null,"errors":[{"message":"You must be authenticated to view this.","locations":[{"line":1,"column":3}],"path":["currentUser"],"extensions":{"code":"UNAUTHORIZED"}}]} ``` - **Pricing page is a heavy, script-laden document requiring custom scraping**: Product behavior/friction: the canonical pricing docs page returned nearly 600KB of markup dominated by repeated per-region pricing tables, forcing the agent to write custom Python HTML-stripping scripts across two separate fetches (/docs/about/pricing/ and /pricing/) just to extract usable numbers. This is friction in consuming the pricing page programmatically, not a credentials issue. Event 34: ```text exit=0 599360 /tmp/pricing.html ``` Event 39: ```text ===== /tmp/pricing.html ===== About Toggle About section Pricing Billing Cost Management Free Trial Support Engineering Jobs Healthcare on Fly.io Extensions Program Extensions API Merch Open Source Using Our Brand ``` #### Suggested Changes - **Publish a lightweight machine-readable pricing summary**: On fly.io/pricing or fly.io/docs/about/pricing/, add a small structured JSON or markdown summary of headline numbers (per-second/per-hour/per-month rates for the most common presets, free trial limits, egress rates) similar to what llms.txt does for doc navigation. Verify by fetching the page with a plain curl and confirming key prices are extractable without full HTML parsing, unlike the current 90KB+ pricing.html that required custom Python scripts to parse in this session (seq 38-55) Event 34: ```text exit=0 599360 /tmp/pricing.html ``` Event 39: ```text CPU(s) RAM Price/second Price/hour Price/month shared-cpu-1x 1 shared 256MB $ ``` - **Provide a documented markdown export path for API reference pages**: The agent guessed at .md and ?format=md suffixes on fly.io/docs/machines/api/ to get a lighter-weight reference, all of which 404'd. If Fly.io wants agents to consume API docs efficiently (as llms.txt suggests), document the actual markdown/JSON export URL pattern (if one exists) directly in llms.txt or on the docs page itself, and confirm by hitting that documented URL and getting a 200 with parseable content instead of the 404s seen at seq 75. Event 75: ```text === https://fly.io/docs/machines/api.md === ... Error 404 · Fly ``` ### Kimi K3 [View public run](https://agents.withgauge.com/p/runs/09350c3e-c697-4580-ad16-0d22ce101a41) · [Read transcript](https://www.ax-check.com/fly.io/sessions/kimi.json) Final output gives a detailed pricing breakdown (shared-cpu-1x ~$2/month, scale-to-zero near-zero, bandwidth $0.02-0.12/GB) explicitly tied to assumptions like machine size, region, and always-on vs scale-to-zero workload. #### End-to-end onboarding - **Onboarding needs a login**: Agent installed flyctl successfully but had no Fly.io account credentials in the sandbox and could not complete the interactive OAuth login flow required by 'flyctl auth login'. It never obtained a real API token, so no authenticated operation against the hosted Fly.io product was attempted or demonstrated. Session ended in a blocked state waiting on a human-supplied FLY_API_TOKEN. Event 21: ```text Error: no access token available. Please login with 'flyctl auth login' ``` Event 23: ```text I installed `flyctl`, but I'm **blocked on authentication**: there are no Fly.io credentials in this environment, and `flyctl auth login` requires an interactive browser OAuth flow (and Fly.io account signup requires a credit card on file, even for the trial). ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No Fly.io account or API token available in sandbox**: The sandbox environment had no pre-provisioned Fly.io credentials, and flyctl auth login requires an interactive browser-based OAuth flow that cannot run headlessly. Additionally, Fly.io account signup reportedly requires a credit card. This is a credentials/environment limitation, not a product code defect — a normal login/signup step blocked further progress on deployment. Event 21: ```text Error: no access token available. Please login with 'flyctl auth login' ``` Event 23: ```text there are no Fly.io credentials in this environment, and `flyctl auth login` requires an interactive browser OAuth flow (and Fly.io account signup requires a credit card on file, even for the trial) ``` #### Suggested Changes - **Document a non-interactive token generation path near the flyctl auth login error**: The agent found no way to obtain a FLY_API_TOKEN without an interactive browser OAuth flow, per its statement at seq 23. If a scriptable path already exists (e.g. generating a deploy token from an existing session on another machine), surface it more prominently near the flyctl auth login error message shown at seq 21, so headless agents/CI users are not immediately blocked. Event 21: ```text Error: no access token available. Please login with 'flyctl auth login' ``` ### Qwen 3.8 Max [View public run](https://agents.withgauge.com/p/runs/aba62910-a190-43ea-a34b-8ee002ec99ff) · [Read transcript](https://www.ax-check.com/fly.io/sessions/qwen.json) #### End-to-end onboarding - **Onboarding needs a login**: Agent confirmed no flyctl binary and no FLY_API_TOKEN in the sandbox, then verified that both the GraphQL API and Machines API return UNAUTHORIZED/401 without a token. No token was ever obtained (no signup flow, no dashboard access, no CLI auth attempted) and no authenticated operation against Fly.io was performed anywhere in the visible transcript. All subsequent work was unauthenticated documentation scraping (pricing page, free-trial page, Machines API docs) rather than any actual resource creation. Event 6: ```text ---TOKEN--- no fly env vars ``` Event 17: ```text {"data":null,"errors":[{"message":"You must be authenticated to view this.","locations":[{"line":1,"column":3}],"path":["apps"],"extensions":{"code":"UNAUTHORIZED"}}]} ``` Event 25: ```text https://api.machines.dev/v1/apps/testapp/machines -> 401 | {"error":"Authenticate: token validation error"} ``` #### Hallucinated URLs None identified in this transcript. #### Blockers - **No Fly.io API token available in sandbox**: The environment has no flyctl CLI installed and no FLY_API_TOKEN set, which blocks any authenticated call to Fly's GraphQL or Machines API. This is a test-environment/credential limitation rather than a product defect — Fly.io correctly returns 401/UNAUTHORIZED for unauthenticated requests, and obtaining a token would normally require a human sign-up/login step that this session could not self-serve. Event 6: ```text ---TOKEN--- no fly env vars ``` Event 17: ```text {"data":null,"errors":[{"message":"You must be authenticated to view this.","locations":[{"line":1,"column":3}],"path":["apps"],"extensions":{"code":"UNAUTHORIZED"}}]} ``` - **Pricing page ships as unrendered template requiring scraping workaround**: Fetching the pricing page returned raw ERB/Ruby template markup (region-select loops, unresolved `<%= %>` tags) rather than final rendered prices, forcing the agent to write a custom Python parser to extract real dollar figures from the served HTML. This is product/docs-site behavior (server-side templating leaking into the fetched HTML for non-browser clients), not agent error, though the agent did successfully work around it. Event 26: ```text The price of a running [Fly Machine](/docs/machines/) VM is the price of a named CPU/RAM preset, plus about $5 per 30 days per GB of additional RAM. ``` Event 26: ```text <% data.pricing.regions.each do |r| %> <% end %> ``` #### Suggested Changes - **Serve pre-rendered pricing tables to non-browser HTTP clients**: A plain curl GET of https://fly.io/docs/about/pricing/ returns unresolved ERB template syntax (region dropdown loops, unrendered price spans) instead of computed values, requiring a custom scraper to recover real numbers. Check by curling the pricing page without a browser user-agent/JS execution and confirming the returned HTML contains final computed prices rather than template directives like data.pricing.regions.each. Event 26: ```text <% data.pricing.regions.each do |r| %> <% end %> ``` - **Clarify Machines API base path in docs to avoid 404 on api.fly.io**: A request to https://api.fly.io/v1/apps/testapp/machines returned a plain 404 while the correct host is api.machines.dev; add a note or redirect on the Machines API introduction page (fly.io/docs/machines/api/) clarifying that REST Machines calls must go to api.machines.dev, not api.fly.io. Verify by curling both hosts and confirming api.fly.io/v1/... now returns a redirect or explicit error pointing to the correct host instead of a generic 404. Event 25: ```text https://api.fly.io/v1/apps/testapp/machines -> 404 | 404 page not found ``` ### Task given to each agent Help me build a simple example using Fly.io. Tell me how pricing works, and briefly tell me whether this product will be easy for you to manage. Let me know if you get blocked. If this product has no developer workflow you can act on, say so plainly and stop. Stay light: use the hosted product through its SDK or API. Do not start local service stacks or wait for long-running commands; if the quickstart requires either, say so plainly and stop. No product credentials were supplied and no purchases were authorized. ## Score: A · 100/100 (provisional) Grades come from completed site checks. Coding sessions and skipped checks do not affect the score. ### Clarity - **Pass** — Homepage answers Markdown requests ```text Homepage returned text/markdown with 200 when requested with Markdown Accept header. ``` - **Pass** — llms.txt provides an actionable documentation index ```text llms.txt links to docs, quickstart, CLI, Machines API, MCP, and organized sections. ``` - **Pass** — llms.txt provides navigation guidance ```text llms.txt groups links under Getting Started, Apps, Machines, Networking, Reference headings. ``` - **Pass** — llms.txt mentions offered API, MCP, and skills ```text llms.txt mentions Machines API, fly mcp CLI, and MCP server blueprints. ``` - **Pass** — A compact guide representation exists ```text Homepage returns text/markdown (117 tokens) when requested with markdown Accept header. ``` - **Pass** — A focused guide is directly retrievable ```text llms.txt is a directly retrievable compact index linking quickstart, flyctl, Machines API, MCP. ``` - **Pass** — Equivalent instructions fit a token budget ```text llms.txt is 9006 tokens but a compact index; quickstart page itself is 3403 tokens. ``` - **Pass** — Product-docs links survive format changes ```text Homepage markdown retains the sprites install link; docs links remain reachable via llms.txt. ``` - **Pass** — The compact guide is independently actionable ```text Quickstart gives concrete steps: install flyctl, fly auth signup, fly launch, fly deploy, then next steps. ``` - **Pass** — Install and next-step links resolve ```text Install and next-step links resolve: flyctl install, essentials, launch, autoscaling pages fetched successfully. ``` ### Onboarding - **Pass** — Docs lead to a relevant quickstart ```text llms.txt and docs link to Quickstart: Launch your app with concrete fly launch steps. ``` - **Pass** — Installation commands are extractable ```text Install page gives brew, curl, and PowerShell commands for flyctl. ``` - **Pass** — Code examples are available without interaction ```text Quickstart and FKS pages show inline code blocks without interaction. ``` - **Pass** — Prerequisites and auth boundaries are explicit ```text Quickstart requires installing flyctl and running fly auth signup/login. ``` ### Pricing - **Pass** — Pricing is readable without interaction ```text Pricing page renders full price tables as static Markdown without interaction. ``` - **Pass** — Prices are stated, not gated ```text Concrete USD prices listed for Machines, Sprites, Postgres, support. ``` - **Pass** — Pricing units and limits are explicit ```text Units explicit: per second, per GB-month, per CPU-hour, region multipliers. ``` - **Pass** — Agents identify pricing and its assumptions ```text 1 of 3 sessions were judged on pricing; 0 fell short. Kimi K3: Final output gives a detailed pricing breakdown (shared-cpu-1x ~$2/month, scale-to-zero near-zero, bandwidth $0.02-0.12/GB) explicitly tied to assumptions like machine size, region, and always-on vs scale-to-zero workload. This behavioural item does not affect the fast grade. ``` ### Activation - **Pass** — An API reference or OpenAPI spec is reachable ```text Machines API page links an OpenAPI 3.0 spec at docs.machines.dev; REST resources documented. ``` - **Pass** — An MCP server is documented and well-formed ```text fly mcp command documented with subcommands (server, proxy, add, launch) and usage. ``` - **Pass** — A CLI install path is documented ```text Install flyctl page documents brew, curl script, and PowerShell install paths. ``` - **Skipped** — SDK packages resolve on their registries ```text No registry lookup result for a Fly.io SDK or packaged CLI package supplied. ``` - **Skipped** — Agent skills are published ```text No agent skills surface offered or fetched in the supplied evidence. ``` [Full report data](https://www.ax-check.com/fly.io/report.json)