The $0.49 Proxy Illusion: How a 200-Line CLI Exposes Real Scraping Costs

A proxy reseller quotes $0.49 per 1,000 requests. The spreadsheet multiplies by 2,000 and lands on $980. The plan looks cheap against every managed scraping API on the market. Then the crawler runs for a month, the invoice arrives at $980, and the dataset is 30% short — challenge pages, 403s, timeouts, and product pages that returned 200 with no product on them.

Nothing in the invoice is wrong. The problem is the unit. You did not buy 2,000,000 pages. You bought 2,000,000 attempts, and only some of those attempts produced a response your pipeline could use. The moment you divide the invoice by usable pages instead of sent requests, the $0.49 number stops being a price and starts being a fiction.

I wrote a small CLI to measure that gap. It is called check-proxy-tax: the measurement core is under 200 lines of standard-library Python, and the shipped command — argument parsing, terminal table, JSON output — is still under 550. It turns "cheap proxy" into a number you can defend in a budget meeting. This article walks through what it measures, what it found against real targets, and how the arithmetic changes a buying decision.

The sticker price is priced in the wrong unit

Proxy plans are sold per request, per IP, or per gigabyte. Crawlers are paid in usable responses. Between those two units sit four failure modes that every scraping team knows and almost no pricing page mentions:

• Hard blocks. 403, 429, or a 503 from an edge rule. The request was billed; the page was not delivered.
• Soft blocks. A 200 containing a Cloudflare interstitial, a consent wall, or a login shell. The status code says success; the parser says otherwise.
• Timeouts. The connection stalls past your deadline. Depending on the plan, the attempt may still consume quota.
• Silent misses. The page loads, but the field you needed — price, stock, rating — is absent because the site served a different variant.

A per-request price treats all four as successful deliveries. Your data pipeline treats all four as failures. The difference is the tax.

What counts as usable

check-proxy-tax uses a deliberately strict definition. A sample is usable only when:

1. the final HTTP status is exactly 200 after redirects, and
2. when --marker is supplied, the response body contains that exact string.

Everything else is blocked: 403, 429, other 4xx, 5xx, timeouts, TLS failures, connection resets, and 200 responses that miss the marker. That last clause is the important one. A challenge page is a 200 until you look for the product title and discover it is not there.

The tool fires a configurable number of concurrent requests at one URL, classifies every response, and reports usable percentage, effective cost per 1,000 usable requests, the tax multiplier, p50/p95 latency, bytes transferred, status counts, and blocked-reason counts. --json emits a single object. Exit codes are stable: 0 all usable, 1 degraded, 2 none usable, 3 bad arguments.

git clone https://github.com/600me/check-proxy-tax
cd check-proxy-tax
python3 -m check_proxy_tax.cli \
  --url "https://example.com/product/123" \
  --requests 20 \
  --marker "Add to cart" \
  --price-per-1k 0.49

No account, no API key, no dependencies, no hosted service. That matters for an audit tool: if measuring the vendor required the vendor's SDK, the measurement would inherit the vendor's assumptions.

Four measurements, including the ugly ones

I ran the CLI on 2026-10-19 from an unproxied connection, with --price-per-1k 0.49. The first target is a control. Three are real sites behind commercial anti-bot stacks. The last is a deterministic local harness that returns seven usable pages and three 403s in every ten requests, so the arithmetic below is reproducible rather than rhetorical.

The control behaved exactly as the sticker price predicts. Ten requests, ten usable responses, $0.49 per 1,000, multiplier 1.00x.

The protected targets did not. G2 and Glassdoor returned 403 on every sample from an unproxied IP. Amazon returned 404 on every sample — a bot wall dressed as a missing page. The CLI reports the effective cost as infinite, not as a large number, because no finite amount of money buys a usable response at a 0% success rate. The JSON sets effective_cost_per_1k and tax_multiplier to null and cost_status to "infinite", which keeps the document valid while refusing to invent a number.

That 0% result is not a verdict on G2, Glassdoor, or Amazon. It is what those edges show a single unproxied client with a plain HTTP stack. A managed scraping API with residential egress and a browser runtime will do better, and that is precisely what you are paying for. The point is that "better" is unknown until you measure it on your targets. A vendor's published success rate is an average over someone else's URL list.

The 70% harness produced the number the rest of this article uses:

{
  "requests": 20,
  "usable_count": 14,
  "blocked_count": 6,
  "usable_pct": 70.0,
  "nominal_price_per_1k": 0.49,
  "effective_cost_per_1k": 0.7,
  "tax_multiplier": 1.428571,
  "status_counts": {"200": 14, "403": 6},
  "blocked_reasons": {"http_403": 6},
  "exit_code": 1
}

Fourteen usable responses out of twenty attempts is a 70% yield. At a nominal $0.49 per 1,000 attempts, the effective price is:

$0.49 / 0.70 = $0.70 per 1,000 usable responses
$0.70 / $0.49 = 1.43x tax multiplier

That is a 43% surcharge on every usable page, invisible on the pricing page and completely real in the invoice.

The arithmetic at production volume

The multiplier is small at pilot scale and brutal at production scale. Suppose the pipeline needs 1,000,000 usable responses per month and the plan charges $0.49 per 1,000 attempts, billing blocked attempts.

At 70% usability the invoice is $700, not $490 — a $210 monthly gap, or $2,520 a year, on a plan the buyer described as "$0.49 per 1,000." At 50% the same 1,000,000 usable pages cost $980, exactly double the sticker. At 10% they cost $4,900, which is more than many managed scraping APIs charge for the same volume with the browser and proxy work included.

This is why comparing a $0.49 proxy plan to a $49 managed API by list price is meaningless. Normalize both to cost per 1,000 usable responses against the same URL list with the same marker, and the ranking often flips. The 403 Tax research page walks through the same normalization across four managed providers, including how credit multipliers for rendering and premium proxies compound the problem.

Retry amplification is the second invoice

Most retry policies were written for transient network errors and then inherited by anti-bot failures. That is a budgeting mistake. A 403 is not a dropped packet; retrying it three times usually produces three more 403s and three more billed attempts.

Take 100,000 URLs, a 30% block rate, and a naive policy that retries every failure three times:

70,000 first-pass successes
30,000 blocked URLs x 3 retries = 90,000 extra attempts
total billed attempts = 160,000
nominal cost at $0.49/1k = $78.40
usable output = 70,000
effective cost per 1k usable = $78.40 / 70 = $1.12
multiplier over sticker = 2.29x

The effective price more than doubled, and the extra spend bought no additional data. A retry budget that distinguishes 429 (back off), 403 (change identity or give up), and timeout (retry once) is worth more than a bigger proxy plan. If you take one operational lesson from the CLI, take this one: log the reason, not just the status. blocked_reasons exists so a 403 storm does not look like a timeout storm in the weekly report.

Latency is a cost line too

The invoice is not the only place the tax shows up. Blocked requests still consume wall-clock time, and tail latency sets your schedule.

At concurrency 20 and a 1.2-second p50, a worker pool sustains roughly 16.7 requests per second. Delivering 1,000,000 usable pages at 70% usability requires 1,428,571 attempts, which is about 23.8 hours of continuous crawling. At a 1.84-second p95 the same work takes about 36.5 hours. The gap between p50 and p95 is more than half a working day, and it is invisible in a per-request price.

The CLI reports p50, p95, min, and max for this reason. A vendor that is cheap per request but adds 800 ms of challenge negotiation to every blocked attempt can lose to a more expensive vendor that fails fast. Throughput is part of unit economics, especially when you are renting workers by the hour.

Twenty samples is not a measurement

The 70% harness result comes from 20 samples. That is enough to demonstrate the arithmetic and not enough to sign a contract. The 95% Wilson interval for 14/20 runs from 48.1% to 85.5%. In other words, the same result is consistent with a 48% yield and an 85% yield, and those imply effective costs of $1.02 and $0.57 per 1,000 usable pages. The confidence interval is wider than most vendor price differences.

At 200 samples with 140 usable, the interval narrows to 63.3%–75.9%. Still not tight, but usable for a buying decision. My rule of thumb:

• 20 samples to catch gross failures and validate the harness.
• 200 samples per target class before comparing vendors.
• Three runs across different hours to catch rate-limit windows and diurnal blocking.
• The hardest 20 URLs in your list, not the homepage. Homepages are the easiest pages a site serves.

Stratify by target class — product pages, search results, profiles, paginated lists — because a vendor that handles static article pages and fails on search is not 90% good. It is 90% good on the traffic you do not have.

A buying workflow that survives contact with production

1. Build a 20-URL sample that mirrors production: the same domains, page types, and required fields. Write down the marker string for each.
2. Run the vendor trial with the same request count, concurrency, timeout, and marker. Do not let the vendor pick the URLs.
3. Capture JSON, not screenshots. --json > vendor-a.json gives you status counts, blocked reasons, and latency percentiles you can diff.
4. Normalize to cost per 1,000 usable responses using the actual plan price and the measured yield. Include retries in the attempt count.
5. Ask how blocked requests are billed. Per-request plans usually charge them; some managed APIs only charge successful requests. The answer changes the multiplier by 1.4x or more.
6. Gate CI on the JSON. A job that fails when usable_pct drops below 95 or cost_status becomes "infinite" catches a proxy pool degrading before the monthly invoice does.
7. Re-run monthly. Anti-bot rules change, IP pools get burned, and the yield that justified the plan in January may not hold in April.

The CLI is designed to be the measurement step, not the whole procurement system. It sends a small diagnostic burst to one URL. It does not execute JavaScript, solve CAPTCHAs, rotate residential IPs, or reconcile the vendor's internal credit accounting. It measures the HTTP outcome your parser sees, which is the part the invoice does not.

What the $0.49 actually means

The $0.49 figure is not a lie. It is a price for attempts, quoted by a seller who knows that attempts and usable pages are different things, sold to a buyer who is thinking in pages. The illusion survives because the gap is invisible until someone counts.

Counting is cheap. The CLI is free, the run takes a minute, and the number it produces — cost per 1,000 usable responses — is the only price that belongs in a budget. Everything else is a sticker.

If you want to run the audit yourself, start with the check-proxy-tax repository. If you want the provider-side view, the live scraping API benchmark normalizes plan pricing and credit multipliers across vendors, and the 403 Tax teardown shows how anti-bot pressure moves the effective rate.

Two relays I use for trials when a target requires managed egress:

• ScraperAPI — request-based plans with premium proxy and rendering modes; run the CLI against your own URLs before choosing a tier.
• Smartproxy / Decodo — residential and datacenter pools; the trial is the only way to know your yield.

FTC Disclosure: Benchmark links may include affiliate tracking. If you buy through the links above I may earn a commission at no extra cost to you. Commissions do not change the price you pay and do not buy a ranking. Every number in this article comes from a local CLI run on 2026-10-19, not from a vendor dashboard. Verify current pricing, trial terms, and billing rules with each provider before you commit.

The next time a plan quotes $0.49 per 1,000, ask one question: per 1,000 of what? If the answer is "requests," you have not been quoted a price. You have been quoted a starting point.