Build a crawler that collects competitor online prices every day

## Role and objective

You are a software engineer designing a crawler that collects competitor online prices every day. Produce an implementation-ready technical specification and development plan for the person who will operate and maintain the crawler. Do not invent the competitor sites, products, prices, access permissions, technology stack, or legal applicability.

The output must follow this order: goal and stack, numbered acceptance criteria, edge cases, and how it is verified. Completion means every required unknown is either confirmed from the input or marked with a specific slot, and every acceptance criterion can be tested through an observable result.

## Scope and given facts

In scope is a crawler for daily collection of online prices from competitors. The input confirms neither the competitor identities nor the pages, products, regions, currencies, variants, or price fields to collect. It also does not confirm whether prices include taxes, shipping, discounts, membership benefits, or availability information.

Treat these as unresolved:

- [FILL IN: competitor websites and permitted access scope] — provide the domains, paths, and written permission or policy basis for access.
- [FILL IN: product matching method] — provide product URLs, SKUs, identifiers, or matching rules.
- [FILL IN: collection time zone and daily run window] — provide the required schedule.
- [FILL IN: storage system and output schema] — provide the destination and required fields.
- [FILL IN: language, runtime, dependencies, and execution environment] — provide the implementation stack.
- [FILL IN: applicable privacy regime] — identify whether GDPR, CCPA/CPRA, HIPAA, or another regime governs any personal data touched.

Do not fill the competitor websites, product list, schedule, or stack with plausible examples.

## Working rules

Use only information confirmed by the input or by sources explicitly supplied for the implementation. Do not claim that a site permits crawling unless its robots.txt, terms, API documentation, contract, or other authoritative permission source has been checked. If access permission is absent, mark the integration as blocked pending review rather than bypassing controls.

Define the collected record before proposing code. At minimum, specify slots for the source URL, retrieval timestamp, product identifier, displayed price, currency, availability, tax or shipping treatment, extraction status, and error details. If a field is not required, state that it is excluded and why. Preserve raw evidence such as the retrieved page, API response, or a permitted snapshot when the approved design allows it; do not represent an inferred or stale value as the current price.

Branch explicitly:

1. If a competitor provides an authorized API, prefer that interface and document authentication, rate limits, pagination, and versioning.
2. If no authorized API exists but automated page access is permitted, use respectful request rates, caching, timeouts, retry limits, and a clear user-agent.
3. If access is prohibited, authentication is unavailable, or anti-bot controls would need to be bypassed, stop that integration and report the condition.
4. If product identity cannot be matched confidently, store the item as unmatched for review instead of assigning a price to the wrong product.

State the language, runtime, dependencies, and execution environment as confirmed values or slots. For each dependency, include its licence and whether copyleft terms are acceptable for this project. Define numbered, observable acceptance criteria. Specify existing behaviour that must not break: [FILL IN: existing system or workflow to preserve].

Define failure behaviour, including error messages, exit codes, retry exhaustion, partial-run handling, duplicate prevention, logging, alerting, and recovery. Do not make unmeasured performance claims.

If the crawler touches personal data, require the governing regime, retention period, and deletion path to appear in the design itself, not only in a comment.

## Output structure

### 1. Goal and stack

State the daily price-collection goal, permitted sources, product scope, schedule, storage destination, and output fields. Mark each item CONFIRMED, PROVISIONAL, or [FILL IN]. List the language, runtime, dependencies, scheduler, persistence layer, observability tools, and execution environment. Include each dependency’s licence and copyleft compatibility status.

### 2. Numbered acceptance criteria

Write testable criteria, numbered consecutively. Cover:

- successful daily execution;
- source authorization and request limits;
- product identification and matching confidence;
- correct extraction of price, currency, availability, tax, and shipping treatment;
- timestamp and source evidence;
- duplicate and stale-record handling;
- partial failures and retries;
- logging, alerting, exit codes, and recovery;
- data retention and deletion when personal data is involved;
- preservation of [FILL IN: existing system or workflow].

For every criterion, state the setup, observable result, and pass condition. Do not provide invented sample prices or competitor names.

### 3. Edge cases

Describe expected behaviour for unavailable pages, changed HTML, API version changes, rate limits, timeouts, malformed prices, multiple currencies, sale prices, unavailable products, login walls, duplicate products, partial runs, empty responses, and clock or time-zone failures. Give the error message or error category, retry policy, exit code, and recovery path where applicable.

### 4. How it is verified

Specify unit, integration, contract, scheduler, data-quality, authorization, security, retention, and end-to-end tests. Identify fixtures and permitted test sources. Require measured timing, request counts, failure rates, and storage results instead of unsupported performance claims. State how a completed daily run is audited and how a failed run is replayed safely.

## Style rules

Use a hybrid style. Use concise, itemized sections for the stack, acceptance criteria, edge cases, fields, and verification checks. Use short narrative paragraphs only to explain architecture decisions, branching conditions, and recovery rationale. Keep a technical, direct register. Avoid vague clichés such as “seamless,” “robust,” “scalable,” or “best practice” unless tied to a measurable requirement.

## Style rules (humanizer v1)

These govern every prose surface in the deliverable. Never alter quotations, code, identifiers, or proper nouns to satisfy them.

- Banned vocabulary: delve, tapestry, testament, showcase, pivotal, crucial, vital, intricate, interplay, meticulous, foster, vibrant, boasts, nestled, groundbreaking, and "landscape" in the abstract sense. Banned inflation phrases: plays a vital role, underscores its importance, evolving landscape.
- Banned constructions: "not just X, but Y" negative parallelism, forced three-item lists, fake ranges ("from X to Y"), signposting ("Let's dive in"), staged staccato ("One goal. Zero compromises."), and synonym cycling. Name a thing the same way every time.
- Punctuation and structure: no em dashes in the final text (rewrite with a period, colon, or parentheses), no emoji, sentence case headings, no heading on every paragraph, no bolding cadence, no "In conclusion" wrap-up. Close on a concrete fact.
- Tone: no flattery ("Great question"), no chatbot residue ("I hope this helps"), no knowledge-cutoff hedging, no stacked hedges. Hold the register the genre calls for and vary sentence length.
- Fact integrity: every instruction to be specific carries one boundary. Use only facts present in the user's input or in a verifiable source. Do not invent details to sound human. Leave anything the user did not supply as a literal [FILL IN] slot instead of a plausible guess.
- False-positive guard: flawless grammar, a single em dash, one "however", or formal wording is not by itself an AI tell. Rewrite only where several signals cluster, and never rough the prose up on purpose.

## Final self-audit

Draft the deliverable in full, then interrogate the draft on two counts. Which passages read as obviously AI-written when checked against the style rules above? Did any line assert a fact absent from the user's input and unverifiable from the sources given? Rewrite what fails and submit only the corrected version. The audit itself never appears in your output.

## Self-verification

1. Confirm that the deliverable is a crawler specification and development plan, not implemented code or collected prices.
2. Confirm that the only confirmed task fact is daily collection of competitor online prices.
3. Check that competitor websites, products, permissions, schedule, storage, schema, and stack remain slots unless supplied.
4. Check that no competitor name, product, price, statistic, dependency, runtime, licence, or legal conclusion was invented.
5. Confirm that the output uses the required order: goal and stack, numbered acceptance criteria, edge cases, and how it is verified.
6. Count the acceptance criteria and verify that each has an observable pass condition.
7. Check that API, permitted page access, and prohibited or blocked access follow the stated branches.
8. Confirm that malformed prices, currencies, availability, stale data, duplicate records, and changed page structures have defined handling.
9. Confirm that error messages, exit codes, retry exhaustion, partial runs, logging, alerting, and recovery are specified.
10. Check that unmeasured performance claims are absent and that measured metrics are assigned to verification tests.
11. Confirm that personal-data handling requests the governing regime, retention period, and deletion path when applicable.
12. Check that the plan does not drift into competitor strategy, market analysis, or price recommendations beyond building and verifying the daily crawler.
13. Confirm that any preserved existing behaviour is represented by “[FILL IN: existing system or workflow]” rather than guessed.
14. Verify that the final text contains no filled slot masquerading as a confirmed implementation fact.