Scrape and store structured data from a large website automatically

flowchart TD trig(("user starts scraping job")):::human s0["fetch page from list"]:::task s1[("cache page locally")]:::store s2["extract data from page"]:::task s3["check against stored data"]:::task s4[("save new or changed records")]:::store trig --> s0 s0 --> s1 s1 --> s2 s2 --> s3 s3 --> s4 lp{"list of pages exhausted"}:::gate s4 --> lp lp -. "move to next page in list" .-> s0 lp -->|"finished"| out out[/"deduplicated database of scraped records"/]:::out pay{{"structured data without manual copying"}}:::pay out --> pay classDef task fill:#e7f6fe,stroke:#34b8f0,color:#2c2a29 classDef svc fill:#f6f8fa,stroke:#7c8795,color:#2c2a29 classDef mi fill:#e7f6fe,stroke:#0079a8,color:#2c2a29,stroke-width:2px classDef human fill:#fff,stroke:#0079a8,color:#0079a8 classDef store fill:#f6f8fa,stroke:#0079a8,color:#2c2a29 classDef trig fill:#00a4eb,stroke:#0079a8,color:#fff,font-weight:bold classDef trigtime fill:#00a4eb,stroke:#0079a8,color:#fff,font-weight:bold classDef trigdata fill:#8ad4f5,stroke:#0079a8,color:#06314c,font-weight:bold classDef gate fill:#fff,stroke:#e8a23d,color:#6b4708,font-weight:bold classDef out fill:#1f9d6b,stroke:#167a53,color:#fff,font-weight:bold classDef pay fill:#06314c,stroke:#021f33,color:#fff