Rachelritzler Siterip Jun 2026

| Step | Action | Tool | Outcome | |------|--------|------|---------| | 1. Permission | Confirmed the CC‑BY‑4.0 license covered full download. | Email to the consortium. | Got explicit written consent. | | 2. Scope | Needed only the CSV files and accompanying metadata. | Defined a URL pattern ( *.csv , *.json ). | Narrowed crawl to < 2 GB. | | 3. Crawl | Wrote a Scrapy spider that followed internal links, filtered file types, and throttled to 1 req/sec. | Scrapy + custom pipeline

| Legal Concept | How It Applies to Site‑Ripping | Practical Takeaway | |---------------|--------------------------------|--------------------| | | Protects the creative expression of HTML, images, text, audio, video, etc. Copying without permission is infringement unless a statutory exemption applies. | Only rip content that is either (a) in the public domain, (b) under a permissive license, or (c) covered by a specific legal exemption (e.g., fair use in a narrow context). | | Terms of Service (ToS) | Violating a site’s ToS can lead to civil claims (e.g., Computer Fraud and Abuse Act in the U.S.) even if the content is public. | Treat the ToS as a contract; if it says “no crawling,” stop. | | robots.txt | Not a law, but many courts treat deliberate ignoring of robots.txt as evidence of intent to violate a site’s policies. | Honor it unless you have explicit written consent. | | DMCA Safe Harbor | Service providers can be shielded from liability if they act upon takedown notices promptly. | If you host a mirror, be prepared to take down infringing material if a legitimate DMCA notice arrives. | | Fair Use (U.S.) / Fair Dealing (other jurisdictions) | Very limited for entire site copies; typically only applies to short excerpts for commentary, criticism, or research. | Don’t rely on fair use as a blanket defense for full‑site rips. | rachelritzler siterip

If you’ve ever searched for the phrase site‑rip you’ve probably seen it in two very different contexts: | Step | Action | Tool | Outcome

| Scenario | Why It’s Usually OK | How RachelRitzler Does It | |----------|----------------------|---------------------------| | (e.g., Project Gutenberg, Government archives) | The content is already free to share. | She mirrors the entire U.S. National Archives site using wget with a 2‑second delay, then uploads the static copy to a nonprofit mirror. | | Open‑Source Documentation (e.g., API docs, language specs) | Licenses (MIT, Apache, CC‑BY) explicitly allow redistribution. | Rachel clones the Rust language reference site with HTTrack , adds a custom search index, and contributes the index back to the community. | | Personal Research (e.g., a conference website that will go offline) | For personal, non‑commercial study, provided the site’s terms of service don’t forbid it. | She downloads the schedule and speaker PDFs of a defunct conference, cites the source, and keeps the copy private. | | Offline Learning (e.g., educational videos released under Creative Commons) | The creator gave permission for redistribution. | Rachel bundles a set of CC‑BY‑SA video tutorials into a single ZIP for students with limited bandwidth. | | Got explicit written consent