Zyte / Oxylabs Integration

[angelotc]

For blogs with many posts, you get rate limited with 429's.

Would be nice to have a proxy rotation service integration. I'm using Zyte in my current setup.

[angelotc]

background on this use-case for this - i've repurposed this script to be ran as a batch job that

1. runs every day at midnight
2. scrapes each substack post via Zyte
3. upserts it into a db if it doesn't exist as a row in the substack_posts table (just using the CDN links from substack for images on my posts, meaning i don't have to host any images).

Also, this is what i'm scraping onto my site -

https://www.nipponhomes.com/blog/yukihomes

But honestly, I probably don't need a proxy rotation service such as Zyte or Oxylabs after the initial scrape.