stealthkit - for python scraping

Hey everyone,

I published my 3rd pypi lib and it's open source. It's called stealthkit - requests on steroids. Good for those who want to send http requests to websites that might not allow it through programming - like amazon, yahoo finance, stock exchanges, etc.

What My Project Does

• User-Agent Rotation: Automatically rotates user agents from Chrome, Edge, and Safari across different OS platforms (Windows, MacOS, Linux).

• Random Referer Selection: Simulates real browsing behavior by sending requests with randomized referers from search engines.

• Cookie Handling: Fetches and stores cookies from specified URLs to maintain session persistence.

• Proxy Support: Allows requests to be routed through a provided proxy.

• Retry Logic: Retries failed requests up to three times before giving up.

• RESTful Requests: Supports GET, POST, PUT, and DELETE methods with automatic proxy integration.

Why did I create it?

In 2020, I created a yahoo finance lib and it required me to tweak python's requests module heavily - like session, cookies, headers, etc.

In 2022, I worked on my django project which required it to fetch amazon product data; again I needed requests workaround.

This year, I created second pypi - amzpy. And I soon understood that all of my projects evolve around web scraping and data processing. So I created a separate lib which can be used in multiple projects. And I am working on another stock exchange python api wrapper which uses this module at its core.

It's open source, and anyone can fork and add features and use the code as s/he likes.

If you're into it, please let me know if you liked it.

Pypi: https://pypi.org/project/stealthkit/

Github: https://github.com/theonlyanil/stealthkit

Target Audience

Developers who scrape websites blocked by anti-bot mechanisms.

Comparison

So far I don't know of any pypi packages that does it better and with such simplicity.