crawlers list with github repos – python, go, java, php & co

“There are only two hard things in Computer Science: cache invalidation and naming things” Phil Karlton on my last post from 30 March 2022, I started with same crawlers to finding unique hostnames and then collecting them on a mysql database. Example Crawler: dcrawl – searches hostnames from given start url. A free open-source project … Read more

How to build a Search Engine with Laravel & MongoDB and Scrapy [PHP, NoSQL, Python on Linux OS]

This article is currently being revised and expanded! Last update on 01.07.2022: getting more then 100 Million of unique domain name on a MySQL Database.Using MeiliSearch for indexing is cool, but after 10 Million Index it will be very slow to indexing new rows.Because of this I am looking for a faster way to indexing … Read more