Web Scraping with Scrapy pipeline to add crawled data to MongoDB collection [Tutorial]

In this tutorial i want to show you how to add the scraped data from scrapy crawler to a MongoDB database. For this we will use the scrapy crawler pipeline with the correct connection to a localhost server. This tutorial will walk you through these tasks: In this Scrapy project I scrape quotes from https://quotes.toscrape.com/ … Read more

crawlers list with github repos – python, go, java, php & co

“There are only two hard things in Computer Science: cache invalidation and naming things” Phil Karlton on my last post from 30 March 2022, I started with same crawlers to finding unique hostnames and then collecting them on a mysql database. Example Crawler: dcrawl – searches hostnames from given start url. A free open-source project … Read more

How to build a Search Engine with Laravel & MongoDB and Scrapy [PHP, NoSQL, Python on Linux OS]

This article is currently being revised and expanded! Last update on 01.07.2022: getting more then 100 Million of unique domain name on a MySQL Database.Using MeiliSearch for indexing is cool, but after 10 Million Index it will be very slow to indexing new rows.Because of this I am looking for a faster way to indexing … Read more

TOP 10+ open-source e-commerce platforms in 2022

comparing open source e-commerce software applications on GitHub and Installation of Cloud Instances like on DigitalOcean, Amazon Web Services(AWS), Google Cloud Platform or on a virtual private server. A comparing of open source e-commerce platforms for unique and clean, and also for fast, modern and e-commerce web applications with awesome interfaces in 2022 PHP, Python … Read more

python string replace in python 3.x

Python String replace() Method There are two ways for replacing a String for python 2.x and also 3.x, use str.replace(). The replace() method in python 3 is used simply. Example for a string replace: or calling the method replace() of  <class ‘str’> in python 3.x: Notes from Python Docs: Method 1: use builtin str’s replace -> str.replace(strVariable, old, … Read more

how to split a string in python

How to split a string into a list of characters in Python?Answer is: You need Lists. Docs: http://docs.python.org/library/functions.html#list You take the string and pass it to list() or You can also do it in this very simple way without list(): If you want to process your String one character at a time. you have various … Read more

python yield keyword

What does the “yield” keyword do in python? First of all you must to understand what generators [ Generators – Python Wiki ] are. And before you understand what generators are, you must understands what iterables [Iterables – PLYMI] are. Iterables Definition: An iterable is any Python object capable of returning its members one at a time, … Read more