How to delete all rows which contains a specific value or character in a mysql database table

On my last posts I had crawled a lot of domain name on a mysql table. And now there is a new problem with specific characters like “:” and port numbers. For example: Now I want to delete all this domain names which contains the port numbers in field “url”. MySQL command for find and … Read more

how to remove duplicate rows in mysql with using temporary table

after i get 100 Million of url’s, i saw a few of duplicates on my mysql table. I used the command line because the MySQL Workbench was not usable with minimum system resources. For login on Terminal on Mac OS X with MySQL Community Server version: 8.0.29 – GPL after this command, u need to … Read more

Web Scraping with Scrapy pipeline to add crawled data to MongoDB collection [Tutorial]

In this tutorial i want to show you how to add the scraped data from scrapy crawler to a MongoDB database. For this we will use the scrapy crawler pipeline with the correct connection to a localhost server. This tutorial will walk you through these tasks: In this Scrapy project I scrape quotes from https://quotes.toscrape.com/ … Read more

Headless ecommerce open source frameworks list [API, JSON, GraphQL]

https://unsplash.com/@mikafinland

headless ecommerce open source framework are very popular on 2022. With newest technologies you can convert or integrate your old ecommerce Shop to a new headless open source ecommerce with API’s and GraphQL. On this post you will get a list of the most popular frameworks with demo links and repositories. A List of open … Read more

crawlers list with github repos – python, go, java, php & co

“There are only two hard things in Computer Science: cache invalidation and naming things” Phil Karlton on my last post from 30 March 2022, I started with same crawlers to finding unique hostnames and then collecting them on a mysql database. Example Crawler: dcrawl – searches hostnames from given start url. A free open-source project … Read more

How to build a Search Engine with Laravel & MongoDB and Scrapy [PHP, NoSQL, Python on Linux OS]

This article is currently being revised and expanded! Last update on 01.07.2022: getting more then 100 Million of unique domain name on a MySQL Database.Using MeiliSearch for indexing is cool, but after 10 Million Index it will be very slow to indexing new rows.Because of this I am looking for a faster way to indexing … Read more

TOP 10+ open-source e-commerce platforms in 2022

comparing open source e-commerce software applications on GitHub and Installation of Cloud Instances like on DigitalOcean, Amazon Web Services(AWS), Google Cloud Platform or on a virtual private server. A comparing of open source e-commerce platforms for unique and clean, and also for fast, modern and e-commerce web applications with awesome interfaces in 2022 PHP, Python … Read more

The Power of Online Marketing in 2022 

Do you want to reach potential customers faster Are you sure that you have implemented the right keywords in your advertising? Are the contacts that are generated to be utilized optimally or are they simply not suitable for your products? SEO advisors can help with the positioning of the website. Let them show you where … Read more

Awesome-list of crawlers

A awasome collection of web crawlers ,spiders and resources in different languages. Tested by myself on Mac OS X with Python 3. Scrapy: The King of Web Crawler Framework written by Python.  A very fast high-level html-content scraping and web crawling framework. A Complete List of all crawlers can be found under: GitHub: https://github.com/BruceDone/awesome-crawler