Potent Pages: Web Applications, Websites & Programming

Checking a Website’s robots.txt Using PHP
When downoading sites, it’s important to make sure that you have permission from the site owner to download pages. This is generally indicated by the robots.txt file. This file specifies which pages crawlers aren’t allowed to download. It can also specify...

How to write a simple PHP web crawler to download an entire website
Do you want to automatically capture an information like the score of your favorite sport, latest fashion style and trend from the stock market from a website for extra processing? If the specific information you need is available on a website, you can...

How to download a webpage using PHP and cURL: an easy-to-use function.
There are a wide range of reasons to download webpages. From parsing and storing information, to checking the status of pages, to analyzing the link structure of a website, web crawlers are quite useful....

{{ server.server }}
{{ domain.domain }}
In this tutorial, there is a lot of find-and-replacing required to help define all of the settings. While certainly not required, we recommend that you create an account if you haven’t already. We will do a lot of the find-and-replace work for you, customizing...