How to Create and Upload a Sitemap: A Step-By-Step Guide

What is a Sitemap?

A sitemap is a resource that lists information about the contents of your website – from videos to text – and the relationships among them. But why do we need sitemaps in the first place? Well, it depends on the type of sitemap – XML or HTML – we’re talking about, as both have different purposes.

XML sitemap

An XML sitemap, which is written in a machine-readable format, is the most relevant for search engines. For example, an XML sitemap lists URLs, the last modified date, change frequency, and priority for a web page on the website. This information helps search engines crawlers index your website more efficiently. 

How to create an XML sitemap?

There are numerous ways to create a sitemap, but first, you need to figure out whether you need one in the first place. If your site is relatively small and linked properly (which should be the case anyway), search engines should be able to crawl and index your site without a sitemap. However, there are some cases where you might need to create one. For example:

• Large sites

The classification of a 'large site' varies on the search engine. For example, Google considers sites with 500 pages or fewer as small. Maintaining proper interlinking across all pages of a large site can be quite challenging. Thus, a sitemap proves to be invaluable, as it assists search engines in effectively crawling and indexing a website. 

• New sites

If your site is new, there’s a big chance that you have few external links leading to your website, making it difficult for search engines to index it. In cases like these, submitting a sitemap to Google Search Console might help.

• Sites containing rich media

You might need to create a sitemap when your website has rich media (video, images, news articles) that you want to appear in search results. Indeed, a sitemap helps Googlebot to find and understand rich files on your website. 

Choosing the right approach

If you decide that your website does need an XML sitemap after all, there are several approaches you can consider. For those using a content management system (CMS), chances are the platform can generate a sitemap automatically. If not, you can also use a third-party XML sitemap generator. Finally, you can also employ a web crawler to perform the job. 

A web crawler is a tool that allows you to select the required website content, crawl all the URLs, or index all the web pages on your website. A web crawler isn’t used for generating sitemaps exclusively, but it certainly does the job well. In fact, the Web Crawler feature that comes with Oxylabs’ Web Scraper API can deliver three types of data output: a URL list (sitemap), parsed results, or HTML files. These outputs can be applied to a variety of use cases, from gathering competitor intelligence to cybersecurity.

Creating XML sitemaps using Oxylabs’ Web Crawler in Python

To make sure you can follow this tutorial as smoothly as possible, install Python 3.7+ or greater on your system. For demonstration, we’ll generate a sitemap for oxylabs.io. 

1. Start by installing the requests module using the following command:

pip install requests

2. Then, create a new payload for the request. You can set multiple parameters depending on your requirements (see complete documentation).

payload = {
   "url": "https://oxylabs.io/",
   "filters": {"crawl": [".*"], "process": [".*"], "max_depth":10},
   "scrape_params": {"source": "universal", "user_agent_type": "desktop"},
   "output": {"type_": "sitemap"},
}

The url parameter sets the base URL of the website for which you want to get URLs. The filters parameters control the scope and extent of the web crawling task. For an in-depth explanation, check out the web-crawler documentation regarding filters.

The scrape_params are used to modify our scraping tasks. For example, you can specify the region for proxies or enable JavaScript rendering when crawling a website. Output describes the output type. Since we need a sitemap, the output type must be set to sitemap.

3. Once the payload is ready, post an HTTP request to the API Endpoint for creating a new crawling job.

headers = {"Content-Type": "application/json"}
response = requests.request(
   "POST",
   "https://ect.oxylabs.io/v1/jobs",
   auth=(USERNAME, PASSWORD), 
   headers=headers,
   json=payload,
)
json_resp = response.json()

The job may take some time to complete and create a sitemap. You may periodically check for the job status by getting the job information. Once the job status is done, you can get the sitemap. 

4. You can get the job-related information by sending a GET request to the (https://ect.oxylabs.io/v1/jobs/{id}) endpoint. Make sure to replace the {id} with your actual job ID.

print("Waiting for Sitemap to create.. This may take a while..")
STATUS = False
while not STATUS:
     url = "https://ect.oxylabs.io/v1/jobs/" + json_resp["id"]
     time.sleep(10)
     info = requests.request(
        "GET",
         url,  
         auth=(USERNAME, PASSWORD),  # Your credentials go here.
     )
     information = info.json()

if len(information["events"]) == 0:
     continue

for event in information["events"]:
     if event["event"] == "job_results_aggregated" and event["status"] == "done":
         STATUS = True

In each while loop iteration, the information variable stores the latest request response in JSON format. The event key in this JSON response contains status information of the job indexing and aggregation.

When the job aggregation is finished, the response JSON will contain an event named job_results_aggregated with the status done. At that time, we set the STATUS flag to True, which causes the control to exit the loop. 

5. Once the crawling job finishes, request at https://ect.oxylabs.io/v1/jobs/{id}/aggregate to get the aggregated results. Again, make sure to replace the {id} with your job ID.

url = "https://ect.oxylabs.io/v1/jobs/" + json_resp["id"] + "/aggregate"
sitemap = requests.request(
   "GET",
   url, 
   auth=(USERNAME, PASSWORD),
)
urls = sitemap.json()

On execution of the above script, the urls variable will now contain the list of aggregated results chunks. This list can be any one of the following types depending on the output chosen:

• A list of URLs (if the output type is “sitemap”)

• A file containing all the parsed results (if the output type is “parsed”)

• A file containing all the HTML results (if the output type is “HTML”)

Since we have selected the sitemap output type, the aggregated result will be a list of URLs. 

6. In the next step, we will loop through these chunks. For each chunk, you must request the relevant chunk endpoint and parse the response to get the URLs. 

no_of_chunks = urls["chunks"]

for n in range(0, no_of_chunks):
   chunk_url = urls["chunk_urls"][n]["href"]


   chunk_data = requests.get(chunk_url,
       auth = HTTPBasicAuth(USERNAME, PASSWORD),
   )

We will send the request to the chunk_url to get the data of that particular chunk. This response data is JSON data lines (in byte format) with a single URL information encoded in each line. Here is what the response in chunk_data looks like: