Search

Getting unique URLs into BurpSuite

Getting unique URLs into BurpSuite

TLDR

Run script, get a list of unique URLs into BurpSuite using the Bing API.

Summary

One of the early tasks when conducting a website penetration test involves Search Engine Discovery. A great well known resources for this is OWASP and this can be found detailed in WSTG-INFO-01. Part of this task involves enumerating various website paths and subdomains that have been publicly indexed. Getting this information into BurpSuite manually is painful so here is the solution I came up with to automate part of the process.

Bing Search API

Why not google? I started with google first and did manage to get results from the API. The only issue was the free tier of the google API seemed to limit the number of results. The few test cases I did showed that Bing, at least at the free tier offered more results.

Create BingSearch API Key

  1. If you haven’t already you will need to create an Azure account. I will be using the free tier of the Bing API so requests will get limited if you use too much, and you *should not be charged but that’s on you to read and make sure.
  2. In the Azure console, search for bing and then select Bing Resources.
image
  1. Select Add then Bing Search.
image
  1. Select all the correct options. The Resource Group can be any existing resource group or just create a new one to keep everything separate. Give the instance a unique name & leave the Region as global. Set the pricing tier to F1 (3 Calls per second. 1k Calls per month). Select the confirmation checkbox then click on Review+Create.
image
  1. Here is an overview of the F1 (Free Tier) for Bing Search. Seems to be more than enough for now much I actually use it.
image
  1. Select Create to complete the creation of the object.
image
  1. Once the deployment has completed. Click on the Go to resource button.
image
  1. At the selected resource select Manage keys.
image
  1. Click on the clipboard icon to copy your key to the clipboard.
image
  1. Test the API from the command line.

PowerShell

Invoke-RestMethod -Uri "https://api.bing.microsoft.com/v7.0/search?q=test&mkt=en-us" -Headers @{'Ocp-Apim-Subscription-Key' = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'}
image

Bash

curl -X GET "https://api.bing.microsoft.com/v7.0/search?q=test&mkt=en-us" -H "Ocp-Apim-Subscription-Key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
image

PowerShell Script

  1. Save the below script to a file named urls.ps1.
function Get-BingURLs {
    param (
        [Parameter(Mandatory = $true)]
        [ValidateScript({
            $_ -match '^[a-zA-Z0-9]{32}$'
        })]
        [string]$ApiKey,

        [Parameter(Mandatory = $true)]
        [string]$Query,

        [int]$Count = 50
    )

    process {
        # Create array
        $total_response = @()       
        # Get the total number of results
        $response = Invoke-RestMethod -Uri "https://api.bing.microsoft.com/v7.0/search?q=$Querymkt=en-us" -Headers @{'Ocp-Apim-Subscription-Key' = $ApiKey}
        # Loop
        for ($Offset = 0; $Offset -le $response.webPages.totalEstimatedMatches; $Offset += 50) {
            # Make the API request using Invoke-RestMethod
            $response = Invoke-RestMethod -Uri "https://api.bing.microsoft.com/v7.0/search?q=$Query&count=$Count&offset=$Offset&mkt=en-us" -Headers @{'Ocp-Apim-Subscription-Key' = $ApiKey}
            Start-Sleep -Seconds 1
            if ($response.webPages -ne $null) {
                $total_response += $response
            }
        }
        # Display search results
        $total_response.webPages.value.url | Sort-Object | Select-Object -Unique
    }
}
  1. Load PowerShell in Kali and then import the function. 
pwsh
Import-Module ./urls.ps1
image
  1. Test the function in PowerShell. Larger sites will take a while to complete due to the loop needing to complete before the URLs can be sorted & reduced to only the unique ones. The site:kfc.com is the exact text passed to Bing. So you can substitute this for other helpful queries like inurl:, intitle:, intext: or whatever else Bing supports. 
Get-BingURLs -ApiKey XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX -Query "site:kfc.com"
image
  1. Usually it’s best to output the list to a text file first so you can review the results. To do that we just modify the command slightly. 
Get-BingURLs -ApiKey XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX -Query "site:kfc.com" | Out-File kfc.txt
head ./kfc.txt
image
  1. The final step is to get it into BurpSuite. Make sure BurpSuite has been loaded and the proxy is operational. 
Get-Content -Path "kfc.txt" | ForEach-Object { Invoke-WebRequest -Uri $_ -Method Get -Proxy "http://127.0.0.1:8080" -TimeoutSec 3 -SkipCertificateCheck }
image
  1. You should see BurpSuite start to populate unique URLs and paths in the HTTP history tab of the proxy.
image
image