Web Scraper API – Documentation

This API allows users to scrape metadata and HTML elements from public websites using FastAPI.

Base URL:

GET /metadata

Description: Extracts <meta> tags from a given webpage.

Query Parameters:

url (string, required): URL of the page to scrape
meta (optional): Comma-separated list of specific meta tag names or properties to extract

Examples:

/api/metadata?url=https://www.example.com/

/api/metadata?url=https://www.example.com/&meta=description,og:title

GET /elements

Description: Extracts specific HTML elements and their attributes from a given webpage.

Query Parameters:

url (string, required): URL of the page to scrape
query (optional): Comma-separated list of HTML tags to include (e.g., h1,p)
key (optional): Comma-separated list of attributes to return per element (e.g., id,class,text)
id (optional): Filter elements by a specific id
class (optional): Filter elements that contain a specific class value

Examples:

/api/elements?url=https://www.example.com/

/api/elements?url=https://www.example.com/&query=h1,p&key=id,class,text

/api/elements?url=https://example.com&query=div&class=section&key=id

GET /all

Description: Fetches both metadata and HTML elements from a given page.

Query Parameters:

url (string, required): URL of the page to scrape

Examples:

/api/all?url=https://www.example.com/

POST /metadata

Description: Allows you to send a POST request to extract <meta> tags from a webpage. This endpoint requires authentication.

Headers:

Authorization (string, required): Bearer token for authentication. The token can be generated inside the app.

Request Body:

{
  "url": "https://www.example.com/",
  "meta": "description,og:title"  
}

Testing with Postman:

Open Postman and select the POST method.
Enter the URL: /api/metadata.
Go to the "Headers" tab and add the following key-value pair:
- Authorization: Bearer YOUR_TOKEN_HERE
Go to the "Body" tab, select raw and choose JSON format.
Enter the request body (example shown above).
Click "Send" to send the POST request.

How to Get a Token:

To make a POST request, you need a token. The token can be generated inside the app. Here's how:

Navigate to tokens.
Click "Create New Token" to get a new token.
Copy the token and use it in your Authorization header for POST requests.

POST /elements

Description: Allows you to send a POST request to extract HTML elements from a webpage. This endpoint requires authentication.

Headers:

Authorization (string, required): Bearer token for authentication. The token can be generated inside the app.

Request Body:

{
  "url": "https://www.example.com/",
  "query": "h1,p",                 
  "key": "id,class,text"           
}

Testing with Postman:

Follow the same steps as for the POST /metadata endpoint above, just use /api/elements as the URL.

POST /all

Description: Allows you to send a POST request to extract both metadata and HTML elements from a webpage. This endpoint requires authentication.

Headers:

Authorization (string, required): Bearer token for authentication. The token can be generated inside the app.

Request Body:

{
  "url": "https://www.example.com/"
}

Testing with Postman:

Follow the same steps as for the POST /metadata endpoint above, just use /api/all as the URL.

GET /api/google-rank

Description: Checks the Google search rank of a specific URL or keyword on Google search results.

Query Parameters:

q (string, required): The query (keyword or phrase) for which to check the rank.
site (optional): The specific site to check within the search results.

Examples:

/api/google-rank?q=web+scraping

/api/google-rank?q=web+scraping&site=example.com

Authorization:

Include the Authorization header with your Bearer token when making the request.

Authorization: Bearer YOUR_TOKEN_HERE

Response Example:

        {
          "user_email": "test@test.com",
          "keyword": "phone",
          "total_results": "15450000000",
          "search_time": "0.22",
          "requested_site": {
            "position": 4,
            "url": "https://www.apple.com/iphone/",
            "display_link": "www.apple.com",
            "title": "iPhone - Apple",
            "snippet": "iPhone 16 screen displaying phone lock screen. Control Center. Totally under control. Customize how your ..."
          },
          "ranking": [
            {
              "position": 1,
              "url": "https://play.google.com/store/apps/details?id=com.google.android.dialer&hl=en_US",
              "display_link": "play.google.com",
              "title": "Phone by Google - Apps on Google Play",
              "snippet": "About this app. arrow_forward. Google's official phone calling app is now available to download for the first time. Phone helps you easily connect with family ..."
            },
            {
              "position": 2,
              "url": "https://www.merriam-webster.com/dictionary/phone",
              "display_link": "www.merriam-webster.com",
              "title": "PHONE Definition & Meaning - Merriam-Webster",
              "snippet": "6 days ago ... 1. a device by which sound (such as speech) is converted into electrical impulses and transmitted (as by wire or radio waves) to one or more specific receivers."
            },
            {
              "position": 3,
              "url": "https://apps.apple.com/us/app/phone/id1146562108",
              "display_link": "apps.apple.com",
              "title": "Phone on the App Store",
              "snippet": "Phone. I'm falling in love with this app and iOS 17 in general. Please don't ever get rid of the new ringtones and text tones because they're so awesome and fun ..."
            },
            {
              "position": 4,
              "url": "https://www.apple.com/iphone/",
              "display_link": "www.apple.com",
              "title": "iPhone - Apple",
              "snippet": "iPhone 16 screen displaying phone lock screen. Control Center. Totally under control. Customize how your ..."
            },
            {
              "position": 5,
              "url": "https://www.phone.com/",
              "display_link": "www.phone.com",
              "title": "VOIP Business Phone Service & Business Phone Systems",
              "snippet": "With Phone.com you get a business phone number that is separate from your personal line and a solution that allows you to work from any device, anywhere."
            },
            {
              "position": 6,
              "url": "https://www.walmart.com/cp/cell-phones/1105910",
              "display_link": "www.walmart.com",
              "title": "Cell Phones, Unlocked & No-Contract Phones, Prepaid Phones ...",
              "snippet": "Shop phones by type. Prepaid. Prepaid Phones Postpaid Contract Phones Refurbished Refurbished Phones Shop all Shop All Phones"
            },
            {
              "position": 7,
              "url": "https://en.wikipedia.org/wiki/Mobile_phone",
              "display_link": "en.wikipedia.org",
              "title": "Mobile phone - Wikipedia",
              "snippet": "In 1979, Nippon Telegraph and Telephone (NTT) launched the world's first cellular network in Japan. ... In 1983, the DynaTAC 8000x was the first commercially ..."
            },
            {
              "position": 8,
              "url": "https://www.bestbuy.com/site/electronics/mobile-cell-phones/abcat0800000.c?id=abcat0800000",
              "display_link": "www.bestbuy.com",
              "title": "Cell Phones: Smartphones and Mobile Phones - Best Buy",
              "snippet": "iPhone is a good choice if you already have other Apple products because your new phone can be easily integrated into the Apple ecosystem. However, you might ..."
            },
            {
              "position": 9,
              "url": "https://dmv.vermont.gov/contact/phone-numbers",
              "display_link": "dmv.vermont.gov",
              "title": "Phone Numbers | Department of Motor Vehicles",
              "snippet": "Mar 6, 2025 ... Vermont Department of Motor Vehicles 120 State Street Montpelier, VT 05603-0001 Monday-Friday: 7:45 am - 4:00 pm email telephone"
            },
            {
              "position": 10,
              "url": "https://www.zoom.com/en/products/voip-phone/",
              "display_link": "www.zoom.com",
              "title": "Cloud VoIP Phone Services | Zoom",
              "snippet": "Enable seamless reliable communications, with an easy-to-use, AI-powered, phone system built with security from the start."
            }
          ]
        }

GET /api/screenshot

Description: Captures a screenshot of the specified webpage with optional highlighting of HTML elements. You can choose between full-page screenshot using Chrome DevTools or viewport screenshot.

Query Parameters:

url (string, required): URL of the page to capture
query (string, optional, default: h1,p): Comma-separated list of HTML tags to highlight (e.g., h1,p,div)
method (string, optional, default: devtools): Capture method, either devtools (faster) or scroll (more reliable)
waitUntilStart (float, optional, default: 0): Additional time in seconds to wait after page load before capturing
width (integer, optional, default: 1920): Viewport width in pixels
height (integer, optional, default: 1080): Viewport height in pixels
full_page (boolean, optional, default: true): Whether to capture the full scrollable page

Examples:

/api/screenshot?url=https://www.example.com/

/api/screenshot?url=https://www.example.com/&query=h1,div,button

/api/screenshot?url=https://www.example.com/&method=scroll&waitUntilStart=2&width=1366&height=768

Notes

If JavaScript content isn't rendered correctly, Selenium will be used as a fallback.
The text key in the key param refers to the visible inner text of the element.
You can pass both query and key as either a single string or a comma-separated list.