> ## Documentation Index
> Fetch the complete documentation index at: https://docs.co-mind.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Scrape URL

> Scrape and extract content from a URL.



## OpenAPI

````yaml /openapi.yaml post /v1/researcher/scrape
openapi: 3.1.0
info:
  title: Co-mind.ai Private AI Platform API
  version: 1.2.4
  description: >
    Co-mind.ai Private AI Platform API.


    ## Features

    - OpenAI-compatible endpoints

    - Tool/function calling support

    - Streaming responses

    - Multiple backend support

    - Personal Access Tokens (PAT) for programmatic access


    ## Authentication

    The API supports two authentication methods:


    **1. JWT Authentication** — Login with email/password to get short-lived
    access tokens.

    Use for interactive sessions (web apps, Postman).


    **2. Personal Access Tokens (PAT)** — Long-lived tokens for programmatic/API
    access.

    Create via `POST /v1/api-tokens` after authenticating with JWT.

    PAT format: `cmnd_<tokenId>.<secret>`


    Both methods use the `Authorization: Bearer <token>` header.
  contact:
    email: support@co-mind.ai
servers:
  - url: http://co-mind-platform-host
    description: Co-mind.ai AI Platform
security:
  - BearerAuth: []
tags:
  - name: Public
    description: Public endpoints (no authentication required)
  - name: Authentication
    description: JWT login, refresh, logout, and user info
  - name: API Tokens
    description: Personal Access Token (PAT) management
  - name: Chat
    description: OpenAI-compatible chat completions
  - name: Knowledge Base
    description: Knowledge base management and RAG-enhanced chat
  - name: Completions
    description: Text completion endpoints
  - name: Embeddings
    description: Text embedding generation
  - name: Discovery
    description: Model and backend discovery endpoints
  - name: Echo Engine
    description: Audio transcription (STT), health, and metrics
  - name: Echo Engine - Recordings
    description: Recording CRUD and job linking
  - name: Echo Engine - TTS
    description: Text-to-speech synthesis
  - name: Sanitizer Admin
    description: Security policy management, health, and testing
  - name: Document Analyzer
    description: Document analysis, extraction, and review workflows
  - name: Researcher
    description: Web search, research sessions, analysis, and synthesis
  - name: Directory Admin
    description: LDAP/AD identity provider configuration and sync
  - name: Entra Admin
    description: Microsoft Entra ID (Azure AD) configuration
  - name: Tenant Admin
    description: Tenant/organization management
  - name: Tenant API Keys
    description: Provider API key management per tenant
  - name: Sub-org Management
    description: Sub-organization management
  - name: Service Endpoints
    description: SSO config discovery, OAuth token exchange
  - name: Audit Logs
    description: Audit log querying and aggregation
  - name: Quota
    description: Usage quota tracking
paths:
  /v1/researcher/scrape:
    post:
      tags:
        - Researcher
      summary: Scrape URL
      description: Scrape and extract content from a URL.
      operationId: researcherScrape
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              required:
                - url
              properties:
                url:
                  type: string
                  format: uri
      responses:
        '200':
          description: Scraped content
          content:
            application/json:
              schema:
                type: object
                additionalProperties: true
        '400':
          $ref: '#/components/responses/BadRequest'
        '401':
          $ref: '#/components/responses/Unauthorized'
components:
  responses:
    BadRequest:
      description: Bad request
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Error'
          example:
            error:
              message: Invalid request parameters
              type: invalid_request_error
              code: bad_request
    Unauthorized:
      description: Authentication required or invalid token
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Error'
          example:
            error:
              message: Invalid authentication credentials
              type: permission_error
              code: unauthorized
  schemas:
    Error:
      type: object
      properties:
        error:
          type: object
          properties:
            message:
              type: string
            type:
              type: string
            code:
              type: string
  securitySchemes:
    BearerAuth:
      type: http
      scheme: bearer
      bearerFormat: JWT or PAT
      description: >
        Bearer token authentication. Supports two token types:

        - **JWT Access Token** — obtained via `POST /v1/auth/login`

        - **Personal Access Token (PAT)** — created via `POST /v1/api-tokens`,
        format: `cmnd_<tokenId>.<secret>`

````