Pull Mechanism

The implementation of the pull mechanism varies depending on the policies of individual facilities and their preferred methods for storing information. We generally lean towards the pull approach over push, as it offers greater control, particularly in back-fill scenarios.

We securely fetch data from your Web API endpoint, normalize it into our internal format ({sensor_id, timestamp, value} for timeseries), and ingest it into our data pipeline.

What We Need From You

  1. Base API URL

    Example: https://your-api.com/api/v1/data

  2. Authentication Details

    • API key, token, OAuth credentials, or Basic Auth
    • Any required headers

  3. Endpoint(s) to Hit

    • List of endpoints with example payloads
    • Documentation or OpenAPI (Swagger) spec if available

  4. Query Parameters / Payload Specs

    • Required parameters (e.g., start_time, end_time, sensor_id)
    • Pagination method (e.g., cursor, page/limit, link-based)

  5. Frequency & Range

    • How often we should poll or pull data
    • Historical backfill duration (e.g., past 7 days, full dump)

Supported Features

  • Support for GET/POST-based APIs
  • Token, API Key, and OAuth authentication
  • Pagination handling (token-based, link-based, offset)
  • JSON/CSV/XML payload parsing
  • Time-based filtering and delta fetches