Trustless Gateway Specification

status: reliable
Editors
Marcin Rataj GitHub
Henrique Dias GitHub
History
Commit History
Feedback
GitHub ipfs/specs (pull requests, new issue, open issues)

Trustless Gateway is a subset of [path-gateway] that allows light IPFS clients to retrieve data behind a CID and verify its integrity without delegating any trust to the gateway itself.

The minimal implementation means:

1. HTTP API

A subset of "HTTP API" of [path-gateway].

1.1 GET /ipfs/{cid}[/{path}][?{params}]

Downloads verifiable, content-addressed data for the specified immutable content path.

Optional path is permitted for requests that specify CAR format (?format=car or Accept: application/vnd.ipld.car).

For block requests (?format=raw or Accept: application/vnd.ipld.raw), only GET /ipfs/{cid}[?{params}] is supported.

1.2 HEAD /ipfs/{cid}[/{path}][?{params}]

Same as GET, but does not return any payload.

1.3 GET /ipns/{key}[?{params}]

Downloads data at specified IPNS Key. Verifiable [ipns-record] can be requested via ?format=ipns-record or Accept: application/vnd.ipfs.ipns-record.

1.4 HEAD /ipns/{key}[?{params}]

Same as GET, but does not return any payload.

2. HTTP Request

Same as in [path-gateway], but with limited number of supported response types.

2.1 Request Headers

2.1.1 Accept (request header)

A Client SHOULD send this HTTP header to leverage content type negotiation based on section 12.5.1 of [rfc9110].

Below response types MUST be supported:

Below response types SHOULD be supported:

A Gateway SHOULD return HTTP 400 Bad Request when running in strict trustless mode (no deserialized responses) and Accept header is missing.

2.2 Request Query Parameters

2.2.1 dag-scope (request query parameter)

Optional, dag-scope=(block|entity|all) with default value all, only available for CAR requests.

Describes the shape of the DAG fetched the terminus of the specified path whose blocks are included in the returned CAR file after the blocks required to traverse path segments.

  • block - Only the root block at the end of the path is returned after blocks required to verify the specified path segments.

  • entity - For queries that traverse UnixFS data, entity roughly means return blocks needed to verify the terminating element of the requested content path. For UnixFS, all the blocks needed to read an entire UnixFS file, or enumerate a UnixFS directory. For all queries that reference non-UnixFS data, entity is equivalent to block

  • all - Transmit the entire contiguous DAG that begins at the end of the path query, after blocks required to verify path segments

When present, returned Etag must include unique prefix based on the passed scope type.

2.2.2 entity-bytes (request query parameter)

The optional entity-bytes=from:to parameter is available only for CAR requests.

It implies dag-scope=entity and serves as a trustless equivalent of an HTTP Range Request.

When the terminating entity at the end of the specified content path:

  • can be interpreted as a continuous array of bytes (such as a UnixFS file), a Gateway MUST return only the minimal set of blocks necessary to verify the specified byte range of that entity.

    • When dealing with a sharded UnixFS file (dag-pb, 0x70) and a non-zero from value, the UnixFS data and blocksizes determine the corresponding starting block for a given from offset.
  • cannot be interpreted as a continuous array of bytes (such as a DAG-CBOR/JSON map or UnixFS directory), the parameter MUST be ignored, and the request is equivalent to dag-scope=entity.

Allowed values for from and to follow a subset of section 14.1.2 from [rfc9110], where they are defined as offset integers that limit the returned blocks to only those necessary to satisfy the range [from,to]:

  • from value gives the byte-offset of the first byte in a range.
  • to value gives the byte-offset of the last byte in the range; that is, the byte positions specified are inclusive.

The following additional values are supported:

  • * can be substituted for end-of-file
    • entity-bytes=0:* is the entire file (a verifiable version of HTTP request for Range: 0-)
  • Negative numbers can be used for referring to bytes from the end of a file
    • entity-bytes=-1024:* is the last 1024 bytes of a file (verifiable version of HTTP request for Range: -1024)
    • It is also permissible (unlike with HTTP Range Requests) to ask for the range of 500 bytes from the beginning of the file to 1000 bytes from the end: entity-bytes=499:-1000

A Gateway MUST augment the returned Etag based on the passed entity-bytes.

A Gateway SHOULD return an HTTP 400 Bad Request error when the requested range cannot be parsed as valid offset positions.

In more nuanced error scenarios, a Gateway MUST return a valid CAR response that includes enough blocks for the client to understand why the requested entity-bytes was incorrect or why only a part of the requested byte range was returned:

  • If the requested entity-bytes resolves to a range that partially falls outside the entity's byte range, the response MUST include the subset of blocks within the entity's bytes.

    • This allows clients to request valid ranges of the entity without needing to know its total size beforehand, and it does not require the Gateway to buffer the entire entity before returning the response.
  • If the requested entity-bytes resolves to a zero-length range or falls fully outside the entity's bytes, the response is equivalent to dag-scope=block.

    • This allows client to produce a meaningful error (e.g, in case of UnixFS, leverage Data.blocksizes information present in the root dag-pb block).
  • In streaming scenarios, if a Gateway is capable of returning the root block but lacks prior knowledge of the final component of the requested content path being invalid or absent in the DAG, a Gateway SHOULD respond with HTTP 200.

    • This behavior is a consequence of HTTP streaming limitations: blocks are not buffered, by the time a related parent block is being parsed and returned to the client, the HTTP status code has already been sent to the client.

3. HTTP Response

Below MUST be implemented in addition to "HTTP Response" of [path-gateway].

3.1 Response Headers

3.1.1 Content-Type (response header)

MUST be returned and include additional format-specific parameters when possible.

If a CAR stream was requested:

  • the response MUST include the parameter specifying CAR version. For example: Content-Type: application/vnd.ipld.car; version=1
  • the response SHOULD include additional content type parameters, as noted in CAR format signaling in Response.

3.1.2 Content-Disposition (response header)

MUST be returned and set to attachment to ensure requested bytes are not rendered by a web browser.

4. Block Responses (application/vnd.ipld.raw)

An opaque bytes matching the requested block CID (application/vnd.ipld.raw).

The Body hash MUST match the Multihash from the requested CID.

5. CAR Responses (application/vnd.ipld.car)

A CAR stream for the requested application/vnd.ipld.car content type (with optional order and dups params), path and optional dag-scope and entity-bytes URL parameters.

5.1 CAR version

Value returned in CarV1Header.version field MUST match the version parameter returned in Content-Type header.

5.2 CAR roots

The behavior associated with the CarV1Header.roots field is not currently specified.

The lack of standard here means a client MUST assume different Gateways could return a different value.

A Client SHOULD ignore this field.

As of 2023-06-20, the behavior of the roots CAR field remains an unresolved item within the CARv1 specification.

5.3 CAR order (content type parameter)

The order parameter allows clients to specify the desired block order in the response. It supports the following values:

A Gateway SHOULD always return explicit order in CAR's Content-Type response header.

A Gateway MAY skip order in CAR response if no order was explicitly requested by the client and the default order is unknown.

A Client MUST assume implicit order=unk when order is missing, unknown, or empty.

5.4 CAR dups (content type parameter)

The dups parameter specifies whether duplicate blocks (the same block occurring multiple times in the requested DAG) will be present in the CAR response. Useful when a deterministic block order is used.

It accepts two values:

When set to y, light clients are able to discard blocks after reading them, removing the need for caching in-memory or on-disk.

Setting to n allows for more efficient data transfer of certain types of data, but introduces additional resource cost on the receiving end, as each block needs to be kept around in case its CID appears again.

If the dups parameter is absent from the Accept request header, the behavior is unspecified. In such cases, a Gateway should respond with dups=n if it has control over the duplicate status, or without dups parameter if it does not. Defaulting to the inclusion of duplicate blocks (dups=y) SHOULD only be implemented by Gateway systems that exclusively support dups=y and do not support any other behavior.

A Client MUST not assume any implicit behavior when dups is missing.

If the dups parameter is absent from the Content-Type response header, the behavior is unspecified, and the CAR response includes an arbitrary list of blocks. In this unknown state, the client MUST assume duplicates are not sent, but also MUST ignore duplicates and other unexpected blocks if they are present.

A Gateway MUST always return dups in Content-Type response header when the duplicate status is known at the time of processing the request. A Gateway SHOULD not return dups if determining the duplicate status is not possible at the time of processing the request.

A Gateway MUST NOT include virtual blocks identified by identity CIDs (multihash with 0x00 code) in CAR responses. This exclusion applies regardless of their presence in the DAG or the value assigned to the "dups" parameter, as the raw data is already present in the parent block that links to the identity CID.

5.5 meta (content type parameter)

The meta parameter allows clients to request the server to include additional metadata about the CAR along with the response body.

The value of this parameter includes both the location where the metadata is given (e.g. eof) as well as the type of data received (e.g. json) separated by a +, to give a value such as meta=eof+json

When the location parameter is set to eof, which is currently the only supported value, the server SHOULD respond with the following response body:

<Response body as CARv1 stream> <0x00 byte> <Metadata>

The only supported value for the data type parameter is json. This signifies that the metadata MUST be a JSON object.

This parameter MUST only be used with CAR version=1.

When the parameter is not set or does not equal eof+json, the server SHOULD not add any extra blocks to the response, neither the 0x00 byte nor any metadata.

When meta=eof+json, the JSON object SHOULD conform to the following JSON schema.

{
  "type": "object",
  "properties": {
    "data": {
      "type": "object",
      "description": "Properties of the response"
    },
    "error": {
      "type": "string",
      "description": "Error message"
    },
    "sig": {
      "type": "string",
      "description": "A signature, using the server's Ed2559 identity, over the `data` object serialized as JSON."
    },
    "required": []
  }
}

The properties object can include any fields that the server would like to implement. The following JSON schema explicitly mentions certain properties fields in order to reach a convention on their definition as they have existing use cases.

{
  "type": "object",
  "properties": {
    "car_bytes": {
      "description": "The total byte length of the CAR stream (excluding the 0x00 byte and the metadata block)",
      "type": "integer"
    },
    "data_bytes": {
      "description": "Total byte length of the flat file before it was encoded into a CAR file",
      "type": "integer"
    },
    "block_count": {
      "description": "Total number of blocks present in the CAR stream (excluding the 0x00 byte and the metadata block, but including duplicates when present)",
      "type": "integer"
    },
    "car_cid": {
      "description": "A hash of the CAR stream giving a CIDv1 with 0x0202 codec",
      "type": "string"
    },
    "b3checksum": {
      "description": "A Blake3 hash (checksum) of the CAR stream (excluding the 0x00 byte and the metadata block). The value should be serialized as a multihash with multibase prefix, preferably using Base58 encoding.",
      "type": "string"
    },
    "content_path": {
      "description": "The url path in the request as executed by the gateway, e.g. `/ipfs/bafy1234/cat.jpg`. The query string MUST BE stripped from the path.",
      "type": "string"
    },
    "dag_params": {
      "description": "A map with DAG params like dag-scope, entity-bytes from [IPIP-402](https://specs.ipfs.tech/ipips/ipip-0402/)",
      "type": "object",
      "properties": {
        "dag-scope": {
          "description": "See [IPIP-402](https://specs.ipfs.tech/ipips/ipip-0402/) for the definition",
          "type": "string"
        },
        "entity-bytes": {
          "description": "See [IPIP-402](https://specs.ipfs.tech/ipips/ipip-0402/) for the definition",
          "type": "string"
        }
      },
      "required": []
    },
    "car_params": {
      "description": "A map with CAR content type params like order and dups from [IPIP-412](https://specs.ipfs.tech/ipips/ipip-0412/)",
      "type": "object",
      "properties": {
        "order": {
          "description": "See [IPIP-412](https://specs.ipfs.tech/ipips/ipip-0412/) for the definition.",
          "type": "string"
        },
        "dups": {
          "description": "See [IPIP-412](https://specs.ipfs.tech/ipips/ipip-0412/) for the definition.",
          "type": "string"
        }
      },
      "required": []
    }
  },
  "required": []
}

5.6 CAR format parameters and determinism

The default header and block order in a CAR format is not specified by IPLD specifications.

Clients MUST NOT assume that CAR responses are deterministic (byte-for-byte identical) across different gateways.

Clients MUST NOT assume that CAR includes CIDs and their blocks in the same order across different gateways.

Clients MUST assume block order and duplicate status only if Content-Type returned with CAR responses includes optional order or dups parameters, as specified by [ipip-0412].

A Gateway SHOULD support some aspects of determinism by implementing content type negotiation and signaling via Accept and Content-Type headers.

In controlled environments, clients MAY choose to rely on implicit and undocumented CAR determinism, subject to the agreement of the following conditions between the client and the gateway:

Mind this is undocumented behavior, and MUST NOT be used on public networks.

5.6.1 CAR format signaling in Request

Content type negotiation is based on section 12.5.1 of [rfc9110].

Clients MAY indicate their preferred block order by sending an Accept header in the HTTP request. The Accept header format is as follows:

Accept: application/vnd.ipld.car; version=1; order=dfs; dups=y

In the future, when more orders or parameters exist, clients will be able to specify a list of preferences, for example:

Accept: application/vnd.ipld.car;order=foo, application/vnd.ipld.car;order=dfs;dups=y;q=0.5

The above example is a list of preferences, the client would really like to use the hypothetical order=foo however if this isn't available it would accept order=dfs with dups=y instead (lower priority indicated via q parameter, as noted in [rfc9110]).

5.6.2 CAR format signaling in Response

The Trustless Gateway MUST always respond with a Content-Type header that includes information about all supported and known parameters, even if the client did not specify them in the request.

The Content-Type header format is as follows:

Content-Type: application/vnd.ipld.car;version=1;order=dfs;dups=n

Gateway implementations SHOULD decide on the implicit default ordering or other parameters, and use it in responses when client did not explicitly specify any matching preference.

A Gateway MAY choose to implement only some parameters and return HTTP 400 Bad Request or 406 Not Acceptable when a client requested a response with unsupported content type variant.

A Client MUST verify Content-Type returned with CAR response before processing the payload, as the legacy gateway may not support optional content type parameters like order an dups and return plain application/vnd.ipld.car.

6. IPNS Record Responses (application/vnd.ipfs.ipns-record)

An opaque bytes matching the Signed IPNS Record for the requested IPNS Name returned as application/vnd.ipfs.ipns-record.

A Client MUST confirm the record signature match libp2p-key from the requested IPNS Name.

A Client MUST perform additional record verification according to the IPNS specification.

A. References

[ipip-0412]
IPIP-0412: Signaling Block Order in CARs on HTTP Gateways. Marcin Rataj; Jorropo. 2023-05-15. URL: https://specs.ipfs.tech/ipips/ipip-0412/
[ipns-record]
IPNS Record and Protocol. Vasco Santos; Steve Allen; Marcin Rataj; Henrique Dias; Gus Eggert. 2023-10-03. URL: https://specs.ipfs.tech/ipns/ipns-record/
[path-gateway]
Path Gateway Specification. Marcin Rataj; Adrian Lanzafame; Vasco Santos; Oli Evans; Henrique Dias. 2023-03-30. URL: https://specs.ipfs.tech/http-gateways/path-gateway/
[rfc2119]
Key words for use in RFCs to Indicate Requirement Levels. S. Bradner. IETF. March 1997. Best Current Practice. URL: https://www.rfc-editor.org/rfc/rfc2119
[rfc9110]
HTTP Semantics. R. Fielding, Ed.; M. Nottingham, Ed.; J. Reschke, Ed.. IETF. June 2022. Internet Standard. URL: https://www.rfc-editor.org/rfc/rfc9110