# ipfs:// URL support in curl

CURL 8.4.0 (opens new window) shipped with built-in support for ipfs:// and ipns:// addresses.

This enables curl to seamlessly integrate with the user's preferred IPFS gateway (opens new window) through the IPFS_GATEWAY environment variable or a gateway file. Best of all, these capabilities are available for immediate use today:

$ export IPFS_GATEWAY="http://127.0.0.1:8080" # local (trusted) gateway provided by ipfs daemon like Kubo
$ curl ipfs://bafkreih3wifdszgljcae7eu2qtpbgaedfkcvgnh4liq7rturr2crqlsuey
hello from IPFS

In this blog post, we will:

# A brief history

Supporting IPFS in CURL has been attempted before (opens new window) as a CURL library feature. Some discussions lead to a belief that this should be implemented in the CURL tool itself, not its library. A renewed implementation attempt (opens new window) took the tool-side approach which ultimately was accepted and is available right now in CURL 8.4.0!

The support of IPFS in CURL is effectively consisting of two implementation details.

  1. CURL tries to find a locally installed or configured gateway.
  2. It then rewrites an ipfs://bafybeigagd5nmnn2iys2f3doro7ydrevyr2mzarwidgadawmamiteydbzi to a gateway URL. This is how curl handles it internally, you see nothing of this URL rewriting.

If you have IPFS installed locally then running curl ipfs:// will Just Work™. If not, CURL will return an error with details about how to set up the gateway preference. This ensures the user agency is respected, no third-party gateway is used as implicit default.

# Why ipfs:// URL support is so important?

Why isn't https://ipfs.io/ipfs/bafybeigagd5nmnn2iys2f3doro7ydrevyr2mzarwidgadawmamiteydbzi equally acceptable?
Or why isn't a local URL http://localhost:8080/ipfs/bafybeigagd5nmnn2iys2f3doro7ydrevyr2mzarwidgadawmamiteydbzi fine?

Both addresses are tied to a specific location.

IPFS is a modular suite of protocols purpose built for the organization and transfer of content-addressed (opens new window) data. It shouldn't matter where the content is. Content Identifier (CID (opens new window)) is all that is required. The "where" part is implementation detail an IPFS system takes care of. Hardcoding a location in addition to a CID (like a specific HTTP gateway) limits end users to IPFS resources available through that one specific, centralized point of entry.

If we pull the URL apart we see:

Users of the IPFS system should not care about the where part, nor be coerced to use a specific, hard-coded entry point into the system.

Public gateways like ipfs.io are always owned by some entity and could get censored or shut down at any time. Many gateways will not allow playback of deserialized videos or only respond to CIDs from allowlists to reduce costs. Other gateways will block specific CIDs from resolving in specific jurisdictions for legal reasons. Community-run public gateways will have limits and throttle usage.

These are not limitations of IPFS but purely a limitation a specific gateway has set through custom configuration. IPFS user should always have ability to avoid such limitations if they choose to self-host and run their own IPFS node with a local gateway (opens new window).

# How does CURL find an IPFS Gateway?

Any IPFS implementation that has support for IPIP-280 (opens new window) exposes an IPFS gateway that CURL (and ffmpeg (opens new window)) can use. At the moment of writing that's just Kubo (opens new window).

CURL 8.4.0 and greater looks for a gateway in the following order:

  1. IPFS_GATEWAY, if set it's used.
  2. The --ipfs-gateway CLI argument.
  3. The ~/.ipfs/gateway file, where it reads the first line.

If a gateway hint is found at any of those places, and if that is a valid HTTP URL, then CURL will use it. If not, then you'll be getting an error message pointing to the CURL documentation related to IPFS (opens new window) to help you further.

One can specify any IPFS gateway that is in compliance with Gateway Specifications (opens new window). It is highly recommended to use a local gateway, as it provides the best security guarantees.

# Malicious gateways and data integrity?

Requesting deserialized responses and delegating hash verification to a third-party gateway comes with risks. It is possible that a public gateway is malicious. Or, that a well-known and respected gateway gets hacked and changed to return payload that does not match requested CID. How can one protect themselves against that?

If deserialized responses are necessary, one should run their own gateway in a local, controlled environment. Every block of data retrieved though self-hosted IPFS gateway is verified to match the hash from CID. For the maximum flexibility and security, find an implementation that provides the gateway endpoint (i.e. Kubo (opens new window)) and run it yourself!

When using a third-party gateway that one can't fully trust, the only secure option is to request verifiable response types (opens new window) such as application/vnd.ipld.raw (opens new window) (a single block) or application/vnd.ipld.car (opens new window) (multiple blocks in CAR archive). Both allow to locally verify if the data returned by gateway match the requested CID, removing the surface for Man-in-the-middle attacks (opens new window).

# CURL Examples

# Deserialized responses

By default, a trusted local gateway acts as a bridge between traditional HTTP clients and IPFS.

It performs necessary hash verification, UnixFS deserialization and return reassembled files to the client, as if they were stored in a traditional HTTP server. This means all validation happens on the gateway, and clients trust that the gateway is correctly validating content-addressed data before returning it to them.

# Downloading a file from IPFS with CURL

$ curl ipfs://bafkreih3wifdszgljcae7eu2qtpbgaedfkcvgnh4liq7rturr2crqlsuey -o out.txt

If curl responds with curl: IPFS automatic gateway detection failure, make sure IPFS_GATEWAY is set (see examples below).

# Explicitly specifying a gateway

To use local gateway on custom port 48080:

$ export IPFS_GATEWAY=http://127.0.0.1:48080
$ curl ipfs://bafkreih3wifdszgljcae7eu2qtpbgaedfkcvgnh4liq7rturr2crqlsuey
hello from IPFS

When setting environment variable is not feasible, one can use --ipfs-gateway instead:

$ curl --ipfs-gateway http://127.0.0.1:48080 ipfs://bafkreih3wifdszgljcae7eu2qtpbgaedfkcvgnh4liq7rturr2crqlsuey
hello from IPFS

# Following subdomain redirects

By default, the URL resolution in curl does not follow HTTP redirects and assumes the endpoint implements deserializing path gateway (opens new window), or at the very least, the trustless gateway (opens new window).
When pointing curl at a subdomain gateway (opens new window) (like https://dweb.link or the http://localhost:8080 provided by a local Kubo node (opens new window)) one has to pass -L in the curl command to follow the redirect.

$ IPFS_GATEWAY=https://localhost:8080 curl -s -L ipfs://bafkreih3wifdszgljcae7eu2qtpbgaedfkcvgnh4liq7rturr2crqlsuey
hello from IPFS

# Piping and streaming responses

Deserialized response returned by CURL can be piped directly to a video player:

$ curl ipfs://bafybeigagd5nmnn2iys2f3doro7ydrevyr2mzarwidgadawmamiteydbzi | ffplay -

# Verifiable responses

By explicitly requesting application/vnd.ipld.raw (opens new window) (a block) or application/vnd.ipld.car (opens new window) (a stream of blocks) responses, by means defined in Trustless Gateway Specification (opens new window), the user is able to fetch raw content-addressed data and perform hash verification themselves (opens new window).

# Fetching and verifying a directory from an untrusted gateway

Requesting trustless and verifiable (opens new window) CAR response via Accept HTTP header:

$ export IPFS_GATEWAY="https://ipfs.io" # using untrusted public gateway
$ curl -H "Accept: application/vnd.ipld.car" "ipfs://bafybeiakou6e7hnx4ms2yangplzl6viapsoyo6phlee6bwrg4j2xt37m3q" > dag.car

Then, CAR can be moved around and imported into some other IPFS node:

$ ipfs dag import dag.car

or verified and unpacked locally, without having to run a full IPFS node, with tools like go-car (opens new window) or ipfs-car (opens new window):

$ npm i -g ipfs-car
$ ipfs-car unpack dag.car --output dag.out
$ ls dag.out
1007 - Sustainable - alt.txt
1007 - Sustainable - transcript.txt
1007 - Sustainable.png

# What's next?

More places supporting IPFS addresses. Everyone can integrate ipfs:// and ipns:// URL support into their application. See specifications proposed in IPIP-280 (opens new window) for technical details. We are tracking potential project (opens new window) where an integration makes sense! If you feel up to the challenge, don't hesitate to drop a comment in one of the potential projects (opens new window) for IPFS URL integration or find us on:

Or one of the other many places where the IPFS community (opens new window) is active.