Today, we are EXCEEDINGLY pleased to announce the Beta of Project Bacalhau (opens new window). Bacalhau is a distributed compute network for IPFS data, where users can run arbitrary docker containers and Wasm images against data stored in IPFS. With Bacalhau, we plan to take a meaningful step towards unlocking distributed, content-addressed compute for everyone and improving how people interact with the ever-growing amount of information available.
Since our initial public launch in July, we have heard tons of exciting ways that researchers and entrepreneurs are using Bacalhau. These include:
- City of Las Vegas & Blocz.io (opens new window): Blocz IO is excited to partner with Bacalhau to process real-time CCTV data for the City of Las Vegas. Our combined services will significantly reduce costs and scale up the existing safety capabilities for its visitors.
- Bacalhau Partners with LabDAO to Accelerate Scientific Progress (opens new window): Our goal is to accelerate progress by making scientific tools more accessible - building distributed compute infrastructure is an essential component on the way there. We are excited to be partnering with the Bacalhau team to integrate this piece of the puzzle together.
- Bacalhau Case Studies (opens new window):
Based on user feedback, we are introducing a breadth of new features to help people adopt compute over data even faster.
# Introducing Bacalhau Beta v1
We are proud to move the Bacalhau API from
beta. With this change, we are committing to both a more stable API as well as backward compatibility for future versions (
alpha jobs will not be supported going forward). For most jobs, this will result in no change, but it will require updating the versions of your jobs from
Note: This only applies if you are using a JSON or YAML job format - if you are using the CLI, you should be all set.
Along with this change, we have several additional features that address some big requests:
- Improvements to network reliability: In particular, by switching the way we are using libp2p gossipsub, you should see significantly reduced network latency.
- Native Filecoin support: via native Lotus integration as well as Estuary integration via an Estuary API key (Estuary publishes results from the network both into IPFS as well as backing them up in Filecoin)
- Support for WebAssembly: Bacalhau can now deterministically run WASM code either uploaded via the client or stored on IPFS.
- Job pipelines from Apache Airflow: Jobs can now be chained together via Airflow, Cron scheduling, and more.
# New Examples
We also have been running Bacalhau through its paces. Below are just a few of the many examples you can already run on Bacalhau against IPFS and/or Filecoin data:
- Python - Hello World (opens new window)
- R - Hello World (opens new window)
- Rust via WebAssembly (opens new window)
- Python - Pandas (opens new window)
- Python - Custom Containers (opens new window)
- Image Processing (opens new window)
- Parallel Workloads (opens new window)
- Blockchain ETL (opens new window)
- Oceanography Analysis (opens new window)
- Stable Diffusion (CPU (opens new window) and GPU (opens new window))
- Object detection - YOLO (opens new window)
- Speech Recognition with Whisper (opens new window)
- Image Generation with StyleGAN (opens new window)
- Molecular Dynamics - Simulation with OpenMM (opens new window)
Our goal is to maintain a quarterly release cadence. By the end of the year, we hope to offer the following:
- Python SDK and SDKs for our API in some other languages too
- FIL+ Dashboard
- Initial simulator framework
- Improved examples
- A draft invocation spec in collaboration with the CoD ecosystem
- Networking design
- Trusted Execution Environment design
- An initial game theoretic analysis of the Bacalhau protocol
# Would you like to learn more? Come help!
If you would like to learn more about Bacalhau or let us know how you'd like the project to change and help you, visit any of the following:
- Website: https://bacalhau.org/ (opens new window)
- Docs: https://docs.bacalhau.org/ (opens new window)
- Mail: https://groups.google.com/g/bacalhau-discuss (opens new window)
- Slack: https://filecoin.io/slack (opens new window) (#bacalhau (opens new window) channel)
- Github: https://github.com/filecoin-project/bacalhau (opens new window)
Thank you so much!