rules_s3

Bazel rules for downloading files from Amazon Simple Storage Service (S3) and S3-compatible object storage.

• Can be used as a drop-in replacement for http_file (s3_file) and http_archive (s3_archive)
• Can fetch large amounts of objects lazily from a bucket using the s3_bucket module extension
• Supports fetching from private buckets using a credential helper
• Uses Bazel's downloader and the repository cache
• No dependencies like the AWS cli (aws) need to be installed

You can find the latest version of rules_s3 on the Bazel Central Registry. Installation works by adding a bazel_dep line to MODULE.bazel.

bazel_dep(name = "rules_s3", version = "1.0.1")

Additionally, you need to configure a credential helper for the S3 endpoint of your choice. In theory, you can use any credential helper that can generate request header signatures for S3. In practice, the recommended credential helper is tweag-credential-helper. The project README will guide you through the required setup. An example is also provided here.

rules_s3 offers two repository rules s3_file and s3_archive for fetching individual objects. If you need to download multiple objects from a bucket, use the s3_bucket module extension instead.

To see how it all comes together, take a look at the full example.

load("@rules_s3//s3:repo_rules.bzl", "s3_archive")

s3_archive(name, build_file, build_file_content, canonical_id, endpoint, endpoint_style, integrity,
           patch_args, patch_cmds, patch_cmds_win, patch_strip, patch_tool, patches,
           remote_patch_strip, remote_patches, rename_files, repo_mapping, sha256, strip_prefix, type,
           url)

Downloads a Bazel repository as a compressed archive file from an S3 bucket, decompresses it, and makes its targets available for binding.

It supports the following file extensions: "zip", "jar", "war", "aar", "tar", "tar.gz", "tgz", "tar.xz", "txz", "tar.zst", "tzst", tar.bz2, "ar", or "deb".

Examples: Suppose your code depends on a private library packaged as a .tar.gz which is available from s3://my_org_code/libmagic.tar.gz. This .tar.gz file contains the following directory structure:

MODULE.bazel
src/
  magic.cc
  magic.h

In the local repository, the user creates a magic.BUILD file which contains the following target definition:

cc_library(
    name = "lib",
    srcs = ["src/magic.cc"],
    hdrs = ["src/magic.h"],
)

Targets in the main repository can depend on this target if the following lines are added to MODULE.bazel:

s3_archive = use_repo_rule("@rules_s3//s3:repo_rules.bzl", "s3_archive")

s3_archive(
    name = "magic",
    url = "s3://my_org_code/libmagic.tar.gz",
    sha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    build_file = "@//:magic.BUILD",
)

Then targets would specify @magic//:lib as a dependency.

ATTRIBUTES

load("@rules_s3//s3:repo_rules.bzl", "s3_file")

s3_file(name, build_file, build_file_content, canonical_id, downloaded_file_path, endpoint,
        endpoint_style, executable, integrity, repo_mapping, sha256, url)

Downloads a file from an S3 bucket and makes it available to be used as a file group.

Examples: Suppose you need to have a large file that is read during a test and is stored in a private bucket. This file is available from s3://my_org_assets/testdata.bin. Then you can add to your MODULE.bazel file:

s3_file = use_repo_rule("@rules_s3//s3:repo_rules.bzl", "s3_file")

s3_file(
    name = "my_testdata",
    url = "s3://my_org_assets/testdata.bin",
    sha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
)

Targets would specify @my_testdata//file as a dependency to depend on this file.

ATTRIBUTES

s3_bucket = use_extension("@rules_s3//s3:extensions.bzl", "s3_bucket")
s3_bucket.from_file(name, bucket, dev_dependency, endpoint, endpoint_style, lockfile,
                    lockfile_jsonpath, method)

Downloads a collection of objects from an S3 bucket and makes them available under a single hub repository name.

Examples: Suppose your code depends on a collection of large assets that are used during code generation or testing. Those assets are stored in a private S3 bucket s3://my_org_assets.

In the local repository, the user creates a s3_lock.json JSON lockfile describing the required objects, including their expected hashes:

  {
      "trainingdata/model/small.bin": {
          "sha256": "abd83816bd236b266c3643e6c852b446f068fe260f3296af1a25b550854ec7e5"
      },
      "trainingdata/model/medium.bin": {
          "sha256": "c6f9568f930b16101089f1036677bb15a3185e9ed9b8dbce2f518fb5a52b6787"
      },
      "trainingdata/model/large.bin": {
          "sha256": "b3ccb0ba6f7972074b0a1e13340307abfd5a5eef540c521a88b368891ec5cd6b"
      },
      "trainingdata/model/very_large.bin": {
          "remote_path": "weird/nested/path/extra/model/very_large.bin",
          "integrity": "sha256-Oibw8PV3cDY84HKv3sAWIEuk+R2s8Hwhvlg6qg4H7uY="
      }
  }

The exact format for the lockfile is a JSON object where each key is a path to a local file in the repository and the value is a JSON object with the following keys:

• sha256: the expected sha256 hash of the file. Required unless integrity is used.
• integrity: the expected SRI value of the file. Required unless sha256 is used.
• remote_path: name of the object within the bucket. If not set, the local path is used.

Targets in the main repository can depend on this target if the following lines are added to MODULE.bazel:

s3_bucket = use_extension("@rules_s3//s3:extensions.bzl", "s3_bucket")
s3_bucket.from_file(
    name = "trainingdata",
    bucket = "my_org_assets",
    lockfile = "@//:s3_lock.json",
)

Then targets would specify labels like @trainingdata//:trainingdata/model/very_large.bin as a dependency.

TAG CLASSES

Attributes

• Credential helper not found

  WARNING: Error retrieving auth headers, continuing without: Failed to get credentials for 'https://malte-s3-bazel-test.s3.amazonaws.com/hello_world' from helper 'tools/credential-helper': Cannot run program "tools/credential-helper" (in directory "..."): error=2, No such file or directory
  

  You need to install a credential helper (either tweag-credential-helper, or your own) as explained above.

• HTTP 401 or 403 error codes

  ERROR: Target parsing failed due to unexpected exception: java.io.IOException: Error downloading [https://s3.amazonaws.com/...] to ...: GET returned 403 Forbidden
  

  Follow the setup instructions of your credential helper (tweag-credential-helper docs).

• Checksum mismatch (empty file downloaded)

  Error in wait: com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException: Checksum was e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 but wanted <actual>
  

  Check if you are using --experimental_remote_downloader. If you do, the remote cache may drop your auth header and silently give you empty files instead. One workaround is setting --experimental_remote_downloader_local_fallback in .bazelrc.

rules_s3 is based on rules_gcs, which was initially developed by IMAX. Both projects are maintained by Tweag.