rules_mayhem

Run Mayhem from your Bazel infrastructure.

To include

You can add the following snippet:

## MODULE.bazel
bazel_dep(name = "rules_mayhem", version = "0.8.3")

rules_mayhem_extension = use_extension("@rules_mayhem//mayhem:extensions.bzl", "rules_mayhem_extension")
use_repo(rules_mayhem_extension, "bazel_skylib", "mayhem_cli_linux", "mayhem_cli_windows", "platforms", "yq_cli_linux", "yq_cli_windows")

## WORKSPACE
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "rules_mayhem",
    strip_prefix = "rules_mayhem",
    urls = ["https://github.com/ForAllSecure/rules_mayhem/releases/download/0.8.3/rules_mayhem-0.8.3.tar.gz"],
    sha256 = "44be6bea9c9462144542331531d18f848e89e6ed69561a52a4d761ffbe10d2e0",
)

load("@rules_mayhem//mayhem:repositories.bzl", "rules_mayhem_repositories")
rules_mayhem_repositories()

Note: Please see the latest release notes for instructions on how to include the latest release of rules_mayhem into your environment.

Pre-requisites

You'll need to modify your .bazelrc to use --spawn-strategy=standalone.

# Enable bzlmod
common --enable_bzlmod

# Spawn strategy - if this is not set, bazel tries to reference files that don't exist
build --spawn_strategy=standalone

Windows settings

If you're running Windows, you'll also need to add the following:

# Enable platform-specific features
common --enable_platform_specific_config=true

# Windows settings
startup --windows_enable_symlinks
startup --output_user_root=C:/tmp  # see https://bazel.build/versions/6.3.0/configure/windows#long-path-issues
common:windows --enable_runfiles

Logging into Mayhem

To use Mayhem, you need to generate a token for your account. You can do this by logging into the Mayhem web interface and navigating to Account Settings -> API Tokens.

When logging in via the CLI, Mayhem will create a configuration file under ~/.config/mayhem/mayhem. This is the default location - you can change it by setting the XDG_CONFIG_HOME environment variable to a different directory.

With Bazel, we simulate the login process by creating a config file located in the XDG_CONFIG_HOME/mayhem directory. You can either pass this value to the bazel build command line directly, like so:

Create a config file with the following content:

[DEFAULT]
url = https://<your_mayhem_instance>
token = AT1.<your_token_here>

By default, this file is created at ~/.config/mayhem/mayhem when you log in via the CLI, but you can choose a different location if you prefer.

Export the MAYHEM_URL and XDG_CONFIG_HOME environment variables in your shell environment.

On Linux:

export MAYHEM_URL="https://<your_mayhem_instance>"
export XDG_CONFIG_HOME="$HOME/.config"

On Windows:

set MAYHEM_URL=https://<your_mayhem_instance>
set XDG_CONFIG_HOME=%USERPROFILE%\.config

Run bazel build with the --action_env flags to specify the Mayhem URL and configuration file location:

bazel build --action_env=MAYHEM_URL --action_env=XDG_CONFIG_HOME //examples:run_mayhemit

Instead of adding these flags to every bazel build command, you can also add them to your .bazelrc file:

# .bazelrc
build --action_env=MAYHEM_URL
build --action_env=XDG_CONFIG_HOME

The remainder of this document assumes that you have set up your Mayhem configuration in your .bazelrc, and omits the --action_env flags from the bazel build commands for brevity.

To build a Mayhemfile

Create a BUILD file:

load("//mayhem:mayhem.bzl", "mayhem_init", "mayhem_run", "mayhem_package")

# Generates a minimal Mayhemfile
mayhem_init(
    name = "factor_mayhemfile",
    project = "bazel-rules",
    target = "factor",
    cmd = "/bin/factor",
    image = "photon:latest",
)

Run bazel build

$ bazel build //examples:factor_mayhemfile

INFO: Analyzed target //examples:factor_mayhemfile (0 packages loaded, 1 target configured).
INFO: Found 1 target...
Target //examples:factor_mayhemfile up-to-date:
  bazel-bin/examples/factor_mayhemfile.mayhemfile
INFO: Elapsed time: 0.128s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action

This should produce a valid Mayhemfile:

$ cat bazel-out/k8-fastbuild/bin/examples/factor_mayhemfile.mayhemfile

# Mayhem by https://forallsecure.com
# Mayhemfile: configuration file for testing your target with Mayhem
# Format: YAML 1.1

# Project name that the target belongs to
project: bazel-rules

# Target name (should be unique within the project)
target: factor

# Base image to run the binary in.
image: photon:latest

# Turns on extra test case processing (completing a run will take longer)
advanced_triage: false

# List of commands used to test the target
cmds:

  # Command used to start the target, "@@" is the input file
  # (when "@@" is omitted Mayhem defaults to stdin inputs)
  - cmd: /bin/factor
    env: {}
extensions: {}

    ## Uncomment to change default dir (/) from which the target is invoked
    #cwd: /

    ## If this is a network target, uncomment the block below and remove
    ## the @@ input file reference in the cmd (you can either test network or
    ## file inputs, not both).
    #network:
    ## Use "127.0.0.1" instead of "localhost" below if you want to test only
    ## for IPv4. For IPv6, use "[::1]". By leaving as "localhost", Mayhem will
    ## attempt to autodetect the one used by the target.
    #  url: tcp://localhost:8080  # protocol, host and port to analyze
    #  client: False           # target is a client-side program
    #  timeout: 2.0               # max seconds for sending data

    ## Max test case length (in bytes) to be taken into account. Test cases over
    ## that length will be truncated. Be very careful about increasing this
    ## limit as it can severely affect your fuzzer performance.
    # max_length: 8192

To run a mayhemfile

Use mayhem_run

mayhem_run(
    name = "run_factor",
    target_path = ".",
    mayhemfile = ":factor_mayhemfile",
)

Then build:

bazel build //examples:run_factor
INFO: Analyzed target //examples:run_factor (0 packages loaded, 0 targets configured).
INFO: From Starting Mayhem run from 'examples':
git version 2.46.0 found
WARNING A default branch has not been configured for this target. Job differences will not be available in this report
Run started: forallsecure-demo/bazel-rules/factor/9
Run URL: https://app.mayhem.security:443/forallsecure-demo/bazel-rules/factor/9
INFO: Found 1 target...
Target //examples:run_factor up-to-date:
  bazel-bin/examples/run_factor.mayhem_out
INFO: Elapsed time: 22.883s, Critical Path: 22.75s
INFO: 2 processes: 1 internal, 1 local.
INFO: Build completed successfully, 2 total actions

Packaging binaries

rules_mayhem also supports the package workflow. In your BUILD file:

cc_binary(
    name = "mayhemit",
    srcs = ["mayhemit.c"],
)

# `mayhem_package` automatically generates a Mayhemfile; a separate mayhemfile rule is not needed
mayhem_package(
    name = "package_mayhemit",
    binary = ":mayhemit",
)

mayhem_run(
    name = "run_mayhemit",
    image = "ubuntu:latest", # or whatever base image your binary should run on; by default, we use debian-buster
    target_path = ":package_mayhemit"
)

Then build and run:

bazel build //examples:mayhemit
INFO: Analyzed target //examples:mayhemit (1 packages loaded, 7 targets configured).
INFO: Found 1 target...
Target //examples:mayhemit up-to-date:
  bazel-bin/examples/mayhemit
INFO: Elapsed time: 0.347s, Critical Path: 0.09s
INFO: 7 processes: 5 internal, 2 local.
INFO: Build completed successfully, 7 total actions


bazel build //examples:package_mayhemit
INFO: Analyzed target //examples:package_mayhemit (0 packages loaded, 1 target configured).
INFO: From Packaging target examples/mayhemit to 'bazel-out/k8-fastbuild/bin/examples/mayhemit-pkg'...:
Generating default configuration under: bazel-out/k8-fastbuild/bin/examples/mayhemit-pkg/Mayhemfile
Packaging complete.
To upload the package do: `mayhem run bazel-out/k8-fastbuild/bin/examples/mayhemit-pkg`.
Before uploading, you may wish to edit the config file at 'bazel-out/k8-fastbuild/bin/examples/mayhemit-pkg/Mayhemfile'.
bazel-out/k8-fastbuild/bin/examples/mayhemit-pkg
INFO: Found 1 target...
Target //examples:package_mayhemit up-to-date:
  bazel-bin/examples/mayhemit-pkg
INFO: Elapsed time: 3.488s, Critical Path: 3.43s
INFO: 2 processes: 1 internal, 1 local.
INFO: Build completed successfully, 2 total actions


bazel build //examples:run_mayhemit
INFO: Analyzed target //examples:run_mayhemit (0 packages loaded, 1 target configured).
INFO: From Starting Mayhem run from 'bazel-out/k8-fastbuild/bin/examples/mayhemit-pkg':
git version 2.46.0 found
WARNING A default branch has not been configured for this target. Job differences will not be available in this report
/tmp/tmp1rd5ih1v/root.tgz   0% |                     | ETA:  --:--:--   0.0 s/B
/tmp/tmp1rd5ih1v/root.tgz 100% |###################| Time:  0:00:00   9.3 KiB/s
Run started: forallsecure-demo/bazel-rules/mayhemit/11
Run URL: https://app.mayhem.security:443/forallsecure-demo/bazel-rules/mayhemit/11
INFO: Found 1 target...
Target //examples:run_mayhemit up-to-date:
  bazel-bin/examples/run_mayhemit.mayhem_out
INFO: Elapsed time: 6.567s, Critical Path: 6.50s
INFO: 3 processes: 2 internal, 1 local.
INFO: Build completed successfully, 3 total actions

Waiting and run reports

You can wait on a Mayhem run by adding the wait = True parameter to mayhem_run():

mayhem_run(
    name = "run_mayhemit",
    image = "ubuntu:latest",
    target_path = ":package_mayhemit",
    wait = True,
)

You can tell Mayhem to return exit code 1 if defects are found. You can also specify to return either a JUnit or SARIF report:

mayhem_run(
    name = "run_mayhemit",
    image = "ubuntu:latest",
    target_path = ":package_mayhemit",
    wait = True,
    fail_on_defects = True,
    sarif = mayhemit.sarif,
    junit = mayhemit.junit,
)

Then build and run:

bazel build //examples:run_mayhemit
INFO: Analyzed target //examples:run_mayhemit (56 packages loaded, 239 targets configured).
INFO: From Packaging target examples/mayhemit to 'bazel-out/k8-fastbuild/bin/examples/mayhemit-pkg'...:
Packaging target: examples/mayhemit
Packaging dependency: examples/mayhemit -> bazel-out/k8-fastbuild/bin/examples/mayhemit
Generating default configuration under: bazel-out/k8-fastbuild/bin/examples/mayhemit-pkg/Mayhemfile
Packaging complete.
To upload the package do: `mayhem run bazel-out/k8-fastbuild/bin/examples/mayhemit-pkg`.
Before uploading, you may wish to edit the config file at 'bazel-out/k8-fastbuild/bin/examples/mayhemit-pkg/Mayhemfile'.
bazel-out/k8-fastbuild/bin/examples/mayhemit-pkg
INFO: From Starting Mayhem run...:
git version 2.46.1 found
WARNING A default branch has not been configured for this target. Job differences will not be available in this report
/tmp/tmpz6srnoug/root.tgz   0% |                     | ETA:  --:--:--   0.0 s/B
/tmp/tmpz6srnoug/root.tgz 100% |###################| Time:  0:00:00   6.1 KiB/s
Run started: bazel-rules/mayhemit/9
Run URL: https://app.mayhem.security:443/forallsecure-demo/bazel-rules/mayhemit/9
INFO: From Waiting for Mayhem run to complete...:
Generating JUnit XML report: mayhemit_junit.xml
  Processing run forallsecure-demo/bazel-rules/mayhemit/9
    Downloading testcase reports...
    Processing 13 testcase reports...
Writing JUnit report to mayhemit_junit.xml
Generating SARIF JSON report: mayhemit_sarif.json
  Processing run forallsecure-demo/bazel-rules/mayhemit/9
    Downloading defect reports...
    Processing 1 defect reports...
Writing SARIF report to mayhemit_sarif.json
ERROR Run completed, defects found
Suggestion: Please review the generated testcase reports and correct the identified defects.
Statistics for forallsecure-demo/bazel-rules/mayhemit/9
    Status: coverage_analysis:running, dynamic_analysis:completed, regression_testing:completed, static_analysis:completed
    Run started: Fri Oct 11 14:46:19 2024 -0400
    Time elapsed: 0:01:14
    Tests performed: 24610
    Test reports: 13
    Crash reports: 1
    Defects: 1
INFO: Found 1 target...
Target //examples:run_mayhemit up-to-date:
  bazel-bin/examples/run_mayhemit.out
  bazel-bin/examples/run_mayhemit.wait.out
INFO: Elapsed time: 87.843s, Critical Path: 87.58s
INFO: 9 processes: 4 internal, 5 local.
INFO: Build completed successfully, 9 total actions

Downloading testsuite and coverage information

rules_mayhem supports downloading testsuite and coverage information. In your BUILD file:

mayhem_download(
    name = "download_mayhemit",
    owner = "forallsecure-demo",
    project = "bazel-rules",
    target = "mayhemit",
    output_dir = "mayhemit_output",
)

Then build and run:

bazel build //examples:download_mayhemit
INFO: Analyzed target //examples:download_mayhemit (6 packages loaded, 15 targets configured).
INFO: From Downloading Mayhem artifacts for download_mayhemit...:
Downloading target data from run forallsecure-demo/bazel-rules/mayhemit/5...
Downloaded: Mayhemfile.
Downloaded: root.tgz.
Downloaded: coverage.tgz.
Downloading testsuite.tar:   0.0 B |#         | Elapsed Time: 0:00:00   0.0 s/B
Downloading testsuite.tar:   8.0 KiB |     #| Elapsed Time: 0:00:01   4.4 KiB/s
Downloading testsuite.tar:  33.0 KiB |     #| Elapsed Time: 0:00:01  17.7 KiB/s
Extracting tests 0 of 23 |                                     | ETA:  --:--:--
Extracting tests 23 of 23 |####################################| Time:  0:00:00
Target downloaded at: 'bazel-out/k8-fastbuild/bin/examples/mayhemit_output'.
INFO: Found 1 target...
Target //examples:download_mayhemit up-to-date:
  bazel-bin/examples/mayhemit_output
INFO: Elapsed time: 6.615s, Critical Path: 6.43s
INFO: 2 processes: 1 internal, 1 local.
INFO: Build completed successfully, 2 total actions

To Do

If you want to help any of the above, feel free to open a PR!