Creating Cloud Native Buildpacks from Classic Buildpacks

Last updated December 03, 2024

Heroku’s Fir-generation apps use a new generation of buildpacks: Cloud Native Buildpacks (CNB). CNBs are an open standard and specification that work well with industry standard tooling. Cloud Native Buildpacks provide some of the same features, functionality, and great developer experience as classic buildpacks, with several improvements including the ability to run locally, robust caching, improved composability, and more. While Heroku’s classic buildpacks and Cloud Native Buildpacks are similar in design and spirit, they aren’t directly compatible. As a result, classic buildpacks are not supported for use on Fir-generation apps.

Developers commonly use community-authored classic buildpacks to customize their builds on Cedar-generation apps. To apply similar customization to a build in Fir, you must use an equivalent CNB. There are a number of classic buildpacks available that don’t yet have an equivalent Cloud Native Buildpack. This article aims to help fill that gap and illustrate the process of creating a CNB for usage on Heroku Fir from an existing classic buildpack.

Example

heroku-buildpack-yq is a simple classic buildpack written in bash that downloads and installs yq. yq is binary that parses json, yaml, and toml files. At the time of writing, there is no equivalent Cloud Native Buildpack.

The bin/detect for this buildpack always passes, so there isn’t any conditional logic:

#!/usr/bin/env bash
# bin/detect <build-dir>

echo 'yq'
exit 0

The bin/compile script for this buildpack has a few features:

  • yq version selection via the YQ_VERSION environment variable
  • Caching of the yq binary, so that it’s not downloaded from the internet on every build.
  • Adding the yq binary to $PATH at so it’s available on the command line — at both runtime and for later buildpacks
#!/usr/bin/env bash
# bin/compile <build-dir> <cache-dir> <env-dir>

set -euo pipefail

BUILD_DIR=$1
CACHE_DIR=$2
ENV_DIR=$3
BUILDPACK_DIR=$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}")")" && pwd)

YQ_BUILD_DIR="$BUILD_DIR/.yq"
YQ_CACHE_DIR="$CACHE_DIR/.yq"

YQ_VERSION="4.44.3"
if [ -f "$ENV_DIR/YQ_VERSION" ]; then
  YQ_VERSION=$(cat "$ENV_DIR/YQ_VERSION")
fi

YQ_URL="https://github.com/mikefarah/yq/releases/download/v${YQ_VERSION}/yq_linux_amd64.tar.gz"

if [[ -f "$YQ_CACHE_DIR/yq-version" ]] && grep -q "$YQ_VERSION" "$YQ_CACHE_DIR/yq-version" ; then
  echo "Using yq $YQ_VERSION from cache"
  cp -R "${YQ_CACHE_DIR}" "${YQ_BUILD_DIR}"
else
  echo "Downloading yq $YQ_VERSION from $YQ_URL"
  mkdir -p "${YQ_BUILD_DIR}"
  curl -sSf --location --retry 3 --retry-connrefused --connect-timeout 10 "${YQ_URL}" | tar -zx -C "${YQ_BUILD_DIR}"
  mv "${YQ_BUILD_DIR}/yq_linux_amd64" "${YQ_BUILD_DIR}/yq"
  printf "%s" "$YQ_VERSION" > "$YQ_BUILD_DIR/yq-version"
  rm -rf "${YQ_CACHE_DIR}"
  cp -R "${YQ_BUILD_DIR}" "${YQ_CACHE_DIR}"
fi

mkdir -p "$BUILD_DIR/.profile.d"
echo "export PATH=\"$HOME/.yq:\$PATH\"" >> "$BUILD_DIR/.profile.d/heroku-buildpack-yq.sh"
echo "export PATH=\"$BUILD_DIR/.yq:\$PATH\"" >> "$BUILDPACK_DIR/export"

To produce an equivalent yq CNB, you must take a number of steps:

This article is only an example, and not an exhaustive guide that covers all possible buildpack use cases. Heroku highly suggests reading the official CNB documentation, like the tutorial or specification, to gain a better understanding of CNB features, capabilities, and usage.

Install pack

The Cloud Native Buildpacks team produces a helpful CLI to interact with CNBs called pack. We’ll need this tool to continue. Installation instructions are available here.

Create a CNB

To create a new CNB, use pack. Build a basic scaffold with pack buildpack new.

$ pack buildpack new heroku-examples/yq --path buildpack-yq
$ cd buildpack-yq

The generated CNB includes a buildpack.toml, and two Bash scripts:bin/detect and bin/build. You can write CNBs in any language, but we’ll use Bash for this CNB to simplify the conversion.

Update buildpack.toml

The basic buildpack.toml generated by pack buildpack new isn’t quite what we’ll need. We can make the following changes:

  • Update the api version to 0.10, the latest CNB buildpack API version supported by Heroku Fir.
  • Add the [[stacks]] table with id = "*" to improve compatibility with older pack versions.
  • Update [[targets]] to include both linux/arm64 and linux/amd64. Heroku Fir apps run on linux/arm64, so arm64 is mandatory for use on Heroku Fir. The amd64 entry is optional, but it can be helpful for testing the CNB locally.

The new buildpack.toml looks like this:

# buildpack.toml
api = "0.10"

[buildpack]
id = "heroku-examples/yq"
version = "0.0.1"
name = "yq"

[[stacks]]
id = "*"

[[targets]]
os = "linux"
arch = "arm64"

[[targets]]
os = "linux"
arch = "amd64"

Copy bin/detect

Because the bin/detect for the yq Heroku Classic buildpack always passes with exit 0, it’s completely compatible with the Cloud Native Buildpacks specification. We can copy bin/detect from the yq Heroku Classic Buildpack to our yq Cloud Native Buildpack.

#!/usr/bin/env bash
# bin/detect

echo 'yq'
exit 0

Copy bin/compile to bin/build

In the Cloud Native Buildpack specification, bin/build is the entrypoint to start the build process. This entrypoint is different from the Heroku’s classic Buildpack API, which uses bin/compile. Copy bin/compile from the classic buildpack to bin/build in our CNB:

#!/usr/bin/env bash
# bin/build

set -euo pipefail

BUILD_DIR=$1
CACHE_DIR=$2
ENV_DIR=$3
BUILDPACK_DIR=$(cd "$(dirname "$(dirname "${BASH_SOURCE[0]}")")" && pwd)

YQ_BUILD_DIR="$BUILD_DIR/.yq"
YQ_CACHE_DIR="$CACHE_DIR/.yq"

YQ_VERSION="4.44.3"
if [ -f "$ENV_DIR/YQ_VERSION" ]; then
  YQ_VERSION=$(cat "$ENV_DIR/YQ_VERSION")
fi

YQ_URL="https://github.com/mikefarah/yq/releases/download/v${YQ_VERSION}/yq_linux_amd64.tar.gz"

if [[ -f "$YQ_CACHE_DIR/yq-version" ]] && grep -q "$YQ_VERSION" "$YQ_CACHE_DIR/yq-version" ; then
  echo "Using yq $YQ_VERSION from cache"
  cp -R "${YQ_CACHE_DIR}" "${YQ_BUILD_DIR}"
else
  echo "Downloading yq $YQ_VERSION from $YQ_URL"
  mkdir -p "${YQ_BUILD_DIR}"
  curl -sSf --location --retry 3 --retry-connrefused --connect-timeout 10 "${YQ_URL}" | tar -zx -C "${YQ_BUILD_DIR}"
  mv "${YQ_BUILD_DIR}/yq_linux_amd64" "${YQ_BUILD_DIR}/yq"
  printf "%s" "$YQ_VERSION" > "$YQ_BUILD_DIR/yq-version"
  rm -rf "${YQ_CACHE_DIR}"
  cp -R "${YQ_BUILD_DIR}" "${YQ_CACHE_DIR}"
fi

mkdir -p "$BUILD_DIR/.profile.d"
echo "export PATH=\"$HOME/.yq:\$PATH\"" >> "$BUILD_DIR/.profile.d/heroku-buildpack-yq.sh"
echo "export PATH=\"$BUILD_DIR/.yq:\$PATH\"" >> "$BUILDPACK_DIR/export"

Remove Classic Variables

Some of the directory variables in the classic buildpack aren’t relevant in a CNB context. You may remove the$BUILD_DIR, $CACHE_DIR, $ENV_DIR, $BUILDPACK_DIR, and $YQ_CACHE_DIR declarations, so that bin/build looks like this:

#!/usr/bin/env bash
# bin/build

set -euo pipefail

YQ_BUILD_DIR="$BUILD_DIR/.yq"

YQ_VERSION="4.44.3"
if [ -f "$ENV_DIR/YQ_VERSION" ]; then
  YQ_VERSION=$(cat "$ENV_DIR/YQ_VERSION")
fi

YQ_URL="https://github.com/mikefarah/yq/releases/download/v${YQ_VERSION}/yq_linux_amd64.tar.gz"

if [[ -f "$YQ_CACHE_DIR/yq-version" ]] && grep -q "$YQ_VERSION" "$YQ_CACHE_DIR/yq-version" ; then
  echo "Using yq $YQ_VERSION from cache"
  cp -R "${YQ_CACHE_DIR}" "${YQ_BUILD_DIR}"
else
  echo "Downloading yq $YQ_VERSION from $YQ_URL"
  mkdir -p "${YQ_BUILD_DIR}"
  curl -sSf --location --retry 3 --retry-connrefused --connect-timeout 10 "${YQ_URL}" | tar -zx -C "${YQ_BUILD_DIR}"
  mv "${YQ_BUILD_DIR}/yq_linux_amd64" "${YQ_BUILD_DIR}/yq"
  printf "%s" "$YQ_VERSION" > "$YQ_BUILD_DIR/yq-version"
  rm -rf "${YQ_CACHE_DIR}"
  cp -R "${YQ_BUILD_DIR}" "${YQ_CACHE_DIR}"
fi

mkdir -p "$BUILD_DIR/.profile.d"
echo "export PATH=\"$HOME/.yq:\$PATH\"" >> "$BUILD_DIR/.profile.d/heroku-buildpack-yq.sh"
echo "export PATH=\"$BUILD_DIR/.yq:\$PATH\"" >> "$BUILDPACK_DIR/export"

Change $YQ_BUILD_DIR

In classic buildpacks, buildpack artifacts are typically stored in a nested build directory, in this example, YQ_BUILD_DIR="$BUILD_DIR/.yq". Artifact storage is a bit different in Cloud Native Buildpacks. Buildpack files are stored instead in “layers“, as nested directories in /layers/.

Update the $YQ_BUILD_DIR assignment to YQ_BUILD_DIR="${CNB_LAYERS_DIR}/yq". This will resolve to /layers/heroku-examples_yq/yq/. The bin/build script now looks like this:

#!/usr/bin/env bash
# bin/build

set -euo pipefail

YQ_BUILD_DIR="${CNB_LAYERS_DIR}/yq"

YQ_VERSION="4.44.3"
if [ -f "$ENV_DIR/YQ_VERSION" ]; then
  YQ_VERSION=$(cat "$ENV_DIR/YQ_VERSION")
fi

YQ_URL="https://github.com/mikefarah/yq/releases/download/v${YQ_VERSION}/yq_linux_amd64.tar.gz"

if [[ -f "$YQ_CACHE_DIR/yq-version" ]] && grep -q "$YQ_VERSION" "$YQ_CACHE_DIR/yq-version" ; then
  echo "Using yq $YQ_VERSION from cache"
  cp -R "${YQ_CACHE_DIR}" "${YQ_BUILD_DIR}"
else
  echo "Downloading yq $YQ_VERSION from $YQ_URL"
  mkdir -p "${YQ_BUILD_DIR}"
  curl -sSf --location --retry 3 --retry-connrefused --connect-timeout 10 "${YQ_URL}" | tar -zx -C "${YQ_BUILD_DIR}"
  mv "${YQ_BUILD_DIR}/yq_linux_amd64" "${YQ_BUILD_DIR}/yq"
  printf "%s" "$YQ_VERSION" > "$YQ_BUILD_DIR/yq-version"
  rm -rf "${YQ_CACHE_DIR}"
  cp -R "${YQ_BUILD_DIR}" "${YQ_CACHE_DIR}"
fi

mkdir -p "$BUILD_DIR/.profile.d"
echo "export PATH=\"$HOME/.yq:\$PATH\"" >> "$BUILD_DIR/.profile.d/heroku-buildpack-yq.sh"
echo "export PATH=\"$BUILD_DIR/.yq:\$PATH\"" >> "$BUILDPACK_DIR/export"

Reference Environment Variables Directly

With classic buildpacks, environment variables are stored as files in $ENV_DIR, but things are simpler with CNBs. You can read environment variables directly from the environment, which greatly simplifies the yq version detection logic. Change this block:

YQ_VERSION="4.44.3"
if [ -f "$ENV_DIR/YQ_VERSION" ]; then
  YQ_VERSION=$(cat "$ENV_DIR/YQ_VERSION")
fi

To this:

YQ_VERSION="${YQ_VERSION:-4.44.3}"-

Remove $PATH Manipulations

The Cloud Native Buildpacks specification automatically adds bin directories in layer directories to the $PATH environment variable. Since we’re installing the yq binary to ${CNB_LAYERS_DIR/yq/bin/, CNB implementations put yq on the $PATH without any custom code. The $PATH manipulations and .profile.d lines from our Heroku Classic Buildpack aren’t required in our CNB. Removing those lines results in a bin/build that looks like this:

#!/usr/bin/env bash
# bin/build

set -euo pipefail

YQ_BUILD_DIR="${CNB_LAYERS_DIR}/yq"

YQ_VERSION="${YQ_VERSION:-4.44.3}"

YQ_URL="https://github.com/mikefarah/yq/releases/download/v${YQ_VERSION}/yq_linux_amd64.tar.gz"

if [[ -f "$YQ_CACHE_DIR/yq-version" ]] && grep -q "$YQ_VERSION" "$YQ_CACHE_DIR/yq-version" ; then
  echo "Using yq $YQ_VERSION from cache"
  cp -R "${YQ_CACHE_DIR}" "${YQ_BUILD_DIR}"
else
  echo "Downloading yq $YQ_VERSION from $YQ_URL"
  mkdir -p "${YQ_BUILD_DIR}"
  curl -sSf --location --retry 3 --retry-connrefused --connect-timeout 10 "${YQ_URL}" | tar -zx -C "${YQ_BUILD_DIR}"
  mv "${YQ_BUILD_DIR}/yq_linux_amd64" "${YQ_BUILD_DIR}/yq"
  printf "%s" "$YQ_VERSION" > "$YQ_BUILD_DIR/yq-version"
  rm -rf "${YQ_CACHE_DIR}"
  cp -R "${YQ_BUILD_DIR}" "${YQ_CACHE_DIR}"
fi

Remove Cache Copies and Exit Early

Caching works quite a bit differently with Cloud Native Buildpacks. Cached files don’t need to be copied around to and from a separate cache directory. Instead, any cached files in cached layers are available during the build. Therefore, the following lines can be removed:

  • cp -R "${YQ_CACHE_DIR}" "${YQ_BUILD_DIR}"
  • rm -rf "${YQ_CACHE_DIR}"
  • cp -R "${YQ_BUILD_DIR}" "${YQ_CACHE_DIR}"

Additionally, you can change the if conditional to exit early when cached files are detected. Afterwards, the bin/build looks like this:

#!/usr/bin/env bash
# bin/build

set -euo pipefail

YQ_BUILD_DIR="${CNB_LAYERS_DIR}/yq"

YQ_VERSION="${YQ_VERSION:-4.44.3}"

YQ_URL="https://github.com/mikefarah/yq/releases/download/v${YQ_VERSION}/yq_linux_amd64.tar.gz"

if [[ -f "$YQ_CACHE_DIR/yq-version" ]] && grep -q "$YQ_VERSION" "$YQ_CACHE_DIR/yq-version" ; then
  echo "Using yq $YQ_VERSION from cache"
  exit 0
fi

echo "Downloading yq $YQ_VERSION from $YQ_URL"
mkdir -p "${YQ_BUILD_DIR}"
curl -sSf --location --retry 3 --retry-connrefused --connect-timeout 10 "${YQ_URL}" | tar -zx -C "${YQ_BUILD_DIR}"
mv "${YQ_BUILD_DIR}/yq_linux_amd64" "${YQ_BUILD_DIR}/yq"
printf "%s" "$YQ_VERSION" > "$YQ_BUILD_DIR/yq-version"

Set CNB Layer Settings

CNB layers have 3 options that determine how and when they are available. You need to set these options in a .toml file. Add the following to the end of bin/build:

cat > "${YQ_BUILD_DIR}.toml" << EOL
[types]
build = true
cache = true
launch = true
EOL

The build = true option ensures that the layer and yq binary are available to buildpacks that run after this one. The cache = true option ensures that the contents of the layer persist both to the cache and get restored from the cache for every build. The launch = true option ensures yq is available when the resulting image launches.

Add Support for Multiple Chip Architectures

Heroku Fir apps run on arm64 instances, while Heroku classic runs on amd64. Classic buildpacks only supported the latter. Add amd64 by updating the $YQ_URL to be conditional based on architecture, which you can read from $CNB_TARGET_ARCH. The updated $YQ_URL assignment looks like this:

YQ_URL="https://github.com/mikefarah/yq/releases/download/v${YQ_VERSION}/yq_linux_${CNB_TARGET_ARCH}.tar.gz"

Testing Locally

At this point, bin/build is in a working state and looks like this:

#!/usr/bin/env bash
set -euo pipefail

YQ_BUILD_DIR="${CNB_LAYERS_DIR}/yq"
YQ_VERSION="${YQ_VERSION:-4.44.3}"
YQ_URL="https://github.com/mikefarah/yq/releases/download/v${YQ_VERSION}/yq_linux_${CNB_TARGET_ARCH}.tar.gz"

if [[ -f "$YQ_BUILD_DIR/yq-version" ]] && grep -q "$YQ_VERSION" "$YQ_BUILD_DIR/yq-version" ; then
  echo "Using yq $YQ_VERSION from cache"
  exit 0;
fi

echo "Downloading yq $YQ_VERSION from $YQ_URL"
rm -rf "${YQ_BUILD_DIR}"
mkdir -p "${YQ_BUILD_DIR}"
curl -sSf --location --retry 3 --retry-connrefused --connect-timeout 10 "${YQ_URL}" | tar -zx -C "${YQ_BUILD_DIR}"
mkdir -p "${YQ_BUILD_DIR}/bin"
mv "${YQ_BUILD_DIR}/yq_linux_${CNB_TARGET_ARCH}" "${YQ_BUILD_DIR}/bin/yq"
printf "%s" "$YQ_VERSION" > "${YQ_BUILD_DIR}/yq-version"
cat > "${YQ_BUILD_DIR}.toml" << EOL
[types]
build = true
cache = true
launch = true
EOL

Try it locally with the pack command:

$ pack build yq-cnb-test --buildpack ./buildpack-yq --builder heroku/builder:24

24: Pulling from heroku/builder
Digest: sha256:2324afe304202e81d452bb203eb4edcc7fed682840d0ec3c82f11fdba96cc199
Status: Image is up to date for heroku/builder:24
24: Pulling from heroku/heroku
Digest: sha256:613aa12fc84c16054be6a9501578e06948d76363e2921e4326447fd0e5a770cf
Status: Image is up to date for heroku/heroku:24
===> ANALYZING
Image with name "yq-cnb-test" not found
===> DETECTING
heroku-examples/yq 0.0.1
===> RESTORING
===> BUILDING
Downloading yq 4.44.3 from https://github.com/mikefarah/yq/releases/download/v4.44.3/yq_linux_arm64.tar.gz
===> EXPORTING
Adding layer 'heroku-examples/yq:yq'
Adding layer 'buildpacksio/lifecycle:launch.sbom'
Added 1/1 app layer(s)
Adding layer 'buildpacksio/lifecycle:launcher'
Adding layer 'buildpacksio/lifecycle:config'
Adding label 'io.buildpacks.lifecycle.metadata'
Adding label 'io.buildpacks.build.metadata'
Adding label 'io.buildpacks.project.metadata'
no default process type
Saving yq-cnb-test...
*** Images (96fff73ff596):
      yq-cnb-test
Adding cache layer 'heroku-examples/yq:yq'
Successfully built image yq-cnb-test

This command builds an image and stores the result in the local Docker daemon. Inspect the resulting image and check that yq works:

$ docker run -it --rm yq-cnb-test yq --version
yq (https://github.com/mikefarah/yq/) version v4.44.3

Packaging and Publishing

Package the buildpack for wider use with the pack buildpack package command.

$ pack buildpack package buildpack-yq --format file
Successfully created package buildpack-yq.cnb and saved to file

The CNB package must be uploaded somewhere publicly accessible. You can find the example CNB file on GitHub here.

Testing on Heroku Fir

After verifying locally and uploading, test the CNB on the Heroku Fir platform. Create a new app in a new directory.

$ mkdir yq-cnb-test-app
$ cd yq-cnb-test-app
$ git init
$ heroku apps:create yq-cnb-test --space <your-fir-space>

Then, set a custom buildpack by creating a project.toml with a reference to the CNB’s url.

# project.toml
[_]
schema-version = "0.2"

[[io.buildpacks.group]]
id = "heroku-examples/yq"
uri = "https://github.com/heroku-examples/buildpack-yq/releases/download/v0.0.1/buildpack-yq.cnb"

Commit the change and deploy the Heroku app as normal:

$ git add -A
$ git commit -m "Add custom yq buildpack"
$ git push heroku main
Enumerating objects: 3481, done.
Counting objects: 100% (3481/3481), done.
Delta compression using up to 14 threads
Compressing objects: 100% (1954/1954), done.
Writing objects: 100% (3481/3481), 7.91 MiB | 8.61 MiB/s, done.
Total 3481 (delta 1329), reused 3477 (delta 1328), pack-reused 0 (from 0)
remote: Resolving deltas: 100% (1329/1329), done.
remote: Updated 1300 paths from 5dfc83a
remote: Compressing source files... done.
remote: Building source:
remote: Waiting on build...
remote: Waiting on build... (elapsed: 19s)
remote: Extracting source
remote: Downloading buildpack https://github.com/heroku-examples/buildpack-yq/releases/download/v0.0.1/buildpack-yq.cnb
remote: Image with name "yq-cnb-test/builds" not found
remote: heroku-examples/yq 0.0.1
remote: Downloading yq 4.44.3 from https://github.com/mikefarah/yq/releases/download/v4.44.3/yq_linux_arm64.tar.gz
remote: Adding layer 'heroku-examples/yq:yq'
remote: Adding layer 'buildpacksio/lifecycle:launch.sbom'
remote: Added 1/1 app layer(s)
remote: Adding layer 'buildpacksio/lifecycle:launcher'
remote: Adding layer 'buildpacksio/lifecycle:config'
remote: Adding label 'io.buildpacks.lifecycle.metadata'
remote: Adding label 'io.buildpacks.build.metadata'
remote: Adding label 'io.buildpacks.project.metadata'
remote: no default process type
remote: Saving yq-cnb-test/builds...
remote: *** Images (sha256:97f83d6f1a1abf9a5973a8afb44f487e36303607e16aed091a0c264168daef34):
remote:       yq-cnb-test/builds:418d86c2-b29f-4a00-871a-e00db35743d6
remote: Adding cache layer 'heroku-examples/yq:yq'
remote: Uploading cache
remote: Launching...
remote: https://yq-cnb-test-97c732604476.herokuapp.com/ deployed to Heroku
remote: Verifying deploy... done.
To https://git.heroku.com/yq-cnb-test.git
 * [new branch]      main -> main

Next Steps

Now that the example CNB from this guide exists publicly and is available for use in any Heroku Fir app. However, this article covered only a small portion of the functionality and feature set of CNBs. For further reading on Cloud Native Buildpacks, see:

Feedback

Log in to submit feedback.