blog.jerryaldrichiii.com

Content for blog.jerryaldrichiii.com

10 KiB

Raw Blame History

layout	title	category	tags	summary
post	Using Policyfile Cookbooks	chef_infra	[chef cookbooks policyfiles]	Policy...what is it good for?

Policyfile? What is that?

A Policyfile (aka Policyfile.rb) is a file that contains information about a node's:

Default source for fetching cookbooks
Run list (or multiple run lists via named_run_list)
Cookbook dependencies and sources
Optional cookbook attributes

Here is an example Policyfile.rb:

name 'mycorp_audit'
default_source :supermarket
cookbook 'mycorp_audit', path: '.'
run_list 'mycorp_audit::default'

This file is a key component in Policyfile cookbooks.

What are Policyfile cookbooks and why should I use them?

Essentially, a Policyfile cookbook is a cookbook that uses a Policyfile.rb to combine the functions of Berkshelf, Environments, and Roles into a single artifact that can be promoted safely through the software development lifecycle.

Without using Policyfiles, special care is needed in order to use Chef safely. This is due to the shortcomings of runtime dependency solving.

Take for example the following scenario:

You have 3 cookbooks: cookbook_a, cookbook_b, and cookbook_c
cookbook_a is deployed and depends on cookbook_b with version ~>1.0.0
cookbook_c is created and depends on cookbook_b with version ~> 1.0.0
cookbook_c is tested/deployed (which deploys newest 1.X.X of cookbook_b)
cookbook_a runs and breaks due to a bug in cookbook_b version 1.1.1

Now, ideally, strict version pinning and adhering to Semantic Versioning would prevent any mishaps...but that is a lot to trust to place on the developer, the pipeline, and the upstream cookbook maintainer.

With Policyfile cookbooks, this is prevented because cookbooks are referenced by hash and not version at runtime. In order to run a new set of cookbooks a new Policyfile.lock must be deployed.

Policyfile.lock

The Policyfile.lock (aka Policyfile.lock.json) is generated from a Policyfile.rb and contains all the same information in it as the Policyfile.rb in addition to a few other key items needed for safely deploying cookbooks.

For the sake of this section of the article, the two most important sections of the Policyfile.lock are the revision_id which contains the hash of the entire lock file and the cookbook_locks section.

These two items together ensure that cookbooks and recipes are applied consistently regardless of the environment that Chef Infra is deployed/ran in.

Below is an example cookbook_locks section:

"audit": {
  "version": "7.5.0",
  "identifier": "27e6ea8e7bc82f5ba44b77834f5472c96da9c27a",
  "dotted_decimal_identifier": "11231419178928175.25794866915266388.126209453834874",
  "cache_key": "audit-7.5.0-supermarket.chef.io",
  "origin": "https://supermarket.chef.io:443/api/v1/cookbooks/audit/versions/7.5.0/download",
  "source_options": {
    "artifactserver": "https://supermarket.chef.io:443/api/v1/cookbooks/audit/versions/7.5.0/download",
    "version": "7.5.0"
  }
}

As you can see above, not only is useful information such as version and origin of the cookbook provided, but also referenced is the exact hash of the upstream cookbook that is being used.

In this case, the audit cookbook with hash 27e6ea8e7bc82f5ba44b77834f5472c96da9c27a

This is critical as it ensures that not only are we running the correct version of the cookbook, but also that the content of that cookbook has not changed since the Policyfile.lock was created.

Benefits over Berkshelf (i.e. runtime dependency solving)

Since the Policyfile.lock is computed prior to uploading to the Chef Infra Server there is no need to compute cookbook dependencies at runtime. This has several emergent benefits. Firstly, by moving this computation to the workstation (or pipeline), Chef Infra is saved from having to do the same computation(s) on each run on every node.

More importantly, since dependency resolution has already been completed and cookbooks are referenced via hash, it is impossible to unintentionally run a cookbook in production.

Deploying/Promoting Policyfiles

Policyfiles are deployed to the Chef Infra Server into what are known as Policy Groups. Conceptually, these Policy Groups take the place of Environments.

This is best illustrated with the output from chef show-policy:

mycorp_audit
=======
* dev:         98ed3a52a5
* staging:     98ed3a52a5
* production:  98ed3a52a5

We can see here that the mycorp_audit policy with the revision ID 98ed3a52a5 is currently deployed in the dev, staging, and production policy groups.

When bootstrapping a node, the policy_name and policy_group are specified, this ensures that the correct policy is applied during runtime.

Promoting Policyfiles through your environments (Policy Groups)

This is done like via chef push POLICY_GROUP POLICY_NAME.

See:

$ chef update # This updates the Policyfile.lock to use the latest deps
OUTPUT REMOVED TO SAVE SPACE

$ chef push dev mycorp_audit
Uploading policy mycorp_audit (bfd3af4697) to policy group dev
Using    mycorp_audit   0.3.0 (f37cdfc3)
Using    audit          7.5.0 (27e6ea8e)

$ chef show-policy
mycorp_audit
=======
* dev:         bfd3af4697
* staging:     98ed3a52a5
* production:  98ed3a52a5

$ chef push staging mycorp_audit
Uploading policy mycorp_audit (bfd3af4697) to policy group staging
Using    mycorp_audit   0.3.0 (f37cdfc3)
Using    audit          7.5.0 (27e6ea8e)

$ chef show-policy
mycorp_audit
=======
* dev:         bfd3af4697
* staging:     bfd3af4697
* production:  98ed3a52a5

$ chef push production mycorp_audit
Uploading policy mycorp_audit (bfd3af4697) to policy group production
Using    mycorp_audit   0.3.0 (f37cdfc3)
Using    audit          7.5.0 (27e6ea8e)

$ chef show-policy
mycorp_audit
=======
* dev:         bfd3af4697
* staging:     bfd3af4697
* production:  bfd3af4697

As you can see the policy with a revision_id of bfd3af4697 is being moved through dev, staging, and production.

For more info see: https://docs.chef.io/ctl_chef.html#policyfile-commands

FAQ

How can I create a Policyfile cookbook?

Older versions of the ChefDK/Chef Workstation support generating Policyfile cookbooks using chef generate cookbook mycorp_app -P. Newer versions will generate Policyfile cookbooks by default.

An easy way to tell if you are using a Policyfile cookbook is that it will not contain a Berksfile.

How can I migrate from the current way I'm doing cookbooks?

At a high level, if you are using a Berksfile:

Swap it out for a Policyfile.rb
Use chef install/push instead of berks install/upload
Run chef-client on your nodes with policy_name/policy_group specified
- Example: chef-client -j policy.json
- Search here for Specify a policy

This is a safe migration since only nodes bootstrapped to use Policyfiles will consume them. Also, since cookbooks used via Policyfiles are stored in a different way than other cookbooks, you cannot accidentally break the old way of using Chef.

Why don't I see Policyfile cookbooks with `knife cookbook list`?

Cookbooks consumed by Policyfiles are stored differently on Chef Infra Server and thus do not show when using Knife. Instead you should use:

chef show-policy POLICY_NAME POLICY_GROUP

Policyfile cookbooks vs plain Policyfiles, which is better?

Ultimately, the Policyfile.lock is the source of truth. With that in mind, it is possible to build a Policyfile.lock using only a Policyfile.rb.

In theory, you would build the top level role cookbook as described in this blog post on writing wrapper cookbooks, then call that cookbook (and set any desired attributes) via a Policyfile.rb, and upload the resulting lock to the Chef Infra Server. This removes the need for a top level cookbook entirely.

That being said, I personally prefer that the top level artifact be a Policyfile cookbook. This way, those that are not familiar with Policyfiles can acclimate to the new pattern and you can set attributes via logic (which cannot be done in a Policyfile.rb)

Attributes, should I set them in the Policyfile.rb?

I personally prefer only specifying dependencies and a run_list in the Policyfile.rb. I prefer this for many reasons.

Mainly:

It matches the older Chef patterns so those who usually look in recipes/attribute files for attributes will not be surprised
It sets the attributes near where they are used
It's more intuitive than methods such as hoisting (personal opinion)

Also, since the Policyfile.lock.json is compiled before runtime it is not possible to set dyanmic attributes that you would normally set during a Chef Infra run (e.g. changing behavior based on the domain, region, or Ohai).

An example of how I do this is as follows:

case node.policy_group
when 'dev'
  node.default['my_cookbook']['api_url'] = 'https://dev.example.com/api'
when 'staging'
  node.default['my_cookbook']['api_url'] = 'https://staging.example.com/api'
when 'production'
  node.default['my_cookbook']['api_url'] = 'https://prod.example.com/api'
end

Chef Infra Server, do I need it if I'm using Policyfiles?

Moving dependency resolution out of runtime has one other major benefit. If you are not relying on state stored in the Chef Infra Server (e.g. data bags, chef search, etc) then you can remove the need for the Chef Infra Server all together.

By using chef export you can create an artifact that contains the policy and all dependent cookbooks. This artifact can then be distributed and executed via chef-client. In fact, this is a core factor in how Effortless Infra and Test Kitchen (when using Policyfiles) functions.

How do I test Policyfile cookbooks in Test Kitchen

By default, the generated kitchen.yml will do what you need. If you'd like to test multiple suites though, please see the following section of the Chef docs:

https://docs.chef.io/policyfile.html#test-w-kitchen

10 KiB Raw Blame History