--- layout: post title: Using Policyfile Cookbooks category: chef_infra tags: [chef, cookbooks, policyfiles] summary: Policy...what is it good for? --- ## Policyfile? What is that? A Policyfile (aka Policyfile.rb) is a file that contains information about a node's: - Default source for fetching cookbooks - Run list (or multiple run lists via `named_run_list`) - Cookbook dependencies and sources - Optional cookbook attributes Here is an example Policyfile.rb: ```ruby name 'mycorp_audit' default_source :supermarket cookbook 'mycorp_audit', path: '.' run_list 'mycorp_audit::default' ``` This file is a key component in Policyfile cookbooks. ## What are Policyfile cookbooks and why should I use them? Essentially, a Policyfile cookbook is a cookbook that uses a Policyfile.rb to combine the functions of Berkshelf, Environments, and Roles into a single artifact that can be promoted safely through the software development lifecycle. Without using Policyfiles, special care is needed in order to use Chef safely. This is due to the shortcomings of runtime dependency solving. Take for example the following scenario: - You have 3 cookbooks: `cookbook_a`, `cookbook_b`, and `cookbook_c` - `cookbook_a` is deployed and depends on `cookbook_b` with version `~>1.0.0` - `cookbook_c` is created and depends on `cookbook_b` with version `~> 1.0.0` - `cookbook_c` is tested/deployed (which deploys newest 1.X.X of `cookbook_b`) - `cookbook_a` runs and breaks due to a bug in `cookbook_b` version `1.1.1` Now, ideally, strict version pinning and adhering to [Semantic Versioning](https://semver.org/) would prevent any mishaps...but that is a lot to trust to place on the developer, the pipeline, and the upstream cookbook maintainer. With Policyfile cookbooks, this is prevented because cookbooks are referenced by hash and not version at runtime. In order to run a new set of cookbooks a new Policyfile.lock must be deployed. --- ## Policyfile.lock The Policyfile.lock (aka Policyfile.lock.json) is generated from a Policyfile.rb and contains all the same information in it as the Policyfile.rb in addition to a few other key items needed for safely deploying cookbooks. For the sake of this section of the article, the two most important sections of the Policyfile.lock are the `revision_id` which contains the hash of the entire lock file and the `cookbook_locks` section. These two items together ensure that cookbooks and recipes are applied consistently regardless of the environment that Chef Infra is deployed/ran in. Below is an example `cookbook_locks` section: ```json "audit": { "version": "7.5.0", "identifier": "27e6ea8e7bc82f5ba44b77834f5472c96da9c27a", "dotted_decimal_identifier": "11231419178928175.25794866915266388.126209453834874", "cache_key": "audit-7.5.0-supermarket.chef.io", "origin": "https://supermarket.chef.io:443/api/v1/cookbooks/audit/versions/7.5.0/download", "source_options": { "artifactserver": "https://supermarket.chef.io:443/api/v1/cookbooks/audit/versions/7.5.0/download", "version": "7.5.0" } } ``` As you can see above, not only is useful information such as version and origin of the cookbook provided, but also referenced is the exact hash of the upstream cookbook that is being used. > In this case, the `audit` cookbook with hash > `27e6ea8e7bc82f5ba44b77834f5472c96da9c27a` This is critical as it ensures that not only are we running the correct version of the cookbook, but also that the content of that cookbook has not changed since the Policyfile.lock was created. ## Benefits over Berkshelf (i.e. runtime dependency solving) Since the Policyfile.lock is computed prior to uploading to the Chef Infra Server there is no need to compute cookbook dependencies at runtime. This has several emergent benefits. Firstly, by moving this computation to the workstation (or pipeline), Chef Infra is saved from having to do the same computation(s) on each run on every node. More importantly, since dependency resolution has already been completed and cookbooks are referenced via hash, it is impossible to unintentionally run a cookbook in production. ## Deploying/Promoting Policyfiles Policyfiles are deployed to the Chef Infra Server into what are known as Policy Groups. Conceptually, these Policy Groups take the place of Environments. This is best illustrated with the output from `chef show-policy`: ```text mycorp_audit ======= * dev: 98ed3a52a5 * staging: 98ed3a52a5 * production: 98ed3a52a5 ``` We can see here that the `mycorp_audit` policy with the revision ID `98ed3a52a5` is currently deployed in the dev, staging, and production policy groups. When bootstrapping a node, the `policy_name` and `policy_group` are specified, this ensures that the correct policy is applied during runtime. ### Promoting Policyfiles through your environments (Policy Groups) This is done like via `chef push POLICY_GROUP POLICY_NAME`. See: ```text $ chef update # This updates the Policyfile.lock to use the latest deps OUTPUT REMOVED TO SAVE SPACE $ chef push dev mycorp_audit Uploading policy mycorp_audit (bfd3af4697) to policy group dev Using mycorp_audit 0.3.0 (f37cdfc3) Using audit 7.5.0 (27e6ea8e) $ chef show-policy mycorp_audit ======= * dev: bfd3af4697 * staging: 98ed3a52a5 * production: 98ed3a52a5 $ chef push staging mycorp_audit Uploading policy mycorp_audit (bfd3af4697) to policy group staging Using mycorp_audit 0.3.0 (f37cdfc3) Using audit 7.5.0 (27e6ea8e) $ chef show-policy mycorp_audit ======= * dev: bfd3af4697 * staging: bfd3af4697 * production: 98ed3a52a5 $ chef push production mycorp_audit Uploading policy mycorp_audit (bfd3af4697) to policy group production Using mycorp_audit 0.3.0 (f37cdfc3) Using audit 7.5.0 (27e6ea8e) $ chef show-policy mycorp_audit ======= * dev: bfd3af4697 * staging: bfd3af4697 * production: bfd3af4697 ``` As you can see the policy with a `revision_id` of `bfd3af4697` is being moved through dev, staging, and production. For more info see: [https://docs.chef.io/ctl_chef.html#policyfile-commands](https://docs.chef.io/ctl_chef.html#policyfile-commands) --- ## FAQ ### How can I create a Policyfile cookbook? Older versions of the ChefDK/Chef Workstation support generating Policyfile cookbooks using `chef generate cookbook mycorp_app -P`. Newer versions will generate Policyfile cookbooks by default. An easy way to tell if you are using a Policyfile cookbook is that it will not contain a Berksfile. ### How can I migrate from the current way I'm doing cookbooks? At a high level, if you are using a Berksfile: - Swap it out for a Policyfile.rb - Use `chef install/push` instead of `berks install/upload` - Run `chef-client` on your nodes with `policy_name`/`policy_group` specified - Example: `chef-client -j policy.json` - Search [here](https://docs.chef.io/ctl_chef_client.html) for `Specify a policy` This is a safe migration since only nodes bootstrapped to use Policyfiles will consume them. Also, since cookbooks used via Policyfiles are stored in a different way than other cookbooks, you cannot accidentally break the old way of using Chef. ### Why don't I see Policyfile cookbooks with `knife cookbook list`? Cookbooks consumed by Policyfiles are stored differently on Chef Infra Server and thus do not show when using Knife. Instead you should use: `chef show-policy POLICY_NAME POLICY_GROUP` ### Policyfile cookbooks vs plain Policyfiles, which is better? Ultimately, the Policyfile.lock is the source of truth. With that in mind, it is possible to build a Policyfile.lock using only a Policyfile.rb. In theory, you would build the top level role cookbook as described in [this](https://blog.chef.io/2017/02/14/writing-wrapper-cookbooks/) blog post on writing wrapper cookbooks, then call that cookbook (and set any desired attributes) via a Policyfile.rb, and upload the resulting lock to the Chef Infra Server. This removes the need for a top level cookbook entirely. That being said, I personally prefer that the top level artifact be a Policyfile cookbook. This way, those that are not familiar with Policyfiles can acclimate to the new pattern and you can set attributes via logic (which cannot be done in a Policyfile.rb) ### Attributes, should I set them in the Policyfile.rb? I personally prefer only specifying dependencies and a `run_list` in the Policyfile.rb. I prefer this for many reasons. Mainly: - It matches the older Chef patterns so those who usually look in recipes/attribute files for attributes will not be surprised - It sets the attributes near where they are used - It's more intuitive than methods such as hoisting (personal opinion) Also, since the Policyfile.lock.json is compiled before runtime it is not possible to set dyanmic attributes that you would normally set during a Chef Infra run (e.g. changing behavior based on the domain, region, or Ohai). An example of how I do this is as follows: ```ruby case node.policy_group when 'dev' node.default['my_cookbook']['api_url'] = 'https://dev.example.com/api' when 'staging' node.default['my_cookbook']['api_url'] = 'https://staging.example.com/api' when 'production' node.default['my_cookbook']['api_url'] = 'https://prod.example.com/api' end ``` ### Chef Infra Server, do I need it if I'm using Policyfiles? Moving dependency resolution out of runtime has one other major benefit. If you are not relying on state stored in the Chef Infra Server (e.g. data bags, chef search, etc) then you can remove the need for the Chef Infra Server all together. By using `chef export` you can create an artifact that contains the policy and all dependent cookbooks. This artifact can then be distributed and executed via `chef-client`. In fact, this is a core factor in how Effortless Infra and Test Kitchen (when using Policyfiles) functions. ### How do I test Policyfile cookbooks in Test Kitchen By default, the generated kitchen.yml will do what you need. If you'd like to test multiple suites though, please see the following section of the Chef docs: [https://docs.chef.io/policyfile.html#test-w-kitchen](https://docs.chef.io/policyfile.html#test-w-kitchen) --- ## Supporting Resources - [More justification for Policyfiles](https://docs.chef.io/policyfile.html) - [Policyfile.rb documentation](https://docs.chef.io/config_rb_policyfile.html) - [Chef Policyfile commands](https://docs.chef.io/ctl_chef.html#policyfile-commands)