Document merging and lookup paths

The basics

Documents (and sub-documents!) can reference other files. These files will be loaded first and then the documents are merged on top of them. This allows you to overwrite and merge configuration files.

The following examples will use the Parent and Example document types from the previous chapters.

Let’s say you have the following configuration file (type Parent):

# fixtures/parent_with_ref.yml
parent:
  $ref: /referenced-document
  name: overwritten
  direct:
    int: 1234
  map:
    key_from_parent_with_ref:
      this: is from parent_with_ref

This document contains a $ref entry. When calling the resolve_and_merge_references() on a YamlConfigDocument Configcrunch looks for files with this relative path in fhe array you provide it. Let’s say you have the following file (note the path):

# fixtures/repo/referenced-document.yml
parent:
  name: this will be lost
  direct:
    this: foo
  map:
    key:
      this: bar
  list:
    - entry1
    - entry2
    - entry3

You can tell Configcrunch to look in the directory fixtures/repo while trying to load parent_with_ref.yml. Configcrunch will notice the $ref key, look for referenced-document.yml and merge the two files:

>>> document = Parent.from_yaml("fixtures/parent_with_ref.yml")
>>> document.resolve_and_merge_references(["./fixtures/repo"]) 
Parent(...)

>>> document.freeze()
>>> print(document['name'])
overwritten
>>> print(document['direct']['this'])
foo
>>> print(document['direct']['int'])
1234

The list passed to resolve_and_merge_references() is a list of lookup paths.

As you can see the resulting document is a combination of the two documents. All values in referenced-document.yml were replaced with values from parent_with_refs.yml. This also spans sub-documents.

The resulting document is:

parent:
  name: overwritten
  direct:
    this: foo
    int: 1234
  map:
    key:
      $name: key
      this: bar
    key_from_parent_with_ref:
      this: is from parent_with_ref
  list:
    - entry1
    - entry2
    - entry3

Chaining references

You can chain $ref-Entries. If a document is $ref’erenced this document can contain a $ref-entry as well. It’s $ref reference will be processed first.

Documents that are referenced can reference other documents relative to their position within the reference paths.

Example document in lookup paths:

# fixtures/repo/referenced-document.yml
parent:
  $ref: ./referenced-document
  map:
    key2:
      this: bar2

Example document that loads this document:

parent:
  $ref: /referenced-document-with-reference
  name: overwritten

Example after merge:

parent:
  name: overwritten
  direct:
    this: foo
  map:
    key:
      $name: key
      this: bar
    key2:
      $name: key2
      this: bar2
  list:
    - entry1
    - entry2
    - entry3

References in sub-documents

Sub documents can also contain $ref entries. They will be merged as expected. For this to work you only need to set up sub-documents as explained in the last chapter.

You place these documents to include also in the lookup paths:

# fixtures/repo/examples/referenced.yml
example:
  this: can be changed
  int: 1

You can then use it with $ref-entries:

parent:
  $ref: /referenced-document
  name: overwritten
  direct:
    $ref: /examples/referenced
    this: was changed

This will result in the following document being merged:

parent:
  name: overwritten
  direct:
    this: foo
    int: 1234
  map:
    key:
      $name: key
      this: bar
    key_from_parent_with_ref:
      this: is from parent_with_ref
  list:
    - entry1
    - entry2
    - entry3

Warning

Sub-documents are loaded after parent documents are merged, so $ref-Entries in sub-documents can be overwritten if they are present in referenced parent documents. They are NOT chained in this case.

As an example, imagine parent_with_ref.yml and referenced-document.yml would both contain a $ref-entry for the sub-document under direct. Only the $ref-entry from parent_with_ref.yml will be processed.

Multiple lookup paths

You can provide multiple lookup paths.

If a document is found in multiple lookup paths, the documents will be processed in the order of the entries in the lookup paths list. They will then be merged as explained under “Chaining references”.

This allows a lookup path to “override” another. The first lookup path is the base lookup path and the documents in the other lookup paths can extend and change definitions in the lookup paths that come before them.

Removing entries

During merging, a document can remove something from a referenced document with the special $remove keyword.

To remove string entries from lists, add an entry to the list which has the original value and prefix it with “$remove::”. Only removing strings from lists is supported.

Example document:

# fixtures/parent_with_ref.yml
parent:
  $ref: /referenced-document
  name: overwritten
  direct:
    int: 1234
  map:
    key: $remove
    key_from_parent_with_ref:
      this: is from parent_with_ref
  list:
    - "$remove::entry2"

Merge result:

parent:
  name: overwritten
  direct:
    this: foo
    int: 1234
  map:
    key_from_parent_with_ref:
      this: is from parent_with_ref
  list:
    - entry1
    - entry3