Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Welcome to the CollectiveAccess support forum! Here the developers and community answer questions related to use of the software. Please include the following information in every new issue posted here:

  1. Version of the software that is used, along with browser and version

  2. If the issue pertains to Providence, Pawtucket or both

  3. What steps you’ve taken to try to resolve the issue

  4. Screenshots demonstrating the issue

  5. The relevant sections of your installation profile or configuration including the codes and settings defined for your local elements.


If your question pertains to data import or export, please also include:

  1. Data sample

  2. Your mapping


Answers may be delayed for posts that do not include sufficient information.

Overriding nested values in configuration files

edited April 2014 in Installation
Hi,

I have started at Gaia Resources working on a project using Collective Access for the Western Australian Museum.  Pretty impressed with what I see so far.

I am wondering about merging configuration files in app/conf and those in app/conf/local, in particular with nested associative arrays.

For example, let's say the base config file (app/conf/foo.conf) contains:

a = {
    b = something,
    c = whatever,
    d = {
        e = nested value
    }
}

If I want to change the value of "e" only, how can I do this?  Or to give a contrived example based on the actual code, let's say I wanted to add an entry for a new jquery plugin, under "jquery", under "packages" in javascript.conf.

The only method that is obvious to me, other than changing the base file (app/conf/foo.conf) directly, is to duplicate the entire top-level key (i.e. "a" and its value) into the local conf file, and then modify the sub-keys that I want to change.  However I can foresee problems when merging changes to those base config files from upstream.  For example, if you added some new keys into this structure, or changed the default value of an existing key, then we would need to look out for this and merge those new keys into our config file in "app/conf/local/foo.conf", otherwise we will be overwriting the changes in our local conf file.  I'm sure you will agree that this process is very error-prone due to changes being incorrectly merged, or missed altogether. 

What I was hoping for was the ability to put something like this into "app/conf/local/foo.conf":

a.c.e = overriding value for e

Or:

a.c = {
    e = overriding value for e
}


Or even:

a = {
    c = {
        e = overriding value for e
    }
}


This last form will completely overwrite the value of a, wiping out b and d, and anything else specified in the base file.  The other two forms seem to be effectively ignored.  I'm guessing the first form, for example, would create a top-level key "a.c.e", which is not the same as values nested under "a", and which is not used anywhere.  Not breaking BC would be a concern here, perhaps it would need a special syntax:

a += {
    c = {
        e = overriding value for e
    }
}


The += here would mean "merge the following into the existing definition of a", and if it was = instead of += then it would overwrite as it currently does.

Is there anything like this, is this already planned functionality, or is it something you would like to have implemented?

Regards,
Ben

Comments

  • Right now configuration files are merged at the top level, as you've noted. This does make for some headaches when trying to override element of files that put a lot under a single top-level key, such as navigation.conf.

    We'd love to implement a solution for this. I have to confess that while it has always bothered me but I've not given too much thought to a solution to now. I like your proposed += notation very much. It seems like a clean solution!
  • Hi Seth,

    I think both the += syntax and the dot notation could be done, and treated as separate enhancements.  Having the += notation would solve the problem of needing to copy and paste whole sections of config just to change one value, but the syntax is quite unwieldy.  Having the "a.b.c = foo" shorthand available as well would make for much more terse local config files.  They could even be nested, so you could have something like:

    a += {
        b.c = foo,
        d.e.f = bar
    }

    Which is both a lot shorter than, and (imo) at least as readable as the equivalent:

    a += {
        b += {
            c = foo
        },
        d += {
            e += {
                f = bar
            }
        }
    }


    In this case, yet another way of writing it would be:

    a.b.c = foo
    a.d.e.f = bar

    I have had a look at the existing config files, and there are some places where there are literal dots in key names.  For example, MIME types as key names in media_processing.conf, and also some key names in media_metadata.conf contain dots.  (These are the only two that I can see).  Perhaps in these cases, we just need to enforce quoting the key name, what do you think?

    I have also had a look at the Configuration class, and I would also like to replace the magic numbers for the states of the state machine, with class constants.  This just makes the code more readable and would be very simple to do (plus having magic constants everywhere is a bit of a code smell :-) ).  The test coverage for that class doesn't look 100% complete (for example there are no unhappy-path tests) but it should be enough to give me confidence that I'm not introducing regressions.

    Ben

  • Oh yes one more thing -- it seems natural to extend the += syntax to [list] types as well as {objects}.  A += would append the list given in app/conf/local/foo.conf to the end of the list given in the default config file (app/conf/foo.conf), whereas a = would replace the list completely.

  • Hi Ben,

    The configuration code is quite old (it dates to 2000 and PHP3), so we need not be shy about changing things. It is one of the odd bits of the CA code that actually hasn't changed all that much in the past 12 years. Constants would be a good idea, maybe along the lines of how the TimeExpressionParser does it.

    Mandating quoting of keys is going to break existing files and for that reason seems like a non-starter.  Periods may be uncommon in keys, but they do happen. Perhaps we could use which should never appears in keys. 

    a > b > c = foo

    I agree that the += should apply to lists as well.
  • edited April 2014
    I did think about the BC-breaking nature of requiring keys with dots to be quoted, but if it was part of a major release (1.5 or 1.6, I haven't looked at your release cycle / plan yet) and well documented in the release notes, I think the risk is pretty minimal.  Especially if those two files that do contain dots in key names (media_processing.conf and media_metadata.conf) are only rarely overridden in app/conf/local -- we can just put the quotes in the base files in app/conf and that will be enough.  But I do take your point that it is introducing potential issues that don't necessarily need to be introduced.  (And furthermore, there may be plugins that others are using, but which are not in the repo, which make extensive use of dots in key names)

    Perhaps, to tweak your suggestion:

    a->b->c = foo

    Where the -> is the same operator as that used for object dereference in PHP..?

  • Hi,

    I'm ok breaking things if there's a really good reason. I agree the the exposure from redefining the period character as reserved in keys is pretty minimal, but I also don't think it's the case that the period is the only (or best) notation for this sort of feature. As far as I'm concerned all three options, ".", ">" and "->" are equally valid. Since ">" and "->" won't break anything I'd tend to go with either of those, but I'm happy to be told that I'm wrong. If you're passionate about "." we can do that :-)
  • Haha, I'm not passionate about dot notation, I've probably just written too much javascript so it's the first thing I go for :-) 

    Let's go with ->, it's PHP-like and readable, and doesn't cause any BC breakage.

  • Ok. Are you going to work on this? Or do we handle it?
  • I'm currently working on a plugin, details to come in a PR, but could probably work on this after that is done.  We're doing project planning today so I'll follow up with something more definite.
  • We've put this into our backlog, but not into our current sprint, which runs for the next two weeks.  I'm hoping to get it into the next sprint but that depends on other project pressures - this is a nice-to-have for us rather than a must-have (the workaround is just to copy the whole config file and change the items that we want to change).  If anyone else gets to it before we do, that's fine, if not, we should be able to work on it some time soon.

  • Ok thanks.
  • edited May 23

    Just read about the override configuration feature, but I don't know if it is integrated in current code. I think it is a very much interesting feature, I have a couple of use case for it (for example, overriding media_processing configuration).

    Concerning dot notation (or > or ->), I agree it is a separate feature, and once overriding is in place, it is just convenient, but not mandatory from my point of view.

Sign In or Register to comment.