Bases: object
A configuration file parser class.
If options is a list or a string, it is assumed to contain the names of one or more configuration files. The list may contain lists that also contain the names of configuration files.
The parsed information is put into self.infos (a list containing elements of ParamStorage type).
The length of self.infos will normally be the same as the length of the list “options”.
If “options” is a string (the name of a configuration file) this is the same as a list with a single element.
If the list contains sub-lists, these are all read into the same part of self.infos.
The raw configuration information associated with each element of self.infos is in a list self.configs.
If a configration file is not found or is invalid, there is a fatal error ONLY of fatal=True.
Configuration files are searched for by absolute path name, if one is given. If there is a failure to read the file referred to by absolute pathname, the local name of the file is assumed, and the search continues for that file through the directory list datadir.
If the path name is relative, the file is searched for through the directory list datadir.
The class is derived from the object class. information via the command line.
See type_convert() for some rules on defining a configration file.
Methods
Utility to send log items to the logger
This method reads one or more configuration files into self.infos and self.configs.
Purpose: parse the information from conf_files into a ConfigParser class instance and return this.
config_error : string giving information on error
Try to eval all terms
Take a ConfigParser instance config and scan info into config.info. This is called recursively if needed.
The string this is split on CSV and each element passed through to be interpreted by the method type_convert. A ‘safe’ split is performed, which doesn’t split on [] or ().
Given an array of names (names) and a dataset (data) convert to a flat array
names: a list of names data: a list of names OR
a dictionary
prev: prev array to load into
If data is an array of names, a boolean numpy array is returned of size len(names) which is True where an element of data appears in names and False elsewhere. This is useful to subset an array of the same size as names into one associated with the data array.
If data is a dictionary then the items associated with the keys in the list names is returned as a numpy array. If an item doesnt exist in data, then None is returned.
This latter use is the main purpose of assoc_to_flat(). In effect it loads items from a dictionary into an array, based on the keys in the list names.
In the class ConfFile it is used to translate any data structure which has a name starting assoc_ into a numpy array of the same name (without the assoc_) at the same level of the hierarchy.
A test call to use ConfFile.
We import the class. Then we initialise a instance of ConfFile with the configuration file “default.conf”
A function to split a string, taking account of [] and () and quotes
Function to make a guess at the type of a variable defined as a string
1. Variables from within the config file can be referred to via $ e.g.: [model] names = eeny,meeny,miney [parameter] names = $model.names,moe
so:
model.names = [‘eeny’,’meeny’,’miney’] parameter.names = [[‘eeny’,’meeny’,’miney’],’moe’]
2. Any parameter names that begin ‘assoc_‘ are special lists/arrays where the parameter values are defined associated with a parameter name. If this mechanism is used, parameter.names must be defined.
For example:
[parameter] names= gamma,xlai, xhc, rpl, xkab, scen, xkw, xkm,
xleafn, xs1,xs2,xs3,xs4,lad
[parameter.assoc_bounds] gamma = 0.01,None xlai = 0.01,0.99 xhc = 0.01,10.0 rpl = 0.001,0.10 xkab = 0.1,0.99 scen = 0.0,1.0 xkw = 0.01,0.99 xkm = 0.3,0.9 xleafn = 0.9,2.5 xs1 = 0.0, 4. xs2 = 0.0, 5. xs3 = None, None xs4 = None, None lad = None, None
Statements in the config file go through an attempt at evaluation. This is partly to set the type (i.e. int, float etc) to something reasonable, but has the by-product of allowing quite a flexible statement definition.
For example, we can apply a logical statement
[general] rule = True unrule = not $general.rule
which sets:
general.rule = True general.unrule = False
By default, you have several python modules that you can call from within the config file.
These are those resulting from:
import sys import numpy as np import os from eoldas_parser import Parser from eoldas_params_storage import ParamStorage
so, for example you can call:
[general] x = np.random.rand(len($parameter.names)) datadir = .,sys.exec_prefix here = os.getcwdu() nice = os.nice(19)
which results in:
general.datadir = [‘.’, ‘/usr/local/epd-7.0-2-rh5-x86_64’] general.here = /data/somewhere/eoldaslib general.nice = 19
Finally, an attempt is made at executing a statement. This might prove a little dangerous (e.g. you could (intentionally or maliciously) use it to delete files when parsing a conf file, but if you want to do that there are plenty of other ways to achieve it). An example of when this might be useful might be if you want to import a python class whilst parsing the config file (for some reason).
[general] np = import numpy as np npp = import somethingThatDoesntExist
which sets:
general.np = import numpy as np : exec True general.npp = import somethingThatDoesntExist
Note that if a statement has been executed (i.e. run sucessfully through exec()).
Then the string ‘ : exec True’ is added to the end of its value as stored.
In the case of ‘import somethingThatDoesntExist’, this was not sucessfully interpreted via an exec() and so is simply assumed to be a string.
You might need to be a little careful in using strings. I suppose it is vaguely conceivable that you might actually want to set general.np to ‘import numpy as np’. Note that items such as those you have imported are not available to other lines of the config, so import serves little purpose, other than testing perhaps.
There might come a point where you might be doing so much coding in the config file that you might be better off writing some new methods/classes, but some degree of processing is of value, and at no great computing complexity or processing cost.