Creating Plots

Howto create a PGM2/SVG plot - Introduction

Output using the PGM2 engine is generated through .gplot files. The name is historical and today files can be generated using gnuplot or SVG graphics. This howto lists the most important aspects of generating a SVG configuration file.

Each graphic generated by FHEM is described by one configuration file. It is currently not possible to generate one page with several graphics from one configuration file.

Taking a high level view for each plot the following is needed:

• Data from a logfile (create it using the FileLog facility)
• Plot configuration (.gplot) file knowing about the internal structure of the logfile
• Association between the FileLog and the plot configuration (using the "logtype" attribute in the FileLog and the LINK in the WebLink object, also see comments below)
• Weblink representing the plot itself

FileLog objects

Digging into the more murky details, first thing is the creation of the FileLog object:

• The FileLog object is always associated with a physical device (but see comments below)
• The creation of a FileLog object together with the corresponding logfile is automatic if the "autcreate" attribute is set in FHEM (it is set by default)
• Manual creation is also possible and requires a statement like:

 define FL_KS550 FileLog /var/log/fhem/KS550_%Y.log KS550:T:.*

• Be it manual or automatic: the definition of a FileLog object determines (through the regular expression) which of the output lines of FHEM end up in the logfile

Plot definition

Once the logfile is accumulating data, the plot configuration itself is needed. There is no object for this, but plain .gplot files in the share/FHEM directory are used by the system. A good start is to copy an existing one and modify it.

On a side note: if you create your own files instead of using the ones already delivered with FHEM, stick to the convention of prepending them with a "my". Files of pattern "my*" are guaranteed not to be overwritten during FHEM updates (or so Rudi promised me).

The following entries are recognized in a SVG context:

• #FileLog holds the information which data need to be extracted from the logfile. See below for details.
• Set commands: title, ylabel, y2label, yrange, y2range, ytics, y2tics, teminal (only "size" parameter is recognized)
• Plot command: see below for details

Placeholders

It is possible to use placeholders in the .gplot files, which will then be replaced by information provided by the actual WebLink attributes listed below. This has the advantage that you can define one .gplot file and re-use it for several devices, changing size, title and lables accordingly instead of maintaining several separated files.

• <OUT> : only relevant for gnuplot
• <SIZE>: attr plotsize
• <TL>  : attr title. The attribute is eval'ed by perl at runtime, be careful and protect by double quotes if you just want to enter strings. This gives access to all internal FHEM values, though admittedly not straightforward
• <Ln>  : attr label. Lables 1-#n can be specified by a double-colon-separated (::) list, see below. This is also eval'ed at runtime, same precautions apply

In general this is good practice and you should especially consider it if you plan to distribute your .gplot files. Using placeholders f.ex. for the title and all the labels and curve descriptors gives other users the possibility to change them without having to touch the .gplot code.

Data Extraction

The central piece of the plotfile is the #FileLog line (yes, it must be prepended by a hash!), which is a special command und internally triggers the use of the FileLog "get" function, which extracts data from the logfile. The general syntax is: <col>:<regexp>:<default>:<fn> .

• col: selects the column of the line (columns are white space separated, numbering starts at 1)
• regexp: validates/selects the line
• default: default value, if none is present
• fn: function which operates on values, pre-defined or eval'ed, see below

The "get" function will set the following %data values (which can e.g. be used in titles, labels, ...) for each requested column (curve), beggining with <x> = 1:

• min<x>, max<x>
• avg<x>, cnt<x>
• sum<x>
• currval<x> (last value)
• currdate<x> (last date)

So you can do something like attr <weblink> title "Min $data{min1}, Max $data{max1}, Last $data{currval1}" Also implemented in this function is code which operates on the data retrieved. This can be specified at the "<fn>" tag. Up till now following functions are impemented:

• int (to cut off % from a number, as for the actuator)
• delta-h / delta-d to get rain/h and rain/d values from continuous data
• And then there is this: the string is evaluated as a perl expression. @fld is the current line splitted by spaces (0-based). So you can do something like $fld[3]=~"on"?0.9:0.8 to map the 4th field which contains an on/off information into numerical values to be plotted in a graph. Be warned though: this string/perl expression cannot (!) contain any spaces.

For even more details see http://fhem.de/commandref.html#FileLogget or the module 92_FileLog.pm.

Debugging Data Extraction

If you do not get the result you are expecting or, worse, no data at all, you may use the debug facility of FHEM to see what the get-command extracts from the logfile. Use telnet to contact FHEM and then use the arguments detailed below ending with the #FileLog argument you are using in the .gplot file:

telnet diskstation 7072
get FileLog_KS550 ?
get FileLog_KS550 - - 2012-01-01 2012-12-31 4::

You should get back the data which you have been asking for, followed by a line holding the .gplot argument. Make sure

• you do not select too much data, this will possibly upset your NAS :-)
• you do select data which really are in the file (beware of year/month changes)

Plot Commands

Once you have extracted the data from the logfile, you need to plot it. There is a plot command which does just that. It has several arguments which you can use:

• axes: not needed by default, but if you want to plot curves which use different y axes, then you must tell the plot command, which y-axis the data should be bound to: "axes x1y1" selects the left, "axes x1y2" the right one
• title: defines the string which is printed in the colour of the curve into the plot. You can (and maybe should) use placeholders like "<Lx>", x being a number
• with: defines how to draw the data. only points, steps, histeps and lines (which acts as a catch all) are currently implemented
• ls: uses CSS definitions, still need to figure out, how this works
• lw: defines the width of a line

Example

A very barebones but functional example of a file using placeholders could be:

set terminal size <SIZE>
set title '<TL>'
set ylabel '<L1>'
set y2label '<L2>'
#FileLog 4:::
plot \
   axes x1y1 title '<L3>' with steps lw 2

It implements the placeholders which can be defined by the user through the WebLink attributes later, selects just one column, without regular expression matches, defaults or subsequent data operations and plot is as a steps diagram. In fact even the "axes x1y1" statement could be left out, it would still work.

Transparent plots

If you need transparent plots, for example in FLOORPLAN, you have to change the following definitions in svg_style.css

Change

.background { fill:#FFFFE7; }

to:

.background { fill:#FFFFE7;fill-opacity: 0; }

Change:

.border { stroke:black; fill:url(#gr_bg); }

to

.border { stroke:black; fill:url(#gr_bg);fill-opacity: 0; }

WebLink objects

Once you have created the FileLog object and written the plot specification, there is not much to be done. Click on the reference to the .gplot file in the FileLog section of a device, then click on the link for WebLink creation and then add the attributes you want to have. Put it into a suitable room and you are done.

If you are using placeholders in your .gplot files and after defining your .gplot file you click onto a FileLog "active" link to create the plot itself and get a message like:

XML Parsing Error: mismatched tag. Expected: </L1>.
...
Line Number 51, Column 95:<text x="12" y="80" text-anchor="middle" class="ylabel" transform="rotate(270,12,80)">'<L1>'</text>

you can safely ignore it. This is due to the fact, that you are using placeholders in the .gplot definition (which is good, because it favors reuseability) but have not yet defined the corresponding attributes. While annoying, this is not your fault: you can only define attributes once you have a weblink. So click on create weblink and define the attributes you have been using placeholders for. The error message should go away.

The attributes which are generally needed are:

• label: should be defined like "Label 1"::"Label 2". Do not omit double quotes and use them individually for each label. Separator is a double (!) colon. If you ever manage to do away with the double quotes on the axes, please let us all know
• title: Can be as simple as "Some Title" or as complicated as "Temperature $data{currval1} ($data{min1}-$data{max1}), Humidity $data{currval2} ($data{min2}-$data{max2}) @ $data{currdate1}"

SVG Programs

The SVG programs can be found in the 98_SVG.pm module. Look for example for the string "histeps". Feel free to adapt and improve. If you ever program a wind rose to plot wind directions, please let me know.

Details

What you also should know:

• All data used in a plot come from one (!) file. It is currently not possible to mix data from several files. So make sure that the regular expression of the FileLog object catches everything which you need in one plot
• While it first seems evident that FileLog objects are associated with a given device, this must be mitigated. Imagine that you want to plot the state (and changes) of some socket adaptors (ZwischenStecker) over time. You then need to gather all the data (from several devices) in one file. So instead of define FileLog_ZwSt_1 FileLog /opt/fhem/var/log/ZwSt_1_%Y.log ZwSt_1 you need to specify define FL_ZwSt_ALL FileLog /opt/fhem/var/log/ZwSt_ALL_%Y.log ZwSt_.*deviceMsg.*BidCoS_RF.*
• Be careful when specifying the regular expression which filters the lines for a given logfile. While often it is sufficient to not specify a specific one at all, consider the following case from the Homematic world, where one action results in 4 lines of FHEM output. The switch communicates not only with the CCU, but also with the remote control, issuing 2 lines for each action:

2012-01-01_07:40:23 ZwSt_3 deviceMsg: on (to Dev.WDisp.Kueche)
 2012-01-01_07:40:23 ZwSt_3 on
 2012-01-01_07:40:26 ZwSt_3 deviceMsg: on (to BidCoS_RF)
 2012-01-01_07:40:26 ZwSt_3 on

• Why one needs to specify a logtype (which is nothing else than a plot definition) for the FileLog has always been a mystery to me. It holds no useful information to the object itself and does not influence the way data is written to the associated logfile. Sure, plots need FileLogs, but not the other way round. And since the WebLink contains (again) the name of the plot definition file, the need to define the logtype attribute seems odd. The master himself says regarding this topic: "Ist eine Eigenschaft, die beschreibt, auf welche Art die Daten angezeigt werden koennen. Steht damit direkt neben den Daten, also da wo ich zuerst suchen wuerde. Ist nicht notwendig, aber so kann FHEMWEB ein 'create weblink' anbieten." I have to admit that this link is useful and I use to use it :-)
• If you meet on your way PERL-code snippets in the .gplot files, you can just ignore them safely. They are only used in the gnuplot world, not in SVG anymore. Now you may ask, where you find the corresponding features in SVG? They are hidden in functions defined in the FileLog "get" primitive.

Documentation

Documentation can be found in several places, it is not complete and you need to piece it together:

• http://fhem.de/HOWTO.html#plotting
• http://fhem.de/commandref.html#FileLogget
• http://fhem.de/commandref.html#label