HTTPMOD

HTTPMOD
Zweck / Funktion
	Extract information from devices with an HTTP interface (or, more generic, from any URL) or send information to such devices
Allgemein
Typ	Gerätemodul
Details
Dokumentation	EN / DE
Support (Forum)	Sonstiges
Modulname	98_HTTPMOD.pm
Ersteller	StefanStrobel (Forum / Wiki)
	Wichtig: sofern vorhanden, gilt im Zweifel immer die (englische) Beschreibung in der commandref!

HTTPMOD provides a generic way to retrieve information from devices with an HTTP Interface and store them in Readings or send information to such devices. It queries a given URL with Headers and data defined by attributes.

From the HTTP response it extracts readings named in attributes using Regexes, JSON or XPath parsing also defined by attributes.

In an advanced configuration the module can also send information to devices. To do this, a generic set option can be configured using attributes.

Introduction

With the help of HTTPMOD you can automatically access websites, retrieve information and send data. In the simplest case, an http call is started regularly and the content of a page, for example, is read and processed further. The URL and the time interval in which the call is to be made must then be specified when the call is made. In order to read information from the called web pages, readings are required. In this readings one specifies Regex, which reads the necessary information. Here is an example.

define PM HTTPMOD  http://www.klack.de/fernsehprogramm/was-laeuft-gerade/0/-1/free.html 20000

This creates a device that accesses the http://www.klack.de/fernsehprogramm/was-laeuft-gerade/0/-1/free.html website with the TV program every 20000 seconds. If neither readings nor other attributes are defined, then only the call of the website takes place and FHEM receives no further data from the web access.

But now a television program is returned when you call the website, which you might want to read out. In order to save the contents in FHEM, which are related to the ARD, you need a reading. This reading must contain a name (so that FHEM knows under which name the information is stored) and a regex (so that you know where the information is stored on the website). Since usually not one, but many readings are defined, these readings are numbered consecutively. This is done by numbers that are given immediately after the word reading. In our example this would be for example

attr PM reading1Name ARD 
attr PM reading1Regex <span class="stationName">ARD<\/span>[\w\W]*?<div class="content">\s*<a[\w\W]*?>\s*(.*?)\s*<\/a> 
attr PM reading2Name ARD_time 
attr PM reading2Regex <span class="stationName">ARD<\/span>[\w\W]*?<td class="time">\s*(.*?)\s*< 
attr PM reading3Name ARD_details 
attr PM reading3Regex <span class="stationName">ARD<\/span>[\w\W]*?<div class="content">\s*<a[\w\W]*?>[\w\W]*?<\/a>\s*<br\/>\s*(.*?)(<img|<\/div)

Reading1 contains the programme, reading2 the time and reading3 further details. The Regex result from the structure of the website and may have to be designed in painstaking detail.

There are cases where this is not enough. For example, it could be the case that the information from the web access is not automated, but manually received. Then a 0 must first be entered at the top of the duration. But how does HTTPMOD then know when the data should be retrieved? This is done with get-commands. If a get-command is executed, HTTPMOD accesses the website, fetches the information and fills the corresponding readings.

Again, it is possible to create not one but several get-commands. Therefore, get is always followed by (at least) one number with which the get commands are numbered. Here is another example. The TV program should be called manually and not time-controlled. To do this we first change the definition to

define PM HTTPMOD  http://www.klack.de/fernsehprogramm/was-laeuft-gerade/0/-1/free.html 0

and add the following to the attributes

attr PM get1Name ARDHolen

With the command "get PM ARDHolen" the above readings are filled. In order for FHEM to know that the other reading2, reading3, etc. are to be read as well as reading1, the (user) attribute

attr PM get1CheckAllReadings 1

needs to be defined.

If the URL is not sufficient when the web page is called, but a special header is to be sent with it, this can be done with the get1Header attribute. If the header is multiline, the individual lines can be entered with get1Header1, get1Header2, and so on.

If you want to evaluate or control different URLs, you can define several get commands (get1, get2, etc.) and provide them with your own URLs. get1 usually reads only the reading1, get2 reads only the reading2 and so on. With CheckAllReadings all readings are evaluated simultaneously.

If only data is to be sent on the web page, but not read, the set command is recommended. It has the same structure as the get-command.

It becomes a bit more complex if the website requires a login check. The sid attributes are used for this purpose.

Availability, prerequisites and definition

This module is part of the regular FHEM distribution and uses the non blocking HTTP function HttpUtils_NonblockingGet provided by FHEM's HttpUtils in a new version published in December 2013. If not already installed in your environment, please update FHEM or install it manually using appropriate commands from your environment. Please also note that FHEM HttpUtils need the global attribute dnsServer to be set in order to work really non blocking even when dns requests can not be answered.

The device is defined as follows

define <name> HTTPMOD <URL> <Interval>

The module connects to the given URL every Interval seconds, sends optional headers and data and then parses the response with regular expressions, xpath or json to set readings.

URL can be "none" and Interval can be 0 if you prefer to only query data with a get command and not in a defined interval.

Example:

define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60

Set-Commands

can be defined using attributes, see advanced configuration

If you set the attribute enableControlSet to 1, the following additional built in set commands are available:

interval: set new interval time in seconds and restart the timer
reread: request the defined URL and try to parse it just like the automatic update would do it every Interval seconds without modifying the running timer.
stop: stop interval timer.
start: restart interval timer to call GetUpdate after interval seconds
upgradeAttributes: convert outdated attributes for older HTTPMOD-Versions that are still defined for this device from the old syntax to the new one.; attributes with the description "this attribute should not be used anymore" or similar will be translated to the new syntax, e.g. readingsName1 to reading01Name.
storeKeyValue: stores a key value pair in an obfuscated form in the file system. Such values can then be used in replacements where the mode is "key" e.g. to avoid storing passwords in the configuration in clear text

Get-Commands

can be defined using attributes, see advanced configuration

simple Attributes

enableControlSet: enables the built in set commands interval, stop, start, reread, upgradeAttributes, storeKeyValue

enableCookies: enables the cookie handling inside HTTPMOD. It is advisable to always set this attribute and allow HTTPMOD to track the state of cookies and set them for following HTTP-requests

enforceGoodReadingNames: makes sure that HTTPMOD only creates readings that are allowd for Fhem (especially if reading names are dynamically created from JSON object names with extractAllJSON. It is advisable to always set this attribute.

handleRedirects: enables the redirect handling inside HTTPMOD which should be used together with the cookie handling of HTTPMOD. HTTPMOD will then automatically follow redirects from a web server and keep track of cookies at the same time. It is advisable to always set this attribute.

requestHeader.*: Define an additional HTTP Header to set in the HTTP request

requestData: POST Data to be sent in the request. If not defined, it will be a GET request as defined in HttpUtils used by this module

reading[0-9]+(-[0-9]+)?Name: the name of a reading to extract with the corresponding readingRegex

reading[0-9]*(-[0-9]+)?OExpr: defines an expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.

reading[0-9]*(-[0-9]+)?OMap: Output Map. Defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb". If specified as readingOMap then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*OMap.

reading[0-9]*(-[0-9]+)?Format: Defines a format string that will be used in sprintf to format a reading value. If specified as readingFormat then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Format.

reading[0-9]*(-[0-9]+)?Decode: defines an encoding to be used in a call to the perl function decode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 instead of utf8.

reading[0-9]*(-[0-9]+)?Encode: defines an encoding to be used in a call to the perl function encode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.

reading[0-9]+Regex: defines the regex to be used for extracting the reading. The value to extract should be in a capture group / sub expression; e.g. ([\d\.]+) in the above example. Multiple capture groups will create multiple readings (see explanation above)

reading[0-9]+XPath: defines an xpath to one or more readings when parsing HTML data (see examples below)

reading[0-9]+XPath-Strict: defines an xpath to one or more readings when parsing XML data (see examples below)

reading[0-9]+JSON: defines a path to the JSON object wanted by concatenating the object names with an underscore as delimiter (see the example below)

noShutdown: pass the noshutdown flag to HTTPUtils for webservers that need it (some embedded webservers only deliver empty pages otherwise)

disable: stop doing automatic HTTP requests while this attribute is set to 1

timeout: time in seconds to wait for an answer. Default value is 2

do_not_notify

readingFnAttributes

Simple Configuration of HTTP Devices

If your device expects special HTTP-headers then specify them as attr requestHeader1 to attr requestHeaderX. If your Device expects an HTTP POST instead of HTTP GET then the POST-data can be specified as attr requestData. To get the readings, specify pairs of attr readingXName and attr readingXRegex, attr readingXXPath, attr readingXXPath-Strict or attr readingXJSON to define which readings you want to extract from the HTTP response and how to extract them. (The old syntax attr readingsNameX and attr readingsRegexX is still supported but the new one with attr readingXName and attr readingXRegex should be preferred. The actual values to be extracted have to be sub expressions within () in the regex (see example below)

Example for a PoolManager 5:

The PoolManager Web GUI can be queried with HTTP POST Requests like this one:

POST /cgi-bin/webgui.fcgi HTTP/1.1
Host: 192.168.70.90
Accept: */*
Content-Type: application/json;charset=UTF-8
Content-Length: 60

{"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value"]}

The resulting HTTP Response would look like this:

HTTP/1.1 200 OK
Content-type: application/json; charset=UTF-8
Expires: 0
Cache-Control: no-cache
Date: Sun, 12 Jan 2014 12:23:11 GMT
Server: lighttpd/1.4.26
Content-Length: 179

{
	"data":	{
		"34.4001.value":	"7.00",
		"34.4008.value":	"0.52",
		"34.4033.value":	"24.8"
	},
	"status":	{
		"code":	0
	},
	"event":	{
		"type":	1,
		"data":	"48.30000.0"
	}
}

To configure HTTPMOD for a PoolManager one would first define a PoolManager device with e.g. the name PM, the URL and an interval of e.g. 60 seconds.

Then the data to be sent in the request needs to be defined because in this example the device expects a POST request so the query is not contained in the URL but in the request data.

Also as seen above the device expects special HTTP headers in the request so these headers also need to be defined as attr PM requestHeader1 and attr PM requestHeader2

Then the names of the readings to be extracted would be set with attributes

Then for each reading value to be extracted a regular expression needs to be set that will match the value in question within ().

Example:

define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60

attr PM enableControlSet 1
attr PM enableCookies 1
attr PM enforceGoodReadingNames 1
attr PM handleRedirects 1

attr PM reading01Name PH
attr PM reading01Regex 34.4001.value":[ \t]+"([\d\.]+)"

attr PM reading02Name CL
attr PM reading02Regex 34.4008.value":[ \t]+"([\d\.]+)"

attr PM reading03Name3TEMP
attr PM reading03Regex 34.4033.value":[ \t]+"([\d\.]+)"

attr PM requestData {"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value", "14.16601.value", "14.16602.value"]}
attr PM requestHeader1 Content-Type: application/json
attr PM requestHeader2 Accept: */*
attr PM stateFormat {sprintf("%.1f Grad, PH %.1f, %.1f mg/l Chlor", ReadingsVal($name,"TEMP",0), ReadingsVal($name,"PH",0), ReadingsVal($name,"CL",0))}

Example for AmbientMonitor

AmbientMonitor is a webbased visualisation for sensors connected to an Arduino device. Its web interface can also be queried with HTTMOD to grab the data into readings.

This example was provided by locutus. The hardware configuration is an Arduino + Ethercard with ENC28J60 Controller + DHT22 Sensor and software can be downloaded from https://github.com/lucadentella/AmbientMonitor

In this example an HTTP GET is sufficent, so no requestData is needed. The device provides temperature and humidity readings in an HTTP response that looks like:

HTTP/1.0 200 OK 
Content-Type: text/html 

myCB({'temperature':22.00,'humidity':46.00})

the definition could be:

define AmbientMonitor HTTPMOD http://192.168.1.221/?callback=? 300

attr AmbientMonitor enableControlSet 1
attr AmbientMonitor enableCookies 1
attr AmbientMonitor enforceGoodReadingNames 1
attr AmbientMonitor handleRedirects 1

attr AmbientMonitor requestHeader Content-Type: application/json
attr AmbientMonitor reading1Name Temperatur
attr AmbientMonitor reading1Regex temperature':([\d\.]+)
attr AmbientMonitor reading2Name Feuchtigkeit
attr AmbientMonitor reading2Regex humidity':([\d\.]+)
attr AmbientMonitor stateFormat {sprintf("Temperatur %.1f C, Feuchtigkeit %.1f %", ReadingsVal($name,"Temperatur",0), ReadingsVal($name,"Feuchtigkeit",0))}

formatting and manipulating values / readings

Values that are parsed from an HTTP response can be further treated or formatted with the following attributes:

(reading|get)[0-9]*(-[0-9]+)?OExpr
(reading|get)[0-9]*(-[0-9]+)?OMap
(reading|get)[0-9]*(-[0-9]+)?Format
(reading|get)[0-9]*(-[0-9]+)?Decode
(reading|get)[0-9]*(-[0-9]+)?Encode

They can all be specified for an individual reading, for all readings in one match (e.g. if a regular expression has several capture groups) or for all readings in a get command (defined by getXX) or for all readings in the main reading list (defined by readingXX):

reading01Format %.1f

will format the reading with the name specified by the attribute reading01Name to be numerical with one digit after the decimal point. If the attribute reading01Regex is used and contains several capture groups then the format will be applied to all readings parsed by this regex unless these readings have their own format specified by reading01-1Format, reading01-2Format and so on.

reading01-2Format %.1f

Can be used in cases where a regular expression specified as reading01regex contains several capture groups or an xpath specified as reading01XPath creates several readings. In this case reading01-2Format specifies the format to be applied to the second match.

readingFormat %.1f

applies to all readings defined by a reading-Attribute that have no more specific format.

If you need to do some calculation on a raw value before it is used as a reading, you can define the attribute readingOExpr. It defines a Perl expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.

Example for an expression:

attr PM reading03OExpr $val * 10

Just like in the above example of the readingFormat attributes, readingOExpr and the other following attributes can be applied on several levels.

To map a numerical value to a name, you can use the readingOMap attribute. It defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb".

Example for a map:

attr PM reading02-3OMap 0:kalt, 1:warm, 2:sehr warm

To convert character sets, the module can first decode a string read from the device and then encode it again. For example:

attr PM getDecode UTF-8

This applies to all readings defined for Get-Commands.

Some help with Regular Expressions

Starting with version 2018-02-10 the internal that holds the HTTP response is no longer called buf but rather httpbody, and it is only displayed when attribute showBody is set to "1".

If HTTPMOD seems not to work and the FHEM Logfile contains a message like

HTTPMOD: Response didn't match Reading ...

then you should check if the value you want to extract is read into the internal with the name buf. Internals are visible when you click on the defined HTTPMOD Device. buf is an internal variable that contains the HTTP Response read. If the value is there and you get the mentioned message then probably something is wrong with your regular expression. Please note that buf might contain special characters like newlines but they are not shown in fhemweb. If you are new to regular expressions then the introduction at http://perldoc.perl.org/perlretut.html might be helpful.

For a typical HTTPMOD use case where you want to extract a number out of a HTTP-Response you can use something like [\d\.]+ to match the number itself. The expression matches the number characters (\d) or a . if one of these characters occurs at least once.

To tell HTTPMOD that the number is what you want to use for the reading, you have to put the expression in between (). A ([\d\.]+) alone would match the longest number in the HTTP Response which is very likely not the number you are looking for so you need to add something to the expression to give it a context and define how to find the number that you are looking for.

If there is a title text before the number or a special text after the number you can put this in the regex. In one of the examples above humidity':([\d\.]+) is looking for the number that immediately follows the text humidity': without any blanks in between. Be careful if the text you are getting from your device contains special characters like newline. You don't see such special characters in the fhem webinterface as contents of the internal buf but they might cause your regular expression to fail.

If you have trouble defining a regular expression that matches a certain name, then many complicated characters and then a number, it might be helpful to use a negation in matching like temp[^\d]+([\d\.]).*. In this examle [^\d]+ means any character that is not a numerical digit, more than once.

Regular Expressions with multiple capture Groups

The regular expressions used in the above example for a Poolmanager will take the value that matches one capture group. This is the part of the regular expression inside (). In the above example "([\d\.]+)" refers to numerical digits or points between double quotation marks. Only the string consiting of digits and points will match inside (). This piece is assigned to the reading.

You can also use regular expressions that have several capture groups which might be helpful when parsing tables. In this case an attribute like

reading02Regex something[ \t]+([\d\.]+)[ \t]+([\d\.]+)

could match two numbers. When you specify only one reading02Name like

reading02Name Temp

the name Temp will be used with the extension -1 and -2 thus giving a reading Temp-1 for the first number and Temp-2 for the second. You can also specify individual names for several readings that get parsed from one regular expression with several capture groups by defining attributes

reading02-1Name
reading02-2Name
...

The same notation can be used for formatting attributes like readingXOMap, readingXFormat and so on.

The usual way to define readings is however to have an individual regular expression with just one capture group per reading as shown in the above example.

Parsing JSON

If a webservice delivers data in JSON format, HTTPMOD can directly parse JSON which might be easier in this case than definig regular expressions. The next example shows the data that can be requested from a Poolmanager with the following partial configuration:

define test2 HTTPMOD none 0
attr test2 get01Name Chlor
attr test2 getURL http://192.168.70.90/cgi-bin/webgui.fcgi
attr test2 getHeader1 Content-Type: application/json
attr test2 getHeader2 Accept: */*
attr test2 getData {"get" :["34.4008.value"]}

The data in the HTTP response looks like this:

{
	"data": {
			"34.4008.value": "0.25"
	},
	"status":       {
			"code": 0
	},
	"event":        {
			"type": 1,
			"data": "48.30000.0"
	}
}

the classic way to extract the value 0.25 into a reading with the name Chlor with a regex would have been

attr test2 get01Regex 34.4008.value":[ \t]+"([\d\.]+)"

with JSON you can write

attr test2 get01JSON data_34.4008.value

which will create a reading with the Name "Chlor" (as shown above) and take the value 0.25 from the JSON string.

or if you don't care about the naming of your readings, you can simply extract all JSON data with

attr test2 extractAllJSON

which would apply to all data read from this device and create the following readings out of the HTTP response shown above:

data_34.4008.value	0.25
event_data	48.30000.0
event_type	1
status_code	0

or you can specify

attr test2 get01ExtractAllJSON

which would only apply to all data read as response to the get command defined as get01.

It might seem very simple at first sight to use extractAllJSON but if you prefer readings with a meaningful name you should instead define these readings with readingXXName and readingXXJSON or getXXName and getXXJSON individually. Of Course it would be possible to create additional user readings outside HTTPMOD but doing calculations, naming and formatting inside HTTPMOD is more efficient.

JSON Lists

imagine the HTTP Response contains:

{ "power":"0",
  "modes":["Off","SimpleColor","RainbowChase"],
  "code1":3,
  "code2":4
 }

then a configuration like

attr device reading01JSON modes 
attr device reading01Name Mode

will create a list of Subreadings just like a regex with multiple matches can create multiple subreadings:

Mode-1	Off
Mode-2	SimpleColor
Mode-3	RainbowChase

if you don't want several subreadings but one reading that contains the list of modes, you can specify a recombine expression:

attr device reading01Name Modes 
attr device reading01RecombineExpr join ",", @matchlist

which will create one reading containing a list:

Modes	Off,SimpleColor,RainbowChase

JSON parsing specifications also don't Need to match exactly. If there is no exact match for a defined reading, the HTTPMOD will try to Interpret the specification as a regex and look for json object paths that match the specification as a regex. For example:

attr device reading01Name CodeElem
attr device reading01JSON code

which will create a list of readings:

CodeElem-1	3
CodeElem-2	4

and of course they could also be recombined into one reading with a RecombineExpr Attribute.

Parsing http / XML using xpath

Another alternative to regex parsing is the use of XPath to extract values from HTTP responses. The following example shows how XML data can be parsed with XPath-Strict or HTML Data can be parsed with XPath. Both work similar and the example uses XML Data parsed with the XPath-Strict option:

If The XML data in the HTTP response looks like this:

<root xmlns:foo="http://www.foo.org/" xmlns:bar="http://www.bar.org">
	<actors>
		<actor id="1">Peter X</actor>
		<actor id="2">Charles Y</actor>
		<actor id="3">John Doe</actor>
	</actor>
</root>

with XPath you can write

attr htest reading01Name Actor
attr htest reading01XPath-Strict //actor[2]/text()

This will create a reading with the Name "Actor" and the value "Charles Y".

Since XPath specifications can define several values / matches, HTTPMOD can also interpret these and store them in multiple readings:

attr htest reading01Name Actor
attr htest reading01XPath-Strict //actor/text()

will create the readings

Actor-1	Peter X
Actor-2	Charles Y
Actor-3	John Doe

Further replacements of URL, header or post data

sometimes it might be helpful to dynamically change parts of a URL, HTTP header or post data depending on existing readings, internals or perl expressions at runtime. This might be needed to pass further variables to a server, a current date or other things.

To support this HTTPMOD offers generic replacements that are applied to a request before it is sent to the server. A replacement can be defined with the attributes

replacement[0-9]*Regex
replacement[0-9]*Mode
replacement[0-9]*Value
[gs]et[0-9]*Replacement[0-9]*Value

a replacement always replaces a match of a regular expression.

replacement[0-9]*Mode: The way the replacement value is defined can be specified with the replacement mode.

If the replacement[0-9]*Mode is reading, then the corresponding replacement[0-9]*Value is interpreted as the name of a reading of the same device or as device:reading to refer to another device.
If the replacement[0-9]*Mode is internal, then the corresponding replacement[0-9]*Value is interpreted as the name of an internal of the same device or as device:internal to refer to another device.
If the replacement[0-9]*Mode is text, then the corresponding replacement[0-9]*Value is interpreted as a static text
If the replacement[0-9]*Mode is expression, then the corresponding replacement[0-9]*Value is evaluated as a perl expression to compute the replacement. Inside such a replacement expression it is possible to refer to capture groups of the replacement regex.
If the replacement[0-9]*Mode is key, then the module will use a value from a key / value pair that is stored in an obfuscated form in the file system with the set storeKeyValue command. This might be useful for storing passwords.

Example:

attr mydevice getData {"get" :["%%value%%.value"]}
attr mydevice replacement01Mode text
attr mydevice replacement01Regex %%value%%

attr mydevice get01Name Chlor
attr mydevice get01Replacement01Value 34.4008

attr mydevice get02Name Something
attr mydevice get02Replacement01Value 31.4024

attr mydevice get05Name profile
attr mydevice get05URL http://www.mydevice.local/getprofile?password=%%password%%
attr mydevice replacement02Mode key
attr mydevice replacement02Regex %%password%%
attr mydevice replacement02Value password

defines that %%value%% will be replaced by a static text.

All Get commands will be HTTP post requests of a similar form. Only the %%value%% will be different from get to get. The first get will set the reading named Chlor and for the request it will take the generic getData and replace %%value%% with 34.4008.

A second get will look the same except a different name and replacement value.

With the command set mydevice storeKeyValue password geheim you can store the password geheim in an obfuscated form in the file system. To use this password and send it in a request you can use the above replacement with mode key. The value password will then refer to the ofuscated string stored with the key password.

The mode expression allows you to define your own replacement syntax:

        
attr mydevice replacement01Mode expression
attr mydevice replacement01Regex {{([^}]+)}}
attr mydevice replacement01Value ReadingsVal("mydevice", $1, "")
attr mydevice getData {"get" :["{{temp}}.value"]}

In this example any {{name}} in a URL, header or post data will be passed on to the perl function ReadingsVal which uses the string between {{}} as second parameter. This way one defined replacement can be used for many different readings.

HTTPMOD has two built in replacements: One for session Ids and another one for the input value in a set command. The placeholder $sid is always replaced with the internal $hash->{sid} which contains the session id after it is extracted from a previous HTTP response. If you don't like to use the placeholder $sid then you can define your own replacement for example like:

attr mydevice replacement01Mode internal
attr mydevice replacement01Regex %session%
attr mydevice replacement01Value sid

Now the internal $hash->{sid} will be used as a replacement for the placeholder %session%.

In the same way a value that is passed to a set-command can be put into a request with a user defined replacement. In this case the internal $hash->{value} will contain the value passed to the set command. $hash->{value} might even be a string containing several values that could be put into several different positions in a request by using user defined replacements.

Other example : steering a pellet stove from Rika

The stove API of Rika on https://www.rika-firenet.com/web/ delivers a JSON string with all settings and values, and can be piloted with a data string containing all the "set" values at once.

Delivered JSON on get https://www.rika-firenet.com/api/client/xxxxxxxx/status : (xxxxxxxx must be replaced with the unique stove ID)

{
    "name": "Vorzimmer",
    "stoveID": "xxxxxxxxx",
    "lastSeenMinutes": 1,
    "lastConfirmedRevision": 1504385700,
    "controls": {
        "revision": 1504385700,
        "onOff": true,
        "operatingMode": 2,
        "heatingPower": 65,
        "targetTemperature": 24,
        "heatingTimesActive": false,
        "heatingTimesActiveForComfort": true,
        "setBackTemperature": 18,
        "convectionFan1Active": false,
        "convectionFan1Level": 0,
        "convectionFan1Area": 0,
        "convectionFan2Active": false,
        "convectionFan2Level": 0,
        "convectionFan2Area": 0,
        "frostProtectionActive": false,
        "frostProtectionTemperature": 5
    },
    "sensors": {
        "statusError": 0,
        "statusWarning": 0,
        "statusService": 0,
        "statusMainState": 1,
        "statusSubState": 3,
        "statusFrostStarted": false,
        "inputFlameTemperature": 21,
        "inputRoomTemperature": 21,
        "inputExternalRequest": true,
        "outputDischargeMotor": 0,
        "outputInsertionMotor": 0,
        "outputIDFan": 0,
        "outputAirFlaps": 0,
        "outputIgnition": false,
        "parameterStoveTypeNumber": 13,
        "parameterVersionMainBoard": 223,
        "parameterVersionTFT": 223,
        "parameterRuntimePellets": 11,
        "parameterRuntimeLogs": 0,
        "parameterFeedRateTotal": 17,
        "parameterFeedRateService": 683,
        "parameterOnOffCycles": 2
    },
    "stoveType": "DOMO MultiAir",
    "stoveFeatures": {
        "multiAir1": true,
        "multiAir2": true,
        "insertionMotor": false,
        "airFlaps": false,
        "logRuntime": false
    }
}

Data string to send to https://www.rika-firenet.com/api/client/xxxxxxxx/controls in order to set values:

heatingTimesActiveForComfort=true&frostProtectionTemperature=3&setBackTemperature=18&targetTemperature=24&convectionFan2Level=0&convectionFan2Active=false&convectionFan1Level=0&onOff=true&convectionFan1Active=false&convectionFan2Area=0&revision=1505550101&heatingTimesActive=false&convectionFan1Area=0&frostProtectionActive=false&operatingMode=2&heatingPower=65

Code in 99_myUtils.pm:

use JSON;
...
sub
replaceJSON ($$) {
   my ($valToReplace, $value) = @_;

   #$value in the parameters is a default value
   #It has to be replaced through the real value nnn passed in the set command "set <device> valToset nnn"
   $value = InternalVal("Ofen", "value", $value);
   Log3 ("Ofen", 3, "replaceJSON Internalvalue: $value");

   #Force an update to avoid outdated revision number
   fhem ("get Ofen revision");

   #Get all the controls as json
   my $json = ReadingsVal("Ofen", "controlsJSON","");
   Log3 ("Ofen", 3, "replaceJSON configsJSON: $json");

   # When starting FHEM or rereading config, the reading controlsJSON is empty
   return if ($json eq ""); 

   my $decoded = decode_json($json);
   my $result;
   for my $key ( keys %$decoded ) {
      $result .= "$key=";
      if ($key eq $valToReplace) {
         $result .= $value."&";
      } else {
         $result .= $decoded->{$key}."&";
      }
   }
   chop($result); #remove last &
   Log3("Ofen", 3, "replaceJSON Result: $result");
   return $result;
}

Define stove in fhem:

defmod Ofen HTTPMOD https://www.rika-firenet.com/api/client/xxxxxxxx/status 60

attr Ofen enableCookies 1
attr Ofen reAuthRegex id="login"|Unauthorized
attr Ofen sid01Data email=xx@xx&password=xx
attr Ofen sid01URL https://www.rika-firenet.com/web/login

attr Ofen reading01JSON sensors_inputRoomTemperature
attr Ofen reading01Name RaumTemp
attr Ofen reading02JSON controls_setBackTemperature
attr Ofen reading02Name Absenkung
attr Ofen reading03JSON controls_frostProtectionTemperature
attr Ofen reading03Name Frostschutz
attr Ofen reading10Name controlsJSON
attr Ofen reading10Regex (?s)controls.*?({.*?})

attr Ofen get09Name revision
attr Ofen get09URL https://www.rika-firenet.com/api/client/xxxxxxxx/status

attr Ofen setURL https://www.rika-firenet.com/api/client/xxxxxxxx/controls
attr Ofen setData {{data}}
attr Ofen replacement01Mode expression
attr Ofen replacement01Regex {{data}}

attr Ofen set11Name frostProtectionTemperature
attr Ofen set11Replacement01Value replaceJSON("frostProtectionTemperature", 2)

attr Ofen set12Name targetTemperature
attr Ofen set12Replacement01Value replaceJSON("targetTemperature", 24)

A detailed explanation (in german) of the login process can be found here: [1] and the explanation of the other parameters here: [2]

replacing reading values when they have not been updated / the device did not respond

If a device does not respond then the values stored in readings will keep the same and only their timestamp shows that they are outdated. If you want to modify reading values that have not been updated for a number of seconds, you can use the attributes

(reading|get)[0-9]*(-[0-9]+)?MaxAge
(reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacementMode
(reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacement

Every time the module tries to read from a device, it will also check if readings have not been updated for longer than the MaxAge attributes allow. If readings are outdated, the MaxAgeReplacementMode defines how the affected reading values should be replaced. MaxAgeReplacementMode can be text, reading, internal, expression or delete.

MaxAge specifies the number of seconds that a reading should remain untouched before it is replaced.

MaxAgeReplacement contains either a static text that is used as replacement value or a Perl expression that is evaluated to give the replacement value. This can be used for example to replace a temperature that has not bee updated for more than 5 minutes with the string "outdated - was 12":

attr PM readingMaxAge 300
attr PM readingMaxAgeReplacement "outdated - was " . $val
attr PM readingMaxAgeReplacementMode expression

The variable $val contains the value of the reading before it became outdated.

Or to show that a device was offline:

attr MyLight reading01Name color
attr MyLight reading01JSON result_02_color
attr MyLight reading01MaxAge 300
attr MyLight reading01MaxAgeReplacement "offline"
attr MyLight reading01MaxAgeReplacementMode text

Note on determining how to send requests to a special device

If you don't know which URLs, headers or POST data your web GUI uses, you might try a local proxy like BurpSuite BurpSuite to track requests and responses. This is a tedious task but probably the best way to achieve a successful result.

Let us consider an example. The Telekom Speedport W724V has a login-site that is famous for being cumbersome. Burp allows to monitor each step in the login procedure. In the case of a speedport the following steps occur:

First burp shows that a get command is issued

################################################################################################## 
GET / HTTP/1.1
Host: speedport.ip
Cache-Control: max-age=0
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4
Cookie: lang=de
Connection: close

In order to mimic the behavior of a real person calling the website HTTPMOD should copy all necessary steps. Host, Cookie and the GET-command are usually necessary. The same cannot be said of the User-Agent because the website can be called from any mobile or desktop computer.

Then, the speedport will answer with a command that consists of several lines. By going through every line for every step in the login procedure one will finally arrive at the information that is necessary to successfully enter the login of the speedport (in case of the W724V, for example, it is necessary to copy a token called _httoken and to include the referer).

Advanced configuration to define a `set` command and send data to a device

When a set option is defined by attributes, the module will use the value given to the set command and integrate it into an HTTP-Request that sends the value to the device. The definitions for URL, headers and post data can contain the placeholder $val which will be replaced by the (numerical) value given to the set command.

This can be as simple as:

# No cyclic requests and no main URL needed in this example
define MyDevice HTTPMOD none 0

attr MyDevice set01Name Licht
attr MyDevice set01URL http://192.168.1.22/switch=$val

A user command

set MyDevice Licht 1

will be translated into the http GET request

http://192.168.1.22/switch=1

In this example a map would also be helpful, that translates on / off to 0 or 1 and allows the user to select on/of in fhemweb:

attr MyDevive set01IMap 0:off, 1:on

This also provides input validation to make sure that only on and off can be used with the set command.

In more complex Scenarios you might need to login before sending a command and the Login might create a session id that has to be part of further requests either in the URL, in headers or in the post data.

Extension to the above example for a PoolManager 5 where a set needs a session id in the URL and the values have to be passed in JSON strings as post data:

attr PM set01Name HeizungSoll
attr PM set01URL http://MyPoolManager/cgi-bin/webgui.fcgi?sid=$sid
attr PM set01Hint 6,10,20,30
attr PM set01Min 6
attr PM set01Max 30
attr PM setHeader1 Content-Type: application/json
attr PM set01Data {"set" :{"34.3118.value" :"$val" }}

This example defines a set option with the name HeizungSoll. By issuing set PM HeizungSoll 10 in FHEM, the value 10 will be sent in the defined HTTP Post to URL http://MyPoolManager/cgi-bin/webgui.fcgi in the Post Data as

{"set" :{"34.3118.value" :"10" }}

The optional attributes set01Min and set01Max define input validations that will be checked in the set function. The optional attribute set01Hint will define a selection list for the FHEMweb GUI.

The HTTP response to such a request will be ignored unless you specify the attribute setParseResponse for all set commands or set01ParseResponse for the set command with number 01. If the HTTP response to a set command is parsed then this is done like the parsing of responses to get commands and you can use the attributes ending e.g. on Format, Encode, Decode, OMap and OExpr to manipulate / format the values read.

If a parameter to a set command is not numeric but should be passed on to the device as text, then you can specify the attribute setTextArg. For example:

attr PM set01TextArg

If a set command should not require a parameter at all, then you can specify the attribute NoArg. For example:

attr PM set03Name On
attr PM set03NoArg

This example defines the set of parameters for Wiesemann&Theiss Web-IO Analog 4.0 #57761. The current limit is set to 16mA (setxxMax)

defmod WuT_Analog1 HTTPMOD none 30
attr WuT_Analog1 room W&T
attr WuT_Analog1 set01Max 16
attr WuT_Analog1 set01Min 0
attr WuT_Analog1 set01Name Output1
attr WuT_Analog1 set01URL http://192.xx.My_IP/outputaccess1?PW=MyPassword&State=$val&
attr WuT_Analog1 set02Max 16
attr WuT_Analog1 set02Min 0
attr WuT_Analog1 set02Name Output2
attr WuT_Analog1 set02URL http://192.xx.My_IP/outputaccess2?PW=MyPassword&State=$val&

Advanced configuration to create a valid session id that might be necessary

In simple cases logging in works with basic authentication. In the case HTTPMOD accepts a username and password as part of the URL in the form

http://User:Password@192.168.1.18/something

However basic auth is seldom used. If you need to fill in a username and password in a HTML form and the session is then managed by a session id, here is how to configure this:

when sending data to an HTTP-Device in a set, HTTPMOD will replace any $sid in the URL, Headers and Post data with the internal $hash->{sid}. To authenticate towards the device and give this internal a value, you can use an optional multi step login procedure defined by the following attributes:

sid[0-9]*URL
sid[0-9]*Data.*
sid[0-9]*Header.*
sid[0-9]*IgnoreRedirects
idRegex
idJSON
idXPath
idXPath-Strict
(get|set|sid)[0-9]*IdRegex
(get|set|sid)[0-9]*IdJSON
(get|set|sid)[0-9]*IdXPath
(get|set|sid)[0-9]*IdXPath-Strict

Each step can have a URL, Headers, Post Data pieces and a Regex to extract a resulting Session ID into $hash->{sid}. HTTPMOD will create a sorted list of steps (the numbers between sid and URL / Data / Header) and the loop through these steps and send the corresponding requests to the device. For each step a $sid in a Header or Post Data will be replaced with the current content of $hash->{sid}.

Using this feature, HTTPMOD can perform a forms based authentication and send user name, password or other necessary data to the device and save the session id for further requests.

To determine when this login procedure is necessary, HTTPMOD will first try to send a request without doing the login procedure. If the result contains an error that authentication is necessary, then a login is performed. To detect such an error in the HTTP response, you can again use a regular expression, JSON or XPath, this time with the attributes

reAuthRegex
reAuthJSON
reAuthXPath
reAuthXPath-Strict
(get|set)[0-9]*ReAuthRegex
(get|set)[0-9]*ReAuthJSON
(get|set)[0-9]*ReAuthXPath
(get|set)[0-9]*ReAuthXPath-Strict

reAuthJSON or reAuthXPath typically only extract one piece of data from a response. If the existence of the specified piece of data is sufficent to start a login procedure, then nothing more needs to be defined to detect this situation. If however the indicator is a status code that contains different values depending on a successful request and a failed request if a new authentication is needed, then you can combine things like reAuthJSON with reAuthRegex. In this case the regex is only matched to the data extracted by JSON (or XPath). This way you can easily extract the status code using JSON parsing and then specify the code that means "authentication needed" as a regular expression.

If for one step not all of the URL, Data or Header Attributes are set, then HTTPMOD tries to use a sidURL, sidData.* or sidHeader.* Attribute (without the step number after sid). This way parts that are the same for all steps don't need to be defined redundantly.

Example for a multi step login procedure:

attr PM reAuthRegex /html/dummy_login.htm 
attr PM sidURL http://192.168.70.90/cgi-bin/webgui.fcgi?sid=$sid
attr PM sidHeader1 Content-Type: application/json
attr PM sid1IDRegex wui.init\('([^']+)'
attr PM sid2Data {"set" :{"9.17401.user" :"fhem" ,"9.17401.pass" :"password" }}
attr PM sid3Data {"set" :{"35.5062.value" :"128" }}
attr PM sid4Data {"set" :{"42.8026.code" :"pincode" }}

In this case HTTPMOD detects that a login is necessary by looking for the pattern /html/dummy_login.htm in the HTTP response. If it matches, it starts a login sequence. In the above example all steps request the same URL. In step 1 only the defined Header is sent in an HTTP get request. The response will contain a session id that is extraced with the regex wui.init\('([^']+)'.

In the next step this session id is sent in a post request to the same URL where tha post data contains a username and password. The a third and a fourth request follow that set a value and a code. The result will be a valid and authorized session id that can be used in other requests where $sid is part of a URL, header or post data and will be replaced with the session id extracted above.

Another Practical Example reading the SOC from a Renault Zoe

define myZoe HTTPMOD https://www.services.renault-ze.com/api/vehicle/VINVINVIN/battery 7000
attr myZoe userattr reading01JSON reading01Name reading02JSON reading02Name reading03JSON reading03Nam\
e reading04JSON reading04Name reading05JSON reading05Name requestHeader02 sid1Data sid1Header01 sid1Id\
JSON sid1IgnoreRedirects:0,1 sid1URL
attr myZoe reAuthJSON message
attr myZoe reading01JSON charge_level
attr myZoe reading01Name SOC
attr myZoe reading02JSON charging
attr myZoe reading02Name charging
attr myZoe reading03JSON plugged
attr myZoe reading03Name plugged
attr myZoe reading04JSON remaining_range
attr myZoe reading04Name remaining_range
attr myZoe reading05JSON last_update
attr myZoe reading05Name last_update
attr myZoe requestHeader02 Authorization: Bearer $sid
attr myZoe sid1Data {"username":"ZEUSERNAME","password":"ZEPASSWORD"}
attr myZoe sid1Header01 Content-Type: application/json
attr myZoe sid1IdJSON token
attr myZoe sid1IgnoreRedirects 1
attr myZoe sid1URL https://www.services.renault-ze.com/api/user/login
attr myZoe timeout 10
#attr myZoe verbose 7

The $sid is read by logging in with username and password from JSON {"token": ..} after detecting an error {"message": ..} from a failed request. This example offers no solution for the reading of the VIN from the login request, which in turn is part of the reading URL. It should be possible to read more than just the $sid from the login sequence.

In the special case where a session id is set as a HTTP-Cookie (with the header Set-cookie: in the HTTP response) HTTPMOD offers an even simpler way. With the attribute enableCookies a very basic cookie handling mechanism is activated that stores all cookies that the server sends to the HTTPMOD device and puts them back as cookie headers in the following requests.

For such cases no sidIdRegex and no $sid in a user defined header is necessary.

Advanced configuration to define a `get` and request additional data with its own request from a device

The normal automatic HTTP request that is done repeatedly after the defined interval has elapsed works well in cases where all required readings can be requested in one common HTTP request. If however a device needs individual requests with different URLs or different POST data for each value, then another method is necessary. For such cases a get option can be defined and the user can either issue FHEM get commands each time he needs the reading or the user can set an attribute to request the reading automatically together with the normal iteration. For each get option attributes define an individual URL, optional headers, and post data as well as individual regular expressions and formatting options.

When a get option is defined by attributes, the module allows querying additional values from the device that require individual HTTP-Requests or special parameters to be sent

Extension to the above example:

attr PM get01Name MyGetValue 
attr PM get01URL http://MyPoolManager/cgi-bin/directory/webgui.fcgi?special=1?sid=$sid 
attr PM getHeader1 Content-Type: application/json 
attr PM get01Data {"get" :{"30.1234.value"}}

This example defines a get option with the name MyGetValue. By issuing get PM MyGetValue in FHEM, the defined HTTP request is sent to the device. The HTTP response is then parsed using the same readingXXName and readingXXRegex attributes as above so additional pairs will probably be needed there for additional values.

if you prefer to define the parsing and formatting of readings individually per get command, you can use attributes like get01Regex, get01XPath, get01Format, get01OMap and so on just like for reading01...

You can also include parameters / values that are passed to the get command in the request just like for set commands. The placeholder $val will be replaced with the value given to the get command or you can specify your own replacement as described above.

If the new get parameter should also be queried regularly, you can define the following optional attributes:

attr PM get01Poll 1
attr PM get01PollDelay 300

The first attribute includes this reading in the automatic update cycle and the second defines an alternative lower update frequency. When the interval defined initially in the define is over and the normal readings are read from the device, the update function will check for additional get parameters that should be included in the update cycle.

If a PollDelay is specified for a get parameter, the update function also checks if the time passed since it has last read this value is more than the given PollDelay. If not, this reading is skipped and it will be rechecked in the next cycle when interval is over again. So the effective PollDelay will always be a multiple of the interval specified in the initial define.

Please note that each defined get command that is included in the regular update cycle will create its own HTTP request. So if you want to extract several values from the same request, it is much more efficient to do this by defining readingXXName and readingXXRegex, XPath or JSON attributes and to specify an interval and a URL in the define of the HTTPMOD device.

Example for a Siemens webserver provided by Lanhydrock:

define ozw672 HTTPMOD https://192.168.178.8/api/auth/login.json?user=test&pwd=test 300

attr ozw672 get1Name tempAussen
attr ozw672 get1URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1960
attr ozw672 get1Poll 1
attr ozw672 get1PollDelay 1800

attr ozw672 get2Name tempAussenGemischt
attr ozw672 get2URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1964
attr ozw672 get2Poll 1
attr ozw672 get2PollDelay 1800

attr ozw672 get3Name tempTWW
attr ozw672 get3URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1996
attr ozw672 get3Poll 1

attr ozw672 get4Name tempKesselSoll
attr ozw672 get4URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1910
attr ozw672 get4Poll 1

attr ozw672 get5Name tempKesselRuecklauf
attr ozw672 get5URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1915
attr ozw672 get5Poll 1

attr ozw672 get6Name tempKesselRuecklaufSoll
attr ozw672 get6URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1916
attr ozw672 get6Poll 1

attr ozw672 get7Name anzahlStartsBrenner
attr ozw672 get7URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1927
attr ozw672 get7PollDelay 1800
attr ozw672 get7Poll 1

attr ozw672 get8Name statusKessel
attr ozw672 get8URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1898
attr ozw672 get8Poll 1
attr ozw672 get8Regex Value": "([a-zA-Zü ]*)"
attr ozw672 get8OMap Aus:0, Nachlauf aktiv:5, Freigegeben für TWW:10, Freigegeben für HK:20, In Teillastbetrieb für TWW:40, In Teillastbetrieb für HK:50, In Betrieb für Trinkwasser:90, In Betrieb für Heizkreis:100

attr ozw672 getRegex Value": "[ ]*([-.0-9]*)"

attr ozw672 reAuthRegex .*session not valid.*
attr ozw672 sid1IDRegex .*"(.*-.*-.*-[0-9a-z]*).*
attr ozw672 sid1URL https://192.168.178.8/api/auth/login.json?user=test&pwd=test

All attributes

reading[0-9]+Name: the name of a reading to extract with the corresponding readingRegex, readingJSON, readingXPath or readingXPath-Strict; Please note that the old syntax readingsName.* does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.

(get|set)[0-9]+Name: Name of a get or set command; If the HTTP response that is received after the command is parsed with an individual parse option then this name is also used as a reading name. Please note that no individual parsing needs to be defined for a get or set. If no regex, XPath or JSON is specified for the command, then HTTPMOD will try to parse the response using all the defined readingRegex, reading XPath or readingJSON attributes.

(get|set|reading)[0-9]+Regex: If this attribute is specified, the Regex defined here is used to extract the value from the HTTP Response and assign it to a Reading with the name defined in the (get|set|reading)[0-9]+Name attribute.; If this attribute is not specified for an individual Reading or get or set but without the numbers in the middle, e.g. as getRegex or readingRegex, then it applies to all the other readings / get / set commands where no specific Regex is defined.; The value to extract should be in a capture group / sub expression e.g. ([\d\.]+) in the above example. Multiple capture groups will create multiple readings (see explanation above); Using this attribute for a set command (setXXRegex) only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.; Please note that the old syntax readingsRegex.* does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.; If for get or set commands neither a generic Regex attribute without numbers nor a specific (get|set)[0-9]+Regex attribute is specified and also no XPath or JSON parsing specification is given for the get or set command, then HTTPMOD tries to use the parsing definitions for general readings defined in reading[0-9]+Name, reading[0-9]+Regex or XPath or JSON attributes and assigns the Readings that match here.

(get|set|reading)[0-9]+RegOpt: Lets the user specify regular expression modifiers. For example if the same regular expression should be matched as often as possible in the HTTP response, then you can specify RegOpt g which will case the matching to be done as /regex/g; The results will be trated the same way as multiple capture groups so the reading name will be extended with -number.; For other possible regular expression modifiers see http://perldoc.perl.org/perlre.html#Modifiers

(get|set|reading)[0-9]+XPath: defines an xpath to one or more values when parsing HTML data (see examples above); Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.

get|set|reading[0-9]+XPath-Strict: defines an xpath to one or more values when parsing XML data (see examples above); Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.

(get|set|reading)[0-9]+AutoNumLen: In cases where a regular expression or an XPath results in multiple results and these results are stored in a common reading name with extension -number, then you can modify the format of this number to have a fixed length with leading zeros. AutoNumLen 3 for example will lead to reading names ending with -001 -002 and so on.

(get|set|reading)[0-9]+AlwaysNum: if set to 1 this attributes forces reading names to end with a -1, -01 (depending on the above described AutoNumLen) even if just one value is parsed.

get|set|reading[0-9]+JSON: defines a path to the JSON object wanted by concatenating the object names. See the above example.; If you don't know the paths, then start by using extractAllJSON and the use the names of the readings as values for the JSON attribute.; Please don't forget to also specify a name for a reading, get or set.; Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.

(get|set|reading)[0-9]*RecombineExpr: defines an expression that is used in an eval to compute one reading value out of the list of matches.; This is supposed to be used for regexes or xpath specifications that produce multiple results if only one result that combines them is wanted. The list of matches will be in the variable @matchlist.; Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.

get[0-9]*CheckAllReadings: this attribute modifies the behavior of HTTPMOD when the HTTP Response of a get command is parsed.; If this attribute is set to 1, then additionally to the matching of the corresponding get specific regex (get[0-9]*Regex), XPath or JSON attribute also all the reading names and parse definitions defined in Reading[0-9]+Name and Reading[0-9]+Regex, XPath or JSON attributes are checked and if they match, the coresponding Readings are assigned as well.; Please note that this does not mean that get01CheckAllReadings will cause a get02Regex to be used. Only the corresponding get01Regex will be used but additionally all the readingXYRegex attributes.; This is automatically done if a get or set command is defined without its own parse attributes.

(get|reading)[0-9]*OExpr: defines an optional expression that is used in an eval to compute / format a readings value after parsing an HTTP response; The raw value from the parsing will be in the variable $val.; If specified as readingOExpr then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Expr.; Please note that the old syntax readingsExpr.* does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.

(get|reading)[0-9]*Expr: This is the old syntax for (get|reading)[0-9]*OExpr. It should be replaced by (get|reading)[0-9]*OExpr. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.

(get|reading)[0-9]*OMap: Map that defines a mapping from raw value parsed to visible values like "0:mittig, 1:oberhalb, 2:unterhalb".; If specified as readingOMap then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Map.; The individual options in a map are separated by a komma and an optional space. Spaces are allowed to appear in a visible value however kommas are not possible.

(get|reading)[0-9]*Map: This is the old syntax for (get|reading)[0-9]*OMap. It should be replaced by (get|reading)[0-9]*OMap. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.

(get|set|reading)[0-9]*Format: Defines a format string that will be used in sprintf to format a reading value.; If specified without the numbers in the middle e.g. as readingFormat then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Format.; Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.

(get|set|reading)[0-9]*Decode: defines an encoding to be used in a call to the perl function decode to convert the raw data string read from the device to a reading.; This can be used if the device delivers strings in an encoding like cp850 instead of utf8.; If your reading values contain Umlauts and they are shown as strange looking icons then you probably need to use this feature.; Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.

(get|set|reading)[0-9]*Encode: defines an encoding to be used in a call to the perl function encode to convert the raw data string read from the device to a reading.; This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.; If your reading values contain Umlauts and they are shown as strange looking icons then you probably need to use this feature.; Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.

(get|set)[0-9]*URL: URL to be requested for the set or get command. If this option is missing, the URL specified during define will be used.

(get|set)[0-9]*Data: Data to be sent to the device as POST data when the get or set command is executed. if this attribute is not specified, an HTTP GET method will be used instead of an HTTP POST

(get|set)[0-9]*NoData: can be used to override a more generic attribute that specifies POST data for all get or set commands. With NoData no data is sent and therefor the request will be an HTTP GET.

(get|set)[0-9]*Header.*: HTTP Headers to be sent to the device when the set is executed

requestHeader.*: Define an optional additional HTTP Header to set in the HTTP request of the main loop

requestData: optional POST Data to be sent in the request of the main loop. If not defined, it will be an HTTP GET request as defined in HttpUtils which is used by this module

get[0-9]+Poll: if set to 1 the get is executed automatically during the normal update cycle (after the interval provided in the define command has elapsed)

get[0-9]+PollDelay: if the value should not be read in each iteration (after the interval given to the define command), then a minimum delay can be specified with this attribute. This has only an effect if the above Poll attribute has also been set. Every time the update function is called, it checks if since this get has been read the last time, the defined delay has elapsed. If not, then it is skipped this time.; PollDelay can be specified as seconds or as x[0-9]+ which means a multiple of the interval in the define command.

(get|set)[0-9]*TextArg: For a get command this defines that the command accepts a text value after the option name. By default a get command doesn't accept optional values after the command name.; If TextArg is specified and a value is passed after the get name then this value can then be used in a request URL, header or data as replacement for $val or in a user defined replacement that uses the internal "value" ($hash->{value}).; If used for a set command then it defines that the value to be set doesn't require any validation / conversion.; The raw value is passed on as text to the device. By default a set command expects a numerical value or a text value that is converted to a numeric value using a map.

set[0-9]+Min: Minimum value for input validation.

set[0-9]+Max: Maximum value for input validation.

set[0-9]+IExpr: Perl Expression to compute the raw value to be sent to the device from the input value passed to the set.

set[0-9]+Expr: This is the old syntax for (get|reading)[0-9]*IExpr. It should be replaced by (get|reading)[0-9]*IExpr. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.

set[0-9]+IMap: Map that defines a mapping from raw to input values like "0:mittig, 1:oberhalb, 2:unterhalb". This attribute atomatically creates a hint for FHEMWEB so the user can choose one of the input values.

set[0-9]+Map: This is the old syntax for (get|reading)[0-9]*IMap. It should be replaced by (get|reading)[0-9]*IMap. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.

set[0-9]+Hint: Explicit hint for fhemWEB that will be returned when set ? is seen. Can be used to get a slider or a list of values to choose from.

set[0-9]*NoArg: Defines that this set option doesn't require arguments. It allows sets like "on" or "off" without further values.

set[0-9]*ParseResponse: defines that the HTTP response to the set will be parsed as if it was the response to a get command.

(get|set)[0-9]*URLExpr: Defines a Perl expression to specify the HTTP Headers for this request. This overwrites any other header specification and should be used carefully only if needed. The original Header is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service.

(get|set)[0-9]*DatExpr: Defines a Perl expression to specify the HTTP Post data for this request. This overwrites any other post data specification and should be used carefully only if needed. The original Data is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service.

(get|set)[0-9]*HdrExpr: Defines a Perl expression to specify the URL for this request. This overwrites any other URL specification and should be used carefully only if needed. The original URL is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service.

ReAuthRegex: regular Expression to match an error page indicating that a session has expired and a new authentication for read access needs to be done.; This attribute only makes sense if you need a forms based authentication for reading data and if you specify a multi step login procedure based on the sid.. attributes.; This attribute is used for all requests. For set and get operations you can however specify individual reAuthRegexes with the (get|set)[0-9]*ReAuthRegex attributes.

(get|set)[0-9]*ReAuthRegex: Regex that will detect when a session has expired during a set operation and a new login needs to be performed.; It works like the global reAuthRegex but is used for set operations.

sid[0-9]*URL: different URLs or one common URL to be used for each step of an optional login procedure.

sid[0-9]*IDRegex: different Regexes per login procedure step or one common Regex for all steps to extract the session ID from the HTTP response

sid[0-9]*Data.*: data part for each step to be sent as POST data to the corresponding URL

sid[0-9]*Header.*: HTTP Headers to be sent to the URL for the corresponding step

sid[0-9]*IgnoreRedirects: Tells the HttpUtils to not follow HTTP Redirects for this Request. Might be needed for some devices that set a session cookie within a 303 Redirect.

clearSIdBeforeAuth: will set the session id to "" before doing the authentication steps

authRetries: number of retries for authentication procedure - defaults to 1

replacement[0-9]*Regex: Defines a replacement to be applied to an HTTP request header, data or URL before it is sent. This allows any part of the request to be modified based on a reading, an internal or an expression.; The regex defines which part of a header, data or URL should be replaced. The replacement is defined with the following attributes:

replacement[0-9]*Mode: Defines how the replacement should be done and what replacementValue means. Valid options are text, reading, internal and expression.

replacement[0-9]*Value: Defines the replacement. If the corresponding replacementMode is text, then value is a static text that is used as the replacement.; If replacementMode is reading then Value can be the name of a reading of this device or it can be a reading of a different device referred to by devicename:reading.; If replacementMode is internal the Value can be the name of an internal of this device or it can be an internal of a different device referred to by devicename:internal.; If replacementMode is expression the the Value is treated as a Perl expression that computes the replacement value. The expression can use $1, $2 and so on to refer to capture groups of the corresponding regex that is matched against the original URL, header or post data.; If replacementMode is key then the module will use a value from a key / value pair that is stored in an obfuscated form in the file system with the set storeKeyValue command. This might be useful for storing passwords.

[gs]et[0-9]*Replacement[0-9]*Value: This attribute can be used to override the replacement value for a specific get or set.

get|reading[0-9]*MaxAge: Defines how long a reading is valid before it is automatically overwritten with a replacement when the read function is called the next time.

get|reading[0-9]*MaxAgeReplacement: specifies the replacement for MaxAge - either as a static text or as a perl expression.

get|reading[0-9]*MaxAgeReplacementMode: specifies how the replacement is interpreted: can be text, expression and delete.

get|reading[0-9]*DeleteIfUnmatched: If set to 1 this attribute causes certain readings to be deleted when the parsing of the website does not match the specified reading. Internally HTTPMOD remembers which kind of operation created a reading (update, Get01, Get02 and so on). Specified readings will only be deleted if the same operation does not parse this reading again. This is especially useful for parsing that creates several matches / readings and the number of matches can vary from request to request. For example if reading01Regex creates 4 readings in one update cycle and in the next cycle it only matches two times then the readings containing the remaining values from the last round will be deleted.; Please note that this mechanism will not work in all cases after a restart. Especially when a get definition does not contain its own parsing definition but ExtractAllJSON or relies on HTTPMOD to use all defined reading.* attributes to parse the responsee to a get command, old readings might not be deleted after a restart of fhem.
get|reading[0-9]*DeleteOnError: If set to 1 this attribute causes certain readings to be deleted when the website can not be reached and the HTTP request returns an error. Internally HTTPMOD remembers which kind of operation created a reading (update, Get01, Get02 and so on). Specified readings will only be deleted if the same operation returns an error.

The same restrictions as for DeleteIfUnmatched apply regarding a fhem restart.

httpVersion: defines the HTTP-Version to be sent to the server. This defaults to 1.0.

sslVersion: defines the SSL Version for the negotiation with the server. The attribute is evaluated by HttpUtils. If it is not specified, HttpUtils assumes SSLv23:!SSLv3:!SSLv2

sslArgs: defines a list that is converted to a key / value hash and gets passed to HttpUtils. To avoid certificate validation for broken servers you can for example specify; attr myDevice sslArgs SSL_verify_mode,SSL_VERIFY_NONE

alignTime: Aligns each periodic read request for the defined interval to this base time. This is typcally something like 00:00 (see the FHEM at command)

noShutdown: pass the noshutdown flag to HTTPUtils for webservers that need it (some embedded webservers only deliver empty pages otherwise)

disable: stop doing automatic HTTP requests while this attribute is set to 1

enableControlSet: enables the built in set commands interval, stop, start, reread, upgradeAttributes, storeKeyValue.

enableCookies: enables the built in cookie handling if set to 1. With cookie handling each HTTPMOD device will remember cookies that the server sets and send them back to the server in the following requests.; This simplifies session magamenet in cases where the server uses a session ID in a cookie. In such cases enabling cookies should be sufficient and no sidRegex and no manual definition of a cookie header should be necessary.

showMatched: if set to 1 then HTTPMOD will create a reading with the name MATCHED_READINGS that contains the names of all readings that could be matched in the last request as well as UNMATCHED_READINGS and LAST_REQUEST.

showError: if set to 1 then HTTPMOD will create a reading and event with the Name LAST_ERROR that contains the error message of the last error returned from HttpUtils.

removeBuf: if set to 1 then HTTPMOD removes the internal named buf when a HTTP-response has been received.; $hash->{buf} is used internally be Fhem httpUtils and in some use cases it is desireable to remove this internal after reception; because it contains a very long response which looks ugly in Fhemweb.

timeout: time in seconds to wait for an answer. Default value is 2

queueDelay: HTTP Requests will be sent from a queue in order to avoid blocking when several Requests have to be sent in sequence. This attribute defines the delay between calls to the function that handles the send queue. It defaults to one second.

queueMax: Defines the maximum size of the send queue. If it is reached then further HTTP Requests will be dropped and not be added to the queue

minSendDelay: Defines the minimum time between two HTTP Requests.

Links

Example: Akkustand einer Zoe und mehr von Renault ZE Services
Example: Extract weather information from WeatherUnderground
Example: Pollen count
Example: Connect Daikin aircondition to FHEM
Example: Extract information from go-eCharger
Example: Integration of energy supplies from sonnen (https://sonnenbatterie.de/en/start)
Example: Miele 3rd party API für Miele Smarthome Geräte
Example: Honeywell Evohome Totalconnect
Example: Tigo Energy Integration
Example: Ecowater
Thread in FHEM Forum that discusses the first version of this module
Thread in FHEM Forum that discusses the second major version of this module
Thread in FHEM Forum that discusses the third major version of this module
Introduction to regular expressions
BurpSuite: Tool (local proxy) to help analyze http traffic

Introduction

Availability, prerequisites and definition

Set-Commands

Get-Commands

simple Attributes

Simple Configuration of HTTP Devices

Example for a PoolManager 5:

Example for AmbientMonitor

formatting and manipulating values / readings

Some help with Regular Expressions

Regular Expressions with multiple capture Groups

Parsing JSON

JSON Lists

Parsing http / XML using xpath

Further replacements of URL, header or post data

replacing reading values when they have not been updated / the device did not respond

Note on determining how to send requests to a special device

Advanced configuration to define a set command and send data to a device

Advanced configuration to create a valid session id that might be necessary

Example for a multi step login procedure:

Another Practical Example reading the SOC from a Renault Zoe

Advanced configuration to define a get and request additional data with its own request from a device

All attributes

Links

Advanced configuration to define a `set` command and send data to a device

Advanced configuration to define a `get` and request additional data with its own request from a device