The  Host sFlow distributed agent article describes how sFlow agents, embedded in applications such as Apache, NGINX, Tomcat, Memcached and Java, coordinate with the Host sFlow daemon (hsflowd) in order to monitor server and application performance.

Implementing sFlow monitoring natively in widely used applications is worth the effort, since the result is a highly scaleable, easily deployed solution with minimal impact on performance. However, this approach is overkill for monitoring application logic implemented in scripting languages such as PHP, Python, Ruby or Perl.

Recently, a simple JSON API was added to hsfowd that makes it easy to integrate performance monitoring in scripting languages. The API is an implementation of the sFlow Application Structures draft, which defines a generalized set of application layer performance metrics.

Note: Even if you instrument your scripted application logic, you should still enable sFlow in the web server since the HTTP information exported from the web server complements application metrics to provide a more complete picture of performance, see HTTP.

Currently the API is only implemented in the Linux trunk. To try it out, you will need to check out the trunk and build hsflowd from sources:

svn co https://host-sflow.svn.sourceforge.net/svnroot/host-sflow/trunk host-sflow
cd host-sflow
make
make install
make schedule

Note: See Installing Host sFlow on a Linux server for additional configuration information. The JSON API will be included in the upcoming Host sFlow 1.21 release.

Uncommenting the following entry in the /etc/hsflowd.conf file opens a UDP port to receive JSON encoded metrics:

jsonPort = 36343

Note: It is recommended that you keep the default port, 36343, since any change to the port number will require a corresponding change in any scripts that send JSON messages. The jsonPort value is written into the /etc/hsflowd.auto file so that it is available to client scripts, along with other sFlow settings, see Host sFlow distributed agent. The hsflowd daemon will only accept messages generated on the same host which shouldn’t be a problem since hsflowd should be installed on every host to report host performance metrics.

The following is an example of the type of JSON message that a script can send to describe the outcome of a transaction:

{"flow_sample:{
  "app_name":"myapp",
  "app_operation":{
    "operation":"user.friend",
    "attributes":"id=123&handle=sith",
    "status_descr":"OK",
    "status":0,
    "req_bytes":43,
    "resp_bytes":234,
    "uS":2000},
  "app_initiator":{"actor":"123"},
  "app_target":{"actor":"231"},
  "extended_socket_ipv4":{
    "protocol":6,
    "local_ip":"10.0.0.1",
    "remote_ip":"10.0.0.23",
    "local_port":123,
    "remote_port":43032}
}}

Note: The names of the structures and attributes in the JSON message mirror the structures defined in sFlow Application Structures.

Many of the structures and attributes are optional, a message can be as simple as:

{"flow_sample:{
  "app_name":"myapp",
  "app_operation":{
    "operation":"user.friend",
    "attributes":"",
    "status_descr":"",
    "status":0,
    "req_bytes":0,
    "resp_bytes":0,
    "uS":0}
}

Constructing and sending JSON messages as UDP datagrams to hsflowd is straightforward. For example, the following PHP app_operation function formats and sends the previous JSON message.

function app_operation($app_name,
                       $op_name,
                       $attributes="",
                       $status=0,
                       $status_desr="",
                       $req_bytes=0,
                       $resp_bytes=0,
                       $uS=0,
                       $sampling_rate=1) {

  if($sampling_rate > 1) {
    if(mt_rand(1,$sampling_rate) != 1) { return; }
  }

  try {
     $sock = fsockopen("udp://localhost",36343,$errno,$errstr);
     if(! $sock) { return; }
     fwrite($sock, '{"flow_sample":{

 "app_name":"'.$app_name.'",
 "sampling_rate":'.$sampling_rate.',
 "app_operation":{

   "operation":"'.$op_name.'",

   "attributes":"'.$attributes.'",
   "status_descr":"'.$status_descr.'",
   "status":'.$status.',
   "req_bytes":'.$req_bytes.',
   "resp_bytes":'.$resp_bytes.',
   "uS":'.$uS.'}}}');
     fclose($sock);
  } catch(Exception $e) {}
}

Note: This function supports transaction sampling, i.e. if you set a sampling_rate of 10 then there is a 1-in-10 chance that the operation will actually generate a measurement. Sampling allows you to reduce the measurement overhead in high transaction rate environments and still generate useful results. You should set a sampling_rate that reduces the impact of monitoring on your application to acceptable levels, although, you probably don’t need to sample at all unless you are handling hundreds of operations per second. Choose a fixed sampling_rate that works for your application – choose the lowest sampling_rate that protects application performance – as you will see later, a low sampling_rate setting will allow hsflowd to maintain more accurate counters and provide a greater range of sampling rates when it exports the data.

Including the app_operation function in your PHP application library makes instrumenting PHP application logic as simple as including a single line of code in a PHP rendered web page:

<?php app_operation("myapp","user.friend"); ?>

When hsflowd receives a JSON message, it increments per application performance counters (scaled by the sampling rate if needed). The counters are periodically exported along with the other sFlow metrics (CPU, memory, disk and network I/O) that hsflowd exports.

The following output from sflowtool shows the contents of an sFlow datagram containing application counters:

startDatagram =================================
datagramSourceIP 127.0.0.1
datagramSize 112
unixSecondsUTC 1336846918
datagramVersion 5
agentSubId 100000
agent 10.0.0.150
packetSequenceNo 2670
sysUpTime 77514000
samplesInPacket 1
startSample ----------------------
sampleType_tag 0:2
sampleType COUNTERSSAMPLE
sampleSequenceNo 4
sourceId 3:150002
counterBlock_tag 0:2202
application myapp
status_OK 23
errors_OTHER 0
errors_TIMEOUT 0
errors_INTERNAL_ERROR 0
errors_BAD_REQUEST 0
errors_FORBIDDEN 0
errors_TOO_LARGE 0
errors_NOT_IMPLEMENTED 0
errors_NOT_FOUND 0
errors_UNAVAILABLE 0
errors_UNAUTHORIZED 0
endSample   ----------------------
endDatagram   =================================

The application counters can be sent to tools like Ganglia and Graphite in order to trend the performance of individual application instances, or of whole clusters of applications.

The transactions are also sampled by hsflowd based on the sampling setting in hsflowd.conf, or DNS-SD. Specific sampling rates can be set based on application name, for example, to override the default sampling rate of 400 and apply a sampling rate of 1-in-100 to the myapp, use the following setting:

sampling.app.myapp=100

Note: If the transactions were sampled before being sent to hsflowd, then they will be sub-sampled to achieve the target sampling rate. For example, if the script used a sampling rate of 1-in-10, then hsflowd would apply a 1-in-10 sampling operation in order to achieve the desired 1-in-100 sampling rate.

The following output from sflowtool shows the contents of an sFlow datagram containing an application transaction sample:

startDatagram =================================
datagramSourceIP 127.0.0.1
datagramSize 136
unixSecondsUTC 1336846925
datagramVersion 5
agentSubId 100000
agent 10.0.0.150
packetSequenceNo 2671
sysUpTime 77521000
samplesInPacket 1
startSample ----------------------
sampleType_tag 0:1
sampleType FLOWSAMPLE
sampleSequenceNo 1
sourceId 3:150002
meanSkipCount 10
samplePool 10
dropEvents 0
inputPort 0
outputPort 1073741823
flowBlock_tag 0:2202
flowSampleType applicationOperation
application myapp
operation user.friend
request_bytes 0
response_bytes 0
status SUCCESS
duration_uS 0
endSample   ----------------------
endDatagram   =================================

The transaction samples provide details that complement the counter samples. For example, if you were to see a rise in the errors_TIMEOUT rate, you could look at the transaction samples and determine the operations associated with the timeouts.

Note: Anyone familiar with Etsy’s StatsD tool will see a similarity in the way sFlow monitoring is embedded in scripts, see Measure Anything, Measure Everything. The main difference is that sFlow application measurements contain additional structure that allows them to be part of a large scale monitoring system linking network switches, hosts and applications together. In addition, sFlow’s inclusion of sampled transaction records allows metrics to be broken out into fine detail, making it possible to see how application instances interact and get to the root cause of performance problems.

Finally, the application metrics extension to the sFlow standard and the implementation in hsflowd are still in the early stages. Please try them out and provide feedback. Any comments or suggestions regarding the sFlow metrics should be directed to the sFlow.org mailing list and comments or questions relating to the scripting API should be directed to the host-sflow mailing list.