Offline Metering via Log Parsing

Allows the productization and metering of APIs without an API gateway

Before investigating an offline/log-based integration, we recommend you review our Linux & container-native metering solution here, which works in a variety of environments and is configurable with only a few lines of additional code.

Offline Metering is useful in situations where an API is being consumed without an API Gateway, or when you wish to meter any type of transaction asynchronously to achieve zero additional latency at the time of transaction (whether using a gateway or not).

This metering method enables a specialized audit trail of API transaction metadata that can be scraped "offline" by Logstash and sent to Revenium for metering.

Offline Metering Architecture

Offline Metering allows the ingest of metering data without, or in addition to, traditional metering via an API Gateway. Using a log scraping frameworks such as Filebeat and Logstash arbitrary log files like web server access logs, firewall logs, loadbalancer logs, etc can be used as sources for API metering data.

Offline Metering of Logfiles Using Filebeat and Logstash

Log entries must contain the following API metadata to be metered:

• The Revenium Source ID OR the URL/URI of the API request

  • The most common method is to use the URI/URL of the request (using the parameterurl), which will automatically create an associated source in Revenium.

  • If you've pre-created sources in Revenium, you can use an existing SourceId instead of url. The parameter to pass in this case is api

• An application identifier (parameter =application), this allows us to track which of your subscribers or applications are using each of your APIs.

• The response code (parameter = responseCode) indicating success or failure (i.e. 200 series responses for success vs. 500 series responses for failures)

  • This is required so that your clients are never charged for failed transactions in the case of monetized APIs.

Example Logstash Configuration

The following documents an example Filebeat and Logstash configuration that parses and submits metering data from a Jetty webserver log in realtime.

Here is the Filebeat configuration to tail the Jetty log file.

filebeat.inputs:

- type: filestream
  id: revenium
  enabled: true
  paths:
    - /jetty/logs/jetty.log

And here is the Logstash configuration to parse and send the metering request to the Revenium platform.

Note you'll need to replace the x-api-key below with your own Integration Key.

input {
  beats {
    port => 5044
  }
}

output {
  http {
    url => "https://api.revenium.io/meter/v1/api/event"
    http_method => "post"
    responseCode => "200"
    application => "12345"
    message => "%{message}"
    content_type => "application/json"
    format => "message"
    retry_failed => true
    headers => [
      'x-api-key', 'hak_123456789'
    ]
  }
}