Offline Metering Policy Configuration

The Revenium Offline Metering Policy enables a specialized audit trail of API transaction metadata that can be scraped "offline" by Logstash and sent to Revenium for metering.

Offline Metering Architecture

The standard Revenium Metering Policy is extremely performant, exhibiting less than 10 milliseconds of additional API request latency. Some use cases, however, require zero or close to zero additional latency. The Revenium Offline Metering is useful when ultra low / no latency is required or the standard Revenium Metering Policy can't be applied.

Offline Metering relies on specialized log entries in the Mule Runtime Engine that can be scraped and shipped in realtime to the Revenium Metering Beacons. This documentation shows how these logs can be shipped to Revenium using Filebeat and Logstash in the following topology:

Offline Metering Architecture

The Revenium Offline Metering Policy is a very simple custom policy that writes JSON formatted log entries on every API request and response with the transaction metadata necessary for Revenium to meter the transactions.

These log entries are polled by Filebeat and shipped to Logstash, which subsequently using the HTTP Output Plugin to send the metering data to Revenium.

Installing the Revenium Offline Metering Policy

The Revenium Offline Metering Policy can be obtained from the Revenium Customer Success Team. Once you’ve obtained the policy, you can uncompress it and follow these steps:

1. Open the pom.xml file and replace the two instances of the "ANYPOINT_GROUPID" placeholder with your Anypoint Organization ID:

   1. "groupId" field

   2. “exchange.url” field

2. Ensure your ~/.m2/settings.xml file has an "exchange-server" defined as follows:

<server>
  <id>exchange-server</id>
  <username>myexchangeusername</username>
  <password>myexchangepassword</password>
</server>

Run the following command

mvn clean deploy

The Exchange Maven plugin requires Java 8 to run properly. You can set your JAVA_HOME for Maven like this:

export JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home ; mvn clean deploy

The Revenium Metering Policy should now be available in the Anypoint Platform API Manager.

You can obtain your Anypoint Organization ID from the URL when you are logged into Anypoint.

For example: https://anypoint.mulesoft.com/home/organizations**/d438148e-8d5f-47eb-98a6-1e5a456d6653**/

Configuring the Revenium Offline Metering Policy

In order to configure the policy we need to obtain the API’s ID from Revenium. The API’s ID in Revenium is required to meter the API through an Anypoint API Gateway.

Navigate to the Manage → APIs tab and click on the “copy” icon to copy the APIs ID. Paste the ID somewhere you can refer to it later:

An upcoming Revenium release will automate the process of obtaining the Asset ID.

In the Anypoint API Manager, navigate to the API to be metered:

Selecting the API to be metered

Navigate to “Policies” and then “Apply New Policy”:

Select the Revenium Offline Metering Policy:

The Metering Policy takes the following configuration parameters:

• Revenium Source ID: The ID of the API. Use the ID of the API we copied earlier from the Revenium Subscriber Interface.

• Metering Expression: An optional DataWeave expression that determines whether or not an API request should be metered. The default behavior is to meter any request that has a 2xx series (successful) response from the API implementation.

• Subscriptions Expression: An optional DataWeave expression to extract the Revenium Subscription from the API request. The default behavior is to determine the header from either the X-REVENIUM-PRODUCT-KEY __ or, if its not present, using the "client_id" header.

• Client ID Expression: An optional DataWeave expression to extract the Anypoint Client ID from the API request. The default behavior is to use the "client_id" header.

• Metadata Expression: An optional DataWeave expression to extract arbitrary metadata from the API request.

The Policy should now be applied and your API is now ready to be metered.

Logstash Configuration

The Offline Metering Policy logs API request and response metadata on the "REVENIUM" log4j category. In order to simplify scraping this data you can modify Mule's log4j configuration to redirect this logging data to a dedicated log by modifying $MULE_HOME/conf/log4j2.xml as follows:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
    <Appenders>
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="%-5p %d [%t] [processor: %X{processorPath}; event: %X{correlationId}] %c: %m%n"/>
        </Console>
        <Console name="ConsoleLogUtil" target="SYSTEM_OUT">
            <PatternLayout pattern="%m%n"/>
        </Console>
        <File name="Revenium" fileName="../logs/revenium.log">
            <PatternLayout>
                <Pattern>%m%n</Pattern>
            </PatternLayout>
        </File>
    </Appenders>

    <Loggers>

        <AsyncLogger name="org.mule.runtime.core.internal.processor.LoggerMessageProcessor" level="INFO"/>
        <AsyncLogger name="com.mulesoft.agent" level="INFO"/>

        <AsyncLogger name="org.mule.runtime.core.internal.logging" additivity="false" level="INFO">
            <AppenderRef ref="ConsoleLogUtil"/>
        </AsyncLogger>

        <AsyncRoot level="WARN">
            <AppenderRef ref="Console"/>
        </AsyncRoot>

        <AsyncLogger name="com.mulesoft.mule.runtime.gw" level="INFO"/>
        <AsyncLogger name="com.mulesoft.extension.policies" level="INFO"/>
        <AsyncLogger name="com.mulesoft.extension.gateway" level="INFO"/>
        <AsyncLogger name="com.mulesoft.analytics" level="INFO"/>

        <AsyncLogger name="REVENIUM" level="INFO">
            <AppenderRef ref="Revenium"/>
        </AsyncLogger>

    </Loggers>

</Configuration>

Now Filebeat can be configured to watch the revenium.log file and send the metering data to a Logstash instance:

filebeat.inputs:

- type: filestream

  id: revenium

  enabled: true

  paths:
    - /opt/mule-enterprise-standalone-4.4.0/logs/revenium.log

Finally a Logstash output plugin must be configured to send the metering log data to the Revenium Metering Beacons:

input {
  beats {
    port => 5044
  }
}

output {
  http {
    url => "https://api.dev.hcapp.io/meter/v1/api/event"
    http_method => "post"
    message => "%{message}"
    content_type => "application/json"
    format => "message"
    retry_failed => true
    headers => [
      'x-api-key', 'hak_123456789'
    ]
  }
}

If its impossible or undesirable to modify the Mule log4j2.xml configuration then a Filebeat Processor can be used to only ship events logged at the "REVENIUM" log4j category.

The Offline Metering Policy is a convenience mechanism to insert metering data into the Mule logs. If, for some reason, you are not apply to apply *any* policy you can still use MuleSoft's logger as follows to inject the lines before and after an HTTP request in any Mule application:

<logger level="INFO" category="REVENIUM" message="{"requestId": "40bfc661-2cc5-4a82-8443-f0104ba6ac3e","eventType": "REQUEST","assetId": "lkpkW3:7d877eb7-19df-4b4c-8500-2958a4b1c734","productKey": "207e9b54-47e6-4198-8d9d-c3ac68578075","method": "GET","currentMillis": 1658196887460}"/>

<logger level="INFO" category="REVENIUM" message="{"requestId": "40bfc661-2cc5-4a82-8443-f0104ba6ac3e","eventType": "RESPONSE","assetId": "lkpkW3:7d877eb7-19df-4b4c-8500-2958a4b1c734","productKey": "207e9b54-47e6-4198-8d9d-c3ac68578075","method": "GET","currentMillis": 1658196887460}"/>