This document is about: SERVER 5

SWITCH TO

SERVER 4 SERVER 5

This page is a work in progress and could be pending updates.

Photon Counters

Photon makes use of Windows Performance Counters as well as custom in-memory counters to track server performance and statistics, e.g. CPU load, memory usage, current connection count, operation execution time, etc.

To display these counters either Windows Performance Monitor or any other external time series database system can be used.
Photon is able to publish its counter values via UDP, PGM or HTTP.

Photon performance counters are easily extensible and the display options can be fully customized.

Performance Counters

In-Memory Counters

Photon's applications are using in-memory performance counters to track application-specific statistics, e.g. the number of current games.
These counters don't show up in the Windows Performance Monitor.

Windows Performance Counters

To track performance data, Photon relies on native
Windows Performance Counters instead.

If you want to make use of these counters, you need to install them before you start Photon.
This can be done in Photon Control:

or by entering this command from your binaries deploy folder (same folder as PhotonControl):

./PhotonSocketServer.exe /installCounters

Besides make sure you have Windows Peformance Counters enabled in the configuration file of your application.
To do so, set EnablePerformanceCounters to True as follows:

XML

<configuration>
  <applicationSettings>
    <!-- other settings -->
    <Photon.LoadBalancing.Common.CommonSettings>
      <!-- other settings -->
      <setting name="EnablePerformanceCounters" serializeAs="String">
        <value>True</value>
      </setting>
      <!-- other settings -->
    </Photon.LoadBalancing.Common.CommonSettings>
    <!-- other settings -->
  </applicationSettings>
</configuration>

Check Windows Performance Monitor to see the installed counters:

Photon is writing data to these counters automatically once it is started.

Usage

Overview

Photon Server uses Windows Performance Counters and in-memory Photon Counters to publish statistics.

Photon native Core writes statistics to Windows Performance Counters only.
They can be read and published by CounterPublisher .NET application to any other endpoint.

Usually, Photon Server carries a few .NET applications on board.

Every Photon .NET application implementing Photon.SocketServer.ApplicationBase contains a default set of counters that can be published through out configurable Photon.SocketServer.Diagnostics.CounterPublisher singleton.

Publishing of .NET application statistics can be enabled in the configuration file of your application.

Publishing rate, used protocol, receiving endpoint and additional options can be configured in place.

Default publisher supports various kinds of protocols.
They can be extended by the plugins.
It is possible to extend counters samples made by the publisher, specifying additional counter schemas in the configuration.

Architecture

Photon native Core writes statistics to Windows Performance Counters only.

Every Photon .NET application implements Photon.SocketServer.ApplicationBase.
The latter uses Photon.SocketServer.Diagnostics.CounterPublisher singleton to sample and publish statistics to the world.
The publisher is configurable with the <Photon> section of the application configuration typed Photon.SocketServer.Diagnostics.Configuration.PhotonSettings, Photon.SocketServer.

The Publisher consists of a "Sampler" and a "Sender".

The Sampler simply reads all the registered counters (from any source that implements ExitGames.Diagnostics.Counter.ICounter interface) and aggregates them during the time.

The Sender uses implementation of ExitGames.Diagnostics.Monitoring.Protocol.ICounterSampleWriter interface to actually send aggregated data to the configured endpoint.
Sender abstracts transport and protocol levels from the Publisher.

There is a default abstract implementation of ICounterSampleWriter to help developers to create new plugins: PhotonBinaryWriterBase.

Configuration

Photon Counters can be configured for any Photon Server application via its configuration file.
The configuration file must contain at least the following lines to have counter publishing enabled with default behaviour:

XML

  <configSections>
    <section name="Photon" type="Photon.SocketServer.Diagnostics.Configuration.PhotonSettings, Photon.SocketServer" />
  </configSections>

  <Photon>
    <CounterPublisher enabled="True" />
  </Photon>

It is important to stick with <Photon> section alias for Photon.SocketServer.Diagnostics.Configuration.PhotonSettings type, because it is an internal name reserved for default configuration of application publisher instance.

List of Settings

Attributes of <CounterPublisher>:

enabled: OPTIONAL.
Default: "False".
Toggles publishing of counters.
Add this and set it to "True" to publish counters.
senderType: OPTIONAL.
Specifies assembly-qualified name of the type for the custom settings class for the attributes of <Sender>.
updateInterval: OPTIONAL.
Default: 10.
Sampling rate.
In seconds.
addDefaultAppCounter: OPTIONAL.
Default: "True".
Controls whether to include default application counters into the publisher by default.
If disabled, CounterPublisher will omit default ApplicationBase counters

Attributes of <Sender> (ExitGames.Diagnostics.Configuration.CounterSampleSenderSettings):

endpoint: REQUIRED.
Default: "http://127.0.0.1".
protocol: OPTIONAL.
Default: "PhotonBinary".
The application level protocol.
Case insensitive ProtocolType (PhotonBinary, GraphiteText, InfluxDB090GraphiteText, StatsDText), or custom Writer type implementation.
senderId: OPTIONAL.
Default: "".
The ID of the sender that will be sent with the counter data.
"{0}" occurrences -if any- will be replaced with Environment.MachineName.
initialDelay: OPTIONAL.
Default: 10.
Period of time to delay first data publishing.
In seconds.
sendInterval: OPTIONAL.
Default: 10.
The publish interval.
In seconds.
maxQueueLength: OPTIONAL.
Default: 120.
How many packages should be enqueued for republishing if an error occurred during publishing.
maxRetryCount: OPTIONAL:
Default: -1.
How many error can occur during publishing before the counter publishers stops publishing.
If a value equal or less than than zero is specified the counter publisher will never stop if an error occurred.

There are also specific settings for per application protocol.

Transport protocols

Supported transport protocols:

UDP
HTTP (or HTTPS)
PGM
Any other custom transport provided by third party clients. TCP is not supported yet.

Transport protocol can be configured through "endpoint" property of ExitGames.Diagnostics.Configuration.CounterSampleSenderSettings.

These transport types are defined by ExitGames.Diagnostics.Configuration.TransportType.
"endpoint" property format is defined by the specification of default constructor of System.Uri type.
Examples: "udp://255.255.255.255:40001" or "https://localhost:4000".

Application Protocols

Photon "CounterPublisher" can write data to the wide set of time series databases.
Actual data publishing done via certain implementation of ExitGames.Diagnostics.Monitoring.Protocol.ICounterSampleWriter.

Usually, for every specific writer, there is corresponding settings class available in ExitGames.Diagnostics.Configuration namespace.

Sender default settings implemented by ExitGames.Diagnostics.Configuration.CounterSampleSenderSettings.

"ExitGamesLibs" contains implementation for:

"CounterPublisher" can use any external implementation of the ICounterSampleWriter.
Thus, it can write to any database via custom Writer implementation.

There are also 2 custom separate implementations of the ICounterSampleWriter enabling support for:

Amazon AWS CloudWatch
(CounterPublisher.AWS.CloudWatch plugin)
NewRelic Platform
(CounterPublisher.NewRelic plugin)

Those custom external implementation can be configured through stating custom "protocol" type, i.e.

XML

  <Photon>
    <CounterPublisher enabled="True" senderType="ExitGames.Diagnostics.Configuration.AWSCloudWatchSettings, CounterPublisher.AWS.CloudWatch">
      <Sender protocol="ExitGames.Diagnostics.Monitoring.Protocol.AWS.CloudWatch.AWSCloudWatchWriter, CounterPublisher.AWS.CloudWatch" />
    </CounterPublisher>
  </Photon>

Important thing is, usually custom protocol implementation requires setting up of specific configuration properties.
It can be done by specifying writer specific "senderType" property class type (like in the example above).

Counters schemas

"CounterPublisher" read and write registered counters.
To register counter, there is a special method CounterPublisher.AddCounter(ICounter counter, string name).

Usually, counters are grouped by some logical reason into classes like Photon.SocketServer.Diagnostics.PhotonCounter.
"PhotonCounter" provides default counters for any ApplicationBase based .NET application:

AvrgOpExecTime
EventsSentCount
EventsSentPerSec
MessagesSentCount
MessagesSentPerSec
RawMessagesSentCount
RawMessagesSentPerSec
InitPerSec
OpReceiveCount
OpReceivePerSec
OpResponseCount
OpResponsePerSec
OperationsFast
OperationsMaxTime
OperationsMiddle
OperationsSlow
SessionCount

Counters are static readonly fields with attribute "PublishCounter" which can define some additional properties like another name, or something.

    public static class PhotonCounter
    {
        [PublishCounter("AvrgOpExecTime")]
        public static readonly AverageCounter AverageOperationExecutionTime = new AverageCounter();

        //...
    }

Counter classes are registered in "CounterPublisher" with CounterPublisher.AddCounterClass or CounterPublisher.AddStaticCounterClass like this:

    if (PhotonSettings.Default.CounterPublisher.Enabled)
    {
        if (PhotonSettings.Default.CounterPublisher.AddDefaultAppCounter)
        {
            CounterPublisher.DefaultInstance.AddStaticCounterClass(typeof(PhotonCounter), this.ApplicationName);
        }

        CounterPublisher.DefaultInstance.Start();
    }

Usually all counter classes are paired with special schema classes which includes logical namespaces and counters names (Photon.CounterPublisher.SocketServerCounterSchema):

namespace Photon.CounterPublisher
{
    public static class SocketServerCounterSchema
    {
        public static class Common
        {
            public static readonly string CategoryName = "Photon Socket Server";

            public static readonly string BuffersInUseCounter = "IO Buffers In Use";

            //...
        }

        public static class Enet
        {
            public static readonly string ACKsInCounter = "Acknowledgements in";

            public static readonly string ACKsInPerSecondCounter = "Acknowledgements in/sec";

            //...
        }

        //...
    }
}

Thus, counter class will describe its counters in terms of its schema:

namespace Photon.CounterPublisher
{
    [CounterSet(Name = "Photon")]
    public static class SocketServerCounter
    {
        /// <summary>
        /// The bytes in per second counter.
        /// </summary>
        [PublishCounter("BytesInPerSecond")]
        public static readonly PerformanceCounterReader BytesInPerSecondCounter =
            CreateCounterReader(Schema.Common.CategoryName, Schema.Common.BytesInPerSecondCounter);

        //...

    }
}

Finally, "CounterPublisher" can be configured to read and write any additional counters via <Counters/Schema> configuration element.

XML

  <Photon>
    <CounterPublisher ... >
      <Sender  ... />
        <Counters>
          <Schema namespace="TestNamespace" type="Photon.CounterPublisher.SystemCounter, CounterPublisher" />
        </Counters>
    </CounterPublisher>
  </Photon>

It is important, to choose metrics names wisely.
Use following rules for naming conventions:

PascalCase matching [a-zA-Z0-9]+ symbol set
1.1. "Per" suffix then format should be: {ValueUnit}{YourMetricInWordsAction}Per{Interval}.
Like System.InBytesRecvPerSecond.
1.2. "Count" suffix then format should be: {CountUnit}{YourMetric}Count.
Like CounterPublisher.EventsSentCount.
Values:
2.1. (Value/Interval): Like "bytes in/sec", "Disconnected Peers +/sec".
2.1. (Value): Like "Time Spent In Server: In (ms)".
{ValueUnit}AnyThingInPascalCase

Standard units types:

Seconds
Microseconds
Milliseconds
Bytes
Kilobytes
Megabytes
Gigabytes
Terabytes
Bits
Kilobits
Megabits
Gigabits
Terabits
Percent
Count.

Publishing Windows Performance Counters

The native Photon Core writes statistics to Windows Performance Counters (WPC).
Therefore, Photon ships with a separate "CounterPublisher" .NET application, which acts as a kind of proxy: it reads the counters from wherever you like (including Photon's Windows Performance Counters), temporarily adds them to in-memory counters and publishes them.

In addition to the Photon Core Counters, the "CounterPublisher" application reads and broadcasts some Windows System Counters, like the CPU load, memory usage etc.

The "CounterPublisher" application is enabled by default; its behavior can be configured in the "CounterPublisher\bin\CounterPublisher.dll.config".
Under the hood, it is usual ApplicationBase based Photon application with its standard "CounterPublisher" instance with some specially designed Counter classes to be able to read WPC.

"CounterPublisher" application registers 2 counter classes using CounterPublisher.AddStaticCounterClass method:

Photon.CounterPublisher.SystemCounter which reports CPU, Memory and Overall Network Statistics and Photon.CounterPublisher.SocketServerCounter which reports specific to native Photon Core WPC counters including network (UDP, ENET, etc.) and various queues statistics.

To see what counters and values are currently published, DEBUG log level for ExitGames.Diagnostics package must be enforced in "CounterPublisher\bin\log4net.config".
Then, take a look into the log file at "Photon\Log\CounterPublisher.log" (this path specified in configuration of LogFileAppender in "log4net.config"):

XML

  <logger name="ExitGames.Diagnostics">
    <level value="DEBUG" />
  </logger>

Publishing to Graphite

Photon supports Graphite.
The "Writer" is implemented inside ExitGamesLibs.
It can operate over any transport layer: UDP, PGM, HTTP.
Default transport layer for Graphite is UDP.

"ProtocolType" is "GraphiteText".
Sender type is ExitGames.Diagnostics.Configuration.GraphiteTextWriterSettings.
Writer type is ExitGames.Diagnostics.Monitoring.Protocol.Graphite.Text.GraphiteTextWriter.

This is text based protocol.
The format is:

metric_path value timestamp\n

metric_path is in format of PATH[.PATH...] where . is a separator of namespace.
value is float.
timestamp is unix time since epoch in seconds.

Example of protocol data:

Plain Old Text

EU.RACK01.FIRST.System.Cpu 1.53993 1428332168
EU.RACK01.FIRST.System.CpuTotal 2.981617 1428332168
EU.RACK01.FIRST.System.Memory 5317 1428332168
EU.RACK01.FIRST.System.BytesTotalPerSecond 8103.247 1428332168
EU.RACK01.FIRST.System.BytesSentPerSecond 1784.697 1428332168
EU.RACK01.FIRST.System.BytesReceivedPerSecond 6383.81 1428332168

When you use unreliable transport like UDP, it is worth configuring "MaxPayloadSize" property to prevent packet fragmentation during network trips.

Be careful to keep the total length of the payload within your network's MTU.
There is no single good value to use, but here are some guidelines for common network scenarios:

Fast Ethernet (1432) - This is most likely for Intranets.
Gigabit Ethernet (8932) - Jumbo frames can make use of this feature much more efficient.
Commodity Internet (512) - If you are routing over the internet a value in this range will be reasonable
You might be able to go higher, but you are at the mercy of all the hops in your route.

Choose application and modify ".config" like this:

XML

<?xml version="1.0" encoding="utf-8" ?>
<configuration>

  <configSections>
    <section name="Photon" type="Photon.SocketServer.Diagnostics.Configuration.PhotonSettings, Photon.SocketServer"/>
  </configSections>

  <Photon>
    <CounterPublisher enabled="True">
      <Sender
        protocol="GraphiteText"
        endpoint="udp://127.0.0.1:2003"
        senderId="{#CloudType#}.{#PrivateCloud#}.{#Region#}.{#Cluster#}.{0}" />
    </CounterPublisher>
  </Photon>

</configuration>

For this "Graphite" protocol implementation, "senderId" will prepend each metric key published.

Publishing to InfluxDB

Photon supports publishing to InfluxDB using Graphite protocol over UDP.
The Writer is implemented inside ExitGamesLibs.
The main difference between original Graphite protocol is how InfluxDB awaits structured keys.

"ProtocolType" is "InfluxDB090GraphiteText".
Sender type is ExitGames.Diagnostics.Configuration.InfluxDb090TextWriterSettings.
Writer type is ExitGames.Diagnostics.Monitoring.Protocol.InfluxDB090.Graphite.Text.GraphiteTextWriter.

This is text based protocol.
The format is:

metric_path value timestamp\n

metric_path is in format of PATH[.PATH...] where . is a separator of namespace.
value is float.
timestamp is unix time since epoch in seconds.

InfluxDB 0.9.0 awaits keys in the following format:

Plain Old Text

ex: region.us-west.hostname.server01.cpu -> tags -> region: us-west, hostname: server01, point name -> cpu
use senderId = "CloudType.{#CloudType#}.PrivateCloud.{#PrivateCloud#}.Region.{#Region#}.Cluster.{#Cluster#}.Machine.{0}"

Example of protocol data:

Plain Old Text

Region.EU.Rack.RACK01.PC.FIRST.NS.System.Cpu 1.53993 1428332168
Region.EU.Rack.RACK01.PC.FIRST.NS.System.CpuTotal 2.981617 1428332168
Region.EU.Rack.RACK01.PC.FIRST.NS.System.Memory 5317 1428332168
Region.EU.Rack.RACK01.PC.FIRST.NS.System.BytesTotalPerSecond 8103.247 1428332168
Region.EU.Rack.RACK01.PC.FIRST.NS.System.BytesSentPerSecond 1784.697 1428332168
Region.EU.Rack.RACK01.PC.FIRST.NS.System.BytesReceivedPerSecond 6383.81 1428332168

InfluxDB allows you to change . separator to something else using "separator" property.
This separator used to split tags from metric key name.
Keys rewriting can be switched off with "rewriteKeys".

When you use unreliable transport like UDP, it is worth configuring "MaxPayloadSize" property to prevent packet fragmentation during network trips.

Be careful to keep the total length of the payload within your network's MTU.
There is no single good value to use, but here are some guidelines for common network scenarios:

Fast Ethernet (1432) - This is most likely for Intranets.
Gigabit Ethernet (8932) - Jumbo frames can make use of this feature much more efficient.
Commodity Internet (512) - If you are routing over the internet a value in this range will be reasonable.
You might be able to go higher, but you are at the mercy of all the hops in your route.

Choose application and modify ".config" like this:

XML

<?xml version="1.0" encoding="utf-8" ?>
<configuration>

  <configSections>
    <section name="Photon" type="Photon.SocketServer.Diagnostics.Configuration.PhotonSettings, Photon.SocketServer"/>
  </configSections>

  <Photon>
    <CounterPublisher enabled="True">
      <Sender
        protocol="InfluxDB090GraphiteText"
        endpoint="udp://127.0.0.1:2003"
        senderId="CloudType.REALTIME.PrivateCloud.BLAH.Region.LOCAL.Cluster.EY.Machine.{0}" />
    </CounterPublisher>
  </Photon>

</configuration>

For this "Graphite" protocol implementation, "senderId" will prepend each metric key published.

Publishing to StatsD

Photon supports StatsD.
The "Writer" is implemented inside ExitGamesLibs.
It can operate over any transport layer: UDP, PGM, HTTP.
Default transport layer for StatsD is UDP.

"ProtocolType" is "StatsDText".
Sender type is ExitGames.Diagnostics.Configuration.StatsDWriterSettings.
Writer type is ExitGames.Diagnostics.Monitoring.Protocol.StatsD.Text.StatsDTextWriter.

The format of exported metrics is UTF-8 text, with metrics separated by newlines.
Metrics are generally of the form <metric name>:<value>|<type>, with exceptions noted in the metric type.

The protocol allows for both integer and floating point values.
Most implementations store values internally as a IEEE 754 double precision float, but many implementations and graphing systems only support integer values.
For compatibility all values should be integers in the range (-2⁵³, 2⁵³).

Example of protocol data:

Plain Old Text

REALTIME/PUBLIC/LOCAL/FIRST.System.Cpu:0.472394|c
REALTIME/PUBLIC/LOCAL/FIRST.System.CpuTotal:4.921546|c
REALTIME/PUBLIC/LOCAL/FIRST.System.Memory:5040|c
REALTIME/PUBLIC/LOCAL/FIRST.System.BytesTotalPerSecond:274.1134|c
REALTIME/PUBLIC/LOCAL/FIRST.System.BytesSentPerSecond:247.9789|c
REALTIME/PUBLIC/LOCAL/FIRST.System.BytesReceivedPerSecond:25.77675|c

When you use unreliable transport like UDP, it is worth configuring "MaxPayloadSize" property to prevent packet fragmentation during network trips.

Be careful to keep the total length of the payload within your network's MTU.
There is no single good value to use, but here are some guidelines for common network scenarios:

Fast Ethernet (1432) - This is most likely for Intranets.
Gigabit Ethernet (8932) - Jumbo frames can make use of this feature much more efficient.
Commodity Internet (512) - If you are routing over the internet a value in this range will be reasonable.
You might be able to go higher, but you are at the mercy of all the hops in your route.

Choose application and modify ".config" like this:

XML

<?xml version="1.0" encoding="utf-8" ?>
<configuration>

  <configSections>
    <section name="Photon" type="Photon.SocketServer.Diagnostics.Configuration.PhotonSettings, Photon.SocketServer"/>
  </configSections>

  <Photon>
    <CounterPublisher enabled="True">
      <Sender
        protocol="StatsDText"
        endpoint="udp://127.0.0.1:8125"
        senderId="REALTIME/PUBLIC/LOCAL/{0}" />
    </CounterPublisher>
  </Photon>

</configuration>

For this "StatsD" protocol implementation, "senderId" will prepend each metric key published.

Publishing to NewRelic Platform

The CounterPublisher plugin for NewRelic is available here.

New Relic's SaaS-based application Performance Monitoring helps you build, deploy and maintain great web software.

To publish metrics to NewRelic Platform with "CounterPublisher", there is a separate implementation of writer plugin in CounterPublisher.NewRelic.

It uses native NewRelic SDK
client to call NewRelic's API over HTTP/HTTPS.

To use this plugin, in addition to modification of "CounterPublisher" configuration, copy all the dependencies into the "bin" folder of the application you are trying to configure.
The dependencies include following:

Plain Old Text

config/newrelic.json

CounterPublisher.NewRelic.dll
CounterPublisher.NewRelic.dll.config

NewRelic.Platform.Sdk.dll
Newtonsoft.Json.dll
NLog.dll

ExitGames dependencies that should be provided by the application itself:

Plain Old Text

ExitGames.Logging.Log4Net.dll
ExitGamesLibs.dll
log4net.dll

Sender type is ExitGames.Diagnostics.Configuration.NewRelicAgentSettings.
Writer type is ExitGames.Diagnostics.Monitoring.Protocol.NewRelic.Agent.NewRelicWriter.

Metric data is sent as an HTTP POST of JSON data using this URI:
https://platform-api.newrelic.com/platform/v1/metrics.

The MIME-type for the POST is application/json.
The New Relic Plugins product is optimized to post the data at a frequency of once every 60 seconds.

POST frequency and limitations from the documentation:

Components: 500 Number of distinct components currently tracked per account.
Metrics per component: 10,000 Total number of unique metrics per component.
Take precautions to ensure metric names are not generated too dynamically.
Even if the number of metrics being sent in each individual post is small, over time they may add up to a large number of unique metrics.
When the number of metrics for a given component exceeds this limit, the server may start aggregating metrics together by globbing segments of the metric name with an asterisk (*).
Metrics per post: 20,000 Number of metrics sent per post.
A post may send data for multiple components in a single request as long as the total number of metrics in the request does not exceed this limit.
Frequency of post: 2 per minute Frequency of update.
Agents are expected to send data no more frequently than 1 per minute.

To start with it:

Register NewRelic account.
Obtain "LicenseKey" in your profile settings (APM -> .NET agent -> Reveal license key)
Copy dependencies to your application.
Files and instructions can be obtained from the github repository.
Setup configuration.

CounterPublisher.NewRelic Specific Settings

agentId: OPTIONAL.
Default: "com.exitgames.plugins.newrelic.agent".
agentName: OPTIONAL.
Default: "".
A human readable string denotes the name of this Agent in the New Relic service.
"{0}" occurrences -if any- will be replaced with Environment.MachineName.
If notconfigured or left empty, it will have the following format: "Agent Name: {agentId}. Agent Version: {version}".
So if both agentId and version are also not configured, the agentName will be: "Agent Name: com.exitgames.plugins.newrelic.agent. Agent Version: 1.0.0.".
licenseKey: REQUIRED.
Get it from NewRelic (APM -> .NET agent -> Reveal license key).
version: OPTIONAL.
Default: "1.0.0".

Note: NewRelic agents are identified by agentId and not by agentName.
Each NewRelic agent can have multiple instances.
Each instance is identified by agentName.

Minimal working NewRelic configuration

XML

  <Photon>
    <CounterPublisher enabled="True" senderType="ExitGames.Diagnostics.Configuration.NewRelicAgentSettings, CounterPublisher.NewRelic">
      <Sender
        protocol="ExitGames.Diagnostics.Monitoring.Protocol.NewRelic.Agent.NewRelicWriter, CounterPublisher.NewRelic"
        licenseKey="YOUR_LICENSE_KEY"
        sendInterval="60" />
    </CounterPublisher>
  </Photon>

Limitations

sendInterval can't be lower then 60 seconds. This makes the sendInterval setting kind of required for the NewRelic plugin configuration.
Plugin always send aggregated data over send interval: min, max, sum of sqr, count.
Plugin do not support metric with empty root namespaces.
Photon CounterPublisher's counter key format is transformed to NewRelic specific metric name format (Component/Label[/Attribute/...]).

Publishing to Amazon AWS CloudWatch

The CounterPublisher plugin for AWS CloudWatch is available here.

Amazon CloudWatch is a monitoring service for AWS cloud resources and the applications you run on AWS.
You can use Amazon CloudWatch to collect and track metrics, collect and monitor log files and set alarms.
Amazon CloudWatch can monitor AWS resources such as Amazon EC2 instances,
Amazon DynamoDB tables and Amazon RDS DB instances, as well as custom metrics generated by your applications and services and any log files your applications generate.
You can use Amazon CloudWatch to gain system-wide visibility into resource utilization, application performance and operational health.
You can use these insights to react and keep your application running smoothly.

To publish metrics to Amazon CloudWatch.
with "CounterPublisher", there is a separate implementation of writer plugin in CounterPublisher.AWS.CloudWatch.

It uses native Amazon SDK CloudWatch client to call API over HTTP/HTTPS.

To use this plugin, in addition to modification of "CounterPublisher" configuration, copy all the dependencies into the "bin" folder of the application you are trying to configure.

The dependencies include following:

Plain Old Text

CounterPublisher.AWS.CloudWatch.dll
AWSSDK.CloudWatch.dll
AWSSDK.Core.dll

ExitGames dependencies that should be provided by the application itself:

Plain Old Text

ExitGames.Logging.Log4Net.dll
ExitGamesLibs.dll
log4net.dll

Sender type is ExitGames.Diagnostics.Configuration.AWSCloudWatchSettings.
Writer type is ExitGames.Diagnostics.Monitoring.Protocol.AWS.CloudWatch.AWSCloudWatchWriter.

Amazon CloudWatch associates the data points with the specified metric.
If the specified metric does not exist, Amazon CloudWatch creates the metric.
It can take up to fifteen minutes for a new metric to appear in calls to the list-metrics action.

The size of a put-metric-datarequest is limited to 8 KB for HTTP GET requests and 40 KB for HTTP POST requests.

Although you can publish data points with time stamps as granular as one-thousandth of a second, CloudWatch aggregates the data to a minimum granularity of one minute.
CloudWatch records the average (sum of all items divided by number of items) of the values received for every 1-minute period, as well as number of samples, maximum value and minimum value for the same time period.

To start with it:

Register Amazon AWS account
Obtain "AccessKey", "SecretKey" in your profile settings
Select service url depending on preferred region and CloudWatch custom metrics namespace (i.e. "ExitGames/PhotonServer")
Copy dependencies to your application.
Files can be obtained from the github repository.
Setup configuration

Limitations:

Send interval can't be less then 60 seconds.
Plugin always send aggregated data over send interval: min, max, sum of sqr, count.
Plugin do not support metric with empty root namespaces.
Plugin tries to detect unit type based on common data size types and "Per" suffix for
choosing interval measures.
It is important, to choose metrics names wisely (which usually: [Namespace.]Key).
Refer to the rules in Counters Schemas for naming conventions and proper formatting of counters.

Choose application and modify ".config" file like this:

XML

<?xml version="1.0" encoding="utf-8" ?>
<configuration>

  <configSections>
    <section name="Photon" type="Photon.SocketServer.Diagnostics.Configuration.PhotonSettings, Photon.SocketServer"/>
  </configSections>

  <Photon>
    <CounterPublisher
        enabled="True"
        senderType="ExitGames.Diagnostics.Configuration.AWSCloudWatchSettings, CounterPublisher.AWS.CloudWatch">

      <Sender
        protocol="ExitGames.Diagnostics.Monitoring.Protocol.AWS.CloudWatch.AWSCloudWatchWriter, CounterPublisher.AWS.CloudWatch"
        senderId="{#CloudType#}/{#PrivateCloud#}/{#Region#}/{#Cluster#}/{0}"

        awsCloudWatchAccessKey = "..."
        awsCloudWatchSecretKey = "..."
        awsCloudWatchServiceUrl = "..."
        awsCloudWatchNamespace = "..."

        sendInterval="60" />
    </CounterPublisher>
  </Photon>

</configuration>

For this "CloudWatch" protocol implementation, "senderId" will dimension each metric key published.

Implementing custom performance counters

On architecture details take a look at Usage/Counter schemas.

At first, your application needs to track the desired values.
We are going to use the in-memory counters from ExitGames.Diagnostics.Counter for this purpose.

Make sure that your application has a reference to "libs\ExitGamesLibs.dll":

Counter Declaration

The next step is to create a counter definition class.
Add "using"-directives for ExitGames.Diagnostics.Counter and ExitGames.Diagnostics.Monitoring and declare your custom counter like shown below.
Add a PublishCounter attribute with a unique name; this makes sure that the counter value will be published by the the counter publisher (see below).

using ExitGames.Diagnostics.Counter;
using ExitGames.Diagnostics.Monitoring;

public static class MyCustomCounter
{
  [PublishCounter("MessagesPerSecond")]
  public static readonly CountsPerSecondCounter MessagesPerSecond = new CountsPerSecondCounter();
}

Refer to the rules in Counters Schemas for naming conventions and proper formatting of counters.

Counter Usage

In the appropriate place in your code, increment or decrement the counter.
For example, to count the number of sent messages per second, add the following code to the method that handles your SendMessage operation:


protected virtual void HandleSendMessageOperation(LitePeer peer, OperationRequest operationRequest)
{
  // [... operation logic ...]

  // increment performance counter:
  MyCustomCounter.MessagesPerSecond.Increment();
}

That's all.
You have done everything that is required to collect your performance data now.

Counter Publisher Initialisation

The next step is to publish the collected performance data.

For this, you need to initialize a "CounterPublisher" during the start-up of our application.
The "Setup" method of your application is a good place for this.

This example shows how to initialize the counter publisher and add your custom counter definition class.
You can choose an arbitrary name for the second parameter.
It's used as the category in which the counter will be displayed by the dashboard later.

protected override void Setup()
{
  // [... setup logic  ...]

  // initialize counter publisher and add your custom counter class(es):
  if (PhotonSettings.Default.CounterPublisher.Enabled)
  {
    CounterPublisher.DefaultInstance.AddStaticCounterClass(typeof(MyCustomCounter), "MyApplication");
    CounterPublisher.DefaultInstance.Start();
  }
}

or you can extended any .NET Photon application configuration to publish those in-memory metrics via <Counters/Schema> configuration element.

XML

  <Photon>
    <CounterPublisher ... >
      <Sender  ... />
        <Counters>
          <Schema namespace="MyApplication" type="...MyCustomCounter, YOUR_LIB_NAME" />
        </Counters>
    </CounterPublisher>
  </Photon>

Counter Publisher Configuration

The last step is to add the appropriate configuration for the counter publisher.
Configure it to send data to the IP / Port where your time series database is running.
Remember, that multicasting is only working in the same subnet; to publish performance data between different networks, use the concrete IP and make sure that the port is opened up in the firewall.

Take a look on any of Usage topics of this manual.

Example:

XML

<?xml version="1.0" encoding="utf-8" ?>
<configuration>

  <configSections>
    <section name="Photon" type="Photon.SocketServer.Diagnostics.Configuration.PhotonSettings, Photon.SocketServer"/>
  </configSections>

  <Photon>
    <CounterPublisher enabled="True">
      <Sender
        protocol="PhotonBinary"
        endpoint="udp://127.0.0.1:40001"
        senderId="servers/Realtime/Public/Local/Test/{0}" />

      <Counters>
        <Schema namespace="MyApplication" type="...MyCustomCounter, YOUR_LIB_NAME" />
      </Counters>
    </CounterPublisher>
  </Photon>

</configuration>

Implementing custom protocol

To take better understanding on how it can be done, take a look at:
CounterPublisher.AWS.CloudWatch project, CounterPublisher.NewRelic project, Application protocols topic.

The basic idea of implementing custom application protocol
is to create a separate .NET library (dll) and:

Implement settings class extension if needed based on ExitGames.Diagnostics.Configuration.CounterSampleSenderSettings.
Implement Writer interface ExitGames.Diagnostics.Monitoring.Protocol.ICounterSampleWriter which is main entry point.
Provide sample configuration to your "App.config" file.

Counter writer interface

ExitGames.Diagnostics.Monitoring.Protocol.ICounterSampleWriter is the main entry point into protocol plugin.
There is a basic abstract implementation of this "Writer" ExitGames.Diagnostics.Monitoring.Protocol.PhotonBinaryWriterBase which provides additional queue to aggregate super sampled data.

Implement ICounterSampleWriter methods:

public interface ICounterSampleWriter : IDisposable
{
    /// <summary>
    ///   Is writer ready to publish any data.
    ///   IsConnected status is subset of this status.
    ///   Default implementation is writer started and connected.
    /// </summary>
    bool Ready { get; }

    /// <summary>
    ///   Start writing.
    ///   Stop is handled by IDisposable implementation.
    ///   Can be called once.
    /// </summary>
    void Start(CounterSampleSenderBase sender);

    /// <summary>
    ///    Called by <see cref="CounterSampleSenderBase"/>
    ///    to write samples out to some storage.
    /// </summary>
    void Publish(CounterSampleCollection[] packages);
}

Configuration

The basic configuration class is ExitGames.Diagnostics.Configuration.CounterSampleSenderSettings.
You can extend it if some additional parameters are required to be setted up for the "Writer".

public class InfluxDB090TextWriterSettings : GraphiteTextWriterSettings
{
    /// <summary>
    ///   Do rewriting of keys provided by sampler to keep them distinct to tags part of the path.
    /// </summary>
    [ConfigurationProperty("rewriteKeys", IsRequired = false, DefaultValue = true)]
    public bool RewriteKeys
    {
        get
        {
            return (bool)this["rewriteKeys"];
        }
        set
        {
            this["rewriteKeys"] = value;
        }
    }

    /// <summary>
    ///   Separator used by InfluxDB instead of default DOT(.).
    /// </summary>
    [ConfigurationProperty("separator", IsRequired = false, DefaultValue = '.')]
    public char Separator
    {
        get
        {
            return (char)this["separator"];
        }
        set
        {
            this["separator"] = value;
        }
    }

    public InfluxDB090TextWriterSettings() { }

    public InfluxDB090TextWriterSettings(CounterSampleSenderSettings settings) : base(settings) { }
}

Usage

To use plugin:

Copy dependencies to the application.
Setup configuration with valid "Writer" type and "Configuration" when needed.

Example:

XML

<?xml version="1.0" encoding="utf-8" ?>
<configuration>

  <configSections>
    <section name="Photon" type="Photon.SocketServer.Diagnostics.Configuration.PhotonSettings, Photon.SocketServer"/>
  </configSections>

  <Photon>
    <CounterPublisher enabled="True">
      <Sender
        protocol="InfluxDB090GraphiteText"
        endpoint="udp://127.0.0.1:2003"
        senderId="CloudType.REALTIME.PrivateCloud.BLAH.Region.LOCAL.Cluster.EY.Machine.{0}" />
    </CounterPublisher>
  </Photon>

</configuration>

Troubleshooting

Is the counter publishing enabled? Check the configuration of your application. i.e. "deploy\CounterPublisher\bin\CounterPublisher.dll.config".
Check the Photon logs files at "\deploy\log*" for errors and destination database log files.
Make sure data is publishing out in logs (set DEBUG in log4net to ExitGames.Diagnostics):
To see what counters and values are currently published, DEBUG log level for ExitGames.Diagnostics package must be enforced in "{appname}\bin\log4net.config".
Then, go to take a look into log file at "Photon\Log{appname}.log" (this path specified in configuration of LogFileAppender in "log4net.config"):

XML
```
<logger name="ExitGames.Diagnostics">
  <level value="DEBUG" />
</logger>
```
Make sure that the appropriate ports are open in the firewall.
Check packets ip / port source / destinations using sniffer and netstat.
Check destination database receives packets using sniffer and extended log levels.
Check format of data / protocol is good inspecting data packets.
Make sure data is publishing out in packets (use wireshark sniffer).

List of counters from ExitGames.Diagnostics.Counter

This section discusses the available types of in-memory counters from the ExitGames.Diagnostics.Counter namespace in greater detail.

All counters implement the ICounter interface:

namespace ExitGames.Diagnostics.Counter
{
    public interface ICounter
    {
        string Name { get; }

        // Gets the type of the counter.

        CounterType CounterType { get; }

        // Gets the next value.
        float GetNextValue();

        // Increments the counter by one and returns the new value.
        long Increment();

        // Increments the counter by a given value and returns the new value.
        long IncrementBy(long value);

        // Decrements the counter by one and returns the new value.
        long Decrement();
    }
}

As you see, there are mainly methods to increment / decrement the counter value.

GetNextValue is the method to return the actual "value" of the counter.
This can be a calculated value - for example, when you call GetNextValue on an AverageCounter, the counter calculates the average of all increments / decrements since the previous call to GetNextValue
and returns that value.
GetNextValue resets internal values - you should never call it in your code manually, otherwise the published data will be incorrect.
The "CounterPublisher" makes calls to GetNextValue
at regular intervals.

NumericCounter

This is a very straight-forward counter.
It holds a single value; GetNextValue returns that single value and does not reset the counter.

Example: At the start of a 10-second-interval, the NumericCounter has a value of 5.
It is incremented 2 times, with values of 1 and 6 and decremented once with a value of 3, resulting in a value of 5+1+6-3 = 9 for this interval.

Typical use-case: The number of users that are currently logged on to your system.

public void AddToCache(ArrayList items)
{
   foreach (var item in items)
   {
     Cache.Insert(item);
   }

   MyCacheItemCounter.IncrementBy(items.Count);
}

public void RemoveFromCache(ArrayList items)
{
   foreach (var item in items)
   {
     Cache.Remove(item);
   }

   MyCacheItemCounter.IncrementBy( -items.Count);
}

// When the CounterPublisher calls MyCacheItemCounter.GetNextValue(), it always returns the current number of items in cache.

AverageCounter

The AverageCounter tracks how often it was called since the last reset and can be incremented by any value.
It returns the average amount per call in the current interval (= total amount / # of calls).

Example: In a 10-second-interval, the AverageCounter is incremented 4 times, with values of 1,2,4,5.
This is 12 in total, resulting in an average of 12/4 = 3 for this interval.

Typical use-case:
The average execution time of a certain operation, or the average text length per sent message.

public void WinGame(Player player, int coins)
{
   player.Account.Add(coins);

   AverageCoinsWonCounter.IncrementBy(coins);
}

public void LoseGame(Player player, int coins)
{
   player.Account.Remove(coins);

   AverageCoinsWonCounter.IncrementBy( -coins);
}

// When the CounterPublisher calls AverageCoinsWonCounter.GetNextValue(), it returns the average amount a player has won in the interval since the previous call to GetNextValue().
// This amount might as well be negative. The internal counter value is reset afterwards - i.e., if no Game was won or lost between two calls to GetNextValue(), the average is 0.

CountsPerSecondCounter

The CountsPerSecondCounter holds a single value, which can be incremented / decremented by any value.
It returns the average amount per second in the last interval.
You only need to increment / decrement the counter, it converts these calls in "per second" values automatically.

Example: In a 10-second-interval, the CountsPerSecondCounter is incremented 3 times, with values of 10, 20 and 70.
This is 100 in total, resulting in a value of 10 per second for this interval.

Typical use-case: The number of sent bytes per second on a network
interface, or the number of times a specific action is taken per second.

public void RaiseEvent()
{
   // do something to raise an event

   EventsPerSecondCounter.Increment();
}

// RaiseEvent can be called at any time - multiple times per second, or only once every few seconds and requires no fixed interval.
// When the CounterPublisher calls EventsPerSecondCounter.GetNextValue(), the counter checks how many seconds have passed since the the last call to GetNextValue().
// It calculates its return value as  "CurrentValue  / SecondsPassed".  The internal value is reset when GetNextValue() is called.

PerformanceCounterReader

The purpose of the PerformanceCounterReader is to read values from underlying Windows performance counters.
It is a read-only counter and can not be incremented / decremented.
This is failsafe, i.e. no errors are thrown if the performance counter does not exist or access is not allowed.

WindowsPerformanceCounter

The WindowsPerformanceCounter is a wrapper class for Windows performance counters; it provides read / write access to the underlying counter.
The performance counter need to exist and the reading / writing application needs to have the appropriate rights, otherwise an exception is raised.

For more information on Windows performance counters, MSDN is a good starting point.