Logplex - Queue Time

"Queue Time is the amount of time (in milliseconds) between the Heroku Router accepting the request and the web dyno starting to process the request."

If you want to auto-scale based on Queue Time percentiles or averages, then you have to create a logdrain and point it at our secure endpoint:

heroku drains:add https://logdrain.hirefire.io

Once created, take note of the drain token:

heroku drains | grep hirefire

The format of the token is  d.00000000-0000-0000-0000-000000000000 . The metrics will become available one minute after adding the logdrain.

Ruby Application Integration

You now have to configure your application to actually log the queue time to STDOUT so that the Heroku Logplex picks it up and forwards it to our logdrain.

If you have a Ruby application backed by Rack (Rails, Hanami, ...), then you can use the hirefire-resource gem for easy setup.

gem "hirefire-resource"

If you're using Rails, it'll automatically inject the required middleware. For any other Rack-based application such as Sinatra or Hanami you'll have to manually add the HireFire middleware at the top of the middleware stack. Open up your middleware configuration (usually located in config.ru) and add the following:

use HireFire::Middleware # at the top
# any other middleware here
run MyApp # at the bottom

Now that the middleware has been set up, all that you have to do is to create a hirefire configuration file. In a Rails application you can simply create an initializer. When working with non-Rails applications, you can just create the file wherever you wish, as long as it's loaded in at application boot time.

Once you've created the file, add the following configuration to it.

HireFire::Resource.configure do |config|
  config.log_queue_metrics = true
end

Your application, once deployed to Heroku, should now be ready to autoscale using the Web.Logplex.QueueTime strategy.

Non-Ruby Application Integration

For applications written in other languages, you have to write the following format to STDOUT on each HTTP request that you receive inside of your application:

[hirefire:router] queue={TIME}ms

For example:

[hirefire:router] queue=162ms

In order to figure out what TIME should be, you need two pieces of information:

• The current time in milliseconds.
• The value stored in the X-Request-Start header. This header is added to the HTTP request by the Heroku Router and represents the time (as a timestamp with millisecond precision) that the request was accepted by the Heroku Router. This header will be available at the application layer.

The following formula will give you the TIME metric you need:

current_time_in_milliseconds - X-Request-Start

In Ruby this can be achieved like so:

(Time.now.to_f * 1000).to_i - env["X-Request-Start"].to_i

Do the equivalent in your programming language.

It's important to do this as high up in your web stack as possible on every HTTP request. If you have a middleware / pipeline layer of some kind, put this logic there (preferably at the top for the best results).

Note that when you write the  [hirefire:router] queue={TIME}ms entry to STDOUT, be sure that you don't run it through a log formatter that prefixes additional data such as, for example, timestamps. Simply puts , print , echo , write  (etc) it directly.

HireFire UI

Once you've made the necessary changes to your application, and the logdrain has been created, and you've taken note of the drain token, log in to HireFire and add your Heroku application if you haven't already. In the application form you'll see a field labeled "Logplex Drain Token". Add the token there and save it.

Now proceed to create a manager, using  web  as its name (must be called web , all lowercase, to reflect the Procfile entry web ), set the Type to  Web.Logplex.QueueTime , configure the rest of the options to your liking and then save and enable the manager.

Once all of that's done, HireFire will be auto-scaling your web dynos.

Details

This strategy allows you to stream your application/platform logs to HireFire's logdrain. The logdrain parses metric data out of the received logs to build a temporary data stream of metrics that HireFire's primary system can utilize to auto-scale your dyno formation.

Logs are consumed, parsed and stored in the logdrain within roughly a second after Heroku emits them, providing HireFire with metrics accurate to the second when it requests them during the minutely manager checkups.

HireFire does not extract any information from your logs other than what is necessary for auto-scaling, which are the metrics emitted by Heroku's Router (service, connect), Application (queue), Runtime Metrics (load1m, load5m, load15m).