Rails auto-scaling on Heroku – Bibliographic Wilderness




	Skip to content

	
		
			
				
					
												Bibliographic Wilderness

										


				
									
						Menu					
				
			

		
	


				
				About
	Contact


		
	
	
	
	
		

	
		

		
			

			
			Rails auto-scaling on Heroku
				
			
				jrochkind			
							
					General				
			
			
			
				January 27, 2021January 30, 2021			

					

	
	
		
We are investigating moving our medium-small-ish Rails app to heroku.  




We looked at both the Rails Autoscale add-on available on heroku marketplace, and the hirefire.io service which is not listed on heroku marketplace and I almost didn’t realize it existed. 




I guess hirefire.io doesn’t have any kind of a partnership with heroku, but still uses the heroku API to provide an autoscale service.  hirefire.io ended up looking more fully-featured and lesser priced than Rails Autoscale; so the main service of this post is just trying to increase visibility of hirefire.io and therefore competition in the field, which benefits us consumers. 




Background: Interest in auto-scaling Rails background jobs




At first I didn’t realize there was such a thing as “auto-scaling” on heroku, but once I did, I realized it could indeed save us lots of money.  




I am more interested in scaling Rails background workers than I a web workers though — our background workers are busiest when we are doing “ingests” into our digital collections/digital asset management system, so the work is highly variable. Auto-scaling up to more when there is ingest work piling up can give us really nice inget throughput while keeping costs low. 




On the other hand, our web traffic is fairly low and probably isn’t going to go up by an order of magnitude (non-profit cultural institution here). And after discovering that a “standard” dyno is just too slow, we will likely be running a performance-m or performance-l anyway — which likely can handle all anticipated traffic on it’s own. If we have an auto-scaling solution, we might configure it for web dynos, but we are especially interested in good features for background scaling. 




There is a heroku built-in autoscale feature, but it only works for performance dynos, and won’t do anything for Rails background job dynos, so that was right out. 




That could work for Rails bg jobs, the Rails Autoscale add-on on the heroku marketplace; and then we found hirefire.io. 




Pricing: Pretty different




hirefire




As of now January 2021, hirefire.io has pretty simple and affordable pricing. $15/month/heroku application. Auto-scaling as many dynos and process types as you like. 




hirefire.io by default can only check into your apps metrics to decide if a scaling event can occur once per minute. If you want more frequent than that (up to once every 15 seconds), you have to pay an additional $10/month, for $25/month/heroku application. 




Even though it is not a heroku add-on, hirefire does advertise that they bill pro-rated to the second, just like heroku and heroku add-ons. 




Rails autoscale




Rails autoscale has a more tiered approach to pricing that is based on number and type of dynos you are scaling.  Starting at $9/month for 1-3 standard dynos, the next tier up is $39 for up to 9 standard dynos, all the way up to $279 (!) for 1 to 99 dynos.  If you have performance dynos involved, from $39/month for 1-3 performance dynos, up to $599/month for up to 99 performance dynos. 




For our anticipated uses… if we only scale bg dynos, I might want to scale from (low) 1 or 2 to (high) 5 or 6 standard dynos, so we’d be at $39/month. Our web dynos are likely to be performance and I wouldn’t want/need to scale more than probably 2, but that puts us into performance dyno tier, so we’re looking at $99/month. 




This is of course significantly more expensive than hirefire.io’s flat rate. 




Metric Resolution




Since Hirefire had an additional charge for finer than 1-minute resolution on checks for autoscaling, we’ll discuss resolution here in this section too. Rails Autoscale has same resolution for all tiers, and I think it’s generally 10 seconds, so approximately the same as hirefire if you pay the extra $10 for increased resolution. 




Configuration




Let’s look at configuration screens to get a sense of feature-sets. 




Rails Autoscale




web dynos




To configure web dynos, here’s what you get, with default values:








The metric Rails Autoscale uses for scaling web dynos is time in heroku routing queue, which seems right to me — when things are spending longer in heroku routing queue before getting to a dyno, it means scale up. 




worker dynos




For scaling worker dynos, Rails Autoscale can scale dyno type named “worker” — it can understand ruby queuing libraries Sidekiq, Resque, Delayed Job, or Que. I’m not certain if there are options for writing custom adapter code for other backends. 




Here’s what the configuration options are — sorry these aren’t the defaults, I’ve already customized them and lost track of what defaults are. 








You can see that worker dynos are scaled based on the metric “number of jobs queued”, and you can tell it to only pay attention to certain queues if you want. 




Hirefire




Hirefire has far more options for customization than Rails Autoscale, which can make it a bit overwhelming, but also potentially more powerful. 




web dynos




You can actually configure as many Heroku process types as you have for autoscale, not just ones named “web” and “worker”. And for each, you have your choice of  several metrics to be used as scaling triggers. 




For web, I think Queue Time (percentile, average) matches what Rails Autoscale does, configured to percentile, 95, and is probably the best to use unless you have a reason to use another.  (“Rails Autoscale tracks the 95th percentile queue time, which for most applications will hover well below the default threshold of 100ms.“)

Here’s what configuration Hirefire makes available if you are scaling on “queue time” like Rails Autoscale, configuration may vary for other metrics. 








I think if you fill in the right numbers, you can configure to work equivalently to Rails Autoscale. 




worker dynos




If you have more than one heroku process type for workers — say, working on different queues — Hirefire can scale the independently, with entirely separate configuration. This is pretty handy, and I don’t think Rails Autoscale offers this.  (update i may be wrong, Rails Autoscale says they do support this, so check on it yourself if it matters to you). 




For worker dynos, you could choose to scale based on actual “dyno load”, but I think this is probably mostly for types of processes where there isn’t the ability to look at “number of jobs”. A “number of jobs in queue” like Rails Autoscale does makes a lot more sense to me as an effective metric for scaling queue-based bg workers. 




Hirefire’s metric is slightly difererent than Rails Autoscale’s “jobs in queue”. For recognized ruby queue systems (a larger list than Rails Autoscale’s; and you can write your own custom adapter for whatever you like), it actually measures jobs in queue plus workers currently busy. So queued+in-progress, rather than Rails Autoscale’s just queued. I actually have a bit of trouble wrapping my head around the implications of this, but basically, it means that  Hirefire’s “jobs in queue” metric strategy is intended to try to scale all the way to emptying your queue, or reaching your max scale limit, whichever comes first.  I think this may make sense and work out at least as well or perhaps better than Rails Autoscale’s approach?




Here’s what configuration Hirefire makes available for worker dynos scaling on “job queue” metric. 








Since the metric isn’t the same as Rails Autosale, we can’t configure this to work identically. But there are a whole bunch of configuration options, some similar to Rails Autoscale’s. 




The most important thing here is that “Ratio” configuration. It may not be obvious, but with the way the hirefire metric works, you are basically meant to configure this to equal the number of workers/threads you have on each dyno. I have it configured to 3 because my heroku worker processes use resque, with resque_pool, configured to run 3 resque workers on each dyno. If you use sidekiq, set ratio to your configured concurrency — or if you are running more than one sidekiq process, processes*concurrency.  Basically how many jobs your dyno can be concurrently working is what you should normally set for ‘ratio’. 




Hirefire not a heroku plugin




Hirefire isn’t actually a heroku plugin. In addition to that meaning separate invoicing, there can be some other inconveniences. 




Since hirefire only can interact with heroku API, for some metrics  (including the “queue time” metric that is probably optimal for web dyno scaling) you have to configure your app to log regular statistics to heroku’s “Logplex” system. This can add a lot of noise to your log, and for heroku logging add-ons that are tired based on number of log lines or bytes, can push you up to higher pricing tiers. 




If you use paperclip, I think you should be able to use the log filtering feature to solve this, keep that noise out of your logs and avoid impacting data log transfer limits. However, if you ever have cause to look at heroku’s raw logs, that noise will still be there. 




Support and Docs




I asked a couple questions of both Hirefire and Rails Autoscale as part of my evaluation, and got back well-informed and easy-to-understand answers quickly from both. Support for both seems to be great. 




I would say the documentation is decent-but-not-exhaustive for both products. Hirefire may have slightly more complete documentation. 




Other Features?




There are other things you might want to compare, various kinds of observability (bar chart or graph of dynos or observed metrics) and notification. I don’t have time to get into the details (and didn’t actually spend much time exploring them to evaluate), but they seem to offer roughly similar features. 




Conclusion




Rails Autoscale is quite a bit more expensive than hirefire.io’s flat rate, once you get past Rails Autoscale’s most basic tier (scaling no more than 3 standard dynos).  




It’s true that autoscaling saves you money over not, so even an expensive price could be considered a ‘cut’ of that, and possibly for many ecommerce sites even $99 a month might a drop in the bucket (!)…. but this price difference is so significant with hirefire (which has flat rate regardless of dynos), that it seems to me it would take a lot of additional features/value to justify. 




And it’s not clear that Rails Autoscale has any feature advantage. In general, hirefire.io seems to have more features and flexibility. 




Until 2021, hirefire.io could only analyze metrics with 1-minute resolution, so perhaps that was a “killer feature”?




Honestly I wonder if this price difference is sustained by Rails Autoscale only because most customers aren’t aware of hirefire.io, it not being listed on the heroku marketplace? Single-invoice billing is handy, but probably not worth $80+ a month.    I guess hirefire’s logplex noise is a bit inconvenient?




Or is there something else I’m missing? Pricing competition is good for the consumer.




And are there any other heroku autoscale solutions, that can handle Rails bg job dynos, that I still don’t know about?




update a day after writing djcp on a reddit thread writes:




I used to be a principal engineer for the heroku add-ons program.
One issue with hirefire is they request account level oauth tokens that essentially give them ability to do anything with your apps, where Rails Autoscaling worked with us to create a partnership and integrate with our “official” add-on APIs that limits security concerns and are scoped to the application that’s being scaled.
Part of the reason for hirefire working the way it does is historical, but we’ve supported the endpoints they need to scale for “official” partners for years now.
A lot of heroku customers use hirefire so please don’t think I’m spreading FUD, but you should be aware you’re giving a third party very broad rights to do things to your apps. They probably won’t, of course, but what if there’s a compromise?
“Official” add-on providers are given limited scoped tokens to (mostly) only the actions / endpoints they need, minimizing blast radius if they do get compromised.





You can read some more discussion at that thread. 


        
    
			

			
			Share this:
	Twitter
	Facebook
	





	


	
			Tagged
	ruby

	


	
			
				
					

		
		
			Published by jrochkind

		


		
						
				View all posts by jrochkind			
		

	

		
			Published
			January 27, 2021January 30, 2021		

	



			
	
		Post navigation

		Previous Post Managed Solr SaaS Options
Next Post Product management


	
			


	
			
			2 thoughts on “Rails auto-scaling on Heroku”		


		
			
			
				
					
												Adam (Rails Autoscale) says:					


					
						January 29, 2021 at 3:29 pm					


									

				
					This is a well-done and fair comparison. I appreciate the write-up and feedback.

Thanks also for the correction regarding multiple worker processes. That’s on me for not having it documented (which I have since updated).

				


							
		
	
			
				
					
												jrochkind says:					


					
						February 1, 2021 at 9:50 am					


									

				
					As far as hirefire’s CPU load metric (available on any type of dyno), note that Google suggests:

> In a majority of cases (although certainly not in all), we’ve found that simply using CPU consumption as the signal for provisioning works well, for the following reasons…

https://sre.google/sre-book/handling-overload/

				


							
		



		
	
	
		
		Leave a Reply Cancel reply




	Enter your comment here...
	





	
		Fill in your details below or click an icon to log in:

			
				
									
			
	
				
									
			
	
			
			
	
				
									
			
	
				
									
			


	


	
		
			
				
			


				
				
					Email (required) (Address never made public)
					

				

				
					Name (required)
					

				

				
					Website
					

				

			

			
		

	


	
		
			
				
			


				
				
				
				
						
			
			You are commenting using your WordPress.com account.			
				( Log Out / 
				Change )
			
			
		

					

	
		

	


	
		
			
				
			


				
				
				
				
						
			
			You are commenting using your Google account.			
				( Log Out / 
				Change )
			
			
		

					

	
		

	


	
		
			
				
			


				
				
				
				
						
			
			You are commenting using your Twitter account.			
				( Log Out / 
				Change )
			
			
		

					

	
		

	


	
		
			
				
			


				
				
				
				
						
			
			You are commenting using your Facebook account.			
				( Log Out / 
				Change )
			
			
		

					

	
		

	



	
		Cancel

		Connecting to %s

	








	 Notify me of new comments via email.
 Notify me of new posts via email.






 






	

	




		
		
	



	Bibliographic Wilderness is a blog by Jonathan Rochkind about digital library services, ruby, and web development. Contact


				
					Search for:
					
				
				
			
Email Subscription

			
									Enter your email address to subscribe to this blog and receive notifications of new posts by email.

						
							
							Join 215 other followers							

						

											
						
							Email Address:						

											

				
				
                    
                    
                    
                    
                    
					                    
	                    Subscribe                    
                

            

			

		
		Recent Posts

			
					Product management
											February 3, 2021
									
	
					Rails auto-scaling on Heroku
											January 27, 2021
									
	
					Managed Solr SaaS Options
											January 12, 2021
									
	
					Gem authors, check your release sizes
											January 11, 2021
									
	
					Every time you decide to solve a problem with code…
											January 10, 2021
									



		Archives
		Archives
		Select Month
 February 2021  (1)
 January 2021  (4)
 December 2020  (1)
 November 2020  (3)
 October 2020  (2)
 September 2020  (3)
 August 2020  (2)
 April 2020  (1)
 March 2020  (1)
 December 2019  (1)
 October 2019  (1)
 September 2019  (1)
 August 2019  (2)
 June 2019  (2)
 April 2019  (3)
 March 2019  (3)
 February 2019  (1)
 December 2018  (1)
 November 2018  (1)
 October 2018  (2)
 September 2018  (4)
 August 2018  (1)
 June 2018  (2)
 May 2018  (1)
 April 2018  (1)
 March 2018  (3)
 February 2018  (1)
 January 2018  (1)
 November 2017  (1)
 October 2017  (1)
 September 2017  (1)
 August 2017  (3)
 July 2017  (1)
 May 2017  (4)
 April 2017  (2)
 March 2017  (9)
 February 2017  (5)
 January 2017  (1)
 December 2016  (7)
 November 2016  (4)
 September 2016  (1)
 August 2016  (4)
 June 2016  (2)
 May 2016  (4)
 March 2016  (2)
 February 2016  (1)
 January 2016  (2)
 November 2015  (2)
 October 2015  (5)
 September 2015  (7)
 August 2015  (5)
 July 2015  (4)
 May 2015  (3)
 April 2015  (5)
 March 2015  (2)
 February 2015  (2)
 January 2015  (4)
 December 2014  (2)
 November 2014  (2)
 October 2014  (6)
 September 2014  (5)
 August 2014  (3)
 July 2014  (3)
 June 2014  (1)
 May 2014  (3)
 April 2014  (5)
 March 2014  (9)
 February 2014  (4)
 January 2014  (5)
 December 2013  (5)
 November 2013  (14)
 October 2013  (4)
 September 2013  (6)
 August 2013  (2)
 July 2013  (7)
 June 2013  (10)
 May 2013  (4)
 April 2013  (5)
 March 2013  (8)
 February 2013  (6)
 January 2013  (16)
 December 2012  (8)
 November 2012  (14)
 October 2012  (6)
 September 2012  (6)
 August 2012  (2)
 July 2012  (5)
 June 2012  (5)
 May 2012  (7)
 April 2012  (12)
 March 2012  (6)
 February 2012  (7)
 January 2012  (6)
 December 2011  (5)
 November 2011  (7)
 October 2011  (5)
 September 2011  (10)
 August 2011  (4)
 July 2011  (5)
 June 2011  (7)
 May 2011  (8)
 April 2011  (5)
 March 2011  (13)
 February 2011  (4)
 January 2011  (12)
 December 2010  (7)
 November 2010  (5)
 October 2010  (5)
 September 2010  (10)
 August 2010  (6)
 July 2010  (7)
 June 2010  (5)
 May 2010  (8)
 April 2010  (8)
 March 2010  (14)
 February 2010  (3)
 January 2010  (3)
 December 2009  (4)
 November 2009  (2)
 October 2009  (3)
 September 2009  (9)
 August 2009  (1)
 July 2009  (4)
 June 2009  (7)
 May 2009  (14)
 April 2009  (17)
 March 2009  (21)
 February 2009  (11)
 January 2009  (16)
 December 2008  (12)
 November 2008  (30)
 October 2008  (12)
 September 2008  (3)
 July 2008  (4)
 June 2008  (2)
 May 2008  (11)
 April 2008  (3)
 March 2008  (4)
 February 2008  (10)
 January 2008  (7)
 December 2007  (4)
 November 2007  (4)
 September 2007  (1)
 August 2007  (3)
 June 2007  (6)
 May 2007  (12)
 April 2007  (11)
 March 2007  (9)




			Feeds
 RSS - Posts
 RSS - Comments

Recent Comments
					
							jrochkind on Rails auto-scaling on Heroku						
	
							Adam (Rails Autoscale) on Rails auto-scaling on Heroku						
	
							On catalogers, programmers, and user tasks – Gavia Libraria on Broad categories from class numbers						
	
							Replacing MARC – Gavia Libraria on Linked Data Caution						
	
							jrochkind on Deep Dive: Moving ruby projects from Travis to Github Actions for CI						
	
							jrochkind on Deep Dive: Moving ruby projects from Travis to Github Actions for CI						
	
							jrochkind on Deep Dive: Moving ruby projects from Travis to Github Actions for CI						
	
							eregontp on Deep Dive: Moving ruby projects from Travis to Github Actions for CI						


				Top Posts
	
					yes, product owner and technical lead need to be different people					
	
					Bootstrap 3 to 4: Changes in how font size, line-height, and spacing is done. Or "what happened to $line-height-computed."					
	
					Dealing with legacy and externally loaded code in webpack(er)					
	
					ActiveRecord: Atomic check-and-update through optimistic locking					
	
					Are you talking to Heroku redis in cleartext or SSL?					

Top Clicks
	bibwild.files.wordpress.c…
	apidock.com/rails/ActiveR…
	github.com/mperham/sideki…
	bibwild.files.wordpress.c…
	opensolr.com

			

			
			


		


		
							
					
						
													
											A blog by Jonathan Rochkind.  All original content licensed CC-BY. 



									

						
											

				

						
				
				Create a website or blog at WordPress.com
				
			
				
							

		
	





	
	



		
		

		
					


	
	
		
			Add your thoughts here... (optional)
			
			Post to
			


			
				Cancel
				
				
				
				
				
				
			


					


		

	


	
				

	
		

		Privacy & Cookies: This site uses cookies. By continuing to use this website, you agree to their use. 

To find out more, including how to control cookies, see here:
				
			Cookie Policy