The Cloud Foundry Blog

Running Resque Workers on Cloud Foundry

We introduced Cloud Foundry’s new “standalone” applications feature in the first post in this four-part series. In this second installment, we will look at the most common use of a standalone application–the worker process. Workers can be used for all kinds of asynchronous background jobs, such as updating search indexes, emailing all users with a password reset approaching, performing a database backup to persistent storage, or uploading new customer data from external storage. In this post, we will walk through an example of deploying workers to Cloud Foundry using Resque.

Resque Workers on Cloud Foundry

Let’s start by cloning the Resque Demo Example.

mycomp:dev$ git clone git://github.com/defunkt/resque.git
mycomp:dev$ cd resque/examples/demo
Let’s add a Gemfile to the example to ensure that Cloud Foundry can find all required gems.

 
source "http://rubygems.org" 
gem 'sinatra' 
gem 'resque' 
gem 'rake' 
gem 'json'

We’ll run “bundle install” and “bundle package” to package the gems in vendor/cache, and we’re ready to deploy.  The resque server is a Rack app, so we’ll deploy it to Cloud Foundry as such.

mycomp:demo$ vmc push resque-server
Would you like to deploy from the current directory? [Yn]:
Detected a Rack Application, is this correct? [Yn]:
Application Deployed URL [resque-server.cloudfoundry.com]: 
Memory reservation (128M, 256M, 512M, 1G, 2G) [128M]:
How many instances? [1]:
Create services to bind to 'resque-server'? [yN]: y
1: mongodb
2: mysql
3: postgresql
4: rabbitmq
5: redis
What kind of service?: 5
Specify the name of the service [redis-2a462]: redis-work-queue
Create another? [yN]:
Would you like to save this configuration? [yN]: y
Manifest written to manifest.yml.
Creating Application: OK
Creating Service [redis-work-queue]: OK
Binding Service [redis-work-queue]: OK
Uploading Application:
Checking for available resources: OK
Processing resources: OK
Packing application: OK
Uploading (21K): OK
Push Status: OK
Staging Application 'resque-server': OK
Starting Application 'resque-server': OK
Let’s have a look at the resque-server app and add some jobs to the queue:
Now that we have some jobs, it’s time to deploy some workers!  First, we need to rename the generated manifest.yml for the Rack app, so it won’t automatically be used in the push.  We can use it again later by doing a “vmc push –manifest server-manifest.yml”.  Now, let’s push the app again as a standalone worker app.
mycomp:demo$ mv manifest.yml server-manifest.yml
mycomp:demo$ vmc push resque-worker
Would you like to deploy from the current directory? [Yn]:
Detected a Rack Application, is this correct? [Yn]: n
1: Rails
2: Spring
3: Grails
4: Lift
5: JavaWeb
6: Standalone
7: Sinatra
8: Node
9: Rack
Select Application Type: 6
Selected Standalone Application
1: java
2: node
3: node06
4: ruby18
5: ruby19
Select Runtime [ruby18]:
Selected ruby18
Start Command: bundle exec rake VERBOSE=true QUEUE=default resque:work
Application Deployed URL [None]:
Memory reservation (128M, 256M, 512M, 1G, 2G) [128M]:
How many instances? [1]:
Bind existing services to 'resque-worker'? [yN]: y
1: redis-work-queue
Which one?: 1
Bind another? [yN]:
Create services to bind to 'resque-worker'? [yN]:
Would you like to save this configuration? [yN]: y
Manifest written to manifest.yml.
Creating Application: OK
Binding Service [redis-work-queue]: OK
Uploading Application:
Checking for available resources: OK
Processing resources: OK
Packing application: OK
Uploading (0K): OK
Push Status: OK
Staging Application 'resque-worker': OK
Starting Application 'resque-worker': OK

So we’ve pushed resque-worker as a standalone app with a Ruby runtime. We gave the command “bundle exec rake VERBOSE=true QUEUE=default resque:work” to start the worker. It is recommended to use bundle exec to ensure that all required gems are available. Since resque-worker does not have a web front-end, we selected “None” for URL. Lastly, we bound the app to the same Redis service used by resque-server. If you’ve perused the resque demo example, you may have noticed that it is setup to connect to a local Redis service. However, we didn’t change the code before we pushed it. How will the app connect to the provisioned Redis service? Since we used the Ruby runtime provided by Cloud Foundry, the app will benefit from the new Ruby auto-reconfiguration support. Cloud Foundry will automatically replace the local Redis connection with a connection to the Redis service we bound to the application!

Let’s check the logs and see if the worker completed that job:
mycomp:demo$ vmc logs resque-worker
====> /logs/stdout.log <====

Loading Redis auto-reconfiguration.
*** Starting worker ubuntu:10245:default
Auto-reconfiguring Redis.
*** got: (Job{default} | Demo::Job | [{}])
Processed a job!
*** done: (Job{default} | Demo::Job | [{}])
*** got: (Job{default} | Demo::Job | [{}])
Processed a job!
*** done: (Job{default} | Demo::Job | [{}])
And we can verify that the worker has registered through the web interface:
We can even scale the workers up.
mycomp:demo$ vmc instances resque-worker +2
Scaling Application instances up to 3: OK
And now the web interface shows three workers:

And there you have it! We can now deploy Resque workers as standalone apps on Cloud Foundry. Clone the Cloud Foundry resque-sample and try it out for yourself!

Conclusion

Cloud Foundry now provides improved Resque support through standalone applications, as well as support for other Ruby worker libraries or apps. If you can package all the bits and provide a start command, you can run it on Cloud Foundry! In the next installment in this series, we will explore another example of workers in action using Spring integration. Stay tuned!

- Jennifer Hickey The Cloud Foundry Team
Don’t have a Cloud Foundry account yet?  Sign up for free today

This entry was posted in CloudFoundry and tagged , , . Bookmark the permalink.

Comments are closed.