The Cloud Foundry Blog

Now You Can Tunnel into Any Cloud Foundry Data Service

Cloud Foundry now enables developers to open a tunnel to any Cloud Foundry data service via a local port. Now you can use familiar client applications to directly analyze, manipulate, or port the data contained in the data services that are bound to your Cloud Foundry applications. This solution is ideal for importing and exporting data when moving application between different clouds running Cloud Foundry (e.g. CloudFoundry.com and Micro Cloud Foundry), for debugging during development, as well as ad-hoc queries and modification to data in a deployed application.

This new feature is made available with a preview release of Cloud Foundry command line tool (‘VMC’). Underlying it is a simple Ruby application, named Caldecott (after the Caldecott Tunnel in California), that gets deployed to your CloudFoundry.com account to connect to services in your account. Caldecott is currently being provided as a preview, as the team is working on making it more robust and performant in the coming weeks. Caldecott provides an HTTP endpoint that facilitate a port forward on your local box. VMC contains built in scripts to run some of the popular client software in the local environment and automatically connect to specified services, or simply create the connection to the service for you to connect using any client software of your choice. Let’s take a look at how this works.

First, install or update your Cloud Foundry command line tool (‘VMC’) to the latest preview version using the following command:

gem install vmc --pre

You can verify that you got the right version using:

vmc -v

which should show the version to be 0.3.14.beta.4 or higher.

You also need the caldecott gem, which you can install using:

gem install caldecott

Note that currently the caldecott gem requires the ‘eventmachine’ gem, which in turn requires native compilation. We are working to simplify this.

The first step is to make sure that you are targeting the Cloud Foundry instance that you want to connect to and log in.  See “Getting Started Guide – Command Line (VMC) users” if you are unfamiliar with VMC.

Next, review all of the services that you have:

vmc services

Under the Provisioned Services section, you will see all of the services that you have created.  For example, here’s an output of my services:

=========== Provisioned Services ============

    +------------------+------------+
    | Name             | Service    |
    +------------------+------------+
    | mongodb-12345    | mongodb    |
    | mysql-12345      | mysql      |
    | postgresql-12345 | postgresql |
    | redis-12345      | redis      |
    +------------------+------------+

Now, you can connect to any service listed through a simple command:

vmc tunnel <servicename> [--port]

The port is an optional parameter that allows you to define a specific port number that you want. By default, VMC will automatically select an available port in the 10000 range. The service name parameter is also optional, you can interactively select the service from a list to establish the connection by using the command without any parameters.

vmc tunnel

As an example, let’s connect to the PostgreSQL service in the list above by typing the following command:

 vmc tunnel postgresql-12345

VMC will first determine if an existing Caldecott application instance has been deployed, if not, it will deploy a new one.

Trying to deploy Application: 'caldecott'.
    Create a password: 

Whenever you create a tunnel, you will be prompted for a password. You will need to furnish a new password initially for the first time the application is deployed, so that VMC can secure future connections to the Caldecott application instance. For additional service connections, you will have to supply the same password that was originally created for security. Following the password creation, you should see the following progress output.

Uploading Application:
 Checking for available resources: OK
 Processing resources: OK
 Packing application: OK
 Uploading (1K): OK
Push Status: OK
Binding Service [postgresql-12345]: OK
Staging Application: OK
Starting Application: OK
Getting tunnel connection info: .OK

You can see that VMC is uploading and starting the Caldecott application and binding the specified service to that application instance. In successive VMC tunnel command calls, it will not be necessary to create a new application instance. Instead, the existing instance will be stopped, the specified service will be bound to the application, and then the same Caldecott application will get restarted.

The next section of the output will display the credentials necessary to connect to the service. These credentials were originally automatically generated by the Cloud Controller when provisioning the service for you. These are the credentials that Cloud Foundry’s staging code will automatically insert into your application’s configurations to connect to the service.

Service connection info:
 username : ua76003139485469989d57b22f9cb83xx
 password : p8214ebd2a7254698913e01c86a4432xx
 name : da281fa2162f947028876d0500ae31cxx

In the case of a database service, the username and password are displayed, along with the database name. Lastly, when the tunnel is established, VMC will offer to start a known client that exists in your execution path or allow you to connect to the local port with your choice of client application.

For the Postgres example, the following prompt is displayed:

Starting tunnel to postgresql-12345 on port 10000.
1: none
2: psql
What client would you like to start?: 

Enter 1 to connect with your own client application and the previously displayed credentials, or 2 to automatically start the client application and pass in the credentials. If you choose the latter, VMC will start the client automatically (in this case psql).

What client would you like to start?: 2
Launching 'PGPASSWORD='p8214ebd2a7254698913e01c86a4432xx' psql -h localhost -p 10001 -d da281fa2162f947028876d0500ae31cxx -U ua76003139485469989d57b22f9cb83xx -w'

psql (9.0.4)
Type "help" for help.
da281fa2162f947028876d0500ae31cxx=> d
                     List of relations
 Schema | Name  | Type  |               Owner
--------+-------+-------+-----------------------------------
 public | hotel | table | ua76003139485469989d57b22f9cb83xx

da281fa2162f947028876d0500ae31cxx> select * from hotel;
  1 | 3555 S. Ocean Drive                    | Hollywood           | USA           | Westin Diplomat                | 199.00 | FL        | 33019
  2 | 890 Palm Bay Rd NE                     | Palm Bay            | USA           | Jameson Inn                    |  60.00 | FL        | 32905
  3 | The Cottage, Southampton Business Park | Southampton         | UK            | Chilworth Manor                | 199.00 | Hants     | SO16 7JF
...

Alternatively, you can supply the desired client to execute at the invocation of the tunnel command to bypass the prompt:

vmc tunnel postgresql-12345 psql

That’s it, at this point, you can choose to view, modify, or export the data from your service. 

Caldecott is currently a preview software, and we are working hard to improve it. Please feel free to send us any feedback via the Cloud Foundry Support Forums

- The Cloud Foundry Team

Don’t have a Cloud Foundry account yet? Signup for free today

This entry was posted in Cloud Foundry. Bookmark the permalink.

22 Responses to Now You Can Tunnel into Any Cloud Foundry Data Service

  1. Pingback: Java Reporting Engine is now available on Cloud Foundry via JasperReports | Blog

  2. Javier Beneito Barquero says:

    I’m trying to connect to a PostgreSQL database through Caldecott using SQLExplorer, a plugin for Eclipse (actually STS) to connect to databases.

    I followed your steps until the selection of the client, when I chose 1 (note, I’m going to use the same values for the database as they appears in this blog, regardless what I’m actually using):

    Starting tunnel to postgresql-12345 on port 10000.
    1: none
    2: psql
    Which client would you like to start?: 1
    Open another shell to run command-line clients or
    use a UI tool to connect using the displayed information.
    Press Ctrl-C to exit…

    However, SQLExporer can’t connect remotely:

    “Connection refused: Check that the hostname and the port are correct and that the postmaster is accepting TCP/IP connections”

    My questions are:
    - What is the URL to connect to? I’m trying api.cloudfoundry.com, the same I have in the target for “vmc”
    Actually, the URL would be something like:
    jdbc:postgresql://api.cloudfoundry.com:10000/da281fa2162f947028876d0500ae31cxx

    Using:
    username : ua76003139485469989d57b22f9cb83xx
    password : p8214ebd2a7254698913e01c86a4432xx

    - Does this host accept the connection I’m trying?

    Thank you very much.

  3. Pingback: MongoDB, Spring data, & Cloud Foundry – The pure JAVA approach (No JAVA Beans) « dittmarconsulting

  4. Your setting up a tunnel to the cloud foundry service, and therefore your also need to use the tunnel to gain access to the service.

    api.cloudfoundry.com should be localhost.

    jdbc:postgresql://localhost:10000/da281fa2162f947028876d0500ae31cxx
    Using:
    username : ua76003139485469989d57b22f9cb83xx
    password : p8214ebd2a7254698913e01c86a4432xx

  5. Pingback: ThemaTweets – Visualizing the french elections buzz on Cloud Foundry | Blog

  6. Pingback: Cloud Foundry Now Supports the Rails Console | Blog

  7. Ritesh says:

    I’m getting an error trying to install caldecott gem, here is what I see in the gem_make.out (I’m running Windows 7 64bit)

    C:/Ruby193/bin/ruby.exe extconf.rb
    checking for rb_trap_immediate in ruby.h,rubysig.h… no
    checking for rb_thread_blocking_region()… yes
    checking for inotify_init() in sys/inotify.h… no
    checking for __NR_inotify_init in sys/syscall.h… no
    checking for writev() in sys/uio.h… no
    checking for rb_thread_check_ints()… yes
    checking for rb_time_new()… yes
    checking for windows.h… yes
    checking for winsock.h… yes
    checking for main() in -lkernel32… yes
    checking for main() in -lrpcrt4… yes
    checking for main() in -lgdi32… yes
    checking for main() in -lssl… no
    creating Makefile

    make
    generating rubyeventmachine-i386-mingw32.def
    compiling binder.cpp
    cc1plus.exe: warning: command line option “-Wdeclaration-after-statement” is valid for C/ObjC but not for C++
    cc1plus.exe: warning: command line option “-Wimplicit-function-declaration” is valid for C/ObjC but not for C++
    In file included from project.h:137:0,
    from binder.cpp:20:
    binder.h:35:34: warning: type qualifiers ignored on function return type
    In file included from project.h:138:0,
    from binder.cpp:20:
    em.h:91:47: warning: type qualifiers ignored on function return type
    em.h:92:76: warning: type qualifiers ignored on function return type
    em.h:93:56: warning: type qualifiers ignored on function return type
    em.h:95:57: warning: type qualifiers ignored on function return type
    em.h:96:60: warning: type qualifiers ignored on function return type
    em.h:97:58: warning: type qualifiers ignored on function return type
    em.h:98:55: warning: type qualifiers ignored on function return type
    em.h:99:36: warning: type qualifiers ignored on function return type
    em.h:101:47: warning: type qualifiers ignored on function return type
    em.h:106:42: warning: type qualifiers ignored on function return type
    em.h:123:45: warning: type qualifiers ignored on function return type
    em.h:132:36: warning: type qualifiers ignored on function return type
    In file included from project.h:145:0,
    from binder.cpp:20:
    eventmachine.h:45:61: warning: type qualifiers ignored on function return type
    eventmachine.h:46:112: warning: type qualifiers ignored on function return type
    eventmachine.h:47:69: warning: type qualifiers ignored on function return type
    eventmachine.h:49:73: warning: type qualifiers ignored on function return type
    eventmachine.h:62:75: warning: type qualifiers ignored on function return type
    eventmachine.h:63:74: warning: type qualifiers ignored on function return type
    eventmachine.h:64:77: warning: type qualifiers ignored on function return type
    eventmachine.h:65:41: warning: type qualifiers ignored on function return type
    In file included from project.h:145:0,
    from binder.cpp:20:
    eventmachine.h:99:60: warning: type qualifiers ignored on function return type
    eventmachine.h:100:58: warning: type qualifiers ignored on function return type
    eventmachine.h:102:60: warning: type qualifiers ignored on function return type
    eventmachine.h:105:41: warning: type qualifiers ignored on function return type
    compiling cmain.cpp
    cc1plus.exe: warning: command line option “-Wdeclaration-after-statement” is valid for C/ObjC but not for C++
    cc1plus.exe: warning: command line option “-Wimplicit-function-declaration” is valid for C/ObjC but not for C++
    In file included from project.h:137:0,
    from cmain.cpp:20:
    binder.h:35:34: warning: type qualifiers ignored on function return type
    In file included from project.h:138:0,
    from cmain.cpp:20:
    em.h:91:47: warning: type qualifiers ignored on function return type
    em.h:92:76: warning: type qualifiers ignored on function return type
    em.h:93:56: warning: type qualifiers ignored on function return type
    em.h:95:57: warning: type qualifiers ignored on function return type
    em.h:96:60: warning: type qualifiers ignored on function return type
    em.h:97:58: warning: type qualifiers ignored on function return type
    em.h:98:55: warning: type qualifiers ignored on function return type
    em.h:99:36: warning: type qualifiers ignored on function return type
    em.h:101:47: warning: type qualifiers ignored on function return type
    em.h:106:42: warning: type qualifiers ignored on function return type
    em.h:123:45: warning: type qualifiers ignored on function return type
    em.h:132:36: warning: type qualifiers ignored on function return type
    In file included from project.h:145:0,
    from cmain.cpp:20:
    eventmachine.h:45:61: warning: type qualifiers ignored on function return type
    eventmachine.h:46:112: warning: type qualifiers ignored on function return type
    eventmachine.h:47:69: warning: type qualifiers ignored on function return type
    eventmachine.h:49:73: warning: type qualifiers ignored on function return type
    eventmachine.h:62:75: warning: type qualifiers ignored on function return type
    eventmachine.h:63:74: warning: type qualifiers ignored on function return type
    eventmachine.h:64:77: warning: type qualifiers ignored on function return type
    eventmachine.h:65:41: warning: type qualifiers ignored on function return type
    In file included from project.h:145:0,
    from cmain.cpp:20:
    eventmachine.h:99:60: warning: type qualifiers ignored on function return type
    eventmachine.h:100:58: warning: type qualifiers ignored on function return type
    eventmachine.h:102:60: warning: type qualifiers ignored on function return type
    eventmachine.h:105:41: warning: type qualifiers ignored on function return type
    cmain.cpp:98:71: warning: type qualifiers ignored on function return type
    cmain.cpp:109:122: warning: type qualifiers ignored on function return type
    cmain.cpp:119:79: warning: type qualifiers ignored on function return type
    cmain.cpp:129:83: warning: type qualifiers ignored on function return type
    cmain.cpp:260:85: warning: type qualifiers ignored on function return type
    cmain.cpp:270:84: warning: type qualifiers ignored on function return type
    cmain.cpp:280:88: warning: type qualifiers ignored on function return type
    cmain.cpp:290:51: warning: type qualifiers ignored on function return type
    cmain.cpp:300:70: warning: type qualifiers ignored on function return type
    cmain.cpp:320:55: warning: type qualifiers ignored on function return type
    cmain.cpp:546:70: warning: type qualifiers ignored on function return type
    cmain.cpp:669:68: warning: type qualifiers ignored on function return type
    cmain.cpp: In function ‘int evma_send_file_data_to_connection(long unsigned int, const char*)’:
    cmain.cpp:752:6: error: cannot convert ‘stat*’ to ‘_stati64*’ for argument ’2′ to ‘int _fstati64(int, _stati64*)’
    make: *** [cmain.o] Error 1

  8. s2reg says:

    Being primarily a Java developer, I do not have Ruby (native) installed – and was instead using vmc over JRuby when required – which works fine ….

    However, because of the “native compilation” gem – caldecott doesn’t work on JRuby :( Is making this work without the native compilation dependency still on the roadmap ?

    Thanks.

  9. Pingback: New Iron Foundry Test Environment .ME - Iron Foundry

  10. Pingback: Caldecott with MS SQL Server! - Iron Foundry

  11. Anuj says:

    I am trying to get a tunnel connection to a mysql service that is created using vmc on a private VCAP instance. i am getting the following exception :

    C:Userskumara>vmc tunnel envt-db
    Deploying tunnel application ‘caldecott’.
    Create a password: ***********
    Error 702: External URIs are not enabled for this account

    Do you what I might be doing wrong?

    Thanks,
    Anuj

  12. xena says:

    I’m trying to execute vmc tunnel but i get error ‘ruby interpreter has stopped working’ . Tryed to find what’s the problem on the Internet but the only solution i saw was to upgrade ruby version to 1.9.3 , but I already have this version.

  13. Pingback: Fiddling around with Cloud Foundry « Teh Tech Blogz0r by Luke Meyer

  14. Vinayak says:

    C:backup>vmc tunnel pot-db psql
    Getting tunnel connection info: OK

    Service connection info:
    username : u10ed33d7f60740cb81372271853b7a16
    password : p592f360ec1f34991a73497562294dfc1
    name : d70d2c36fa89b4350bce5ae95a7e26fb0

    Starting tunnel to pot-db on port 10000.
    Launching ‘psql -h localhost -p 10000 -d d70d2c36fa89b4350bce5ae95a7e26fb0 -U u1
    0ed33d7f60740cb81372271853b7a16 -w’

    Error: ‘psql’ execution failed; is it in your $PATH?

    • Andy Piper says:

      So the error message here is telling you what is wrong – vmc tunnel is attempting to run the psql command-line tool to administer Postgres, but it cannot find psql in your PATH. Do you have psql installed?

  15. radojko says:

    If you are having problems installing caldecott on Win64, execute first following:
    gem install eventmachine –pre
    gem install thin

    and only then
    gem install caldecott

  16. Pingback: Cloud Foundry 在 SpringOne 2GX 大会上预展新功能 | 博客

  17. Pingback: IaaS vs. PaaS: Deploying a Web Application « Richard Seroter's Architecture Musings

  18. Pingback: Cloud Foundry Integration for Eclipse Now Supports Tunneling to Services | Blog

  19. Pingback: VMware`s Cloud Foundry Eclipse Integration Supports Tunneling - Fundamental Technology Partners Inc.

  20. henry says:

    Please use this command to update your vmc
    gem install vmc -v 0.3.14.beta.4 –pre
    For more details, please refer https://rubygems.org/gems/vmc/versions/0.3.14.beta.4

  21. Membera Aburu says:

    Hello,

    Does this mean that I’m able to tunnel into mysql from my on-premise applications?

    Thanks in advance!

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title="" rel=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>