A New Blog is born: Tweetegy

Please note that I am no longer updating “Never Say Never” blog here on WordPress.com. Instead, I have launched a new blog which I am hosting myself.

Welcome to Tweetegy!

The blog is called “Tweetegy” and can be found at the above link.This new blog basically continues where this one left off! Enjoy.

Never Say Never: One Year On!

It has been a year since I started this blog. It was really a test for myself to see if I can post regularly (at least once a month) to my own, personal blog. I think the experiment has been a success! Not only have I achieved this posting frequency but I am actually generating a modicum of traffic flow to this blog (about 30 visits per day). OK, that is not too high but I have plans for the future now anyway!

What I am going to do is move this blog to a dedicated host (I’ll probably use Slicehost’s 256 Slice) and I’ll probably re-launch under a new name (not sure yet what to call the new blog).

Anyway, my new plan is to post at least once a week and try and achieve a traffic flow of 100+ unique visits per day (for starters). I hope to blog about anything that I am currently working on or am interested in. So this would include a mixture of programming (Linux/Ubuntu, Rails, Ruby, Flex, JavaScript, RDF, MySql, NoSql, Graph Databases), business ventures that I am involved in (Yoolk, MangoMap, Upstart and AdFlickr) and personal activities (Piano, Mountain Biking, Eating Out and Reading).

Next month, I’ll post to this blog for the last time, which will probably be an announcement of the launch of my new blog.

Using Google Analytics API in a Ruby on Rails project

Ingredients

• A new Rails project
• The Garb gem
• A Google Analytics Account

Preparation

I assume you know about Rails already….Garb can be installed as follows:

gem install garb

Go to your Google Analytics account and fetch the Account Id and Profile Id that your interested in. You will find your Account Id on your account home page under the link “Edit account settings”, it is the number after the “UA-”. You will find your Profile Id, by going back to your account home page, finding the profile in the table, then click the “Edit” link in that row….the Profile Id is at the top of this page….phew!

If you are not familiar with the Google Analytics API, then you will find the Data Feed Query Explorer tool useful.

The Task: Record and display Unique Pageviews from a set list of pages

Now for the fun part! Lets build something that records Unique Page Views from a list of pages in a local database. We need to create a model to store these pages and the unique pageviews. Change into your rails project directory and run the following scaffold script:

script/generate scaffold page url:string unique_pageviews:integer

Open up the pages controller that was just generated and add the following method, which sets the class instance variable @profile to an instance of your Google Analytics profile based on the Account Id and Profile Id.

 def get_analytics_profile
    Garb::Session.login(your_google_analytics_username,your_google_analytics_password)
    accounts = Garb::Account.all
    #Loop all Accounts for account_id
    accounts.each do |account|
      if account.id == your_google_analytics_account_id
        #Loop all Profiles for profile_id
        account.profiles.each do |profile|
         if profile.id == your_google_analytics_profile_id
           @profile = profile
           break
         end
       end
      end
    end
  end

Basically, Garb::Session.login is called passing your username and password to authenticate the session. Next we fetch all accounts and iterate the array until we find the account that matches your Account Id. Then we need to iterate all the profiles within that account until we find the profile that matches your Profile Id.

Now add this method which will update all the unique pageviews in your list of pages:

def update_all
    @pages = Page.all
    @pages.each do |page|
        get_analytics_profile if !@profile
        report = Garb::Report.new(@profile)
        report.metrics :unique_pageviews
        report.dimensions :page_path
        report.filters :page_path.contains => page.url
        page.unique_pageviews = report.results.first.unique_pageviews
        page.save
      end
    end 
  end

Since you have used a scaffold to create this controller, there should already be an index method present. Add a call to the update_all method at the top of the index method so that everytime the index method is called the update_all method is called. Here is how your index method should look:

 def index
    update_all
    @pages = Page.all

    respond_to do |format|
      format.html # index.html.erb
      format.xml  { render : xml => @pages }
    end
  end

Final Touches

Now all that remains is a little configuration, add some fixtures and we’re done! You should already have fixtures for your page model. Note that the url does not need to include the domain name, so /pricing/ or /sign-up/ are the sorts of values you should put here (obviously ones that exist in your Analytics profile!). The configuration is simply to add:

config.gem "garb"

in the appropriate place of your environment.rb file.

Get those metrics!

Now fire up your rails app and navigate to the index for your pages http://localhost:3000/pages/. The call to update_all will be made which will update your local database with the metrics and then render the data in the browser in the usual Rails table format.

Conclusion

Using the Garb gem makes it extremely easy to work with the Google Analytics API in a Rails project. Obviously this example application can be improved to include caching, error handling (it will bomb out if the Id’s or urls are not found in your Analytics account) as well as other improvements that you might want to make.

Hudson setup to execute scripts with root privileges (safely)

There is a lot out there on the web about how to setup and install Hudson. In fact that process is very easy so I will just provide a script to explain that (see the end of this post). What I found to be an interesting challenge is getting Hudson to execute my scripts with root privileges…

The first time I execute the script (without sudo), my job fails and I am informed that the user hudson does not have permission to execute apt-get. So naturally, I prefix the script with sudo and my job fails again, this time with an incorrect password. So how to get Hudson to execute my script with root privileges?. After a lot of searching I managed to find one blog post which mentions adding the user (in my case user hudson) to the sudoers file and giving the user full access without being promted for a password. Using visudo, I added this line to my sudoers file:

hudson  ALL=(ALL) NOPASSWD:ALL

However, I was most concerned about giving hudson the rights to do what it wanted without requiring a password as this leaves an obvious security hole. So I posted my concern on ServerFault.com and fortunately I was informed that it is possible to lock down the commands and the parameters that can be used by a user without requiring a password. So I changed my entry in sudoers file to this:

hudson  ALL=(ALL) NOPASSWD:/var/scripts/the-script-I-want-to-run.bash

Now all hudson can do (without being prompted for a password) is to execute sudo /var/scripts/the-script-I-want-to-run.bash. This makes be feel safe and my build works a treat each and every time too! I thought I would share this since I could not find a clear solution to this issue.

By the way, if you want to install hudson on your linux server, here is the script I prepared earlier!

#Install Hudson
#NOTE: You MUST manually add hudson user to the sudoers account using visudo,
#specifying all the scripts that you want hudson to be able to run without requiring a root password, for example:
#hudson  ALL=(ALL) NOPASSWD:/var/scripts/the-script-I-want-to-run.bash
wget -O - http:// hudson-ci.org/debian/hudson-ci.org.key | sudo apt-key add -
if ! grep -q "http:// hudson-ci.org/debian binary/" /etc/apt/sources.list ; then
    echo "deb http:// hudson-ci.org/debian binary/" >> /etc/apt/sources.list
fi
#NOTE: On a fresh install, you will have to run this script twice since the first time the script will not know about sources.list update
apt-get install hudson
#Install Hudson Plugins
if test ! -f /var/lib/hudson/plugins/mercurial.hpi ; then
    wget -P /var/lib/hudson/plugins http:// hudson-ci.org/download/plugins/mercurial/1.28/mercurial.hpi
    chown hudson:nogroup /var/lib/hudson/plugins/mercurial.hpi
fi

Additionally, if you want Hudson to be accessible at http://hudson.yourserver.com while keeping Hudson running on port 8080 and you are running Apache then run the following script!

SITE_CONFIG_PATH="/etc/apache2/sites-available/default"

if ! grep -q "ServerName hudson.yourserver.com" $SITE_CONFIG_PATH
then
	echo "Configuring..."
	echo "" >> $SITE_CONFIG_PATH
	echo "# Hudson virtual host" >> $SITE_CONFIG_PATH
	echo "<VirtualHost *:80>" >> $SITE_CONFIG_PATH
	echo "  ServerName hudson.yourserver.com" >> $SITE_CONFIG_PATH
	echo "  ProxyPass / http:// localhost:8080/" >> $SITE_CONFIG_PATH
	echo "  ProxyPassReverse / http:// localhost:8080/" >> $SITE_CONFIG_PATH
	echo "  ProxyPreserveHost on" >> $SITE_CONFIG_PATH
	echo "  <Proxy *>" >> $SITE_CONFIG_PATH
	echo "   allow from all" >> $SITE_CONFIG_PATH
	echo "  </Proxy>" >> $SITE_CONFIG_PATH
	echo "</VirtualHost>" >> $SITE_CONFIG_PATH

	#Create symlink to proxy module
	ln -s ../mods-available/proxy.load /etc/apache2/mods-enabled/proxy.load
	ln -s ../mods-available/proxy.conf /etc/apache2/mods-enabled/proxy.conf
	ln -s ../mods-available/proxy_http.load /etc/apache2/mods-enabled/proxy_http.load

	echo "Restarting apache"

	#Restart Apache
	/etc/init.d/apache2 restart
else
	echo "Already configured!"
fi

How to return content from an application running on a specific port in apache

Continuing on from the previous post where I explain how to show content from a different root folder depending on the sub domain e.g. http://foo.example.com will actually return content from http://example.com/foo (without changing the URL). Today I will extend this idea to show content from a separate service running on a different port within the same server. So in this case a call to http://foo.example.com returns http://example.com:port.

As in the previous post, we assume that you already have your sub domain DNS settings correctly configured and the subdomain now points to the IP address of your server, you should do the following in Apache running on an Ubuntu server.

Open the configuration file for your site. In this example, we will use the default site.

nano /etc/apache2/sites-available/default

Add the following additional VirtualHost configuration to the end of the file (change the port number on localhost if you need to, of course!)

<VirtualHost *:80>
  ServerName foo.example.com
  ProxyPass / http:// localhost:3000/
  ProxyPassReverse / http:// localhost:3000/
  ProxyPreserveHost on
  <Proxy *>
   allow from all
  </Proxy>
</VirtualHost>

Now restart apache and your all done!

 /etc/init.d/apache2 restart

How to subdomain redirect using Apache Virtual Host

Do you want to setup a website subdomain so that, for example, http://foo.example.com will actually, without the end user realizing, return content from http://example.com/foo ? In this short, but useful blog post, I will show you how you can redirect a subdomain to a server directory without anyone (except you) knowing!

Assuming that you already have your subdomain DNS settings correctly configured and the subdomain now points to the IP address of your server, you should do the following in Apache running on an Ubuntu server.

Open the configuration file for your site. In this example, we will use the default site.

nano /etc/apache2/sites-available/default

Add the following additional VirtualHost configuration to the end of the file.

<VirtualHost *:80>
  ServerName foo.example.com
  DocumentRoot /var/www/foo
</VirtualHost>

Now restart apache and your all done!

 /etc/init.d/apache2 restart

If you point your browser to http://foo.example.com you will see the content from http://example.com/foo ! Note: You do, of course, have to create the folder /var/www/foo and put a index.html file there with some content like “Here be foo!” so you can test it actually works!

Extra Bonus Script Time!
As an added bonus, I have included a setup script that you can use to perform the above in one shot. Use it if you wish. Change foo and example.com to your domain and subdomain names.

#!/bin/bash

SITE_CONFIG_PATH="/etc/apache2/sites-available/default"

echo "Configure default site with foo virtual host"
if ! grep -q "ServerName foo.example.com" $SITE_CONFIG_PATH
then
	echo "Configuring..."
	echo "" >> $SITE_CONFIG_PATH
	echo "# foo virtual host" >> $SITE_CONFIG_PATH
	echo "<VirtualHost *:80>" >> $SITE_CONFIG_PATH
	echo "  ServerName foo.example.com" >> $SITE_CONFIG_PATH
	echo "  DocumentRoot /var/www/foo" >> $SITE_CONFIG_PATH
	echo "</VirtualHost>" >> $SITE_CONFIG_PATH
	echo "Restart apache"
	/etc/init.d/apache2 restart
else
	echo "Already configured!"
fi

Step by step guide to setting up a client server workstation on Ubuntu and VirtualBox

I work on a number of different software projects. During the week I work on project called Yoolk and during the evenings and weekends I might work on AdFlickr or some other new project! Either of these main projects might also have sub-projects. The cleanest way to separate all of these projects is to develop the software using a guest OS running in VirtualBox. It makes sense, of course, that the guest OS matches your production environment as much as possible. Some of the benefits of doing this are:

• Complete separation of all your project dependencies. Change a setting on the guest development server for one project has absolutely no affect on your other projects.
• Allows much more freedom to experiment with different dependencies, try out new software and generally just screw around without worrying about having to clean up your mess after (just take snapshots or re-install your guest OS!).
• During development your are running your application in an environment which (at OS level at least) should be exactly the same as production. Therefore, this minimizes surprises during deployment.
• Allows you to keep your client OS the way you want it so that your development experience is just the way you like it. For example, installing different text editors, music libraries, video libraries, browsers etc etc without worrying that it will affect the application your building.

While the benefits are great, there is a little work to be done to set this up initially (but it’s not too much). The longest step is installing the guest OS on VirtualBox. Obviously this depends on which guest OS you install. I use Ubuntu 9.04 Server Edition. By the way, my host OS is currently Ubuntu 9.10 but I know that this also works on Ubuntu 9.04 (since I used to run that before).

Below is a step by step guide to setting this up. Notice that you will be alternating between the client (your desktop OS) and the server (the guest OS in VirtualBox):

1. client: install server on virtualbox (make the default user the same name as client user name)
2. client: configure virtualbox to use bridged networking and start server
3. server: sudo apt-get update
4. server: sudo apt-get install openssh-server
5. server: ifconfig (and note the ip address)
6. client: ssh into the server using the IP address you recorded in the previous step
7. server: mkdir .ssh
8. client: ssh-keygen (don’t enter any values, press return three times, yes passwords should be blank)
9. client: cat ~/.ssh/id_rsa.pub – copy the output to the clipboard (very carefully, no pre/trailing white space)
10. server: touch .ssh/authorized_keys
11. server: sudo nano .ssh/authorized_keys – paste clipboard contents
12. client: sudo nano /etc/hosts – add line: 192.168.???.??? myappname.dev http://www.myappname.dev (get ip from ifconfig)
13. client: sudo /etc/init.d/networking restart (refresh new data in /etc/hosts)
14. client: In nautilus go to ssh://myappname.dev then add the location to bookmarks
15. client: In your web browser, go to http://myappname.dev and add the location to bookmarks (you will not get a response unless you have a web server running your application on your guest OS, of course!).

After you have completed the above steps when logging in via ssh you shouldn’t be prompted for a password! Note, if IP address of the server changes, you should edit /etc/hosts to reflect that change.

command to ftp in linux bash script

Command line ftp scripts are often unappreciated. Often, when a requirement to ftp files arises it makes sense to write a ftp command file which automates this. The question is what is the best approach? Out of the box, most Linux and Unix ftp commands are limited. In particular, recursive folder creation on the ftp server is not possible with the basic linux / unix ftp tool. It lacks a command in ftp that offers this recursive functionality. So what is the solution if you require a bash script ftp for upload of files and folders?

I recommend you download ,install and use ncftp. This is a far superior option compared with the basic set of ftp commands that comes with the installed ftp program with linux.

It would not be fair to leave without providing an example so here is one! Imagine that you have a set of nested folders, some which are empty and some which contain files. You want to ftp the entire folder structure (including the files, of course!) to a remote ftp server. Here is a ftp bash script to do just that (replace with your settings where appropriate).

FTP_HOST=ftp.yoursite.com
FTP_LOGIN=yourusername
FTP_PASSWORD=yourpassword

ncftp <<EOF 
open -u $FTP_LOGIN -p $FTP_PASSWORD $FTP_HOST
cd somefolder/onyourserver
lcd "somefolder/onyourclient"
put -R *
bye
EOF

Save the above script in a file named ftp-example.sh and execute it in your terminal as you would any shell script. The key command in this example which creates all the folders recursively is put -R *. The rest of the example is pretty straight forward.

Create a WCF Rest Application using .NET 3.5

In this post, I will explain how to create a simple read only Rest service using WCF .NET 3.5 WebGet endpoints. I will also explain a little bit about the wonderful UriTemplate class.

First create a new “WCF Service Application”

Create a new “WCF Service Application” project. This will provide you with the standard example files for WCF that contain the string GetData(int value) and CompositeType GetDataUsingDataContract(CompositeType composite) signatures.

Now this application is not yet Rest enabled. You need to do a few more things. No doubt, of course you can turn this into your own Visual Studio project template later on if you need to.

Rest enable the WCF Application

Go ahead and add a reference to System.ServiceModel.Web dll. Now add the WebGetAttribute on the method you want to expose. Let’s just use the supplied GetData method to keep things simple. However, notice that I have to change the data type of the input parameter to string since this is what the UriTemplate expects. This method signature in the IService1 Interface should look something like this:

[OperationContract]
[WebGet( UriTemplate = "somepath/{value}")]
string GetData(string value);

Then, in the web.config file that we create, we must also include the following entry (put this within the behaviors tag):

<endpointBehaviors>
<behavior name="WebBehavior">
<webHttp />
</behavior>
</endpointBehaviors>

Now look within the services tag for the endpoint tag that describes the service you are building. You need to change the binding to “webHttpBinding” and add the attribute behaviorConfiguration=”WebBehavior” to this tag. The endpoint tag for your service should now look something like this:

<endpoint address="" binding="webHttpBinding"
contract="WcfService1.IService1"
behaviorConfiguration="WebBehavior" />

At this point you are ready to run the service, so hit F5. Your browser window should open to the service description page. Please then change the path following the “localhost” root to: ./Service1.svc/somepath/sqlserverdotnet and you should then see an XML response from the service as follows:

<string>You entered: never say never</string>

Now you are free to modify this service method as you want. Perhaps to connect to a database and return a collection as serialized XML for example. Obviously you can remove the .svc extension from the url by using a HttpModule. There are many examples of url re-writing on the web which demonstrate exactly how to do this.

The easy way to create a new Ruby Gem!

If you want to make a Ruby Gem and want to do it in a very easy way then here’s is one very good option – use Jeweler! Thanks to John Nunemaker at RailsTips for pointing me in the right direction with his post Building API Wrapping Gems

1. Install git core
   

   apt-get install git-core

2. Install jeweler gem
   

   gem install jeweler

3. Now add the following line to your bashrc file to ensure that you always use spec (that if you do use spec). Otherwise the default is shoulda and if your happy using that then you can skip this part.
   

   export JEWLER_OPTS = "--spec"

4. Setup your global username and email for git
   

   git config --global user.name "Your Company Name"
   git config --global user.email "yourname@yourcompanyname.com"

5. Script your new gem using jeweler
   

   jeweler your_shiny_new_gem

6. Set your init version. I do this so that I can easily generate the gemspec later.
   

   rake version:write MAJOR=0 MINOR=1 PATCH=0

7. Modify the new gem rake file to add the meta info for your project
   

   begin
     require 'jeweler'
     Jeweler::Tasks.new do |gem|
       gem.name = "your_shiny_new_gem"
       gem.summary = %Q{A simple gem to access the Shiny API}
       gem.description = %Q{Use this gem to access our Shiny API !!}
       gem.email = "yourname@yourcompanyname.com"
       gem.homepage = "http://bitbucket.org/yourrepo/your_shiny_new_gem"
       gem.authors = ["Your Company Name"]
       gem.add_development_dependency "rspec", ">= 1.2.9"
       # gem is a Gem::Specification... see http://www.rubygems.org/read/chapter/20 for additional settings
     end
     Jeweler::GemcutterTasks.new
   rescue LoadError
     puts "Jeweler (or a dependency) not available. Install it with: gem install jeweler"
   end
   

8. Generate the gemspec
   

   rake gemspec

9. Write the code that goes into your gem. This is where your own imagination comes in! See the Google Weather API Gem for an example. When your gem is finished (including all your tests passing!) move on to building and testing the installation of the Gem.
   

   code.write!

10. Build the gem
    

    rake build

11. Install the gem
    

    rake install

You can deploy to github (the easiest solution when using Jeweler). However, you can still deploy to other repositories like Bitbucket for example. Then, as recommended by the author of Jeweler, Josh Nichols, have a delicious beverage!

Happy Gem stone cutting!