Adding Apt GPG Key When Using Ansible

I’m using Ansible to provision, setup, and manage a server. If you want to install MariaDb, Postgresql, and many other packages you will want to use the best source. Often this is source is provided by projects themselves, in the form on an Ubuntu PPA. In the case of Postgresql and MariaDb you’ll use a PPA like I have. This normally means doing two things with Ansible.

First is to use the apt_key module to install the gpg key of the PPA repo. Second you use the apt_repository module to install the repository, after which apt-get will be able to install software using the repo.

The Ansible apt_key module is fairly simple, unlike the apt-add-key command you’d likely use on Ubuntu. You have to have a URL which directly references the gpg key. In the case of Postgresql this isn’t an issue, if you visit their download page the URL of the pgp key is given; not so lucky with MariaDb.

If you read MariaDb’s docs they’ll tell you to use the apt-key command with advanced options to install the gpg key. This doesn’t work in Ansible. It doesn’t support these options, so you need a real URL you can use. So how did I figure out what this URL is?

I started by going to the server listed on MariaDb’s site. That will give you a page where you can search for the gpg key. For the search string use the id, which in my case is ’0xcbcb082a1bb943db’. Make sure to also check the ‘Only return exact matches’ box so that you don’t get a bunch of similar but incorrect results. After you hit search you’ll should see exactly one match, which will include a bunch of information on the signing of the key. At the top is a line that looks like this:

pub  1024D/1BB943DB 2010-02-02

You will want to click that link. You end up with this page, which is the URL you want for the Ansible ‘add_key’ command. Plug that in as the ‘url’ key to the command and you’ll be all set.

Go VIM Features (on os x)

Go is a very good programming language which has Ken Thompson‘s fingerprints all over it. If you are writing any system level applications it’s worth taking a look at. If you really dig Python’s clean syntax, but want your app(s) to be able to better utilize machine resources this is the language for you. It is static, but in a good way (think Haskell), it allows management of memory (but also GC) and has a concurrency model which doesn’t punish the uninitiated.

Vim of course is the text editor of choice for system programmers, it runs on everything and can be extended in very powerful and useful ways. It’s an uber tool which stands the test of time.

The following is how I setup my Mac Book Pro (snow leopard) with both Go and VIM.

Download the latest Go package from here and install it on your mac you’ll find all the files in /usr/local/go. In that folder you also should have a collection of VIM features (highlighting for one) which you’ll want. Here is what I recommend.

If you aren’t already using Pathogen you first install it. It allows much cleaner management of VIM add ons than possible by default. You can go here and figure out how it all works, along with download and install it. Here is another article detailing Pathogen a bit more. Read both these resources, install Pathogen. Now you are ready for installing the add ons for Go.

Once Pathogen is installed simply do the following to install the add ons:

cd ~/.vim
mkdir -p bundle/golang
cp -R /usr/local/go/misc/vim/* bundle/golang

Now you should be able to open a .go file and see the syntax highlighter in effect.

Grails and UrlMappings.groovy

Doesn’t work…

http://jira.grails.org/browse/GRAILS-6587

Yeah that right there is a major flaw that needs fixing ASAP. Can’t believe it’s still an issue at 1.3.7…

I mean c’mon this totally breaks a ton of things, especially REST where “this+that” is much different than “this that”.

Grails, Validation of Map on Domain Class

Grails has a bunch of can’d validators such as maxSize, blank, nullable and so forth. These are hard workers, they’ll check values, report errors which can easily be internationalized, and can even influence auto-schema generation (a feature I advise you ignore).

If you are reading this maybe you googled something like “grails map validation” which means that you want to know how you can properly check a map of something inside of a domain class. Well this post may or may not help you. What I was doing today was working on a way to allow arbitrary key values (only strings) to be set on my domain class. I figured I would allow 255 characters for key and value.

This ends up not being all to difficult but does require that you know how the underlying Spring Framework mechanisms work.

First off is the Errors interface which in Grails uses the concrete implementation BeanPropertyBindingResult.

When using a custom validator you can pass a code block, or closure, that take 3 parameters. In order they are the value being validated, the object that value is part of, and the Errors implementation being used for the object to handle validation and error messaging. When, in your validator code, you find an error you will want to tell the Errors object. There are a number of methods in the Errors interface which allow this. Look at the various flavors of reject and rejectValue which are found. It’s important to really understand how these work because you’ll rely on getting things right here to be able to use property files to externalize and i18n your error messages.

What I ended up doing in this closure was iterating the key/value pairs, and checking if the lengths of the keys and values. If any key or value was more than 255 characters I added an error for that field on that object to the Errors object by calling rejectValue. This worked out perfect and the validator works just like the ones baked into Grails. I can put a version of my message in any language and it will be properly built and reported at the proper layer of my application.

Here is the code for my validator:

attributes(validator: { attributes, obj, errs ->
	attributes.each { k, v ->
		if (k.size() > 255) {
			errs.rejectValue("attributes", "key.toobig", [k] as Object[], "Key size too big, max is 255" as String)
		}
		if (v.size() > 255) {
			errs.rejectValue("attributes", "value.toobig", [k] as Object[], "Value for key ${k} too big, max is 255" as String)
		}
	}
})

Notice that the ‘as Object[]‘ and ‘as String’ are there to make sure the types are cast to the proper ones which the methods of the Errors interface require. Remember Groovy gotcha #1739, if you use interpolation in a literal string you will get a GString (yes really that is what they called it) and not a String type back, and any method that requires String will be unhappy if you pass it a GString (yeah imagine that).

Install MongoDB on OS X

I’ve already had MongoDB installed for years on my Mac, but I found this article recently and followed the instructions to setup a better install. Using these steps you can use OS X’s launch control to stop/start/restart MongoDB and to have it run on system startup.

It’s worth doing IMO since I have had to manually run the server which can get tedious. Also I’m kinda a sucker for doing things the ‘right’ way in this case meaning using launch control which is the preferred Mac way to handle daemons.

MongoDB OS X Boost Lib Fails

Recently I was working on upgrading the MongoDB on my OS X Lion machine. When I upgraded then tried the mongo client I got this:

dyld: Symbol not found: __ZNK5boost15program_options16validation_error4whatEv
  Referenced from: /opt/local/bin/mongo
  Expected in: /opt/local/lib/libboost_program_options-mt.dylib
 in /opt/local/bin/mongo

So I did a little googling and found this bug report. If you follow the directions and install boost version 1.49.0_0 using Subversion as shown it should fix the issue for you. Here are the commands to install the older and needed boost version:

svn co  -r 93341 'http://svn.macports.org/repository/macports/trunk/dports/devel/boost/'
cd boost
sudo port install

Once you’ve finished try ‘mongo’ again and see if it doesn’t work. If it fails you should try ‘port installed | grep boost’ and then check that boost 1.49.0_0 is the active version. It will be obvious because it will say ‘active’ next to it :D If not you can do the following:

sudo port activate boost @1.49.0_0

Now try ‘mongo’ again, if this fails I am not sure exactly how to help, sorry!

Install gevent + greenlets on OS X

The last few years a lot more people have been focusing on async IO in Python. This is also referred to as “non-blocking IO”. Gevent is one project that makes it easy to do this sorta stuff. How it works is simple, you spawn up “greenlets” which are sorta like threads but very lightweight. You are spawning these in a single Python process, and single thread (which because of the GIL really can only have one synchronous flow of execution anyhow). So this normal Python process will start processing your greenlet(s) and when one hits a point where IO is performed, say it goes out and talks to Redis, execution yeilds back while IO is being performed and another greenlet is picked and then put into the running state. If you had a bunch of greenlets all waiting for IO then they could come back and start processing as soon as one got it’s output. The important thing to grasp here is only one process, and single thread of execution is happening, but if one of the greenlets hits IO rather than block we can start executing some other code, and process the piece of code waiting for IO to return when it is ready.

If you’re looking to try Greenlets and gevent and you are on a mac you should be able to follow these directions.

First install libevent via Mac Ports. If you do not already have Mac Ports then first install it. I hear you can also use homebrew but I’ve been a ports user and it’s more like the apt-get I’m used to.

sudo port install libevent

Now you should be ready to install gevent (which’ll install greenlets).

First set this flag so that gevent can build:

export CFLAGS=-I/opt/local/include

That’ll point the C compiler to the ports folder where the event.h (and other) files are.

I created a virtualenv for this project, therefore to install gevent I simply did a:

pip install gevent

After which I had gevent and greenlets ready to go! I advise you go try this example to get a very simple start using greenlets and gevent.

Grails broken list() on domain object, fetchMode eager

I am using Grails 1.3.7 and found a very confusing issue. You could possibly run into this problem, maybe you have and that’s why you’re here! A colleague of mine setup a domain class that had

fetchMode = [aClass: 'eager']

This is almost always wrong, as GORM when using ‘eager’ or the synonym ‘join’ for this value will do a very dumb thing. It will use an outer join from parent to children, then limit on that cartesian product!  So what you say? Well if you do this:

SomeDomainClass.list(max: 10)

You will see some odd results. Basically say SomeDomainClass has many AnotherDomainClass and that is set up with eager/join fetch. Well since the query works this way you could get very odd behavior. If the parent class has ten children expect to only get the one parent back from list(), not ten instances of the parent! Indeed, that sucks.

So you are saying “Ouch, this one hurts!” There probably should be a giant red stop sign in the documentation that makes you stop, and then be drilled on the pitfall(s) this encompasses, followed by a multiple choice quiz, or something.

For completeness here is the SQL GORM/Hibernate generates for the ‘eager’ case:

select ... from test_case_execution_context this_
left outer join test_case_execution_context_test_case_execution_step_status stepstatus2_
on this_.id=stepstatus2_.test_case_execution_context_step_status_id
limit ?, ?

You see how that is not what you wanted, right?

Luckily the fix is pretty easy, though you might not like it for any odd reasons. Use the property lazy: false!

static mapping = {
    aMember lazy: false
}

BooYahhh! Now you will get the results you are expecting.

With the lazy set to false you get results like this:

select ... from test_case_execution_context this_ limit ?, ?

Oh but where are the child domain results? Well, they get mapped in with the JDBC connection open, right away, in separate selects. Yes you are going to be making a bunch of db calls, but it’s all up front using one JDBC connection. This is the way it works folks. On the bright side now offset and max work!

Charity: Nginx, Bind, Grails, SSL etc

I was given an opportunity to help my wife setup a website for a charity she is involved in. It wasn’t anything particularly earth shaking, just a single page with a form for donating money via the web. Her charity wanted to run credit cards manually, since they have a credit card terminal. We would be collecting card info and thus we needed to have the traffic encrypted.

First things first I went and registered the domain packsoflove.com which is the domain my wife had chosen. Next I needed to setup DNS information for the new site, basically point the domain to my name servers so that our domain would resolve to the proper IP address. I used the registrars web based software to input the two name servers I manage. To finish the DNS setup I went and configured BIND on my servers so that they properly resolved packsoflove.com and www.packsoflove.com to my server’s IP. Now we had a place in the world wide web, but there wasn’t anything there!

At this point we needed an app. One that would be located at the domain we just setup. Since we where going to be processing form data it made sense to use a web application framework as opposed to a static site. The question was which framework to use? These past 10 or so years I’ve dived into a ton of them, writing apps of all size in numerous languages. For this task I choose Grails, partly because I’m going to be using it at my new job, but also because it’s very modular. You can install plugins for doing things like email or security and be done with it in minutes. Other choices that may have worked are Django, or Rails.

So I built the webapp, and it took a couple hours. I was having to prime myself a bit, I hadn’t done any Grails in over two years. I built a form for entering the card info that posts to a simple action which then emails off the data (securely of course) to my wife. I also built one more very very simple page that basically says thanks for donating and links back to the first page. So now I built the application and uploaded it to my server.

Grails runs on the Java language, and also uses most of the standards that Java has made famous (or is it infamous?) Anyhow you build a WAR file (short for webapplication resource file) which you install in an application container which serves up the app. In this case I used Jetty. It’s pretty easy to install via apt-get if you are using Ubuntu. Once installed you can put applications in the /usr/share/jetty/webapps folder. So I put the WAR file into the webapps/ folder, then rebooted Jetty. All good, except Jetty runs on localhost, which means it’s not accessible from the internet! What I had to do now is setup our proxy server, Nginx, so that it handles web requests for packsoflove.com and proxied them back to the Grails application running in Jetty, Pheeew!

My Vim

Vim, let me count the ways I love thee. If not for ye I would have fell so short in my text editing.

I use pathogen, which makes installing, updating and especially uninstalling\ plugins a breeze. You can even checkout plugins from Git or Mercurial repos real easy and then when you need to update you just use the client to pull down the latest updates. To use it follow the directions on the site, putting the bundle in .vim/autoload.

I also use the pyflakes-vim plugin, which makes keeping python files clean. Unused imports and variables will be highlighted and explanations given, so you don’t end up with a ton of cruft when you refactor. To install it just get the latest zip file and unzip it into .vim/bundle (if you are using pathogen).

Anyways here is my config.

filetype off
call pathogen#helptags()
call pathogen#runtime_append_all_bundles()

filetype plugin indent on

set nocompatible

set modelines=0

syntax on

set tabstop=4
set shiftwidth=4
set softtabstop=4
set expandtab

set encoding=utf-8
set scrolloff=3
set autoindent
set showmode
set showcmd
set hidden
set wildmenu
set wildmode=list:longest
set visualbell
set ttyfast
set ruler
set backspace=indent,eol,start
set laststatus=2
set relativenumber
set undofile

nnoremap / /\v
vnoremap / /\v

set ignorecase
set smartcase
set gdefault
set incsearch
set showmatch
set hlsearch
nnoremap <leader><space> :noh<cr>
nnoremap <tab> %
nnoremap <tab> %

set nowrap

nnoremap <up> <nop>
nnoremap <down> <nop>
nnoremap <left> <nop>
nnoremap <right> <nop>
inoremap <up> <nop>
inoremap <down> <nop>
inoremap <left> <nop>
inoremap <right> <nop>

nnoremap j gj
nnoremap k gk

nnoremap <leader>b :buffers<cr>