I just paid, through Paypal, for the next Yow! night. And then paypal told me I had paid $10 to “Bedarra Research Labs Pty Ltd”. Huh? Who are they? Who just got my money?

Turns out they’re the right people after all. Isn’t that nice.

Probably.

First, an admission: the comparison I did was neither thorough nor comprehensive, and provided only a slim reason to go one way or the other.

I was looking at a proposal for some work, and it specified that the system would have 300 million records, and some information about the likely structure of those records. Looking at that number and the likely structure, I had two questions:
1. Will we have to worry about how PostgreSQL goes with those sorts of numbers, spending time sharding and distributing and generally making life harder than it is with a single postgres instance?
2. Would MongoDB make our lives better?

The structure of the data is up to 30,000 buckets, with up to 10,000 records in each bucket. The obvious relational thing is to have a buckets table, and a records table, where the records table has a bucket_id and some other data. In this case, I used two decimals for the “some other data”, assuming that using more than just two decimals would change the constant factors but the basic performance characteristics would be the same. From what I know of the actual application, there’ll be another layer of indirection – something will have a bucket_id, and we’ll go from the something to the bucket to the records. Most operations will be either “append to bucket” or “get contents of bucket”, with the latter sometimes involving additional sorting/filtering.

My main interest is in looking ahead to a possible future where we are under deadline, and we have a performance problem, and we need the datastore to just do those append and get-contents operations as quickly as possible. If only one of them can be quick, I want it to be getting the contents of a bucket, because we can potentially queue appends.

So, I knocked up a little database schema for postgres, put appropriate indexes on it, and instructed the computer to invent 300,000,000 records, 10000 for each of 30000 buckets. An hour and a half later, I had them. I ran two tests, one for adding records to a bucket, and one for getting the contents of a bucket. They took 6 and 7 seconds of wall-clock time each (running the former 10000 times and the latter 100 times).

Then I produced a similar-ish thing in mongo. I created a collection, and to that collection I added an entry for each bucket. Each entry contained an index integer and a list of records. Trying approximately the same operations, append-to-bucket and retrieve-bucket, yielded 10 and 24 wall-clock seconds respectively.

Mongo and Postgres both coped fine with the volume of data. I suspect mongo’s relatively-slow performance is a result of the document size, and am curious about if arranging the data into smaller documents would work better. It’s all a bit irrelevant, though, because we have lots of experience of doing rails apps with postgres, and it looks like that’ll work fine at this scale, at least as far as the database is concerned.

Hey, caveats! I compared the mongo driver to ActiveRecord::Base.connection.execute; I didn’t compare mongo mapper to activerecord. The mongo document arrangement I chose may well have been suboptimal.

So, what did I learn from the experiment?
1. We can recommend postgres and expect not to get burned by that choice.
2. While mongo is shiny and interesting, it’s not obviously better for this project than postgres.

This is an update to Stubbing out OAuth or OpenID, on how everything hangs together now.

I have a file “spec/stub_openid.rb”, which contains:

require 'openid'
require 'openid/extensions/ax'
require 'gapps_openid'

class OpenID::Consumer
  def begin(*args)
    o = Object.new
    class << o
      def add_extension(*args)
      end

      def redirect_url(*args)
        "/session/create"
      end
    end
    o
  end

  def complete(*args)
    OpenStruct.new(:status => OpenID::Consumer::SUCCESS)
  end
end

class OpenID::AX::FetchResponse
  def self.from_success_response(response)
    OpenStruct.new(:data => { "http://schema.openid.net/contact/email" => User.first.email })
  end
end

This has changed slightly from the previous version. We’re now doing some fancy meta stuff because the gapps_openid extension – required to make the openid library find the accounts hosted under Google Apps for Your Domain – changes the way the extension works a bit.

Because the library names are a bit nutty, we have these two gemfile lines:
gem “ruby-openid”, “2.1.8”, :require => [‘openid’, ‘openid/extensions/ax’]
gem “ruby-openid-apps-discovery”, “1.2.0”, :require => ‘gapps_openid’

That stub_openid.rb file is ignored in production, and those two lines mean the library gets included properly. Thanks, bundler!

In order to make the library not be hit in the cucumber environment, our config/environments/cucumber.rb file requires ‘spec/stub_openid’.

And now it’s time for rainbows and unicorns.

Daring Fireball, huh? Often interesting, sometimes Completely Damn Infuriating.

Today is one of the completely infuriating days, as he talks about jailbreaking.

Defending Apple’s institution of measures to prevent downgrading from one official OS version to another, he says:

Apple isn’t going to support downgrading to an older version of the OS with known security vulnerabilities.

The thing is, Apple isn’t simply “not supporting” downgrading. They could do that, and not provide any mechanism to downgrade, perhaps by making iTunes refuse to downgrade a device. Instead, the device itself will only install firmware that Apple signs as appropriate for the device, and Apple take away the mechanism for signing firmware as appropriate to the device when they want to make a release un-installable. That is to say, they actively prevent the installation of past versions.

There is a difference between “we don’t support that, it voids your warranty” and devoting substantial technical effort to make sure something never happens. And Gruber’s confusion between the two has brought me to gruber rage today.

In his most recent post on the matter, Gruber writes

I didn’t say Apple doesn’t have a stance regarding jailbreaking. But judging by their actions to date, they’re treating it as nuisance, not a significant problem

That clarified version of his argument is not one that I particularly want to argue with. I just object to his claim that deprecating old versions necessitates preventing downgrades.

(incidentally, stuff like the whole SHSH nonsense is why I was never able to bring myself to jump on the iPhone bandwagon)

We have an application that uses OAuth to authenticate users as having valid google identities. It works, but there was a few conditionals in there so that in development mode it just skipped that step, which offends one of my co-workers. I wonder if it’ll offend me when I’ve got more experience in this game.

Goal: Put in place a little service, using something like sham_rack or WebMock to impersonate one or other of the services. This is a story of 2 failures and 1 success, more or less. I talk in some detail about what the process is when you use OpenID or OAuth, and it’s worth noting that this is detail I’ve gleaned from looking at the actual traffic, with occasional checks of the standards, to see how I could make one particular thing work in one particular case. If you want a general description of how things work, then run away, run away now.

At the moment, the application uses OAuth to get access to a user’s contact list (ie, address book). Then it reads the owner of the contact list, fetching all of the user’s gmail contacts in the process, and authenticates the user as that person. Awesome. The process has many steps:

1. User -> Application: I want to login
2. Application -> Google: I want some contacts
3. Google -> Application: okay, have this crazy number
4. Application -> User: Login over there. Use this crazy number
5. User -> Google: I’ve got this crazy number. And a username. And a password.
6. Google -> User: Cool cool. Go over here
7. User -> Application: Hi. Do you love me?
8. Application -> Google: Pssst, about that crazy number..
9. Google -> Application: Yeah, yeah, it’s cool.
10. Application -> Google: Gimme contacts, kthxbye.

Our goal is to meddle with the conversation between the application and google, such that the user goes from our “start logging in” page directly to our “login successful” page without seeing any googlestuff. The only way I could think of to do this is to change step (4), so that when the OAuth library thinks it’s sending the user to the “login over there” url, it’s actually sending them to the “login successful” page. Accordingly, I started looking at the response from Google to the application in step 3, thinking to mock our login-successful URL in there.

FAIL. The OAuth thing is for the application to know what google’s login page is, not for Google to tell the application where to go. So it can’t be done for OAuth just by messing with the wire traffic.

But that’s okay, because using OAuth for this is silly anyway! We want to send a user to google with a question of: “what’s this guys email address?”, and google to answer that without actually giving us any of the poor bastard’s details. And this, happily, is what OpenID is for.

Now, the OpenID way:

1. User -> Application: I want to log in
2. Application -> Google: Where you at?
3. Google -> Application: I’m at here
4. Application -> User: Login here. Give ’em these details. Make sure they give me your email address. Go back to the donespot when you’re done
5. User -> Google: Hi. I’m me. Can you tell them my email address.
6. Google -> User: Cool. You’re you. Go to the donespot, and tell it that your email address is foo@bar.com.
7. User -> Application: Hello, donespot. Here’s a number. And my email address.
8. Application -> Google: Hey, here’s this number. They say you authenticated them. And that they are foo@bar.com. And some other stuff.
9. Google -> Application: Yeah, I said that

The notable difference between the two is that in OAuth, we throw an authentication token back and forth until google says that dude’s authenticated, then we pull information directly from the server to google. In OpenID, we throw an authentication token back and forth, but the information we want goes from google, to the client, then to us, and we then throw it to google with the token and ask for confirmation that the authentication token and the information actually line up.

How do we achieve our goal in this case? How about we change google’s response in step (3), so google claims to be at our login-successful page? Then we can forge the confirmation in step 9 to say it’s all good. The fail this time comes from the query strings which are used to keep this whole show on the road. There’s a query string that indicates “login successful, here’s the juice”, and a query string that indicates “I want to login, what happens now?”. The ruby-openid library manages those query strings. We can put a /sessions/create?openid.mode=id_res in as the page to log in to, but the library chops off the query string and sends the user to /sessions/create?openid.mode=checkid_setup. At which point the openid library rightly says that it’s not being asked to confirm a session’s creation, it’s being asked to start the process, and what the hell are you doing you crazy fool?

So, in both cases, the working of the library prevents the only ways I can think of to prevent the user from getting across to google. My man-in-the-middle attacks, stymied! That’s actually not such a bad thing, in retrospect.

What, then, are we to do? Looking at my code that used OpenID to talk to google, I realised that it the library had a very small surface area. Then I wrote the following file, which I saved to “lib/open_id.rb” (because this is rails-land, referring to OpenID will hit the missing constant stuff, constantize OpenID to open_id, then require the right file, all by magic).

require 'openid'
require 'openid/extensions/ax'
require 'gapps_openid'

class OpenID::Consumer
  def redirect_url(*args)
    "/session/create"
  end

  def complete(*args)
    OpenStruct.new(:status => OpenID::Consumer::SUCCESS)
  end
end

class OpenID::AX::FetchResponse
  def self.from_success_response(response)
    OpenStruct.new(:data => { "http://schema.openid.net/contact/email" => User.first.email })
  end
end

Now any time we talk to the library, expecting it to talk to google, it just answers back nice and fast with something that works. Hooray. Success! Achieved with only a slight modification of the original goal.

Now I have a way of stubbing the library out. All I need now is a way of conditionally including it for development and test environments in a way that won’t offend my co-worker…

I bought a shiny new Android phone – it’s a HTC Desire, I like it, and having the internet in my pants makes me feel like I’m living in the future – and I’ve been doing a little bit of development for it. There’s another app idea I want to work on, but I don’t want to be working on it in java. I want something more functionalish, you see, because doing real stuff in functional languages has appealed to me for a long time.

So, I’ve been trying to get an app written in Clojure to happen. Thankfully, someone else has already blazed this particular trail. Stupidly, I thought that meant it would be easy. One attempt at compiling a fresh-baked android app with clojure later, my expectation was no longer one of ease. What ensued was a process of ambling forwards and backwards through the process until victory. This blog post is mostly about hopefully making error messages line up with solutions in google.

The first error message?

-dex:
[echo] Converting compiled files and external libraries into /Users/iain/Code/chordtastic/bin/classes.dex…
[apply]
[apply] UNEXPECTED TOP-LEVEL ERROR:
[apply] java.lang.OutOfMemoryError: Java heap space
[apply] at com.android.dx.util.IntList.(IntList.java:87)

I went back to the very start. Grabbed the hello-dalvik.sh script that somebody posted on the mailing list. Got it working (for me, it works trivially with the emulator but fails against the actual device). If you’re having trouble, that’s probably a good place to start. In my case, I discovered that I could solve the out of memory error by using a “target=android-8” line instead of “target=android-3” in the default.properties file. It looks like editing this file is discouraged, and you should have the sdk do it for you by issuing this command (which git tells me makes precisely that change in that file, no more, no less):

$ android update project -p $PROJECT_DIR -t 7

And then, like magic, dex stops running out of memory. Trumpets! Parades! A litter bearing the king!

Not yet. It’s very slow. There are some ideas on how to fix it, but I haven’t looked at them yet. Make it work, then make it fast.

So, making it work. At this stage if I do “ant install” in my project directory (I used this as a starting point of sorts), the clojure code gets compiled and pushed into the emulator just fine, but it just crashes when I try to start it. Taking a look at our friend Mr Adb Logcat, we see the following:

E/AndroidRuntime( 433): Uncaught handler: thread main exiting due to uncaught exception
E/AndroidRuntime( 433): java.lang.ExceptionInInitializerError
E/AndroidRuntime( 433): at java.lang.Class.newInstanceImpl(Native Method)
E/AndroidRuntime( 433): at java.lang.Class.newInstance(Class.java:1479)
E/AndroidRuntime( 433): at android.app.Instrumentation.newActivity(Instrumentation.java:1021)
E/AndroidRuntime( 433): at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2409)
E/AndroidRuntime( 433): at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2512)
E/AndroidRuntime( 433): at android.app.ActivityThread.access$2200(ActivityThread.java:119)
E/AndroidRuntime( 433): at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1863)
E/AndroidRuntime( 433): at android.os.Handler.dispatchMessage(Handler.java:99)
E/AndroidRuntime( 433): at android.os.Looper.loop(Looper.java:123)
E/AndroidRuntime( 433): at android.app.ActivityThread.main(ActivityThread.java:4363)
E/AndroidRuntime( 433): at java.lang.reflect.Method.invokeNative(Native Method)
E/AndroidRuntime( 433): at java.lang.reflect.Method.invoke(Method.java:521)
E/AndroidRuntime( 433): at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:860)
E/AndroidRuntime( 433): at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:618)
E/AndroidRuntime( 433): at dalvik.system.NativeStart.main(Native Method)
E/AndroidRuntime( 433): Caused by: java.lang.ExceptionInInitializerError
E/AndroidRuntime( 433): at clojure.lang.Namespace.(Namespace.java:34)
E/AndroidRuntime( 433): at clojure.lang.Namespace.findOrCreate(Namespace.java:176)
E/AndroidRuntime( 433): at clojure.lang.Var.internPrivate(Var.java:94)
E/AndroidRuntime( 433): at net.remvee.android.hello.ClojureHelloAndroid.(Unknown Source)
E/AndroidRuntime( 433): … 15 more
E/AndroidRuntime( 433): Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: Could not locate clojure/core__init.class or clojure/core.clj on classpath:
E/AndroidRuntime( 433): at clojure.lang.RT.(RT.java:305)
E/AndroidRuntime( 433): … 19 more
E/AndroidRuntime( 433): Caused by: java.io.FileNotFoundException: Could not locate clojure/core__init.class or clojure/core.clj on classpath:
E/AndroidRuntime( 433): at clojure.lang.RT.load(RT.java:412)
E/AndroidRuntime( 433): at clojure.lang.RT.load(RT.java:381)
E/AndroidRuntime( 433): at clojure.lang.RT.doInit(RT.java:416)
E/AndroidRuntime( 433): at clojure.lang.RT.(RT.java:302)

So, that’s awesome. I spent a while being stupid here. I read that “Could not locate clojure/core__init.class” message to mean that it couldn’t find clojure at all, so I took the clojure code from the shell script and dragged it into ClojureHelloAndroid.clj, compiled it with the ant task, and ran it on the device with the dalvikvm command (if you try to do this, bear in mind that you need to slap a namespace on the class that you’re asking it to start: dalvikvm -classpath /data/app/net.remvee.android.hello.apk net.remvee.android.hello.ClojureHelloAndroid). At that point I was satisfied that the build process in the project was baking clojure in in a functioning sort of way.

Google google mailing list newsgroup off-topic posts later, I find some talk about classloaders in relation to that error message. And am reminded of dalvik’s non-javaness. And realised that the last four lines of that stack trace clearly indicated that clojure was, in fact, coming out to play, it was just that it was having some trouble loading other bits of itself in.

To the source code! In the clojure source tree, there’s a file src/jvm/clojure/lang/RT.java. It has a line like this:
final static public Var USE_CONTEXT_CLASSLOADER =
Var.intern(CLOJURE_NS, Symbol.create("*use-context-classloader*"), T);

And if you change that T to an F, then rebuild clojure, drop the new jar into your project’s directory, rebuild and reinstall, you might just find that it works. Clearly this is an awful cludge. I have five things now to do:
1) Tinker with dex stuff, to try to get the dexing time down
2) Get the app startup time down in the emulator
3) See if the whole house of cards stays up on the actual device
4) Do some actual development
5) Find a way of making it work without that particular awful cludge. I suspect this involves learning about what class loader is in play when the app launcher kicks off my application.