Skip to content

LambdaConf, Yarvin and Moldbug

I’ve spent the last weekend enjoying the excellent National Folk Festival in Canberra and thinking a lot about the question of whether or not Curtis Yarvin should be invited to LambdaConf, and if so, whether or not I want to attend. This post is about the latter.

I am only interested in going to LambdaConf if I can do so in opposition to the ideas that Curtis Yarvin published under the name of Mencius Moldbug. Even if Yarvin is right when he protests that he is not a racist (and I’m not sure that he is right when he makes that claim), I believe his ideas have fueled and provided cover for a large number of people who do have racist views. He has the power to speak and write in such a way as to push back against those people, but he has not done so. He has the power to say that slavery as an institution in the USA was evil and wrong, but as far as I know he has not done so. He has the power to say that any movement in the direction of a return to slavery in the USA would be evil and wrong, but as far as I know he has not done so.

For a while, I thought I could attend LambdaConf and do so explicitly in opposition to Moldbug: I could wear a badge that had the word “Moldbug” in a red circle with a line through it, and I could have “I Stand Against Moldbug” in the footer of all my slides. Since Curtis Yarvin has said that he, Yarvin, is attending in a professional capacity, and that the Moldbug identity has no connection to his professional capacity, I thought this might be a viable option. Unfortunately, then Yarvin published a post in a Reddit AMA clarifying a Moldbug post, writing as Curtis Yarvin, and using the first person pronoun. This suggests that the line between Yarvin and Moldbug is not as clear as all that, and that to attend in opposition to Moldbug would be to attend in opposition to Yarvin, which would be an ugly and nasty sort of thing to do. So I don’t want to do that.

My next hope was that I could attend LambdaConf in opposition to Moldbuggian ideas. I could wear #BlackLivesMatter merchandise and RISE logos and Black Girls Code tees and put a rainbow sticker on my laptop and find unmoldbuggian examples to thread through my talk. But as I thought more about the way this would work in practice, I remembered something about John de Goes’ original post, in particular the proposed addition to the pledge of conduct:

That in consideration of the professional nature of this event, I will refrain from discussing potentially offensive and divisive topics unrelated to the topic of programming (specifically religion, morality and politics); except in the company of willing participants to such conversations, and even then, only in a manner consistent with the pledge;

My reading is that attending in opposition to Moldbuggian ideas might be viable this year, but if they could change the code of conduct for this year they would have, and it would make this approach unviable. The presence of people who do not stand entirely behind all aspects of the pledge of conduct means talking about the things it promotes can be divisive, and means even people who want to boost the inclusion of minorities (the list in the pledge includes women, people of colour, disabled individuals, and non-cis sexual orientations and gender identities) have to leave that desire at home. The general direction here is that people should attend exclusively in their professional capacities, and leave the personal and political sides of themselves at home.

There’s a sense in which this years LambdaConf is now an experiment in a particular way of dealing with issues of this sort. I could go in support of unmoldbuggian ideals this year, and possibly contribute to the success of the experiment, but attending in the same manner the next year would probably be impossible. I’d like to avoid this outcome.

In fact, I think the only viable way to attend in opposition to Moldbug is to go proudly and strongly in support of the idea of that split between the personal and professional. There are two problems with this: the first is that I have no idea how to attend strongly and proudly in support of that distinction; the second is that I am not proudly and strongly in support of that idea. It’s one thing for Yarvin to leave his political views at home, but somebody whose ancestry didn’t give them fair skin doesn’t get to leave that behind and present as white for the weekend. Somebody who is queer might be able to choose to pass as cis and straight for the weekend, but I can’t endorse asking them to do so. There are so many aspects of the personal that cannot be left behind that I cannot say I am proudly and strongly in support of separating it from one’s professional identity.

Since I do not currently believe I can attend LambdaConf in opposition to Moldbuggian ideas, I do not currently plan to attend LambdaConf this year.

There are many events; their organisers make choices. There are many potential attendees; we choose which events we want to participate in. I choose not to participate in this one.

That I can make that choice is indeed a luxury for which I am grateful. I hope that De Goes succeeds in making LambdaConf into an event where people who are frequently marginalized by our industry are welcome, and can find a foothold to get into or advance within our industry. I hope that MoonConf succeeds in creating a space where people who feel that they cannot attend an event that sponsors Curtis Yarvin can connect, learn, and share ideas.

There’s a number of things I am deliberately not arguing here:

  • that LambdaConf must exclude Yarvin. I think De Goes and the committee are free to try this experiment in requiring their attendees to leave objectionable views at the door.
  • that a government should
    • exclude Yarvin from participating in official events. Governments excluding people for their beliefs has not worked terribly well on either side of the aisle.
    • allow Yarvin to participate in official events. I do believe that Right and Wrong exist, and that it is entirely possible that Yarvin holds, continues to hold, and has worked to propagate views, and to argue for the implementation of views, that are Wrong.
  • that Yarvin holds, continues to hold, and has worked to propagate, and to argue for the implementation of, views that are Wrong. I think much of his writing is disingenuous at best, but I haven’t read enough of it to conclude that he’s wrong in such a manner that he ought to be excluded. I think I can make a decision on this matter without putting myself through that. I do, however, trust David Nolen’s, Sara J Chipps’, and Bodil Stokke’s reactions to his writing enough to think that I’d probably want him to be excluded. Please don’t @mention these folks in relation to this blog post, they have said what they want to say on this matter and are quite capable of jumping in if they wish to discuss it further
  • that people need to be objectively and evilly Wrong for them to be excluded. Dan P and Alissa (Please don’t @mention these folks in relation to this blog post, they have said what they want to say on this matter and are quite capable of jumping in if they wish to discuss it further) make strong arguments that building an inclusive community can require choosing to exclude people who will produce too much discomfort amongst the people who make up that community.
  • that people must remain eternally responsible for things they claim once. Yarvin has not given any sign of having changed his mind on the problematic things in question, so the notion that he’s being attacked for something he said once in the distant past is not an argument that has any force for me.
  • that if Yarvin were to say All The Right Things after reading this blog post I would change my position. I haven’t read enough of his writing to know how he’d need to change his positions to make me think they didn’t need to be opposed.
  • That this list contains all the things I’m not trying to say with this post.

I’d also like to thank Amar Shah for shedding light on the feedback-gathering process and to Julia Moronoki for her thoughtful post about why she and Christopher Allen are continuing to sponsor the conference (again, please don’t @mention them). In general, I’d like to thank everyone for making most of what I’ve seen of this conversation into a respectful argument and not a screaming bunfight.

Matilda, Cambridge Theatre, London

Where 1984 was gripping, exhilarating and exhausting, Matilda was lovely, light and fun. The opening number riffs somewhat on Monty Python’s “we’re all individuals” as all of these kids sing about how special and unique and wonderful they all are. Where the Pythons play the lone ordinary person for laughs, this one is decidedly darker, with the music and melody going sad as Matilda introduces herself, singing about how she’s not special and not wanted. The joke of how everyone is special and unique keeps the scene from getting too dark, but it’s definitely got an edge to it. This is a pattern throughout most of the musical, with the darkness of Matilda’s home situation and the grimness of her school being lightened by various jokes. In particular, the people that inflict most pain (Matilda’s parents, the Headmistress) are comical figures.

The Headmistress is played by a man in drag, which works exceedingly well, although it may not be entirely politically correct these days. Her evil is played to the audience as a joke, but its consequences are played as real and serious, which works really well as a device for leavening comedy with humour.

I was surprised by how late in the piece Matilda’s ability to move things with her mind is introduced. It only appears three times. The emphasis is really on Matilda being special because she’s so smart and already knows how to read when she starts at school and likes going to the library and making up stories. In other words, she’s special in ways that I expect resonate for lots of kids in the audience. That she’s special in ways that make lots of people in the audience all feel special in the same way is unexamined, and I wouldn’t want it otherwise. It’s not that sort of show.

As you’d expect with music by Tim Minchin, the lyrics have plenty of clever rhymes and are generally top-notch. I wished (as I usually do with musicals) that I could hear the words better; in particular, the kids bits were sung in a pretty high register that I had trouble with. I was generally impressed by the kids, though: there were a lot of them and they were actually good, which is not what I expect of child actors.

The writing generally was fun, with three figures deliciously representing various kinds of evil: the bumbling father, the uncaring mother, and the sinister headmistress. The choreography was fun too, and it all ended happily ever after. It was a bit strange, though: there’s a moment when Matilda has the chance to be whisked away to a happier life, but she seems to want to stay with her parents. Then there’s a scene where nothing relevant changes, and then she makes the opposite decision and chooses to be whisked away. I guess the glimmer of Matilda’s parents liking her and Matolda wanting to stay with her parents makes leaving into a genuine choice, which makes the departure lighter in a way that would be lacking if we still thought her parents wholly indifferent. And that makes the happy ending even happier. Hooray!

1984, Playhouse, London

This was a stunning piece of theatre, which I went to largely on the strength of this review.

The show opens on Winston Smith, sitting at a table and opening a book. He muses to himself about the consequences of the diary he is about to start, the consequences of thoughtcrime.

Then, snap, the show twists into another time, another place. A group of people discuss a book and the reality that book describes. Smith is still present, still sitting at the table, but not interacting with the scene. In his presence on the stage he mirrored the way 1984 is present in the background of any discussion of Big Brother-esque behaviour. We don’t know this scene’s time and place, and the scene nurtures that unknowing. They could be in the fictional universe of 1984, talking about Orwell’s novel. They could be in the fictional universe talking about Smith’s account. Or they could be in our universe, in the future, looking back at the present day where the events of 1984 are in the process of taking place. It’s the play on this final possibility, that 1984 is actually coming true right now in the world outside the theatre, that gives these scenes their power. The play is made up of scenes that are taken from the novel interspersed with scenes discussing the book in this ambiguous other time.

There are two other scenes that I want to talk about. One is a scene that repeated 3 or 4 times, with characters in the scene being unpersoned between repetitions. After the first two times we know how the scene goes, so it keeps making a sort of grisly sense even as people are removed from it. The most macabre aspect is that one of the players in the scene is talking about how proud he is of his daughter for informing on someone, and when he vanishes we know exactly who reported him to the Ministry of Love. Later, in the Ministry of Love, he talks again about how proud he is of his daughter, and it is heartbreaking.

The other scene is at the end, when Winston Smith is in the Ministry of Love and they are having what we might enigmatically call a civilized discussion on the matter of what two plus two is equal two. The whole scene is brilliantly done, with the treatment that Smith is receiving being conveyed by smart deployment of a sound design full of low buzzing and high cracking, and a light design involving columns of high-powered light delivering singular potent strobe-like flashes. That technology works to complement the acting, which is just brilliant. The most powerful moment for me was when Smith, as the rat cage is moved closer to his face, breaks the fourth wall – which has been maintained for the entire production – and pleads, exhorting the audience not to sit there and watch, not to let this happen. To have what feels like a personal request from a man in such very great distress makes for extraordinarily powerful theatre.

It also a moment that, without preaching or getting on a high horse, brings to mind a key question that we would do well to have on our minds as we leave the theatre: are we going to sit there and watch, are we going to let this happen, as the world around us develops more and more attributes that look like something Big Brother dreamed up? That this call was delivered primarily as a way of heightening the drama impressed me a lot. That it also serves to highlight that point is a very clever bit of theatrecraft.

You may disagree that the world around us is developing more and more attributes that look like something Big Brother dreamed up. I consider the matter more or less proven by the Snowden disclosures, but if you’re not, remember that this production was held in London, which is one of the most heavily surveilled cities in the world in terms of CCTV. And if that doesn’t strike you as being amiss, the play remains an extraordinary play, weaving together straightforward depictions of the book’s text and scenes that render those depictions ambiguous on the way to a gripping climax when Smith finally cries out a broken man.

4 days in Edinburgh

image

(Not shown: It Might Get Ugly late on the Saturday night)

Sh*t-faced Shakespeare

This is a show with a very simple premise: take a cast of 6 people, condense a Shakespeare play down to an hour, prepare it as a serious Shakespeare production, then pick one of the cast and get them very very drunk before the play starts. Then you have 5 people trying very hard to do the play properly and one person who is completely incompetent: coming onto stage before their cues, missing the cues, telling other people to kiss, telling the audience that she’s going to do some acting now, and generally messing stuff up. It was good fun but not super-hilarious. How good it is probably depends a lot on who they get drunk and what whims flit across that person’s mind on the night.

Boris & Sergey’s Astonishing Freakatorium

Really impressive puppetry: most of the puppets took three people to operate (one person on legs, one on arms, one on head, I think) and they were able to use that to give the puppets a really expressive physicality. It also meant that despite having 6 puppeteers they were limited to two characters on stage at a time, which is a bit ridiculous. They did have a fun (if very ambitious) audience participation bit where they called upon two volunteers to assume the characters of spirits summoned in a séance. The puppetry was technically amazaing and I was charmed by the two main characters, but on the whole the show was a bit excessively bleak.

Abandoman: Hot Desk

Good fun, but not the heights of brilliance I was hoping for (this show cane highly recommended by several people). The setup is that the cast have to write a hit song in the next hour or one of them will be killed. We the audience supply ideas that they then improvise a rap around. They start with some love songs (one about a couple in the audience who they quizzed, one about someone’s crush on Lara Croft), then some songs for old people (about hobbies, investments AMD reminiscences), then a song about a first world problem (soggy croutons, in our case). The guy worked the crowd incredibly well, but the show didn’t really click for me.

The Horne Section: Milk the Tenderness

I liked this. Alex Horne is an impresario leading a 5 piece band for a sort of variety hour. They have a guest come in and do a bit (we got a standup who ended his bit by remarking to Horne: “you know, I don’t think I’ve made an audience that angry with for a very long time”. The bit was a shambolic one but I couldn’t work out if it was meant to be or not). Anyway, Horne and the band do a bunch of musical comedy bits and Horne drives the whole thing along brilliantly.

Viewmaster

This was good, too. You start in the Summerhall courtyard, which is buzzing with food, drink and merriment. Then you are led down some passageways, past queues of people chatting quietly as they mill around waiting for their show, and up the stairs to the quietness of the loft, perched above and at a remove from all that chaos. You fill in a questionnaire and enter Dan and Ryan’s den. They look at the questionnaire, analyse the results, and pick a journey for you to go on.

Dan plays the music for the journey, Ryan reads the poetry, and a viewmaster provides the visuals of the journey. It’s a beautiful little experience (the whole show is for one person and only takes 15 minutes) that draws you out of Edinburgh and into the locale of the particular journey with remarkable vividness.

Nofit State: Bianco

This was a very intriguing show. I think it was probably the biggest show that I saw, in terms of both ambition and size of cast. It was held in the Nofit State big top, which was operating as a venue without seats. They had four wheeled truss structures that they moved to provide various settings for the action to take place on, and as the action moved the audience was moved to follow it. This was both a strength and a weakness of the show, with quite a lot of the total running time being taken up with rigging changes as they moved stuff around. They were going for this very dynamic thing of the action shifting and changing and the audience coming after it, which was a beautiful dynamic, but it was hindered by the time taken to move stuff around and lock it into place. The overall vibe of the show was wonderful, feeling like a celebration brim-full of joyful chaos (I know I said something similar about Akoreacro. Once is a coincidence. Twice is a genre). And they had some immensely skilled people, particularly a tightrope walker who did a frontsault.

The Jest

The Jest are a five person sketch comedy crew. My relationship with sketch comedy is a complicated one (I don’t usually like it but I keep going to see it). A lot of their stuff was pretty good. I particularly liked their wordplay where they’d pick a particular actor and work a ton of titles of their films into a sentence.

Jamie Adkins: Circus Incognitus

I think the last one-man circus show I saw was the brilliant ‘Kaput’ by Tom Flanagan. This was a reminder that a show that good is quite unusual. There was nothing wrong with this show, but it felt very formulaic and workmanlike.

Mark Grist & MC Mixy: Dead Poets Death Match

This was fun, although very silly. The show ends with the two guys adopting the characters of two dead poets for a rap battle. On the way, the guys talked about various dead poets and the influence they had felt from them, and did poems inspired by those figures. This was kind of in the intersection of spoken word, poetry and rap, and I think I would have liked it to be more on the straight poetry side, but it was a fun and engaging hour based on dead poets.

Torsten the Bareback Saint

This was pretty bad. It was billed as a song cycle about a Dorian Gray figure, which I guess it might have been, but that didn’t come through for me in the songs. For me the songs were a bunch of disparate points that I was unable to link back into that supposed central theme. If the songs had grabbed me, that might not have mattered, but they didn’t. They’d made a weird choice to have a pianist on stage but a lot of the music was coming through on a backing track. Either have a backing track, or have an accompanist, or make a joke out of having both. This show failed there.

And I haven’t even mentioned the biggest problem with the show, which is that it felt like the performer was taking the audience for granted. This was an side project for him, and I suspect he’s used to playing much bigger stages. His mode and manner might work for playing a bigger room but just didn’t translate to the smallish room he was in.

Or maybe he’s just not very good.

Shaun Usher: Letters Live

This was the only non-fringe event I went to, being part of the book festival. It was a curiously lifeless event hosted by two people, one of whom is behind the ‘Letters of Note’ project and the other is the author of a history of letters and letter-writing called ‘To the Letter’. Through their work these guys have discovered a lot of letters written by and to various people, some with great gravitas and some entirely frivolous. The event consisted of a cast of actors, comedians and so on coming up and reading a selection of these letters. It didn’t work for me, with the length of the letters being part of the problem. Much of the beauty of a letter is the intimacy of it, the strength of authorial voice. And here we had people standing up and reading letters that were not their own. The very sterile context of a book festival stage and the rather short length of the letters did not give the readers the time or the space to claim the author’s voice as their own. That the readers were introduced before they read, thus bringing them into focus as themselves when they needed to be present only as the author’s avatar, added to the problem.

Nice idea, though.

It Might Get Ugly

This was an honesty themed stand-up show with a lineup that changes every night. The brief to the comedians is super-simple: be honest to the point of regret. I think it could be really magical when a skilled comedian really crosses the line into over-sharing, because most of them have practiced walking that line. The night I went was pretty good, but nobody really crossed the line. One of them told a lovely story about the lengths he was willing to go to to visit his girlfriend when she goes to study in Oman, one I have forgotten all details except the two words ‘Canadian’ and ‘lovely’, (oh, a third word: ‘bisexual’), and one told a hilarious story about going back to a girl’s house after a gig, going into her room (in her parents house) and seeing posters of himself on the wall, insisting on seeing ID, then sleeping with her. That last guy was the closest to fulfilling the ‘honest to the point of regret’ brief, but still wasn’t really sharing more about his life than I expected him to be. Still, a good night, even if the MC was a pretty awkward dude. He kept talking about how much the producers had paid him to take the show, which is a regular event in London, to Edinburgh. I assume the producers plan on some shows going well and some going poorly, and seeing the crowd this one drew on a Friday night I assume this one falls onto the ‘poor’ bucket. Which is a shame, because as shows with a lineup of comedians go, I’ve seen a lot worse.

And that was my 4 days in Edinburgh.

Progress report

I remain captivated by the idea of a large-scale software system implemented entirely in purely functional style, and have been noodling away for a while on a much simpler implementation of some of this folly.

The old way was to take pure functions and try to magically incrementalize them. The problem with this – apart from it being really quite difficult – was that I don’t want purely functional style all the way through for its own sake. I want to reduce the number of sources of surprise in day-to-day software development. In practice, a magical incrementalizer can only magically incrementalize some functions, and the rules around which functions get nice big-O behaviour in the incrementalized versions are surprising indeed. So maybe that’s not such a good idea.

Instead of writing pure code and turning that into incremental code, I’m now writing code in terms of a set of primitives (lists, maps, structs) that know about acting incrementally. It works pretty well and I’ve just finished a big reworking so the guts of the thing are much simpler.

The implementation takes the user’s function (written in terms of the below primitives), and turns it into a network. The difference between this and every other FRP or related system I know of is that the change propagation network is completely distinct from the value network, and the primitives and operations are designed so that any change in the input can be pushed through to the system state with nice big-O behaviour. This is accomplished by evaluating the user’s function once – with a funny notion of evaluation – then walking all changes through the resulting network.

Every invocation of a primitive becomes a node in the network. Each node has some inner nodes; changes are propagated from each node to the nodes that have it as an inner node, and a node is only fully compiled once its inner nodes have been compiled. In the current design, change propagation goes like this:

  1. A change arrives at a node
  2. The node takes the change and produces a (possibly-empty) list of changes that result
  3. The new list of changes are propagated from the node.

This is likely to change in a future reworking so that a node can propagate a change differently according to where it arrived from (in the current design, StructElems exist solely to work around this lack).

A change is an impulse applied at a location. A location is a path through the structs and maps that make up the value being changed. Once a change is propagated up to the top-level value, it is ready to be applied.

Primitives and Operations

Currently the following primitives and operations exist:

Lists

With lists, the overall list is a changeable evolving thing, but the individual values are not.

map :: (a -> b) -> [a] -> [b]

If the change is adding an element e, map produces a change that adds the element created by applying e to the function. If the change is removing an element e, map produces a change that removes the element created by applying e to the function.

filter :: (a -> bool) -> [a] -> [a]

For both adding and removing an element e, when applying e to the function returns true filter returns the same change. It produces no changes (effectively swallowing its input) if applying e to the function returns false.

shuffle :: (a -> b) -> [a] -> Map b [a]

Shuffle takes a list and sorts it into buckets. For both adding and removing an element e, shuffle causes that change to happen at a path determined by passing e to the function.

inputList

An inputList is necessary to feed the monster. The idea is that you have one (or more, I guess?) streams of inputs to your application. Constructing impulses that add elements to your input list allows you to have your application’s state evolve over time.

Structs

Structs are intended to hold related values. They are largely a way of translating product types into this space of incrementally-evaluated functions. Their structure and keys are fixed, but their values are changeable evolving things.

Dicts

With a dict, the overall structure is a changeable evolving thing, as are the individual values, but the keys are fixed.

map :: (a -> b) -> Dict k a -> Dict k b

This takes a map and applies the function to all the values in the map. I use this to take a shuffled input list and produce a map of structs of interesting things resulting from that list of values.

Interesting aspects of the implementation

There are two wrinkles that might be interesting if you’re trying to do something like this. The first is not a big deal: if you don’t want to split structs into a struct node plus a node for each element, then you need to be able to handle incoming changes differently depending on which node they came from.

The more interesting one is around mapDict. The arguments passed to mapDict are a function f and a dictionary d. The hopefully unsurprising goal is for every elem (k, v) in d to become (k, f v). One approach might be to generate a change propagation network for f for every element in d, but that’s going to get us a huge change propagation network over time. What can we do? It turns out that changes coming out of d are already changes at a location in d. That means that if we construct a single change propagation network for f and feed changes coming out of d into it, we’ll get suitable changes for the mapDict coming out of it.

The question then is how to do that. We can’t just apply dict to f because f is a function from a -> b, and dict is of type dict k a. Therefore, we introduce the ability for the internals to create an arbitrary node of any type, and we create such a node (which we call a) and pass it to f. At this point we have two choices:

  1. Manually introduce the plumbing so that changes coming out of d get propagated to the a, and cause changes coming out of f a to get propagated out of the mapDict call. This turns out to be pretty hairy, but the hairiness is all in MapDict.
  2. Declare that any arbitrary node a has an inner node (which is any watchable thing), and changes in that watchable thing appear as changes in a. This turns out to be a whole lot simpler, with the caveat that each type now needs to not just provide a mechanism for producing an arbitrary node, but also to allow that arbitrary node to have an inner node and compile it appropriately.

Spoiler alert: the second approach is better.

Future Work

There’s so much future work! So much. So much.

Right now, the distinctions between what are changeable evolving things and what aren’t is rather ad-hoc. This could stand some more rigor.

It could be quite useful to have operations like

  • keys :: Map k a -> [k]
  • values :: Map k a -> [a]
  • elems :: Map k a -> [(k, a)]

Intriguingly, the elements of the list returned by values could be changeable evolving things (as could the second elements of elems‘ tuples), unlike other lists. This feeds back into the rigor around distinctions mentioned above.

It would probably be useful to have a way of looking up elements in a struct or map.

The syntax for invoking all of this stuff is hairy. I need either a new language or a much prettier embedding into haskell.

I have integers, but the only operation is addition, and the only way to get one is by summing a list. Generalizing that sum to a fold might be useful.

There’s a growing diamond problem under the hood. I’d like to get that under control, but it feels like it needs to get worse before it can get better. It may not turn out to get worse. There’s definitely some work to be done around removing duplication, especially around inners, ids, and the ability to be a tube.

And all of that is without looking at pushing back towards integration with a datastore or pushing forwards into the browser.

An Oversight

Object Relational Mappers have a dream. Their dream is that the mechanics of creating, reading, updating and deleting can be abstracted away and need not be thought about.

I have a dream, too. My dream is that the fact that creating, inserting, updating and deleting is going on can be abstracted away and need not be thought about.

The ORM dream has a big ol’ flaw, to the extent that I am sceptical about the use of ORMs in general. The flaw is that the right way to do CRUD operations depends rather heavily on the domain and the application in question.

My dream should not be vulnerable to that particular flaw, though, because I’m trying to abstract away those operations entirely. Or so I thought right up until yesterday, when I realised that since my approach is to first convert the programmer’s code into a form that does have CRUD operations, I’m going to have the ORM problem from then onwards.

And it’s worse, because when you call the CRUD operations yourself you can exercise a fair degree of control over what goes on. When the CRUD invocations are automatically generated, that control would have to be exercised by hints and indirect suggestions.

Bugger.

3 Approaches

In today’s fantabulous outing, marvel at three approaches to efficient implementation!  Ride the wave of perplexity that is the function from a list of inputs to the current state! Gaze in wonder at the persistent storage of that current state!

Option 1 – incrementalisation

In this approach, we take a state function f, that takes a list of inputs and produces current state, and produce an incrementalised state function, that takes a change in the list of inputs to the resulting change in the current state. Liu gives a systematic approach for deriving the incrementalised versions of functions.

Advantages

  • It enables a neat way of making a highly responsive UI. As well as incrementalising the state function, you also incrementalise the view function (a function from state to HTML). On the client, you can run the incrementalised state function, pass its output to the incrementalised view function, and apply the resulting change to the DOM. (this works)
  • It lends itself to a really cute way of integrating with a data store: write a function that takes the system state and returns a relational representation of that state. Then incrementalise that function. Any time the system state changes, you can pass that change into the incrementalised relational representation function and hit the database with its output. Shiny! (I haven’t done this yet)

Disadvantages

  • The wording of Liu’s paper is very careful. What they have is a systematic approach for deriving incrementalised versions of functions. What they don’t have is a fully-automated approach. There needs to be some user interaction or some hinting (in the general case) in order to produce an incrementalised version of a function.
  • When you have an inner function that uses free variables, and the values of those free variables can be affected by a change, the incrementalised versions start to get pretty hairy and potentially rather inefficient. The current version of my code assumes that any free variables used will not change, which is obviously not true in general.
  • At the moment I have an informally specified bug ridden implementation of less than half of Liu’s technique.

Option 2 – Partial Evaluation

Partial evaluation is where you take a program of n inputs, then provide some subset of those inputs and evaluate the program with respect to those inputs. This isn’t immediately applicable to this state-updating shenanigans, because there’s no provision there for changing inputs.

Instead of running the partial evaluator directly on a program that takes a list of inputs and returns a state, we can augment the program with a new top level function:

return state(cons(new_input, inputs))

Assuming the partial evaluator unfolds that cons, we’re in with a chance of getting a program that incorporates a new input without doing too much stuffing around. It’s not at all clear how you might get the state into a persistent data store with this approach, but worse then that, the resulting program won’t know how to capitalize on existing bits of state. This is probably surmountable by memoizing computation of bits of the state, but if you’re going to go there…

Option 3 – Adaptive Computation

Adaptive computation (after Acar) is where you think of a program as building up a dependency graph of values. The final output depends on a bunch of intermediate values, which depend on other intermediate values, which ultimately depend on the program’s inputs.

There’s no obvious barrier to storing that whole dependency graph in some sort of persistent store. Obviously, this is going to be a much weirder database than Option 1 gives you; but on the other hand, this approach naturally gives you much more fine-grained change propagation behaviour. That means you don’t have to worry about free variables making stuff confusing. More importantly, it is fully automatable. You just feed your program into the madhouse, and off it goes.

Similar AJAXyness to that of Option 1 is imaginable, but vastly more intricate: you must ship the portions of the dependency graph relevant to the client’s computation down to it. Determining in advance which portions will be relevant is quite likely to be vulnerable to the halting problem, so you’ll have to either send too little (and have a lazy loading strategy to get other data that you need on the client) or too much (and clog the intertubes, OMG).

This option is closest to the approach taken by Asana’s Luna and by Meteor.

Parenthetically: I was amused to discover that with Option 1, the efficient thing to do is to cons a new input onto the head of the list. With this option, the efficient thing to do is to replace the empty list at the end of the input list with a singleton list containing the new input.

Conclusion

I’m sticking with Option 1 for now. I’m about to attempt integrating with a database; if that goes badly, Option 3 is going to start looking much better. Again.