The Unwashed Meme

The problem

We use the trac TimingAndEstimationPlugin to record a lot of our time and git, of late, has been our preferred source control system. As we’ve been using more topic branches in git this poses a bit of a headache: patches can show up multiple times.

We’ve traditionally had the post-receive hook only select patches that update the master branch. This mostly works, but sometimes ticket comments and hours don’t appear for days/weeks until that branch gets merged back. What I really want is for a patch to be posted onwards only when it is new to this repo. As such we can develop in a topic branch, push that topic up to the central repo so it is backed up and hours/comments are recorded for the project manager to see, and then merge it into the mainline at some point later and have all of this work as expected.

The only thing this doesn’t support is rebase… but we all know that’s dangerous already, right? [1]

git rev-list to the rescue

During post receive we want to find commits that are now reachable from one of the refs, but was not previously reachable.

It turns out git-rev-list can give us exactly that, but crafting the call to it is a bit tricky. When you call git rev-list CommitA ^CommitB it will give you back the set difference, i.e. commits reachable from A that are not reachable from B. So we just need to ask git what’s reachable now, that wasn’t before this receive.

Algorithm

The post-receive runs after the database has been updated but you are given a log of what has been updated.

1. Find the set, OriginalRefs, of refs reachable in the database now.
2. foreach ref that was updated: remove it from OriginalRefs; exclude the old value and include the new value of that ref.
3. exclude all refs left in OriginalRefs

There’s a bit of trickiness in watching for new or deleted branch names but that’s pretty much it. We feed these to rev-list over stdin and read the list of commits on stdout.

We implemented it in python and run this regularly. Speed is pretty good even for repositories that have several hundred named refs in them (mostly publish tags); a couple seconds of overhead perhaps[2], but compared to the ssh push it doesn’t feel like the process is slower.

[1] The next day after we deployed this we ran into a case of someone having the rebase on pull flag set that caused problems.
[2] This also includes time to load a trac environment and make a database call in there for every new commit we’ve found; this could be improved but good enough.  I suspect the time in git rev-list to be even less.

Scenario

I ssh into a bunch of different servers all the time. I would like to set my EDITOR variable on those servers to invoke emacs on my local machine using emacsclient.

This is confounded somewhat because my machine is behind NAT so my host is not remotely addressable. If it was I could tell emacs server to listen publicly, and maybe even get the server to use a consistent auth key and port.

The setup

• A wrapper around ssh to send the configuration to the remote server whenever I start a session there. This script will also add the remote port forward so that emacsclient on the remote host can get back to my machine.
• A script on the remote hosts to invoke emacsclient with the right parameters.
• export EDITOR

The Details

ssh wrapper

Stick this in a script on the local machine and invoke it instead of ssh.

editor scrpt

Put this on the remote machines, marked executable, and export the EDITOR variable to point at it:

EDIT (6/23/10 10:40AM EST): Updated the editor script to include some error checking.

Here at Acceleration.net we love Mozilla Firefox and wanted to share a few tips that make it even better. Firefox is a powerful, secure, and friendly web browser. If you’ve only ever used Microsoft’s Internet Explorer before you will probably find the interface fairly familiar; power users can take advantage of an abundant ecosystem of add-ons to really make it their own.

Smart Location Bar

The bar at the top of the browser where the address is (e.g. http://www.acceleration.net/) went through a big upgrade in Firefox 3. The improvement was so great that during development it was dubbed the “Awesome Bar.” A name we feel it continues to merit even after using it for over a year!

When you type something into the bar, just a couple words you remember from the title, it will search through your browsing history and bookmarks to try to find the best matches. Separate the words you are searching for with spaces; they don’t have to be in order or complete.

Example 1. I was reading an interesting news article the other day about netbooks and I want to now show it to someone else. Just type ‘netbook’ in and there it is.

Example 2. I want to lookup a ticket on our work tracking system– named trac.

Note how it bolds the matching part of the title and address– this makes it easier to see why that result shows up.

If Firefox shows no results then hitting enter will use Google’s Browse By Name feature. Google performs a search, if it thinks the first result is a good enough match it sends you directly to that page. Otherwise the standard page of Google results is displayed. Mozilla Reference.

Persistent Sessions

If the power goes out or Windows is nagging me to apply updates it is nice to have persistent sessions in Firefox. The same tabs and pages that were there when you closed Firefox will reappear next time you open Firefox.

It’s really easy, go to

1. Tools > Options > ‘Main’ Tab
2. In the top section ‘Startup’
3. From the drop down “When Firefox starts: “, select “Show my windows and tabs from last time.”

If you want more control over session saving and restoring, e.g. have Firefox prompt at startup for which session (or parts of one) to restore, install the Session Manager add-on.

Smart Keywords

If you frequently use a site’s built in search you can easily create a Firefox shortcut to search there right from the Awesome Bar. This works great even for internal company webapps that Google can’t get to.  E.g. We’ve got this setup so we can find a client record as quickly as possible when handling support calls. Another one of my favorites is to use this to search wikipedia by just typing.

1. Ctrl-L This puts my cursor in the address bar up top.
2. w Levon Helm w is my keyword for wikipedia and we’re currently listening to The Band so I wanted some more information[here].
3. Enter submit the search

Alternatively if you prefer the Search Bar (the one at the top right) then Ctrl-E is the speedy shortcut to get there and the community has already built up a big list for the popular public sites.

Automatically Block Ads

Are the flashing ads a distraction while you are trying to read an article? Adblock Plus to the rescue. Adblock Plus helps keep the web sane by making it easy to remove ads from pages (a new option in the right-click menu) but even better you can subscribe to a list of filters the community works to keep up to date. I recommend EasyList which is over 3000 carefully targeted filters that remove the most obnoxious ads you are likely to hit frequently.

What is the difference between Mozilla and Firefox?

Mozilla is the organization that has as its premiere (free) product Firefox. Firefox is the web browser (the piece of software that lets you read pages on the internet). Mozilla also publishes an email client, and supports many more projects as well.

Other tips from Mozilla

I put together a spreadsheet at google of all the setlists the dead have performed so far. Grabbed the info from http://www.dead.net/dead09.  They’ve got very little repetition in their set lists so far.

We travelled to the next town via game trail, making great time. Ate a delicious lunch of venison and then made it to the next town by dusk.

We talked to the guard who didn’t have much to report. At the Red Door Inn we saw a petrified wyvern and secured a decent room for the night. Talked to the weaponsmith a bit, found that the local mayor was upgrading the towns old weapons and that a local gambler had reported being attacked by goblins. No one really believed him though, they thought he was trying to plea out of debt.

Went to a lovely tavern and listened to the bard sing for a while. He bought us a shot of some mediocre fire water. Nothing has the true savoriness of a fey stream.

Midway through the evening, right as I was conversing with the paladin about scouting out the gambling hole that we’d heard of, a bunch of hobgoblins and goblins burst in through the door. While I was facing the window by the door, such was the delight of the music and my discussion that I too was (moderately) surprised.

As the first among us to recover, I upturned a table at one of the intruders. It missed him but did obstruct the doorway somewhat. I then somersaulted out through the window landing on my feet with longsword drawn in front of a crowd of hobgoblins trying to push their way into the inn. As my teammates recovered I clove at two of the dirty creatures, but missed! I was so fired up that I took another swing and this time they both fell.

A couple of the goblins slipped away from me into the tavern and one threw a firebrand at the bar almost setting off all the moonshine. Our quick thinking druid shifted into his mighty bear form, hefted the cask of beer, and launched it at the fire almost completely extinguishing it.

The rest of the battle went fairly quickly. The goblins were cut down while the fled. One of the patrons at the bar showed an amazing will; he ripped off a table leg and went after the would be arsonists.  He barely managed to dodge what would have been a serious blow and then returned even better. The goblins did try to throw two more torches, neither of which were as well placed, and which were put out soon after the goblins were put down.

Leaving the tavern, we ran into a guard captain who thanked us for our help. Looking around, we can see other places in the town were on fire, and there is a dreadful commotion coming from the Northern side of the town…

One of the projects we’ve been working on recently has a lot of dynamic data driven content that doesn’t change much. Knowing this we made sure the backend was setting cache control headers appropriately and then tried to turn caching on in apache using mod_disk_cache– so far so good. Pages were speedier; cpu load was lighter; we were happy. Then tritchey started insulting us.

I didn’t take it personally; I already knew these were good ideas and I saw this as an opportunity. Surely we could match his feat– I should only need to add one line to the config and enable a module. Ah the foolishness of youth.

I turned on mod_deflate, cleared the cache, and hit refresh. Hrm, nothing happens. Firebug[1] is reporting that the page is definately being delivered zipped, but the cache wasn’t ever being filled. It would successfully zip and cache the static javacript and css files. The html coming through the reverse proxy was never put in the disk cache and was requiring a re-request to the backend every time.

I saw some references to AddOutputFilterByType not working with reverse proxies from not being able to correctly identify the content type but that seemed to be fixed and deprecated at the same time. Suggestions were to use mod_filter instead but the docs on mod_deflate still pointed to the old style and there weren’t many good examples of this type of setup with mod_filter (good example coming below). Someone else claimed that apache was stripping off the cache control headers the backend needed, but my logging showed that wasn’t my problem. I’d largely given up and just left it at the caching without the deflate since that gave me better performance.

At the same time we were trying to bring up a new server for this to be hosted on using Ubuntu JEOS (8.04) rather than the Debian Etch we had been using. Once we got all deployed on the new box, I decided to give the caching another try. It works great!  The difference for us seems to primarly reside somewhere between the Apache 2.2.3 we had been using on Debian Etch, and the 2.2.8 with Hardy Heron. To me this somewhat justified our decision to give Ubuntu a shot over Debian. I’ve been a fan of Debian for a while but found quite a few cases where I hit a bug that has been fixed in a newer version that wasn’t in stable. Backports were sometimes available; more frequently it would only be available in testing which through libc6 would require everything to be upgraded at once.

Here’s an example of the config:

    RewriteEngine On # we are actually using mod_rewrite to implement it
    ProxyRequests Off # don't be a proxy, just allow the reverse proxy

    #see if a static file exists in the webroot first and serve it from there.
    RewriteCond /var/www/gainesville-green.com/current/www/%{REQUEST_FILENAME} -f
    RewriteRule ^(.+) /var/www/gainesville-green.com/current/www/$1 [L]

    #if not forward it to the lisp process listening locally
    RewriteRule ^/(.*)$ http://127.0.0.1:3434/$1 [P]

    #set up caching, enabled for the entire site.
    CacheRoot /var/cache/apache2/mod_disk_cache/gainesville-green.com
    CacheEnable disk /

    #Declare a filter named gzipping
    #The 2nd parameter is type of filter. I believe this is saying
    # that the filter operates on the content body, as opposed to
    # the url or some other part.
    FilterDeclare gzipping CONTENT_SET
    #in filter gzipping use deflate when content type equals text/html
    FilterProvider gzipping deflate Content-Type text/html
    FilterProvider gzipping deflate Content-Type text/css
    #'$' here is substring match, match both text/javascript application/x-javascript
    FilterProvider gzipping deflate Content-Type $javascript
    #insert the filter into the chain, by default at the end.
    FilterChain gzipping

[1] I’m using the beta which has got a lot of nice improvements to an already great extension. Also Clear Cache Button is a nice Firefox exension that aided in the testing here.

We’ve been working for a long time to resolve an error in our ldap setup. Whenever we tried to use the LDAPVerifyServerCert option to verify the ldap server we were talking to is correct, it didn’t work. Always failed with the unhelpful error:

[LDAP: ldap_simple_bind_s() failed][Can't contact LDAP server]

We had set the appropriate CA cert with the LDAPTrustedGlobalCert option. We could use openssl s_client to verify the certificate chain.  We couldn’t figure out why it didn’t work; it was always just simple bind failed.

I finally found it today: The certificate file needs to be readable by others. Aparently the apache process reads that file separately than the rest of the config. SSL certificates for mod_ssl appear to be fine to only have root read on it, but not the LDAPTrustedGlobalCert.  It would’ve been nice if the log message had said something like “Permission denied reading …” or even just “Couldn’t read certificate.”  Unfortunately it falls back to the most generic error there.

I just read an article in The Washington Post about Wikiality.

Society is running up against the truth of what ‘Peer Review’ actually means and finding it disconcerting. The notion that truth by committee as practiced on wikipedia is something different than truth by committee as practiced by a journal is getting tiresome[5]. When society actually uses this idea of Peer Review to create[1] the most powerful and easy to use system of edit and review so far, articles like this bemoan that we have somehow let the Golden Days Of Research[4] slip away. That article was decent but when they complain that Google is linking to wikipedia instead of Peer Reviewed research I say it is working perfectly. A key point: google operates by Peer Review; that is the fundamental basis for their algorithm– someone linking to a website is a vote of confidence[2]. The fact that Wikipedia has risen to the top in this process shows how fundamentally useful it is.

The truth[3] remains though: research is hard, it always has been and probably always will be.  Maybe what these authors are really bemoaning is more people think they are good at research because it has actually gotten easier for once.

Maybe I just need to follow their advice:
Wikipedia is awesome.
Wikipedia is the best source online.
Didn’t you hear how great Wikipedia is?
There is nothing better than Wikipedia.
O O
<       “Wikipedia works for me.”
\__/
And for fun, here are my contributions to this great endeavor, anyone care to review them? What about yours?

[1] By Clay Shirky’s estimate, this–all of wikipedia– was created in less time than American’s spend watching TV every year. Just wait till we put a couple more years into it.
[2] The more people that link to a website, the more weight an outgoing link from that website counts for. So an expert– someone with lots of incoming links– will be counted more than one without. http://www.rose-hulman.edu/~bryan/googleFinalVersionFixed.pdf
[3] In my eyes, do you agree?
[4] This may be a bit of a straw man argument– they never said as much in that article– but it is a common theme I’ve seen about.
[5] The main difference, one side is much less about the reviewing and more about designated authority.

Peter Seibel wrote a Common Lisp tutorial that is the best introduction for Common Lisp that I’ve seen. I’ve poked around at a few different books, but that one was the one I actually learned Common Lisp from. It is clearly written giving you what you need to know without getting bogged down. I also find myself revisiting it as a reference from time to time even though I now use the Common Lisp Hyperspec much more. Common Lisp rocks.

Even If you aren’t looking to learn Common Lisp, I would recommend Object Reorientation: Generic Functions for a non-Java-like look at how object orientation can be done nicely. Multimethods rock.

• Carptacular: Something is fishy
• Carpe Diem: The day of the fish
• Carpentry: Do fish really build?
• Carpal Tunnel Syndrom: When a fish burrows into your wrists.
• Carport: Where fish park.
• Carpathian: Mountain range made entirely of fish bones.
• Carpet: Flooring of fish scales.
• Carpooling: How a school gets to school.
• Escarpment: A wall of fish.
• A Carpella: a bunch of fish singing.