Skip to content

CouchDB: the last RESTful JSON store you’ll ever need

Today I gave a talk about CouchDB at the company I work for. I titled the talk “CouchDB: the last RESTful JSON store you’ll ever need”, which is of course an exaggeration, but our shop is heavy into Oracle and JSON services, so I geared the talk around presenting CouchDB as a viable alternative. I demoed Couch Crawler, as well as an internal app that tracks server metadata. Hope you like it. Also, please let me know if anything is wildly inaccurate. I did a fair amount of research and prototyping on CouchDB, I’m still a bit of a n00b.

CouchDB: the last RESTful JSON store you’ll ever need

Posted 04 February 2010 † Carlo § CouchDB ‡ Comments (0) ° Tagged:

Announcing Couch Crawler, a CouchDB search engine/crawler

Hi! So, for fun, I made couch-crawler, a search engine and crawler on top of the very excellent couchdb-lucene. I wanted to create a hackable search engine for my work intranet using modern tools. Lucene is great, but the Nutch search engine/crawler was kind of annoying to work with. I couldn’t figure out how to get it to update the search indexes without a restart of the server, which sucks. Also, I just really, really like CouchDB.

There’s no real web tier, CouchDB hosts static JavaScript/HTML/CSS files and the UI gets built up dynamically with AJAX calls to CouchDB. It’s kind of nice to be able to cut out a whole layer of glue code.

Templating is done with mustache.js, a JavaScript templating language that does a good job of being a dumb template language, making you define your presentation logic in JavaScript, where it should be.

On the indexing side of things, there’s a crawler written in Python that pulls down html, parses it with BeautifulSoup, extracts useful text content to be indexed then follows links within the page to a specified max depth. It probably could be smarter and parallel-er, but I wanted to start with a simple design and iterate over it.

The couchdb-lucene indexer indexes the title, url and contents, and saves the first 140 characters from the contents in the index to display with search results.

Ch-ch-check it out and let me know what you think.

P.S. If you use Homebrew for your OS X packaging needs, I have a fork of homebrew with a couchdb-lucene formula for easy installation.

Posted 17 January 2010 † Carlo § CouchDB § Lucene § Python ‡ Comments (5) ° Tagged: , ,

Javascript sucks at sorting integers

In general, I heart Javascript. It’s one of the most misunderstood languages, but it has definitely made a comeback in a big way, not just with sweet client-side frameworks like Prototype and jQuery, but also on the server-side with CouchDB, MongoDB and Node.js.

But in its current form (Javascript 1.5), it sucks at sorting integers:

js> [10,5,3].sort(); // should be [3, 5, 10]
[10, 3, 5]

Suck! It’s trying to sort the integers alphabetically, as if they were strings. Fortunately, you can override the default comparator:

js> [10,5,3].sort(function(a, b) { return a - b });
[3, 5, 10]

That works nicely, but you still need to know to do that with integers, violating the principle of least surprise.

Posted 13 January 2010 † Carlo § Javascript ‡ Comments (0) °

Merlke: a native Erlang build tool

After seriously diving into my pet Erlang project, I found myself clinging to Rake to do my builds. It was kind of unsatisfying though, especially once my builds became more complicated (hooray for code-generated parsers, leex/yecc). Why can we have Erlangs all the way down? Futhermore, OTP provides most of the build steps you’d wanna do to an Erlang project, like generating documentation, running tests, and of course compilation. However, there was no nice script to tie them all together, so I wrote Merlke, a native Erlang build tool. (The name is a bastardization of Make and Erl)

Initially I had Merlke require a build file (merlkefile.erl) but I noticed that the vast majority of Erlang projects followed the same file and directory name conventions, so I just provided reasonable defaults for Merlke, letting you override them if you want with a merlkefile.

You can run Merlke like this:

merlke test edoc

Which will execute the test task’s dependencies, the test task, the edoc task’s dependencies, and the edoc task, in that order. If a dependency has been run earlier, it won’t be run again. This is all basic dependency-based programming ala make/rake/ant etc.

Right now it’s just a static set of tasks defined, but I hope to be able to let you create your own tasks and dependencies. Also, the dialyzer and dist (release) tasks aren’t implemented yet. I’ll implement those once I figure out how they work. Patches welcome.

Posted 10 December 2009 † Carlo § Erlang ‡ Comments (0) ° Tagged: ,

Idea: Ikea for computer nerds

How does one make vga/dvi cabling look sheik? Where can I get a designer wifi router pedestal? What’s the best way to mount a dynamically updating grocery list screen onto my refrigerator?

These are the questions of the time. Who will step up to answer them?

Posted 01 December 2009 † Carlo § Wishful thinking ‡ Comments (0) °

Tis the season to go quad core

I was briefly tempted by the various Black Friday marketing to get a PS3 to play some of the sweet exclusives coming out, like Uncharted 2 or Demon Souls, but instead I think I’m going to upgrade my home app server. Currently I’m running Ubuntu Jaunty on an old school Pentium 3 that my old company was going to throw out and it’s kind of showing it’s age. It can’t play Flash videos without st-st-stuttering, and for some reason doing anything causes 100% iowait.

So I read up on the state of the consumer CPU market and hit up good ol Newegg so see what I could come up with. It seems like quad core CPUs are within reasonable price, so I tried to see if I could whip up a nice system.

My goal was lots of CPUs (for ERLANG) and something that wouldn’t break the bank if I left it on all day, so low power usage. That brought me to AMD’s new 45nm Phenom II X4, which benchmarked well for energy efficiency and good price/performance ratio. Here’s what the rest of my rig is gonna look like: Newegg wishlist.

Posted 29 November 2009 † Carlo § Hardware ‡ Comments (1) ° Tagged: , ,

iGhost ride the iWhip

So cool to be so nerdy!

video and title stolen from Giles Bowkett

Posted 11 November 2009 † Carlo § Uncategorized ‡ Comments (0) °

A sneak peak at a future Python feature

$ python
>>> from __future__ import braces
File “<stdin>”, line 1
SyntaxError: not a chance
>>>

Nice.

Posted 22 September 2009 † Carlo § Uncategorized ‡ Comments (0) °

Big Mac Math

The Economist shows an interesting chart comparing how much working time it takes to buy a Big Mac in different cities around the work. The New York City value looks like it’s about 17 minutes of work per Big Mac? Assuming 8 hour work days, that means New York earns about 28 Big Macs per day per person. If we convert that into calories, we get 14,000 calories at 514 calories a Big Mac. Given that 2,000 calories is your recommended daily caloric intake, we can say that New Yorkers earn enough to feed 7 people per day including themselves, assuming no other expenses. The average household size in the New York metropolitan area is about 2.7 people, so where does that other 4.3 people’s worth of food go? I’m assuming cocaine.

By contrast, it takes people in Nairobi about 160 minutes to earn a Big Mac. That’s only 1500 calories per day per person, not even enough to meet their daily recommended value. Is there no justice in the world? I bet they don’t even have Sweet Tea.

Posted 21 August 2009 † Carlo § Uncategorized ‡ Comments (0) °

Tokyo Tyrant: The magic little database

I’ve been playing around with Tokyo Tyrant master-master replication and I have got to say that it is a magical little database. Here’s some sample code to configure two Tokyo Tyrant instances in master-master replication mode, as well as a script to show off some of Tyrant’s capabilities, including memcached protocol support.

#!/bin/bash
 
HOST=127.0.0.1
PORT=1978
SID=1
UPDATE_LOG_DIR=ulog/$SID
MASTER_HOST=127.0.0.1
MASTER_PORT=1979
 
mkdir -p $UPDATE_LOG_DIR
 
ttserver -host $HOST \
         -port $PORT \
         -ulog $UPDATE_LOG_DIR \
         -sid $SID \
         -mhost $MASTER_HOST \
         -mport $MASTER_PORT
#!/bin/bash
 
HOST=127.0.0.1
PORT=1979
SID=2
UPDATE_LOG_DIR=ulog/$SID
MASTER_HOST=127.0.0.1
MASTER_PORT=1978
 
mkdir -p $UPDATE_LOG_DIR
 
ttserver -host $HOST \
         -port $PORT \
         -ulog $UPDATE_LOG_DIR \
         -sid $SID \
         -mhost $MASTER_HOST \
         -mport $MASTER_PORT
import pytyrant
import time
 
t1 = pytyrant.PyTyrant.open('127.0.0.1', 1978)
t2 = pytyrant.PyTyrant.open('127.0.0.1', 1979)
 
print t2.get('foo', 'nothing here') # 'nothing here'
 
# Writing to t1 replicates to t2
t1['foo'] = 'bar'
print t2.get('foo', 'nothing here') # 'foo'
 
 
# Tokyo Tyrant also speaks the memcached protocol 
 
import memcache
mc = memcache.Client(['127.0.0.1:1978', '127.0.0.1:1979'])
 
print mc.get('foo') # 'bar'
mc.set('yo', 'sup') # Assigns ('yo', 'sup') pair to one of the servers in the pool 
 
# Both Tyrant servers should have ('yo', 'sup') because of master-master replication 
print t1.get('yo', 'nothing in t1') # 'sup'
print t2.get('yo', 'nothing in t2') # 'sup'

For more on Tokyo Tyrant and Tokyo Cabinet, check out this slideshow:

Introduction to Tokyo Products
Posted 15 August 2009 † Carlo § Database § Python ‡ Comments (0) ° Tagged: