Politics and Budgets

(I had a lot of time to think this week while sitting in airports, hence the posting blitz.)

While watching the recent economic turmoils such as the collapse (so to speak) of Greece, and the pending collapse of the state of California, I've come to the conclusion that Democracy is failing.  Now, I'm not much of a political science guy, so I really don't have a better system on the whole.  The really nice thing about Democracies is that they tend to be stable governments (in the sense that they resist violent-revolution) since malcontents and prospective revolutionaries find it easier to work within the system than to overthrow it.  But they seem to be failing the "rational self-interest" test for short-sightedness across the board (ie, "Will I later regret the costs of getting what I want now?").

There is a trade-off here between short-sightedness, and the length of time you are willing to tolerate bad leaders.  Giving elected leaders longer terms will force them to plan farther ahead, since they will suffer more consequences when they are held responsible for the results.  So, to some extent, short-sightedness is the price you pay for avoiding being unhappily misgoverned by a leader you cannot dispose for a while.  But I think there are more basic problems with the control systems that might be fixable without such a trade-off.  (Yeah, I'm a computer engineer, so ideal governments are just fancy control systems with lots of inputs.)

The most basic problem I see, is that there is the lamentable lack of an educated, and self-disciplined populace.  This seems to be indirectly asserted by the prevalence of political advertising.  Everyone who puts up a political sign with "Yes on Blah" and "Vote for Me," seems to be implicitly admitting that they expect a large enough portion of the voting population to be swayed merely by repeated exposure to their signs that it is worth the expense and effort of placing them.  This to me, is the antithesis of a well-educated and self-disciplined populace.  But short of disenfranchising such folk (which severely reduces the stability benefits mentioned above), or pumping lots of money into education (which has been shown to increase both education and discipline, but takes a good 20-40 years to kick in), I have no real solution, so I will limit my whining.

The solvable part of what I think is behind the economic troubles, is that legislatures experience what Frederick Brooks calls the "committee effect."  When you get a group of people together with varying interests, there is a strong incentive to pass anything you do not strongly object to.  To use a game theory explanation, if each committee member has an item he or she personally wants passed, and fighting against another's item will cause them to retaliate by trying to block yours, it is in your best interest to make friends and support everyone's agenda, so that they will support yours.  In engineering, this results in "design by committee," where the feature list is so bloated as to be unrealizable.  The problem here is that while each individual item may be doable, there is an inherent trade-off in constrained resources.  However, the game-theory dynamics of each individual actor in the standard committee promote neither the conservation of those resource, nor any consideration of the trade-offs.  The result is that while each actor succeeds at the committee game by getting their feature on the list, they ultimately lose the real-world game, when the project fails (or random features get cut, if you have a particularly aggressive engineering department).  For legislative committees, the items are often government programs, and the constrained resource is budget dollars for their implementation.  When everyone's programs pass, the government overspends their revenue, and eventually the bottom falls out and someone loses the reality game (usually the voters).

The underlying economic problem seems to be requiring a balanced budget (or at least requiring the budget to fit some fixed figure, determined by the financial policy gurus who may calculate the optimal level of overspending).  There are a few ways to do this, and I think the best solution is a hybrid approach that uses all of them.  First, you could require the legislatures to balance the budget.  This effectively moves the resource constraint problem inside the committee.  Unfortunately, it also destabilizes the equilibrium of the game theory, which quickly results in deadlocks.  The California legislatures inability to pass a budget every year is a prime example of this.  So while I ultimately think that the solution is to move the resource constraints inside the control system, you have to be slightly more clever.

Another way to get around this is to add a second, post-committee control system that intelligently handles the over-spending problem.  If you can excuse the Computer Science reference, this would be a second-pass on the output.  This is akin to the engineering department that intelligently drops what they consider to be the less important features in order to make it fit into the schedule and budget.  You've now separated the decision making from the people who should rightly be making the decisions, and lost some subtleties about the true priorities and rankings in the process, but at least you have the problem solved somewhat intelligently by an interested party.  The improvement here is that otherwise, the real-world could impose constraints randomly, as whatever features fail to be finished on-time are cut, or when the project fails and everyone loses.  In the legislature, the place to vest this power seems to be with the governor.  Any budget which passes the legislature that over-spends the expected revenue, will be trimmed down to size by the governor however he sees fit.

While I think this is an improvement, I see it as falling into one of two possible traps.  If the legislature is consistently unable to pass a controlled budget, then you have just given the governor basically arbitrary control over the budget and program funding (up to the amount the legislature goes over).  To me this seems like a lot of power to stick in one place, even if it's only there as a check when the legislature has "failed".  Alternatively, the legislature could realize that it's in their rational self-interest to balance the budget themselves, and the dynamics reduce to the previous case where the legislature deadlocks.  So this system seems to oscillate between misrepresenting the true priorities (by concentrating the power in one man), or deadlocking.  But at least now we're only deadlocking sometimes.

The third approach is to change the game, so that the priorities of the actors promote balancing the budget more than (or at least appropriately with) their own personal (or constituent if you are less cynical) agendas.  This is kind of tricky, especially given the term-length/responsibility trade-off I've mentioned above.  But, I think that one way to improve this, is to give voters an incentive to care about the state budget, by raising taxes across the board to fund any budget overages.  Especially if this is done on a yearly basis, so when the taxes shoot up, or a new unpopular program gets started, it's fairly easy for an individual voter (or their favorite political media) to figure out why, and calculate their expected return for canceling the program.  Once this is well-understood enough that political candidates can use a rivals spending preferences against him, it should help coalesce the legislative agendas with the true cost to society, while avoiding the long-term overspending trap.  It might even produce better-educated, engaged and disciplined voters, but that's just a pipe-dream of mine...  I don't see a particular downside to this (besides the difficulties of getting it implemented), but it's unclear to me how much of an effect this would have by itself on the voters.  If nothing else, it would reduce the debt-incurring costs of overspending, even if the government itself is not made more efficient with respect to societies' true wishes.

But, I think that you can combine the strengths of all of these into a hybrid system.  First, require tax increases to cover any spending deficits, putting an overall constraint on the system.  Then, allow the governor the power to cut funding at will from any program if an over-spending budget passes the legislature.  To avoid the power issues, give the legislature the power to override the "budget veto" with a super-majority.  Together, this would establish hard-cap on the over-spending from the tax increases (with whatever accompanying changes in voter priorities that come out of it as a secondary benefit), allow intelligent decisions to be made about where to cut over-committed resources during legislative failure, move the constraining problem inside the committee deliberations (with the "out" of passing an over-spending budget to avoid deadlocks), and not concentrate too much power in the governor by allowing to a legislative override.

I realize, the current economic situation is much more complicated than government overspending, but this seems to be a recurring theme in modern democracies.  But, I'm approaching this from a control systems angle with a little bit of game theory thrown in for politics.  Anyone have a better idea?

Fun with Fedora

I don't want to be the guy who rants all the time, because life's too short to spend it complaining.  But I had such a negative experience that I'm going to share it anyway.  Thumper's Mother can scold me later...

I had to test some things for work on the new shiny Fedora 14 release, and decided that it was the most brain-dead and hard to setup Linux distribution I've ever worked with.  (After some reflection, the only other serious contender was Topologilinux, who's built-in upgrade system used to leave the installation unbootable.  But that was back in 2002, and since they're essentially some fancy installation scripts around Slackware, I'll cut them some slack (no pun intended), since I don't expect them to maintain the entire Slackware repository.)    All I wanted to do was get VMware tools installed properly, and to do that I needed to install gcc and some kernel headers, which sounded simple enough.  I'm willing to overlook (what I perceive as) the grave crimes of any distro that doesn't ship with gcc and the kernel headers included, because I realize that not everyone uses systems the same way I do.

Now, granted, I've never worked with Fedora or RedHat before, and Gentoo's package management system has pretty much spoiled me for life, so I was prepared for a bit of a learning curve to get everything setup properly.  The first task was networking.  I'm behind the firewall-proxy setup at work, so I didn't expect networking to work out of the box.  I'm even willing to overlook the fact that there is no happy graphical utility to set a system-wide proxy configuration.  Nobody else seems to do this either, despite the fact that it would just have to dump two environment variables to a config file.  But, on a modern Linux distribution, and especially on standard hardware environments like a VMware virtual machine, I expect my network devices to come up under DHCP without any user intervention.  So I wasted several minutes trying to play with proxy settings, before I realized that I had the more fundamental problem: eth0 does not start by default.

The next thing I expect from a software system, is reasonably descriptive error messages. If there's a network problem, I expect to see something to the effect of "Unable to connect to fedora.com," or even "Unable to download package database" or even "Plug in your network cable you moron."  But, the message yum gave me was something like "Error: Cannot retrieve repository metadata (repoman.xml)," which just doesn't help me at all.  Now, maybe they tried, since the word "retrieve" is in there, which sounds vaguely networky, but it's not terribly descriptive.  It sounds like the kind of generic error message you get when the programmers were either too lazy to implement proper error handling, or the errors are obscure enough that the programmers could not have reasonably predicted them.  So I went digging around with yum, trying to rebuild package databases, and running obscure maintenance-looking commands, trying to figure out what could possibly be weirdly screwed up on my stock install.  After fixing my proxy settings (aka, copying the environment variables so many places, yum has to get it right), I was confronted with a different error message.  Now yum was listing websites and telling me it was failing to connect with them.  Somehow, after giving yum access to the Internet, it was finally giving me an error message that looked definitively network related.... go figure.  Clearly something else was wrong.  I managed to convince myself that the sites in question really did exist (at this point I was praying they weren't dumb enough to ship with dead mirrors), and that I really could connect to them through the proxy and the firewall at work.

After longer than I'd like to admit, and double and triple checking the network connectivity and proxy settings, I gave up and started digging around on Google.  I finally found some other poor saps looking at the same error message who had managed to find a fix.  Apparently, and for reasons I cannot adequately explain, yum is unable to access HTTPS sites out of the box.  This is, as you might imagine, a severe limitation, when all of the default mirrors include https:// in the URL.  So, I went and hand-edited my repo list, and converted all of the https:// to http://, and prayed that nobody was running a https only mirror.  That finally worked for me, and I was able to start the onerous process of matching up real-world package names like "gcc" and "kernel headers" to the cryptic, numbered formulas that seem to be the best non-Gentoo package managers can offer.  I'm really hoping that this was an issue with the Squid proxy at my work, and that the Fedora folks didn't ship a release that was unable to validate SSL certificates.  For that matter, I'm not completely sure what added security running over HTTPS really gives you.  I'm hoping they checksum the binaries, so the only advantage I see to HTTPS is to make it so that people upstream from me can't tell exactly which packages I'm downloading...

Meanwhile, I had another problem: yum was deadlocking itself.  Apparently, yum will happily spin for over 10 minutes, trying to acquire some internal yummy lock.  Furthermore, Unix file-locking being what it is, killing the competing processes doesn't release the lock.  So, I had to go Google around for the lock files yum uses, so I can kill all the yums on my system, free the locks, and try again.  (I had by this point accumulated quite a number of hung yums.)  Somehow, this was unsuccessful.  So I logged out, and restarted my X session.  No luck.  I rebooted.  No luck (after reworking the solutions for eth0, and the proxy above).  I finally figured out that whatever system-tray icon their default session launches to inform me of all the wonderful updates I'm missing was also running yum, and hanging in some new and interesting way.  I never solved that one.  But, if I killed it, and all my hung yums, and cleaned all the lock files, finally I could install packages.  However, mistyping a shell command would cause my shell to hang indefinitely... I can only assume it was asking yum what I meant to type, so it could offer to install it for me, but that yum was still horribly broken in some unusual way.

This, is the Linux distribution that gets all the money?  I fear for our novices...

Fun fact of the day: Ping

Pressing "Ctrl+\" while running ping will display the current statistics without quitting.

You may now return to your regularly scheduled programming...

The Design of Design

I recently finished reading The Design of Design: Essays from a Computer Scientist by Frederick P. Brooks, (better known for his classic text The Mythical Man-Month: Essays on Software Engineering).  I would highly recommend it for any computer science students, and even any other technical discipline that deals with design work.  While most of what he says about design isn't anything particularly new or sensational, he manages to cover the breadth of design wisdom and put it all into a coherent framework, with lot's of examples and stories to back it up.  It's also a fairly quick read.  My only complaint is that about half the book is an account of several iterations of him designing his family beach house.  That was boring enough that I skimmed or skipped most of it.  He used it mainly to illustrated some of his design principles in a way that was accessible to a general audience, and I'm sure it would be more useful to people who are hearing some of his design philosophy for the first time.  More useful, was his analysis of some of the early IBM computer architectures that he worked on, and some discussion on the constraints and trade-offs involved.

Another chunk of the book was devoted to discussing the design process as it interacts with an institutional framework such as a business.  Having recently started a full-time job as a computer programmer, I found this to be the most interesting part of the book.  He balanced theory with examples and lots of practical advice, which is fairly rare among most of the people I've seen try and cover the topic.

Of uids and shell scripts ...

So, I've got this extensive collection of shell scripts that I've written sitting in ~/bin, that I use all time.  In fact, I like them so much, that I wanted to be able to call them while  I was root.  But that's a huge security hole, because it means that anyone who has become me and can edit my scripts, now has a pretty good chance of me running them as root.  Granted, if someone actually owns my user account, I already have some serious problems, because in all reality I care about my data much more than I care about "root access" by itself.  But, seeing as I wrote all of these scripts myself, there's a reasonable chance that there's some egregious bug in them that I don't want to run as root anyway.

I used to just put my user scripts dir in my path as root, figuring that I'm the only one who's ever me, and I usually have physical security of my box.  But as I started copying my scripts to more and more computers (including my linode, which I don't  have physical control of), this started making me nervous.   So eventually I solved this problem by making two copies of my scripts, one to sit in /root and one to sit in ~/bin.  This worked great for security, but now I had to maintain and sync two copies of this stuff, and bug fixes weren't propagating quickly.  But I didn't have a better way to do things.


I was talking to one of my friends yesterday, who told me that he uses "su -c" for that problem.  That let's you run a command as another user, so you can drop root privileges.  If the syntax was a little different, this is almost exactly what I wanted, but unfortunately I play enough games with arguments, that I wanted to be able to pass ./script 1 2 3 'a b c' correctly, and I couldn't figure out how to do that on a shell level.  I also don't want to worry about sanitizing arguments with spaces or escape sequences or anything.

So I finally convinced myself that the right way to do this was in C (after a failed attempt to find perl commands that would let me do this).  It actually turned out to be way easier than I expected.  Essentially I can just call setuid, pull off the command line arguments, and pass them to exec, and I have a script which runs a command as the specified user.  I ended up pulling some library routines out of the "su" utility to do some environment sanitization , but other than that the code is a few system calls and some error checking.  If I ever clean it up, and make sure I've preserved all the copyright notices I'll stick it up here.

But here's the gist of it (error checking removed):
Called as "suwrap user command args"

int main(int argc, char ** argv)
{     
    int x;
    uid_t uid;
    char **args;
    passwd *p;
    int argsize; 
     
     
    p = getpwnam(argv[1]);
     
    addenv("HOME", p->pw_dir);
    addenv("USER", p->pw_name);
    addenv("LOGNAME", p->pw_name);
    
    uid = p->pw_uid;
    
    setuid(uid); 
         
    argsize = argc-COMMAND_INDEX+1;
    args = malloc(argsize*sizeof(char*)); 
     
    for(x=COMMAND_INDEX;x<argc;x++) { 
        args[x-COMMAND_INDEX] = argv[x]; 
    }
         
    args[argsize-1] = NULL; 
    
    execvpe(argv[COMMAND_INDEX], args, newenvp); 
     
    //No one will ever see this... 
    printf("\nDone!\n");
    return 0;
} 

I don't know if I should feel proud or disturbed that I've progressed from writing shell scripts to C code, but I definitely feel that I have crossed a Unix threshold here...

My poor neglected Blog...

Ok, the Amazon context ads for my blog are showing "Eric Clapton" and books on massage.  Clearly I need to add some more geeky posts.  I'll see what I can do.  (Of course posting this message will only convince the AI that it was right...)

Book Review: Priests and Programmers

I recently finished reading Priests and Programmers by J. Stephen Lansing, a book on irrigation practices and native religion in Bali.  To be honest, I found this book in the $1 used book bin at a textbook store outside UC Irvine, and I thought they were talking about computer programmers.  So I bought it expecting to find some Orwellian tale of a totalitarian government using religion and technology to blind it's populace.  Turns out they mean "programmers" as in "people who make up schedules," in this case, the priests setting the irrigation schedules.  Once I started reading it though, it was so interesting I kept going.

Apparently, the island of Bali (located in Indonesia) has an amazing irrigation system that has to be carefully maintained in order for anyone to grow any rice.  During the rainy season, there's water enough for everybody, but during the dry season, the water usage has to be regulated so that all of the farmers get enough water for their rice.  This gets slightly tricky since rice needs lots of water at the beginning of it's growth cycle, and very little at the end, so if you stagger the planting of the various farmers in the region, you can minimize the water shortages and maximize the rice.  The real trouble comes from the rice pests, which hop from field to field as long as somebody nearby is growing rice.  So to control the pest population you want to coordinate nearby fields so that they all lie fallow at the same time and the bugs and things die.  Striking the correct balance between synchronizing and staggering the fields is a complicated process that has to be tuned to the current conditions like rainfall and local weather.

In Bali the farmers attach mystical significance to the river and irrigation waters as being the property of the various gods and goddesses that inhabit the island.  As such, all of the water usage is coordinated by priests that inhabit temples built at the source of each canal, river, and lake.  These are called water temples, and serve to coordinate the water usage and planting cycles of all of the farms that belong to a particular watershed.  Lansing also argues that the religious rituals of the Bali people that involve the mingling of holy water from the various springs and lakes attached to the temple, symbolize the reliance of all these separate farmers and peoples on each other, and create a sense of group consciousness of the responsibility and social implications associated with water use.

Besides offering detailed accounts of the relevant religious ceremonies and an analysis of a computer model demonstrating the impact of water temple coordination with pest cycles, Lansing also tells the history of the Bali island and its water temples.  The Dutch arrived and eventually conquered the island in the 1850's or so, and tried to manage the local populace and the rice production.  However, they failed to notice the role of the water temples in coordinating the farming activities, because they tended to discount anything related to the local religions.  So the system kept functioning undisturbed until the "Green Revolution" hit Bali in the 1970's and 1980's.  Bali was now being run by Western-educated Indonesians, who didn't understand the roles of the water temples, and started disrupting the traditional farming practices by introducing new high-yield crops and farming technologies.  This quickly resulted in disaster as pest populations swarmed out of control, and devastated rice yields.  Farmers who resisted the transformation were written off as religious conservatives who were fighting change.  (Granted, it's not clear to me that the farmers or the priests completely understood their role in regulating the ecology of the rice fields, so I don't think you can completely write off the Western influence as being unduly dismissive of the traditional practices.)  At the time of the writing, the government of Bali was starting to study and understand the role the water temples play in managing rice production, and is beginning to allow them greater control in regulating the rice production.

All in all it was a good read.  There was a little bit of social-science style "Marxian Analysis" at the beginning and the end, but the rest of it was just anthropological study, history, and ecology.  I think it's definitely worth the few hours it takes to read it.  (It's not quite 200 pages altogether).