Linux Journal | December 2011 | Issue 212 - ACM Digital Library

Learn Drupal & jQueryFROM THE COMFORT OFYOUR LIVING ROOMThe Lullabot Learning Series includes everything you need to become aDrupal & jQuery expert from the comfort of your living room! The videosare available in both DVD format and high-definition video download.Purchase the videos at http://store.lullabot.com

CONTENTSDECEMBER 2011ISSUE 212FEATURE50 Readers’ Choice Awards 2011See how your favorites align with other readers.Shawn Powers50ON THE COVER• OpenRISC: an Introduction, p. 98• Use Mercurial for Version Control, p. 94• Readers' Choice Awards 2011, p. 50• How to: Read Linux Journal on the Command Line, p. 36• Scale Web Apps with Amazon's Simple Queue Service, p. 24• Best Practices for Designing Monitoring Systems, p. 68• EFI Features and How They Impact Linux, p. 86• Fix Broken Protocols on the Fly with Netfilter, p. 106 COVER IMAGE: © Can Stock Photo Inc. / beholdereye4 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

COLUMNS24 Reuven M. Lerner’s At the ForgeMessage Queues32 Dave Taylor’s Work the ShellPlaying with Twitter Stats36 Kyle Rankin’s Hack and /Read Linux Journal from theCommand Line114 Kyle Rankin and Bill Childers’Tales from the Server RoomZoning Out118 Doc Searls’ EOFReality Fidelity Field36 READ LINUX JOURNAL ON THE COMMAND LINEIN EVERY ISSUE8 Current_Issue.tar.gz10 Letters14 UPFRONT40 New Products44 New Projects119 Advertisers IndexINDEPTH68 Complexity, Uptime and the Endof the WorldTake a look at how to build robustmonitoring scripts in your data centerand in the cloud.Michael Nugent76 MariaDB/MySQL, PostgreSQL andSQLite3: Comparing Command-LineInterfacesInteracting with databases on thecommand line.Daniel Bartholomew86 Using Linux with EFIA look at the overall features andprinciples of EFI, and why you mightwant to use it.Roderick W. Smith94 Mercurial—Revision ControlApproximatedMercurial provides some of the featuresof systems like Git and some of thefeatures of systems like CVS or Subversion.Joey Bernard98 The OpenRISC Processor:Open Hardware and LinuxOpen-source hardware is now readyfor prime time.James Tandon106 Fixing Broken Protocolswith NF_QUEUEMangling packets for fun and profit.Paul AmaranthLINUX JOURNAL (ISSN 1075-3583) is published monthly by Belltown Media, Inc., 2121 Sage Road, Ste. 310, Houston, TX 77056 USA. Subscription rate is $29.50/year. Subscriptions start with the next issue.WWW.LINUXJOURNAL.COM / DECEMBER 2011 / 5

Subscribe toLinux JournalDigital Editionfor only$2.45 an issue.Executive EditorSenior EditorAssociate EditorArt DirectorProducts EditorEditor EmeritusTechnical EditorSenior ColumnistSecurity EditorHack EditorVirtual EditorJill Franklinjill@linuxjournal.comDoc Searlsdoc@linuxjournal.comShawn Powersshawn@linuxjournal.comGarrick Antikajiangarrick@linuxjournal.comJames Graynewproducts@linuxjournal.comDon Martidmarti@linuxjournal.comMichael Baxtermab@cruzio.comReuven Lernerreuven@lerner.co.ilMick Bauermick@visi.comKyle Rankinlj@greenfly.netBill Childersbill.childers@linuxjournal.comContributing EditorsIbrahim Haddad • Robert Love • Zack Brown • Dave Phillips • Marco Fioretti • Ludovic MarcottePaul Barry • Paul McKenney • Dave Taylor • Dirk Elmendorf • Justin RyanProofreaderGeri GaleENJOY:Timely deliveryOff-line readingPublisherAdvertising Sales ManagerAssociate PublisherWebmistressAccountantCarlie Fairchildpublisher@linuxjournal.comRebecca Cassityrebecca@linuxjournal.comMark Irgangmark@linuxjournal.comKatherine Druckmanwebmistress@linuxjournal.comCandy Beauchampacct@linuxjournal.comEasy navigationPhrase searchand highlightingAbility to save, clipand share articlesEmbedded videosAndroid & iOS apps,desktop ande-Reader versionsLinux Journal is published by, and is a registered trade name of,Belltown Media, Inc.PO Box 980985, Houston, TX 77098 USAEditorial Advisory PanelBrad Abram Baillio • Nick Baronian • Hari Boukis • Steve CaseKalyana Krishna Chadalavada • Brian Conner • Caleb S. Cullen • Keir DavisMichael Eager • Nick Faltys • Dennis Franklin Frey • Alicia GibbVictor Gregorio • Philip Jacob • Jay Kruizenga • David A. LaneSteve Marquez • Dave McAllister • Carson McDonald • Craig OdaJeffrey D. Parent • Charnell Pugsley • Thomas Quinlan • Mike RobertsKristin Shoemaker • Chris D. Stark • Patrick Swartz • James WalkerAdvertisingE-MAIL: ads@linuxjournal.comURL: www.linuxjournal.com/advertisingPHONE: +1 713-344-1956 ext. 2SubscriptionsE-MAIL: subs@linuxjournal.comURL: www.linuxjournal.com/subscribeMAIL: PO Box 16476, North Hollywood, CA 91615-9911 USALINUX is a registered trademark of Linus Torvalds.SUBSCRIBE TODAY!

2U Appliance:You Are the CloudExpansionShelvesAvailableStorage. Speed. Stability.With a rock-solid FreeBSD® base, Zettabyte File System (ZFS)support, and a powerful Web GUI, TrueNAS pairs easy-to-manageFreeNAS software with world-class hardware and support foran unbeatable storage solution. In order to achieve maximumperformance, the TrueNAS 2U System, equipped with the Intel®Xeon® Processor 5600 Series, supports Fusion-io’s Flash MemoryCards and 10 GbE Network Cards. Titan TrueNAS 2U Appliancesare an excellent storage solution for video streaming, file hosting,virtualization, and more. Paired with optional JBOD expansionunits, the TrueNAS System offers excellent capacity at anaffordable price.For more information on the TrueNAS 2U System, or to requesta quote, visit: http://www.iXsystems.com/TrueNAS.CloneSnapshotAll VolumesKeY feATUreS:. Supports One or Two Quad-Core or Six-Core, Intel® Xeon® Processor 5600 Series. 12 Hot-Swap Drive Bays - Up to 36TB ofData Storage Capacity*. Periodic Snapshots Feature Allows You toRestore Data from a Previously GeneratedSnapshot. Remote Replication Allows You to Copy aSnapshot to an Offsite Server, forMaximum Data Security. Software RAID-Z with up to Triple Parity. 2 x 1GbE Network interface (Onboard) +Up to 4 Additional 1GbE Ports or Single/Dual Port 10 GbE Network CardsJBOD expansion is available on the2U System* 2.5” drive options available; pleaseconsult with your Account ManagerCreate Periodic SnapshotCall iXsystems toll free or visit our website today!1-855-GREP-4-IX | www.iXsystems.comIntel, the Intel logo, Xeon, and Xeon Inside are trademarks or registered trademarks of Intel Corporation in the U.S. and/or other countries.

Current_Issue.tar.gzSHAWN POWERSThe CustomerIs Always RightThis is the time of year when theLinux Journal staff turns to you,our readers, for insight on the bestprograms in the Linux world. I love thistime of year. No, not because you all domost of the work, but rather because Iget to see how my preferences compareto those of our readership. You get todo the same. Whether you’re lookingfor validation with your software choicesor hoping to fill a gap in your digitalrepertoire, this issue should please.Along with the Readers’ Choice winners,we have an issue full of “choice” articleswe’ve picked to go along with this month’stheme. Reuven M. Lerner shows us aneasy way to scale Web applications withAmazon’s Simple Queue Service (SQS).Amazon makes scaling services simple, andWeb applications are no exception. DaveTaylor describes how to make a scale ofour own for rating Twitter accounts. Usingscripting (Dave’s specialty), extracting dataabout a Twitter account is pretty simple.Come up with your own formulas for whatmakes tweets terrific, and you can make ascript that is judge and jury all in one.Our other command-line guru, KyleRankin, teaches us to laugh in the face ofE Ink and scoff at the Kindles of Amazon.In the same way Kyle chats with Irssi,e-mails with Mutt and system-administersfrom an xterm, this month he shows howto read Linux Journal with his e-readerof choice: a terminal window. If you’re aminimalist like Kyle or just like to out-geekthe person next to you, you’ll want to readKyle’s article. At the very least, it will makeyou thankful for your digital e-reader!Michael Nugent addresses a problem thismonth that is near and dear to me. Everysysadmin should have a monitoring system,but what happens when that monitoringsystem is more annoying than helpful? Iget daily e-mail messages from several ofmy systems with reports on their success orfailure. After 20–30 days of “all normal”,the messages tend to slip past my radar.Then one day when they stop arriving, theirabsence goes unnoticed. The opposite can8 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

CURRENT_ISSUE.TAR.GZbe true though as well. How may timeshave you been woken up by your pagerbeeping incessantly over a false positive? At3 o’clock in the morning? Michael discussessome best practices for making yourmonitoring system effective at doing its jobwhile not driving you insane in the process.If you’re a software developer, you willwant to check out Daniel Bartholomew’sarticle on databases. Sure, databases aren’tthe most exciting things in the world, but ifyou’re a programmer, interfacing with themis important. Add to that Joey Bernard’sarticle on Mercurial for revision control, andit’s like soup for the programmer’s soul.We realize not everyone is intoprogramming though, and for you hardwarehackers, we have a couple exciting articlesas well. James Tandon shows us the opensourceprocessor OpenRISC and teaches ussome tricks for utilizing it. As a communitythat historically has struggled with workingwith proprietary hardware, the open-sourcehardware idea is very attractive. RoderickW. Smith helps us stay ahead of thehardware transition game this month too.He describes the new EFI boot mechanismthat is slowly taking over the role of BIOS incomputers. Since hardware manufacturerswill be moving more and more toward EFI,it’s important for us to understand. Afterall, “booting up” is a pretty important partof any operating system—even if it is onlyonce every few years for Linux users.Networking folks haven’t been left outthis month either. Paul Amaranth showsus a pretty neat method of fixing brokenNAT protocols using NF_QUEUE. NATworks so well anymore, most of us takeit for granted. Sometimes it doesn’t workas magically as we expect, however, andPaul shows us how to do some magic ofour own. Bill Childers and Kyle Rankinclose off the issue with a scary, buteducational, story about wiping out theirdata center—over and over. It’s scary thedamage we can do accidentally when wework on production servers. Bill and Kyleare two people I turn to when I have issuesI can’t solve, and as you’ll read this month,they’ve learned much of their knowledgeat the school of hard knocks.We’d like to thank you, our readers, formaking this issue fun for us. It’s great tohear from you regarding what softwareand hardware you prefer. It not only helpsus produce a magazine that will fit yourneeds, it also gives us a chance to learnfrom you. So sit back with your Kindle,prop up an iPad or flip through digitalpages with your Android. This month,you get to see how you line up with otherLinux Journal readers. We hope you enjoy.■Shawn Powers is the Associate Editor for Linux Journal. He’s alsothe Gadget Guy for LinuxJournal.com, and he has an interestingcollection of vintage Garfield coffee mugs. Don’t let his silly hairdofool you, he’s a pretty ordinary guy and can be reached via e-mailat shawn@linuxjournal.com. Or, swing by the #linuxjournal IRCchannel on Freenode.net.WWW.LINUXJOURNAL.COM / DECEMBER 2011 / 9

lettersHome RouterHackingI thoroughlyenjoyed KyleRankin’sarticle“PracticeHacking onYour HomeRouter” in theOctober 2011issue. Usuallysystem cracking gets a bit deep for me,but this example was both practical andeasy to follow—nice! One minor nit: dataencoded into a URL is actually GET datarather than POST data. (I’ll give him a passthis time on “hacking” vs. “cracking”.)—Grant Rootissue]. My own experience is with anunder-powered Acer Aspire One Netbook.One note, to further improve disk IO timeand SSD wear, I load all partitions withnoatime and nodiratime options in fstab.Acer Aspire 532h Fedora 14 i686:OCZ 60GB:n Cold boot to GNOME login: 27 secondsn Log in to usable desktop: 2 secondsn Open Firefox to cursor at URL bar:2 secondsn Open LibreOffice Write to blank doc:2 secondsKyle Rankin replies: Hey, precision isimportant, so thanks for the nitpick! Youare right. The way I was submitting datato the router was GET data. GET or POST,the lesson is that if you accept input,sanitize it. If you are curious about whyI use “hacker”, I explain it in detail in myNovember 2010 column “Some Hacksfrom DEF CON”.Return to Solid StateI enjoyed getting a little comparative data onSSD performance [see Kyle Rankin’s “Returnto Solid State” review in the October 2011Seagate ST9250827AS:n Cold boot to GNOME login: 47 secondsn Log in to usable desktop: 16 secondsn Open Firefox to cursor at URL bar: 5secondsn Open LibreOffice Write to blank doc: 7secondsThanks.—rathomas10 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

[ LETTERS ]Kyle Rankin replies: Thanks forthe feedback—a good point aboutatime. I can understand that it makessense to disable atime completely onsome filesystems for speed (such asmountpoints dedicated to databasestorage), but for me, I find such benefitin atime for forensics that if speed is aconcern, I like to use the new relatimeoption in the fstab. That way, I still getatime; however, atime writes are cachedso it gives much better performance.The Digital SubscriptionI want to thank you for converting toall-digital download for Linux Journal. Isee that in the past the magazine keptshrinking slowly over time, and now thatyou have gone 100% digital, it has grownin size! I think that is a plus for you. Atfirst, I was worried about whether I wouldlike the digital version, but after twomonths, I find I have begun to like it. Keepup the good work and the great articles!—Jim BrownYou bring up a point I hadn’t evenconsidered in regards to the “thickness”of the magazine. We can be a little moreflexible now than we could be with paper.Thanks for the kudos as well. In all thechaos involved with the digital transition,the one thing we wanted to maintain wascontent.—Ed.

[ LETTERS ]Digital Format SurpriseI’m a longtime reader, first-timefeedbacker. I just finished reading my firstdigital copy of Linux Journal, and althoughI thought I would hate the experiencecompared to reading in dead-tree format,I was pleasantly surprised. I spend all daystaring at LCD pixels, and I rather dislikedthe idea of including my casual readinghere. With the paper version of the journal,I often would read about somethingcool and say to myself, “Neat project Ishould check this out next time I sit at mycomputer.” Only I would never rememberto remember to check it out. I love theHTML links built in to the digital versionof the journal. It is so much more of aninteractive experience. And that got methinking, take one more step by creatinga wiki-like version of the journal with usercomments that show up next to articles,videos and so on—just a thought. Keep upthe good work.—DanCool idea Dan. As we learn to do thiswhole digital thing better and better everymonth, it will make new ideas feasible.Now that the entire production processis in-house (no more sending off to theprinter), it makes experiments mucheasier as well. I totally agree regardinghyperlinks too. In the past, I’ve donemy best to include short URLs in printedarticles, but now we can use the real linksand not worry about someone typingthem in by hand. My favorite thing aboutdigital though? Searching.—Ed.epubI tried out the epub version of theSeptember Linux Journal on my NookColor and enjoyed reading it on thatdevice—until I came to the graphics. Itseems that images, such as a Bash screenin an article, are too small on the Nookto be read. They don’t magnify either.I tried turning the Nook on its side forlandscape view, which it will do withbooks and Web pages, but the pageswouldn’t landscape. Yuck!So, please study the epub version for away to fix landscape or magnify images(perhaps embed a clickable biggerpicture). Perhaps the problem is reallywith the Nook and epubs. In general,the Nook has problems with PDFsalso. Usually they will magnify, but thelandscape mode tends to be busted oruseless with it still in a smaller portraitmode while turned 90 degrees.I guess for now I won’t be able to read LJon my Nook with much success. I’ll readit on my computer instead.—DaveYou are absolutely correct. Althoughwe’ve been doing magazine layoutforever, this whole “flowing text” thingis new for us. Trying to make epub filesthat work well on all devices is reallychallenging. We are working hard every12 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

month to make the experience a little better, andI think we’ve succeeded a little bit each month.Another challenge is example code. Because devicesare so varied in size, it’s rough to make the code“perfect” on each reader.Like I said, we’re working on honing our skills, soyou should see improvement every month. Sending incomments like this really helps us determine what tofocus on, so thank you again!—Ed.Please Write about epub ToolsThis month, I received Linux Journal in pure electronicform. I got it in PDF and mobi formats, and I can readit directly on my Kindle. I realized that since LinuxJournal decided to deliver its content in e-formats,maybe it can write about it. For instance, do anoverview of available software tools, techniques fortext formatting (especially code formatting suitablefor gadgets), tools for transforming from one formatto another and so on. I think it would be great topicfor readers.—ValentinI agree! I want to see some articles on epub creationmyself, so hopefully we can find some experts on thetopic. I know there are many conversion tools outthere, but what I’m looking for (and I think you’relooking for) are tools to create them or edit them.The epub format offers so many cool features liketable of contents, chaptering, graphic placement andso on, and I want to learn how to make or tweak myown. Thanks for sharing my interest.—Ed.WRITE LJ A LETTER We love hearing from our readers. Please send usyour comments and feedback via http://www.linuxjournal.com/contact.At Your ServiceSUBSCRIPTIONS: Linux Journal is availablein a variety of digital formats, including PDF,.epub, .mobi and an on-line digital edition,as well as apps for iOS and Android devices.Renewing your subscription, changing youre-mail address for issue delivery, paying yourinvoice, viewing your account details or othersubscription inquiries can be done instantlyon-line: http://www.linuxjournal.com/subs.E-mail us at subs@linuxjournal.com or reachus via postal mail at Linux Journal, PO Box16476, North Hollywood, CA 91615-9911 USA.Please remember to include your completename and address when contacting us.ACCESSING THE DIGITAL ARCHIVE:Your monthly download notificationswill have links to the various formatsand to the digital archive. To access thedigital archive at any time, log in athttp://www.linuxjournal.com/digital.LETTERS TO THE EDITOR: We welcome yourletters and encourage you to submit themat http://www.linuxjournal.com/contact ormail them to Linux Journal, PO Box 980985,Houston, TX 77098 USA. Letters may beedited for space and clarity.WRITING FOR US: We always are lookingfor contributed articles, tutorials andreal-world stories for the magazine.An author’s guide, a list of topics anddue dates can be found on-line:http://www.linuxjournal.com/author.FREE e-NEWSLETTERS: Linux Journaleditors publish newsletters on botha weekly and monthly basis. Receivelate-breaking news, technical tips andtricks, an inside look at upcoming issuesand links to in-depth stories featured onhttp://www.linuxjournal.com. Subscribefor free today: http://www.linuxjournal.com/enewsletters.ADVERTISING: Linux Journal is a greatresource for readers and advertisers alike.Request a media kit, view our currenteditorial calendar and advertising due dates,or learn more about other advertisingand marketing opportunities by visitingus on-line: http://ww.linuxjournal.com/advertising. Contact us directly for furtherinformation: ads@linuxjournal.com or+1 713-344-1956 ext. 2.WWW.LINUXJOURNAL.COM / DECEMBER 2011 / 13

UPFRONTNEWS + FUNdiff -uWHAT’S NEW IN KERNEL DEVELOPMENTYou may have heard of the recent securitybreach that took place on kernel.org.The attacker gained root access to theservers and modified a kernel source-treerelease candidate, in the hope of infectinglots of users.Since then, the kernel.org systemadministrators have been workinglike mad, cleaning out the servers andimplementing security measures that mighthopefully prevent another attack. Onesuch measure involves restricting accessto kernel.org itself. In the past, peoplemaintaining a git tree on kernel.org couldget a shell account on that system. Thosedays are gone. H. Peter Anvin announcedthat the gitolite tool would be used toupdate git trees from now on, and shellaccess would no longer be handed out asfreely as it was.The kernel.org folks also are institutinga cryptographic “web of trust”, so thatpeople maintaining a git tree will be ableto establish their identity when doingupdates. If you’re a developer who hacksthe kernel in your spare time or for youremployer and typically submits patches viae-mail, you won’t need to be part of theweb of trust; in fact, your work flow cancontinue unchanged. Only folks involvedin maintaining projects on kernel.org areaffected by these new policies.Linus Torvalds has expressed somedoubt that cryptographic signatures areas important as others believe. He said,“Realistically, I checked a few signaturesthis time around due to the kernel.orgissues, but at the same time, the thing thatmade me trust most of it was just lookingat commits and the e-mail messages—the unconscious and non-cryptographic’signature’ of a person acting like youexpect a person to act.”Andi Kleen has resubmitted his patchthat makes 3.0 kernels pretend to be2.6 kernels, so binary-only softwareexpecting to run on a 2.6 kernel still willrun correctly under 3.0 kernels. It’s anugly pill to swallow. This time around,Linus Torvalds asked which binary-onlysoftware actually was breaking under3.0, and a number of people replied,listing off several applications. SomeHP management tools were amongthem. There also were a lot of nonbinaryapplications, including a number of Pythonscripts that performed an incorrect test forthe current kernel version number.Linus seems very reluctant to adopt thispatch, especially considering that Andi14 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

[ UPFRONT ]has stated positively it’s not just a shorttermfix, but that 3.0 kernels would haveto continue to masquerade as 2.6 kernelsfor the long term, in order to maintaincompatibility with those binary-only tools.—ZACK BROWNGoodbye GNOME 2, Hello GNOME 2?Many Linux users who havebeen GNOME fans for yearsfind themselves in a suddenquandary. GNOME 3.0 hascompletely abandoned thedesktop experience we’vecome to love during theyears. That’s not to saychange is bad, it’s just thatmany folks (even LinusTorvalds) don’t really wantto change.As an Ubuntu user for several years,I’m accustomed to how well Canonicalmakes Linux on the desktop “just work”.Unfortunately, Ubuntu’s alternative to theGNOME 3 switch is Unity. I want to likeUnity. I’ve forced myself to use it to see if itmight grow on me after a while. It hasn’t.And, to make matters worse, version 11.10won’t have a classic GNOME option, whichmeans I either need to bite the bullet andget used to Unity or go with an alternative.Thankfully, XFCE has all the featuresI love about GNOME. No, XFCE isn’texactly like GNOME, butit feels more like GNOME2 than GNOME 3 does!If you are like me anddesperately want to havethe old GNOME interfaceyou know and love, Irecommend checking outXubuntu (the version ofUbuntu that uses XFCE).With minimal tweaking,it can look and feel likeGNOME 2. Plus, XFCE has the ability tostart GNOME (or KDE) services on login,which means GNOME-native apps usually“just work”.The time may come when we’re forcedto adopt a new desktop model. For thetime being, however, alternatives likeXFCE or even LDXE offer familiar andhighly functional desktop experiences.If you fear GNOME 3 and Unity, tryXFCE. Download Xubuntu and check itout: http://www.xubuntu.org.—SHAWN POWERSWWW.LINUXJOURNAL.COM / DECEMBER 2011 / 15

[ UPFRONT ]Get More from Youre-Reader: InstapaperIf you use adedicatede-reader to readLinux Journalevery month,chances are youwant to readother material onit as well. Thanksto a free servicecalled Instapaper,if you have ane-reader like theLinux-poweredKindle, you can take yourfavorite Web articles withyou on the go, even ifyour destination doesn’thave Internet access!Instapaper works bygiving you a bookmark toclick anytime you want toview a Web page later onyour e-reader. Then, fromyour Internet-enabledmobile device, you canread the articles at yourleisure. For those folkswith a Kindle, Instapaperprovides a free deliveryservice that sends batchesof articles to your Amazonaccount. It’s importantto realize Amazon willcharge you for 3G-baseddeliveries, but if yourKindle supports Wi-Fi,delivery is free as well.Instapaper is great ifyou don’t like to do thebulk of your reading ina Web browser, but stillwant to read the latestnews from the Internet.The service is currentlyfree, but you canbecome a subscriberfor $1 a month andsupport the company.Instapaper is available athttp://www.instapaper.com.—SHAWN POWERSThey Said ItIf I’m not back in fiveminutes...just waitlonger.—Ace Ventura,Ace Ventura: Pet DetectiveThis is space. It’ssometimes called thefinal frontier. (Exceptthat of course youcan’t have a finalfrontier, becausethere’d be nothing forit to be a frontier to,but as frontiers go, it’spretty penultimate...)—Terry PratchettI can picture in my minda world without war,a world without hate.And I can picture usattacking that world,because they’d neverexpect it.—“Deep Thoughts”with Jack HandeyFantasy is the impossiblemade probable. Sciencefiction is the improbablemade possible.—RodSterling, The Twilight ZoneThere’s that word again,“heavy”. Why are thingsso heavy in the future?Is there a problem withthe earth’s gravitationalpull?—Emmet Brown, Back ToThe Future16 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

[ UPFRONT ]good programs exist thatalready do that very well.You can use avogadroto output files that canbe used by these otherprograms as input filesto do such higher-levelcomputations. Under theExtensions menu item, youwill find entries to buildinput files for GAMESS,Gaussian, MOLPRO,MOPAC, NWChem andQ-Chem. Each of thoseentries pops open a newdialog window where you can selectthe extra options for the to-be-createdinput file. This includes things like thenumber of processors to use, the type ofcalculation to do or the theory to use, forexample. A preview window shows youwhat this export file will look like, so youcan be sure you’re getting what you wereexpecting. Once you’re happy, click onthe generate button at the bottom of thewindow to generate the input file for theexternal program of interest.Once these calculations are done, youcan import them back into avogadro to dosome analysis. You can import trajectoryfiles or vibration files. Once that data isimported, functions are available to graphthis data in order to extract informationand see what’s happening. For files thatcontain molecular vibration information,you even can graph the results as a movie.You have some options in terms of theFigure 4. Here I’m ready to upload spectral data from actualexperimental measurements.frame rate and so on, and once you aresatisfied, you can save it as an AVI file.You also can import data fromexperiments too. When studyingchemicals, one common experiment isto look at the spectra of the chemical ofinterest. To start, click on the menu itemExtensions→Spectra. This pops open a newwindow in which to do the analysis. Atthe bottom of the screen, you can click onLoad Data. This lets you import data in twoformats: either PWscf IR data or TurbomoleIR data. You can select how this data isdisplayed, including producing publicationqualitygraphics.Hopefully, this short introduction givesyou some ideas for getting started. Lots ofother chemistry programs exist that youcan look at to offer more functions andcalculations. Now you’re ready to go outand do some quantum chemistry.—JOEY BERNARD20 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

Silicon Mechanics Announces Research Cluster GrantWe are pleased to announce our sponsorshipof a unique grant opportunity:a complete high-performancecompute cluster using the latestAMD Opteron processors and NVIDIA ® GPUs.This grant program is open to all US and Canadianqualified post-secondary institutions, university-affiliatedresearch institutions, non-profit research institutions, andresearchers at federal labs with university affiliations.To download the complete rules, application, andhardware specification, visitwww.siliconmechanics.com/research_cluster_grantor emailresearch-grant@siliconmechanics.comSilicon Mechanics would also like to thank the manyhardware partners that have made this grant possible.When you partner with Silicon Mechanics,you get more than affordable, high-qualityHPC — you get a team of Experts dedicatedto the advancement of scientific research.Ocean Row Solo Expedition UpdateWave Vidmar has adjusted his schedule. He will be shipping “Liberty” to Portugal for a February launch to row theNorth Atlantic East-to-West route, heading for an area north of the Caribbean Islands. He is currently planning toundertake the West-to-East Row in May. We will continue to follow his expedition at siliconmechanics.com/ors.

[ UPFRONT ]Lowjack Your Body with RunKeepersearch for “exercise” in the Marketplace,and you’ll find a plethora of options. Keepin mind, however, that GPS-based exercisetrackingprograms aren’t much good inthe northern winters, when running movesindoors to a treadmill.Get the RunKeeper Android app athttp://www.runkeeper.com/android.—SHAWN POWERSThis past summer, I went to a beach resortin Mexico with my wife. It made sense toget into a little better shape so as not tocause any beached-whale rumors while Isoaked in the rays. Typical geek that I am,I wanted to track my every move so I couldsee how much exercise I really was doing.And, I wanted to do that with technology.Thankfully, RunKeeper is available forAndroid. RunKeeper is an exercise-tracking appthat uses GPS to track your exercise. Thanks togeographical information over GPS, RunKeeperwill track your distance, pace, time and evenelevation. The free version provides lots ofawesome features, and its social features alsocan help keep you accountable. (Although yourTwitter followers might get tired of hearingabout your daily walks to the park.)Several other exercise apps are available,so if RunKeeper isn’t your cup of tea, just22 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

[ UPFRONT ]Non-Linux FOSSWe’ve covered the cross-platform videoplayer VLC in past issues, but if you’rean OS X user, it’s often preferable to useQuickTime. Because it’s built in to theoperating system, QuickTime integrateswith almost every aspect of the system.Unfortunately, QuickTime has limitedplayback support. Enter Perian. Perian is anopen-source plugin for QuickTime that givesthe native player the ability to play mostpopular video formats. The Perian Web sitelists the following supported formats:Is Perian better than VLC? No, it’s notbetter or worse; it’s different. If you wantto play unsupported multimedia formatswith Apple’s QuickTime player, Periancan make that happen. If you’d ratheruse a completely open-source playerapplication, VLC fits the bill. Either way,open source saves the day with the abilityto play back just about any file you’dever want to play. Check out Perian athttp://www.perian.org.—SHAWN POWERSn File formats: AVI, DIVX, FLV, MKV, GVI,VP6 and VFW.n Video types: MS-MPEG4 v1 and v2, DivX,3ivx, H.264, Sorenson H.263, FLV/SorensonSpark, FSV1, VP6, H263i, VP3, HuffYUV,FFVHuff, MPEG1 and MPEG2 video,Fraps, Snow, NuppelVideo, TechsmithScreen Capture and DosBox Capture.n Audio types: Windows Media Audiov1 and v2, Flash ADPCM, Xiph Vorbis(in Matroska), MPEG Layer I and IIAudio, True Audio, DTS CoherentAcoustics and Nellymoser ASAO.n AVI support for AAC, AC3 Audio,H.264, MPEG4, VBR MP3 and more.n Subtitle support for SSA/ASS, SRTand SAMI.EMBEDDEDSERVER• Fanless x86 500MHz/1GHz CPU• 512MB/1GB DDR2 RAM On Board• 4GB Compact Flash Disk• 10/100 Base-T Ethernet• Reliable (No CPU Fan or Disk Drive)• Two RS-232 Ports2.6 KERNEL• Four USB 2.0 Ports• Audio In / Out• Dimensions: 4.9 x 4.7 x 1.7” (125 x 120 x 44mm)Standard SIB(Server-In-a-Box)Starting at $305Quantity 1.Since 1985OVER25YEARS OFSINGLE BOARDSOLUTIONS• Power Supply Included• Locked Compact Flash Access• Analog SVGA 3D Video• Optional Wireless LAN• EMAC Linux 2.6 Kernel• Free Eclipse IDEEQUIPMENT MONITOR AND CONTROLPhone: (618) 529-4525 · Fax: (618) 457-0110 · Web: www.emacinc.comWWW.LINUXJOURNAL.COM / DECEMBER 2011 / 23

COLUMNSAT THE FORGEMessageQueuesREUVEN M.LERNERAmazon’s Simple Queue Service (SQS) provides an easyway to scale your Web applications.This might come as a surprise, given that Ihave spent much of my professional careerworking with, writing on and teaching aboutthe use of databases, but there was a period(mostly during and right after college) whenI really didn’t understand why people everwould need them. After all, I thought, youcan just store and retrieve information in afile on your disk, no? My attitude back thendemonstrated not only profound ignoranceabout databases themselves, but also aboutthe types of problems people need to solveand the ways in which database technologyhad, even then, been developed to solvethose problems.Now I’m not quite as dismissive oftechnologies as I was back in my collegedays. But, it’s true that although I’ve longheard of “message queues”, it’s been onlyin the last year or so, while working on aproject, that I’ve come to realize just whata useful innovation they are. Sure enough,now that I understand how and why Iwould want to use them, I see uses formessage queues everywhere.In this article, I introduce the idea ofmessage queues and give several examplesof how you can install and use them. I alsodiscuss why you might want to use a messagequeue, particularly on a Web application,which people typically think of as consistingonly of an HTTP server and related software.Message QueuesIf you’ve been programming for any periodof time, you know that a fundamental datastructure is the queue, or FIFO (first in, firstout). Like a queue at the post office, the firstitem stored also is the first item removed.Different queue implementations havedifferent capabilities, but the general ideais that you put something in the queue andthen retrieve it when it’s ready. In Ruby (andmany other languages, such as Python), youcan implement a queue as follows:class Queuedef initialize@queue = [ ]end24 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

COLUMNSAT THE FORGEdef enqueue(thing)@queue > require 'queue' #=> true>> q = Queue.new #=>>> q.enqueue('a') #=> ["a"]>> q.enqueue('b') #=> ["a", "b"]>> q.enqueue('c') #=> ["a", "b", "c"]>> q.dequeue #=> "a"Because queues in Ruby can hold any value,you don’t need to worry about what will beon the queue. You just know that you canstick whatever you want on there, and thateventually you can retrieve it, in order.(I should note that if you’re activelyprogramming in Ruby, I hope you’re not reallyusing a queue class like this, but that you’rerather just using arrays, which support all thebasic operations you’re likely to need to workwith simple queue data structures.)Queues are great, but I’m sure you canalready imagine all sorts of horrible scenariosif you were to use this simple one forsomething important, such as a list of banktransfers to execute. My banker’s desk is fullof piles of papers that she needs to handle,and she (presumably) works through themfrom top to bottom, dealing with each one inturn, but it would be pretty unforgivable forone or more of those papers to get lost. And,although it’s easy to say that Ruby arrays arepretty stable, what happens if the power goesout? In such a case, the entire queue is lost,causing untold problems for the people whoexpected safe delivery.The difference between the simple-mindedqueue I showed above and a true messagequeue is that the latter guarantees deliveryof every message. This means that basicallyno matter what happens, you can be surethe message eventually will be delivered,despite power outages and other issues. But,message queues are even better than that.Not only do they guarantee delivery, but theyalso work quickly, allowing you to queueup a number of messages or actions thatrequire attention, but for which you lack theresources to handle immediately.For example, consider a Web applicationthat is designed not to provide immediatefeedback to users, but rather to receiveand process information sent from othercomputers or mobile devices. This type ofapplication typically doesn’t require givingthe user immediate feedback (other than anacknowledgement that data was received). AllWWW.LINUXJOURNAL.COM / DECEMBER 2011 / 25

COLUMNSAT THE FORGEof the messages sent are of great importance(and should not be lost), but the number ofmessages can vary greatly from minute tominute, let alone from hour to hour. Whenthe data is processed and eventually placed inthe database, however, doesn’t matter nearlyas much.There are more mundane examples as well.Consider a Web application that needs tosend e-mail updates to people, such as froma calendaring application. If the applicationwere to send e-mail each and every time anevent were changed, the response time—orthe number of server processes available toreceive new incoming messages—might wellsuffer. Instead, the application can stick themail-sending task on a message queue andthen let a process on a separate computerretrieve the order and send the actual e-mail.Offloading the retrieval of messages to aseparate computer offers another performanceadvantage. It allows you to scale up theprocessing as necessary, by adding additionalback-end computers. Given that a messagequeue is transactional (that is, all-or-nothing),you can have as many back-end machinesretrieving from the queue as you want. Youdon’t have to worry that the same message willbe delivered twice or that two processes willhave to fight over the retrieved data.Amazon SQSSo, now that I’ve convinced you that youwant to have a message queue, how doyou go about using one? The first questionis which one to choose. Many messagequeues exist, and each has its advantagesand disadvantages. I’ve been using Amazon’sSimple Queue Service on a project for the pastnumber of months, and although it certainlyhas its downsides—it costs money, and it cantake a bit of time for messages to percolatethrough the system—the advantageshave been fairly clear, including Amazon’swillingness to store messages for up to twoweeks and its impressive uptime statistics.And, although I certainly could have set upmy own message-queueing system, I’ve beenworking on other aspects of the project andappreciated that someone else, out there in“the cloud”, was dealing with the various ITrelatedtasks associated with running a queue.If you have used any of Amazon’s previouscloud offerings, its queue service will notbe a surprise. You need to have an Amazonaccount and sign up for a unique accesskey that will identify you to Amazon foridentification and billing purposes. In additionto the access key, which you can think of asa user name, you also need a secret (akin toa password), which is sent to Amazon alongwith each request.Once you have set yourself up with SQS,you need to connect to it, preferably (but notnecessarily) using one of the many SQS clientlibraries that have been developed. Most of mywork nowadays is in Ruby, and when I startedmy project, I found that the best-known Rubygem for SQS access was from RightScale, inthe “right_aws” package. I have been using26 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

COLUMNSAT THE FORGEthis driver without any problems, but it’s truethat Amazon has since released its own driversfor Ruby. I hope to experiment with that driverin the near future and to compare it with theRightAws modules—although to be honest, Idon’t expect to see any significant differences.If you’re using another language, therealmost certainly are libraries you can use aswell. For the Python community, there arethe boto packages. See Resources for moreinformation.By the way, it’s true that SQS costs money.However, queueing systems exist to handlelarge quantities of data, which means they’regoing to charge you very little per message.How little? Well, according to the pricesposted at the time of this writing, sendingmessages is absolutely free. Receivingmessages is free for the first GB each month.After that, you pay nothing for the first GB,and then 12 US cents for each GB, up to1TB. Now, Amazon does have a number ofdifferent server centers, and each might haveits own pricing. Also, pricing is applicableonly when going in or out of Amazon’sserver systems. This means if you’re using ahosting solution, such as Heroku, which sitson Amazon servers, transfer to and from SQSis completely free. Actually, that isn’t quitetrue—data transfers are free only if you staywithin the same geographic server cluster. Mypoint, however, is that for most people andprojects, the pricing should not be an issue.I’m using SQS for a Web application thatis (by the time you read this, if all goes well)intended to receive JSON data from mobiledevices, sent via an HTTP POST request. TheJSON data then needs to be parsed and stuckinto a relational database, but that doesn’tneed to happen right away. The architectureof the application, thus, consists of twoseparate parts. The main Web app receivesthe data and puts it onto the messagequeue with minimal parsing and validation.A separate Web app, running on a separateserver, retrieves the JSON data, parses andvalidates it, and then puts it into the database.From the perspective of SQS, the fact that I’musing different servers really doesn’t matterat all; as long as I connect to SQS with theright user name and password, and use theright queue name for sending and receiving,everything will be just fine.Connecting to SQSBefore you can send to or receive fromSQS, you first must connect to it. Since I’musing the right_aws gem for Ruby, I need todownload and install that:$ gem install right_awsNote that because I’m using RVM, the Rubyversion manager, I installed this gem as myown user. If I were installing it for the entiresystem, or if I were not using RVM, I wouldneed to log in as root or use sudo to executethe command as root.With the right_aws gem installed and inplace, I now can use it to connect to the SQSWWW.LINUXJOURNAL.COM / DECEMBER 2011 / 27

COLUMNSAT THE FORGEserver. Note that RightScale’s gem providesaccess to several different APIs, includingseveral different “generations” of SQS. I amusing the second-generation API, via theRightAws::SqsGen2 class.I’ve put my Amazon keys in a separateYAML-formatted configuration file, allowingme to change and update keys as necessary,as well as keep track of separate keys fordifferent environments. I then read theconfiguration information into my programwith the following line:SQS_CONFIG = YAML.load_file("/Users/reuven/➥Desktop/config.yml")['defaults']The above takes each of the namevaluepairs in the “defaults” section ofmy config.yml file and puts it into a hashnamed SQS_CONFIG. (Note that I’ve usedall caps to indicate that this is a constantand should not be modified by otherprogrammers unless they have a really,really good reason for doing so.)I then can get a connection to SQS with thefollowing code:require 'right_aws'sqs = RightAws::SqsGen2.new(SQS_CONFIG['aws_access_key_id'],SQS_CONFIG['aws_secret_access_key'],{ :server => SQS_CONFIG['sqs_server'] })As you can see from the above call,Right::SqsGen2.new takes threeparameters: the AWS key, the AWS secretand a hash of options that help configurethe queue object. The most important oneto pass is the name of the SQS server youwant to use. If you don’t specify it, you’ll getqueue.amazonaws.com, but to be honest,I haven’t really thought about it muchsince checking with Heroku (our hostingprovider) about which server to use.Once you’re connected to SQS, youmust create (or retrieve) a queue. Youcan think of a queue as a single array towhich you can store or retrieve data, justas I did in my simple Queue class earlierin this article. The difference, of course, isthat the actual data storage is happeningacross the network, on servers to whichyou have no direct access. You can haveany number of queues, each with its ownname, containing alphanumeric characters,hyphens and underscores. So, if you wantto use a queue called “testq”, just say:sqs_queue = sqs.queue('testq')This returns an instance ofRightAws::SqsGen2::Queue, an object thatrepresents an Amazon message queue. Anumber of methods are defined on thisobject, including creation (which I do viathe above call, rather than directly), deletion(which will remove all of your data, so I reallywouldn’t suggest it unless you have to),and the sending and receiving of messages.You also can set the visibility timeout on28 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

COLUMNSAT THE FORGEthis object, which tells Amazon how long amessage should be invisible once it has beenread, but before it has been deleted. You evencan get the size of the message queue, usingthe size method.Sending MessagesIn my simple, non-distributed message queueexample, you saw that new messages areadded to the queue using an enqueuemethod, taking a single object as a parameter.The same is true in this case; if you want tosend a message to the queue, you simply say:my_message = 'hello!'sqs_queue.send_message(my_message)This will turn your string into an SQSmessage and send it to the queue. So long asthe message is less than 64KB in size and is intext format (including JSON or XML), Amazonprobably will accept it. (The RightScale gemclaims to support messages only up to 8KB insize, just as Amazon used to do, but it’s notclear to me whether the gem enforces theselimits or if Amazon’s updates are reflected bythe gem’s behavior.) Trying to send a messagethat’s too long for Amazon’s limits will result inan exception being thrown. There is an explicitlist on the SQS FAQ page of which UTF-8characters are acceptable in an SQS message.One nice thing about SQS is that you canhave any number of messages in a queue ata time; there is no defined limit. By default,messages are kept in a queue for four days,but you can configure that to be anywherefrom one hour to two weeks.Receiving MessagesSo, you’ve sent a message to the messagequeue. How do you receive it? After goingthrough the initial configuration, connectionand queue creation/opening displayedabove, you can retrieve the first waitingmessage on the queue with:message = mothra_queue.receiveIn RightScale’s Ruby library, message isset to nil if no messages were available. Thus,before you can operate on the message, youmust first check to ensure that it’s non-nil.Assuming that message is not nil, youcan get contents by transforming themessage into a string—in other words,by invoking .to_s on the message:print message.to_sWhen you retrieve a message, Amazonkeeps a note of that and makes it invisibleto other processes that might try to retrieveit. In other words, if you’ve queued a singlemessage and then retrieve that message, otherprocesses trying to retrieve from the queuewill be told that no messages are available.However, this is true only for a short time.Once the visibility timeout has passed, themessage is once again available to retrievingprocesses. So, in order to ensure that aWWW.LINUXJOURNAL.COM / DECEMBER 2011 / 29

COLUMNSAT THE FORGEmessage is not read twice, it must be deleted:message.deleteUnder most circumstances, you will wantto retrieve and then delete a message almostright away.ConclusionIf you’re saying, “Well, that seems quitesimple”, you’re right. Message queues area dead-simple idea, particularly if you’refamiliar with queues as data structures.Distributed message queues can be quitedifficult to get to work in a distributed andpersistent way, but Amazon has done justthat and makes its queue available for avery reasonable price, often ending up freefor small organizations and sites.The advantages that a distributedmessage queue can bring to the table areoverwhelming though, particularly when youhave tasks or pieces of data that are comingin too rapidly to handle, but which could beprocessed by a large number of back ends.It’s easy to imagine a large number of backendcomputers picking messages off andinserting them into a database, after parsingand checking them for validity. Indeed, that’swhat I’m doing on my current project, and ithas been working like a charm.Now, there are issues with Amazon’squeues. For starters, they have longer latencythan you would get with a local queue,and they also are sitting on third-partyservers, which might not sit well with somecompanies. But for the most part, it hasworked without a hitch and has become acore part of the infrastructure on my project.During the course of this work, I’ve started tofind all sorts of uses for message queues, andI’m starting to incorporate them into otherprojects on which I work. The day may comewhen it’s an exceptional project that doesn’tuse a message queue, rather than the otherway around.■Reuven M. Lerner is a longtime Web developer, architect andtrainer. He is a PhD candidate in learning sciences at NorthwesternUniversity, researching the design and analysis of collaborativeon-line communities. Reuven lives with his wife and threechildren in Modi’in, Israel.ResourcesThe home page for Amazon’s SQS ishttp://aws.amazon.com/sqs. This site, likeall the other Amazon Web Services sites,has extensive documentation, tutorials andexamples, as well as forums that let developershelp one another. I’ve never needed to usethe help, because the documentation alwayshas been sufficient for my needs, but I’veread through the forums on some occasionsand have been impressed with the degreeof both community involvement and officialanswers from Amazon’s staff.You can find, learn about and downloadRightScale’s AWS-related gems fromhttp://rightaws.rubyforge.org.If you’re a Python programmer, you candownload code from the boto Project for AWSaccess from http://code.google.com/p/boto.30 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

DON’T MULTITASK.MEGATASK.Imagine what you can do with an expert workstation.Take iteration, visualization, and simulation data to the next level of productivitywith the supercharged performance of an expert workstation powered by theIntel® Xeon® processor 5600.Deskside and portable workstations from Ace Computers feature the Intel® Xeon®processor 5600 series.www.acecomputers.comAce Computers877-ACE-COMP (877-223-2667)Inspiring creativity means using tools that don’tdisturb your train of thought. Ace presents theNEW LogiCAD 45525SQ workstation, whichcan handle the highest workloads possible, butdoes it at the ambient noise level of a regulardesktop PC.With the Ace ® Raptor 6 portableworkstation, engineers have the abilityto adjust designs in the fi eld directly. TheRaptor 6 is equipped to meet the capabilitiesof professional desktop workstations but in aportable notebook-like form factor.Ace ® LogiCAD 45525SQSuper-quiet Workstation• Dual Intel ® Xeon ® 5600 series processors with up to 384GB memory• Super-quiet chassis—28dB maximum noise level—less thana desktop• Up to 4 graphics cards for extreme CAD/graphical simulationsStarting at $1,995.00Ace ® Raptor 6 PortableWorkstation with Built-in UPS• Intel ® Xeon ® processor 5600 series• Up to 4 hard drives with hardware RAID/up to 24GB DDR3 memory• Up to 2 graphics cards for extreme CAD/graphical simulationsStarting at $2,995.00© 2011, Intel Corporation. All rights reserved. Intel, the Intel logo, Intel Xeon, and Xeon Inside are trademarks of Intel Corporation in the U.S. and/or other countries.*Other names and brands may be claimed as the property of others.© 2011 – Ace Computers. All Rights Reserved. Ace Computers, Raptor, and LogiCAD are trademarks of Ace Computers.

COLUMNSWORK THE SHELLPlaying withTwitter StatsDAVE TAYLORIt’s not easy to determine whether someone’s worthfollowing on Twitter, but Dave takes on the task witha shell script that extracts account stats for a givenTwitter account, then calculates their follow value.He also explains the philosophy behind the project andfinds that Twitter has some weirdnesses in its HTMLthat make parsing the results interesting.So, you’ve been using Twitter sinceit was all about the fail whale and notabout the corporate sponsorships andback-end analytics. Me too. The problemis, Twitter also has become even morecrazy and hard to understand as it hasgained its millions of followers and itsutility ecosystem has expanded andcontracted variously.One thing that’s always interestedme though is whether there’s a wayto calculate a numeric value for givenTwitter users based on both their visibilityand engagement. How do you measurethose? Visibility could be calculatedsimply by looking at how many followerssomeone has, but most Twitter usersfollow lots of random people, so thatthey can have lots of followers.This behavior is based on what DrRobert Cialdini calls the Principle ofReciprocity in his brilliant book Influence,wherein he observes that if someone doessomething for you, you feel an inherentobligation to return the favor. Think HareKrishnas at the airport giving you a flowerbefore they ask for a donation. Thinkof the self-appointed pundits and gurustelling you their rules of netiquette, orof your own reactions—“if this person’sfollowing me on Twitter, I should followthem back. It’s only polite, after all.”The upside is that if you just look at howmany followers someone has without alsochecking how many people they follow,you can be duped into thinking somethingalong the lines of “25,000 followers?Impressive.” without ever noticing that the32 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

COLUMNSWORK THE SHELLperson follows 30,000 people in turn.One way to differentiate these differenttypes of Twitter users, therefore, is tocalculate the ratio of followers to following.That’s half the calculation.Engagement is trickier to calculate, but ifyou examine someone’s Twitter stream, youcan separate out broadcast messages fromthose that are either an at-reply (as in“@DaveTaylor nice column!”) or a retweet.It’s another ratio. If the majority oftweets from someone are broadcasttweets, their level of engagement is low,whereas a Twitter user whose messagesalmost always are responses is high onthe engagement scale.One more criterion: gross numbers. Howmany followers does someone have overall?How many tweets has the user sent? Anaccount with a high engagement but onlyseven tweets in the last six months is lessinteresting than one with lower engagementbut an average of 20 tweets a day. Agreed?So, how do we calculate these sortsof figures?Understanding a Twitter Profile PageTwitter offers up quite a bit of informationfor its public profiles (and just about everyTwitter profile is public), including thekey stats we want to start with: followercount and following count.To get them, we don’t even need tonegotiate the OAUTH login. We can justuse curl from the command line:$ curl -s http://twitter.com/davetaylor |grep 'stats_count numeric'566 10,187 790 You can see that my Twitter account,@DaveTaylor, has 10,187 followers, whileI’m following 566 people. The “list”figure suggests popularity too, but sincemost Twitter users I know eschew lists,let’s just ignore that for now.We’d also like to grab the raw tweetcount to see if it’s an account that actuallyhas sent some tweets or is dormant.Examining the HTML closely reveals thatalthough the previous items are put intothe class stats_count, the number oftweets sent is put in a similar, but not quiteidentical, class called stat_count. Typo?Maybe. Meanwhile, it forces us to tweakour regular expression:$ curl -s http://twitter.com/davetaylor |grep -E '(stats_count|stat_count)'566 10,187 790 30,055TweetsIt’s a bit ugly, but it’s not much work toextract and reformat the data in a script.The challenge really is just to strip awayWWW.LINUXJOURNAL.COM / DECEMBER 2011 / 33

COLUMNSWORK THE SHELLThe challenge really is just to strip away all the HTMLjunk, because once we’ve used it to select the linesin question, we don’t actually need it anymore.all the HTML junk, because once we’veused it to select the lines in question, wedon’t actually need it anymore.My first attempt is this:curl -s http://twitter.com/davetaylor | grep -E➥'(stats_count|stat_count)' | sed 's/]*>/ /g'56610,187790$ echo "hello" | sed 's//-/g'-That didn’t work. We want “hello”as the result, because we don’t wantto lose the non-HTML values. Here’s mysecond try:30,055 TweetsWe still need to get rid of those peskycommas, but that’s a small addition to thesed statement, right? Let’s use this instead:sed 's/]*>/ /g;s/,//g'.The results are ready to be parsed:$ echo "hello" | sed 's/]*>/-/'-helloAha! That’s what we need—a regularexpression that basically says “< followedby as many characters as are present otherthan the ’>’ character”.To strip all the HTML, simply make it aglobal search and replace by appending a“g” to the sed statement:5661018779030055 TweetsThat can be done with one of my favoritescripting commands, cut. The wrinkle,however, is that when we drop this into ashell script, the results are a bit surprisingif we look at my @FilmBuzz movie newsTwitter profile. First, the script snippet:$ echo "hello" | sed 's/]*>/-/g'helloThat’s great. Now we can turn themess of results into something hopefullya bit more useful:stats="$(curl -s $twitterurl/$username | grep -E➥'(stats_count|stat_count)' | sed 's/]*>/ /g;s/,//g')"echo $statsAnd, the results:34 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

COLUMNSHACK AND /Read LinuxJournal from theCommand LineKYLE RANKINKindles? Nooks? It just takes your trusty command lineand a few command-line tools to read Linux Journal.In this day and age, there are more waysto read than ever before. Even though LinuxJournal no longer publishes on paper, you stillcan read it with Web browsers, PDF software,e-book readers and cell phones. I don’t havean e-book reader myself, but I think you couldmake the argument that the one true way toread Linux Journal is from the command line.After all, I read my e-mail, chat, check Twitter,do most of my day job and write my articlesfrom the command line (okay, it’s true I usegvim too; it frees up a terminal window), sowhy not read Linux Journal from the placewhere I spend most of my time?The Text, the Whole Text and Nothingbut the TextThe first format I’m going to cover is thePortable Document Format (PDF). AlthoughPDFs are aimed at capturing a document sothat it looks the same to everyone, it turnsout you also can strip out the text and imagesFigure 1. pdftotext’s Default Output for MyColumnfrom a PDF file. The first program I use for thisis the aptly named pdftotext. This programis part of a group of PDF utilities that arepackaged as the popper-utils package underDebian-based systems, but you should be able36 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

COLUMNSHACK AND /to find them under a similar name for yourdistributions. The most basic way to executepdftotext is the following:$ pdftotext input_document.pdf output_document.txtBy default, pdftotext does not attempt topreserve all the formatting of the document,which is nice because you don’t have toscroll up and down multiple columns of apage. The downside is that it doesn’t knowto strip out all the extraneous text, headers,pull-quotes and other text you will find in amagazine article, so the result is a bit limited,as you can see in Figure 1.Text Plus ColumnsSo although I suppose pdftotext’s defaultoutput is readable, it’s less than ideal. That’snot to say I’m out of tricks though. Amongits command-line options, it provides a-layout argument that attempts topreserve the original text layout. It’s still notperfect, as you can see in Figure 2, but ifyou size your terminal so that it can fit a fullpage, it is rather readable.Text Plus ImagesThere is a bit of a problem, you’ll find, if youdo read Linux Journal in text-only mode:there’s no pictures! Although some articlesstill are educational in pure text, with others,it really helps to see a diagram, screenshotor some other graphical representation ofwhat the writer is saying. You aren’t withoutoptions, but this next choice is a bit of ahack. Because there are versions of thew3m command-line Web browser that candisplay images in a terminal (the w3m-imgpackage on a Debian-based system providesit), what you can do is convert the PDF toHTML and then view the HTML with w3m. Todo this, you use the pdftohtml program thatcame with the same package that providedpdftotext. This program creates a lot of files,so I recommend creating a new directory foryour issue and cd-ing to it before you run thecommand. Here’s an example of the steps toconvert the September 2011 issue:$ mkdir lj-2011-09$ cd lj-2011-09Figure 2. pdftotext with the Layout Preserved$ pdftohtml -noframes /path/to/linuxjournal201109-dl.pdf➥lj-2011-09.htmlWWW.LINUXJOURNAL.COM / DECEMBER 2011 / 37

COLUMNSHACK AND /Although it’s nice to see the images in a terminal,it would be better if everything was arranged so itmade a bit more sense.was arranged so it made a bit moresense. Like with pdftotext, pdftohtmlhas an option that attempts to preservethe layout. In the case of pdftohtml,you add the -c option:$ mkdir lj-2011-09$ cd lj-2011-09$ pdftohtml -noframes -c /path/to/linuxjournal201109-dl.pdf➥lj-2011-09.htmlFigure 3. A More Negative Version of MeOn the one hand, this commandgenerates some really good-lookinggraphical pages. On the downside, itOnce the command completes, youcan run the w3m command against thelj-2011-09.html file, and if you have thespecial version that loads images, youwill start to see the images load in theterminal. Now, by default, this output ismuch like the original output of pdftotext.There is no attempt to preserve formatting,so the output can be a bit of a mess toread. Also, as you can see in Figure 3, myheadshot looks like a photo negative.Text Plus Images Plus ColumnsAlthough it’s nice to see the images in aterminal, it would be better if everythingFigure 4. It’s an improvement for image quality,but worse for readability.38 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

COLUMNSHACK AND /looks like the images are displayed overthe top of the text, and as you can seein Figure 4, there’s a whole graphicalsection of my column with no text onit. As you scroll down the page, you stillcan read a good deal of the text, but itstands independent of the image. On theplus side, it no longer shows a negativeheadshot.Go with the ReflowSo PDF conversion technically worked, butthere definitely was room for improvement.As I thought about it, I realized that epubfiles work really well when it comes toreflowing text on a small screen. I figuredthat might be a better source file for mycommand-line output.The tool I found best-suited to the jobof converting epub files to text is Calibre.In my case, I just had to install a packageof the same name, and I was providedwith a suite of epub tools, includingebook-convert. Like with pdftotext,all you need to do is specify the inputfile and output file, and ebook-convertgenerates the output file in the formatyou want based on the file extension(.txt in this case). To create a basic textfile, I would just type:$ ebook-convert /path/to/LJ209-sept.epub LJ209-sept.txtFigure 5. Even with Indents, a Quite ReadableLJ Articleof the terminal. That said, I would saythat so far, it was the most readable ofthe output, as you can see in Figure 5.So, with all of those different ways toread Linux Journal from the commandline, two methods stand out to me rightnow. If you don’t need images, I thinkthe epub-to-text conversion works thebest, with the pdftotext that preserveslayout coming in second. If you do needto see images though, it seems like yourmain choice either is to convert fromPDF to HTML and then use w3m, or justuse w3m to browse the Linux Journalarchives directly.■I found the resulting text file quitereadable actually, although it did like toindent all of the headers and a lot of therest of the text, so it started at the centerKyle Rankin is a Sr. Systems Administrator in the San FranciscoBay Area and the author of a number of books, including TheOfficial Ubuntu Server Book, Knoppix Hacks and Ubuntu Hacks. Heis currently the president of the North Bay Linux Users’ Group.WWW.LINUXJOURNAL.COM / DECEMBER 2011 / 39

NEW PRODUCTSFluendo’s Codec PackFluendo, the folks whose mission is to make everythingrun on Linux, recently announced Codec Pack Release15. In addition to the existing Windows Media, MPEG,DivX and other decoders, the new version 15 now offerssupport for AMD GPUs through the XvBA using OpenGLfor efficient video rendering. This new addition follows onthe heels of recently added support for VDPAU (NVIDIA)and VAAPI (Intel). Fluendo says its products are utilized bymost OEMs for devices like desktops, laptops, thin clients,tablets, digital signage, set-top boxes, connected TV andmedia centers. Also, Fluendo solutions have been used by companies who’ve adoptedLinux internally, allowing them to provide their employees with complete multimediacapabilities while remaining in full compliance with laws and patents.http://www.fluendo.comAnderson and Mutch’s Preventing Good PeopleFrom Doing Bad Things (Apress)“Good fences make good neighbors” is the old adage behind Brian Anderson and JohnMutch’s new security book, Preventing Good People From Doing Bad Things: ImplementingLeast Privilege. Many corporations have had to learn the hardway that they can have the slickest security software moneycan buy, while the greatest threats lurk from within. The mainfocus of the book, published by Apress, is to show how firmscan remove these internal weaknesses by applying the conceptof least privilege. The authors point out the implications ofallowing users to run with unlimited administrator rights anddiscuss the implications when using Microsoft’s Group Policy,UNIX and Linux servers, databases and other apps. Other topicsinclude virtual environments, compliance and a cost-benefitanalysis of least privilege. Auditors, geeks and suits will allfind useful information in this book.http://www.apress.com40 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

NEW PRODUCTSMcGuire and O’Leary’s Book: AFuturist’s Manifesto (O’Reilly Media)Linux Journal’s own complete migration from the Gutenbergrealm to the digital one is a fine case study on the powerfultrends in publishing that are explored in a novel new project titledBook: A Futurist’s Manifesto by Hugh McGuire and Brian O’Leary.The core content of this O’Reilly project is a collection of essaysfrom thought leaders and practitioners on the developmentsoccurring in the wake of the digital publishing shake-up brought on by the Kindle, iPhoneand their kindred devices. The essays explore the new tools that are rapidly transforminghow content is created, managed and distributed; the critical role that metadata plays inmaking book content discoverable in an era of abundance; the publishing projects thatare at the bleeding edge of this digital revolution and how some digital books can evolvemoment to moment, based on reader feedback. This particular project will do just that,incorporating reader feedback as the book is produced in hybrid digital-print format anddetermining in what ways the project will develop.http://www.pressbooks.com and http://www.oreilly.comQueplix’s QueCloud FREEMIUM VersionQueplix recently announced the availability of the FREEMIUM version ofQueCloud, a product the company bills as “the first data managementcloud [that] enables companies to securely integrate cloud applications suchas Netsuite, Google and SAP with unprecedented speed and simplicity”. Thebenefits, says QueCloud, include an 80%+ TOC reduction vs. traditional ETL datamanagement tools. QueCloud’s core technology is Queplix’s flagship Virtual DataManager architecture, which enables the configuration of a series of intelligentapplication software blades that identify and extract key data and associatedsecurity information from many different target applications. The blades identifyand extract key metadata and associated security information from the datastored within these applications, then bring it into the Queplix Engine to supportdata integration with other applications. The FREEMIUM version provides freeaccess to users, developers and independent software vendors.http://www.queplix.comWWW.LINUXJOURNAL.COM / DECEMBER 2011 / 41

NEW PRODUCTSArkeia’s BackupAppliancesThe new third generation ofArkeia Backup Appliancespromises to “reducecosts with larger diskcapacities, embeddedSSD, and deduplication”says the well-knownprovider of backup anddisaster recovery products.Deduplication delivers effective storage that is many times the capacity of internal drives.The company also touts new features, such as RAID-6, faster network connectivity, fullyintegrated and optimized Arkeia Network Backup v9 software, integrated bare-metaldisaster recovery and support for VMware vSphere virtual environments. Target customersfor the appliances are mid-sized companies or remote offices.http://www.arkeia.comOpenStack DiabloThe new Diablo (4th) release of the OpenStack cloudcomputingplatform recently went live, containingenhancements across the three existing projects:OpenStack Compute, Object Storage and ImageService. Key new features include new networkingcapabilities, such as “networking as a service”and unified authentication across all projects.Diablo extends existing API support and acceleratesnetworking and scalability features, allowing enterprises and service providers todeploy and manage OpenStack clouds with larger performance standards and withease. Two new projects for the next OpenStack software release, “Essex”, have beeninitiated, currently code-named Quantum and Keystone.http://www.openstack.org42 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

NEW PRODUCTSeyeOS Professional EditioneyeOS bills itself as “the most-advanced Webdesktop in the world and the largest open-sourcesoftware project in Spain”. The new eyeOSProfessional Edition takes the original edition to anew level, enabling private clouds that are accessiblevia any browser or device. All employees’ and customers’ workspaces can be virtualized in thecloud. The company says that the solution is highly scalable, easy to manage, does not requirea large investment and includes all types of applications, including virtualized ones. BecauseeyeOS is developed in PHP and JavaScript and compiled in Hip-Hop, the system is not onlyhigh-performance, but also no software needs to be installed on the computer in order towork with it. Furthermore, the company says that eyeOS can integrate existing SaaS, virtualizedlegacy applications and other in-house apps served up as Web services.http://www.eyeos.orgGrammaTech’s CodeSonarKill programming bugs dead with CodeSonar, GrammaTech’s static-analysis toolthat performs a whole-program, interprocedural analysis on code and identifiescomplex programming bugs. The breakthrough feature in this release relates tothe new program-analysis algorithms that identify data races and other seriousconcurrency defects. The process involves symbolic execution techniques toreason about many possible execution paths and interleavings simultaneously.The concurrency analysis can be applied to multithreaded software writtenfor both single-core and multicore architectures. Another new feature, codelevelmetrics, is built on CodeSonar’s existing code-analysis and reportingframework, which enables project managers to track popular metrics, such ascyclomatic complexity, or even define new metrics. Warnings can be generatedautomatically when metrics are outside an expected range. CodeSonar runs onLinux, Windows, Solaris and Mac OS and supports most compilers.http://www.grammatech.comPlease send information about releases of Linux-related products to newproducts@linuxjournal.com orNew Products c/o Linux Journal, PO Box 980985, Houston, TX 77098. Submissions are edited for length and content.WWW.LINUXJOURNAL.COM / DECEMBER 2011 / 43

NEW PROJECTSFresh from the LabsArduino—Open Hardware andIDE Combohttp://arduino.cc/enThis article is a bit different from my usualcolumn in two ways. First, it’s startingwith a hardware and software combo—something I’ve not done before. Second,the projects are linked to each other andcome recommended to me by Perth LUGmember, Simon Newton.Given the mostly hardware-basedinformation for the project, here are somecarefully selected bits of information fromthe Web site:software running on a computer.The open-source Arduinoenvironment makes it easy to writecode and upload it to the I/O board.It runs on Windows, Mac OS X andLinux. The environment is written inJava and based on Processing, avr-gccand other open-source software.Arduino is an open-sourceelectronics prototyping platformbased on flexible, easy-to-usehardware and software. It’s intendedfor artists, designers, hobbyistsand anyone interested in creatinginteractive objects or environments.Arduino can sense the environmentby receiving input from a varietyof sensors and can affect itssurroundings by controlling lights,motors and other actuators. Themicrocontroller on the board isprogrammed using the Arduinoprogramming language and theArduino development environment.Arduino projects can be standaloneor they can communicate withArduino: this hardware/software combo allowsyou to program chips and test them on thefly—a real advantage of open hardware.Installation For those who are happywith a binary, the Web site makes thingsvery easy with 32- and 64-bit binarytarballs at the download page, and ifyou’re lucky, the Arduino IDE may evenbe in your repository. If you’re going withthe binary tarball, just download the44 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

NEW PROJECTSOne of the wacky creations possible withArduino, this spider-like robot is made byCurtin University student, Phillip Lawrence.latest from the Web site, extract it, andopen a terminal in the folder. To run theprogram, enter the command:$ ./arduinoIf you’re running from source instead,instructions are available on the Website with a link from the Downloadspage, although I don’t have the spacehere to cover its somewhat unusualinstaller method. Nevertheless, it doesrecommend a series of packages thatshould help troubleshoot mishaps withboth the source and binary tarballs. TheWeb site says you need the following:Sun Java SDK, avr-gcc, avr-g++, avr-libc,make, ant and git.If you have a local repository versioninstalled, chances are the program canbe started with this command:$ arduinoUnder my Kubuntu installation, the ArduinoIDE was available in the KDE menu underApplications→Electronics→Arduino IDE.However, I must stop you here beforeactually running the program, and Iapologize if I led you astray in the last fewparagraphs (don’t worry if you’ve alreadystarted it, you can close and re-open itwith no worries). Obviously, before youcan do anything with an Arduino boardand the software, you first have to plug inyour Arduino device. This will help in theconfiguration of your hardware, especiallyif you’re using a USB connection.Once that’s out of the way, you nowcan start the program with any of themethods above.Usage With the program runningand the device plugged in, let’s set itup. Inside the main window, click on theTools menu and navigate your way to theBoard menu. From there, choose yourArduino device (I had the Arduino Uno).Now you have to choose your serial port,which is under Tools→Serial Port. If youhad a USB device and the program foundit, a USB option should appear here (inmy case, /dev/ttyUSB0).With all of that boring stuff out of theway, I’m sure you’re keen to sink yourteeth into this hardware/software combo.The IDE makes things simple with a seriesof examples in an easy-access menu.Look under File→Examples, and checkout the impressive list of examples fromwhich to choose. I recommend startingwith Blink under the 1.Basics menu.WWW.LINUXJOURNAL.COM / DECEMBER 2011 / 45

NEW PROJECTSWith Blink, you can start with the mostbasic of basics and come to grasp thesyntax with well laid-out code includingdocumentation for each line. To try out thiscode, click Upload, which is the sixth buttonalong in the blue menu, with the rightfacingarrow. If all goes well, you shouldsee your device start blinking from an LED,perhaps with a board reset in the process.If your board has an enabled resetfacility like the Uno I was using, youshould be able to make code changesby uploading them, watching the boardnext to you reset and start again withthe new program. In fact, I recommendyou try it now. Change one of thelines, perhaps one of the lines dealingwith the delay time, and then upload itagain. Now this may seem lame, but to ahardware “n00b” like myself, changingaround the program and updating therunning hardware in a visible way wasquite a buzz!If you want to check out your codebefore uploading it, the start andstop buttons are for verifying thecode, with the stop button obviouslyallowing you to cancel any compilingpartway through. Although I’m runningout of space for the software side, Irecommend checking out more of theexamples in the code, where genuinelyreal-world uses are available. Somehighlights include ChatServer, “a simpleserver that distributes any incomingmessages to all connected clients”; areader for barometric pressure sensors;and a program for demonstrating andcontrolling sprite animations.However, I’ve been neglecting one ofArduino’s real bonuses, and that is theability to use a board to program anynumber of chips, remove them from themain Arduino board, and use them torun external devices. The nature of openhardware really makes this a roboticenthusiast’s wet dream, with exampleslike my close mate Phil’s robotic spidershowing some of the cool things you canachieve with this suite.Nevertheless, I do have one specificuse of Arduino in mind to tie this columntogether, and that is Simon Newton’sArduino RGB Mixer: a six-channel colormixer that interfaces with OLA. Checkout the following link for instructionson how to make this simple device thatSimon Newton’s RGB Mixer is a great way to useboth Arduino and OLA together.46 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

NEW PROJECTSshows off both of these projects at thesame time: http://www.opendmx.net/index.php/Arduino_RGB_Mixer.OLA—Open Lighting Architecturehttp://www.opendmx.net/index.php/OLAWhether you’re into concert lighting, flashydisplay stalls or Christmas lighting, or maybeyou take mood lighting a little too seriously,you’ll want to check out OLA. Combinedwith Simon’s RGB Mixer, hopefully we canexplore this project with relative ease.According to the Web site:The Open Lighting Architecture(OLA) provides applications witha mechanism to send and receiveDMX512 and RDM commands usinghardware devices and DMX overIP protocols. This enables softwarelighting controllersto communicate withhardware either viaEthernet or traditionalDMX512 networks.(./examples/ for C++ and ./python/examples/ for Python).OLA supports some of the newestlighting control protocols, includingRemote Device Management orRDM. Like the name suggests,this allows lighting devices tobe remotely configured, andinformation like temperature, fanspeeds, power consumption and soon to be fed back to the lightingcontroller. The Arduino RGB Mixerimplements basic RDM functionalityallowing the user to set the DMXStart address and invert the outputsignals in order to support differenttypes of LEDs. With the addition of atemperature sensor, it can report thisinformation back to OLA.OLA also providesC++ and PythonAPIs so it’s easyto build your owncustom lightingcontrol software. TheWeb site containsdocumentation onthe APIs and thereare code examplesprovided in the repoInside the OLA Web interface, these sliders will let you control yourlights in real time.WWW.LINUXJOURNAL.COM / DECEMBER 2011 / 47

NEW PROJECTSOne of the jaw-dropping creations using OLA: theNixie Mixie Matrix @ Gare St. Sauveur de Lille(FR) by Artist Boris Petrovsky (http://petrovsky.de).Photo: stereomorph.net (http://stereomorph.net).Installation Before I continue, I mustforewarn you of a deviation from the officialdocumentation. The official docs recommendyou use a program called QLC; however, Isimply couldn’t satisfy the library dependencies.Because of the library issues, I had to strayfrom the recommended applications in theLinux how-to and use some recommendations,amusingly, from a Windows how-to—ahow-to that consisted of using VMware torun Linux under Windows (perhaps one ofthe more bizarre troubleshooting methodsI’ve employed). And, may I also give a bigthanks to the authors of this guide, most ofwhich will be guiding the following process.Anyway, this guide still uses Linux inthe end, and it gave the following helpfulcommand for installing the needed libraries:$ sudo apt-get install libcppunit-dev libcppunit-1.12-1➥uuid-dev pkg-config libncurses5-dev git libtool➥autoconf automake g++ libmicrohttpd-dev libmicrohttpd5➥protobuf-c-compiler libprotobuf-lite6 python-protobuf➥libprotobuf-dev zlib1g-devApologies for any “Ubuntuization” withthat command, but the package namesshould point you in the right direction foryour own distro, and if your distro doesn’tuse sudo, with any luck, you might be able tolog in as root and simply drop the sudo fromthe start of the command. Depending onhow your distro works, you may need to runldconfig manually at this point (note, thisrequires root privileges to work properly).From here, let’s download the sourcecode with git and compile it. To save timeand space, I combine these steps into onestream of commands. From your console,enter the following:$ git clone https://code.google.com/p/linux-lighting/ ola$ cd ola$ autoreconf -i$ ./configure --enable-python-libs$ make$ make check(Note: if you have errors after thatlast command, you still may be able tocontinue regardless.)If your distro uses sudo:$ sudo make install$ sudo ldconfigIf your distro uses root:$ su(enter password)# make install# ldconfig48 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

NEW PROJECTSAlthough this obviously is a modest example ofOLA’s potential, if you look at the screenshots,there are impressive examples of using OLA ona much larger and dramatic scale.Usage OLA uses a dæmon that variousprograms then can interact with, the easiestof which is a localized Web interface. To getstarted, most people should be able to getaway with using this fairly simple command:$ olad -l 3If all goes well, your device should benamed at the bottom of the text output(if not, see the basic documentation formore information on switches). Again,going with the best-case scenario, andwhat most users will be getting away with,let’s now look at the Web interface forcontrolling the Arduino RGB Mixer.In your favored Web browser, enter“localhost:9090” into your addressbar. If all goes well, a local OLA Webpage should appear with a redirectedURL that resembles something likehttp://localhost:9090/ola.html.To start interacting with your Arduinodevice, click the Add Universe button.This brings up a list of device ports, with“Open Lighting—RGB Mixer” most likelyat the bottom. Check its box, and assigna number and name of your choosing forthe Universe Id and Universe Name fields,respectively. And presto, you now can playwith your device.Click on the Console tab, and you’llsee the vertical sliders. These turn thebrightness of individual LEDs up and downin real time. Again, perhaps it’s a bit lame,but it gave me the same thrill of interactionI experienced with the Arduino examples.Although this obviously is a modestexample of OLA’s potential, if you lookat the screenshots, there are impressiveexamples of using OLA on a much largerand dramatic scale. Hopefully, this isone of those projects that gains realdevelopment and maturity, becoming anunderground hero. For a personal wishfulfillment, I’d like to see concert lightingbecome significantly cheaper, and ifsmaller independent bands can use this inparticular to realize their artistic visions,OLA will make me a happy man.■John Knight is a 27-year-old, drumming- and bass-obsessedmaniac, studying Psychology at Edith Cowan University in WesternAustralia. He usually can be found playing a kick-drum far too much.BREWING SOMETHING FRESH, INNOVATIVE OR MIND-BENDING? Send e-mail to newprojects@linuxjournal.com.WWW.LINUXJOURNAL.COM / DECEMBER 2011 / 49

READERS’CHOICEAWARDS2011SEE HOW YOUR FAVORITES FARED IN THIS YEAR’S VOTE.SHAWN POWERSThe votes are in, the tallies are counted, the hanging chads have beenevaluated, and we have our winners. This year holds a few surprises,a couple dominant players and as much open source as you can handle.We don’t encourage gambling here at Linux Journal, but if you had an office poolgoing for pizza money, it’s officially too late to make your wager.50 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

Best Linux DistributionUbuntuRunner-up: Debianapt, apt and more apt this year in the distribution category.Although it’s no surprise that Ubuntu remains king of the distros,it’s nice to see Debian, the “father” of Ubuntu gaining some ground. Whether it’s because LinuxMint is making Debian more user-friendly or because folks are drawn to the appeal of Debian’sstability, it got just about half the votes of all the Ubuntu variants combined. Way to go Debian!Oh, and of course, congratulations to the winner and still-champion, Ubuntu.Best Distribution for Netbooks/Limited HardwareUbuntu Netbook RemixRunner-up: Android and Debian (tie)Although Ubuntu is streamlining its versions and making the desktop screen function similarlyto the Netbook screen, Ubuntu Netbook Remix still garnered the most votes this year. Will thepush to Unity make next year’s Readers’ Choice look a little different? The future awaits. Ourrunner-up last year was Android, and this year, Android is still our runner-up, but it shares thesilver medal with Debian. Why is Debian getting so much attention this year? For the samereason soda-pop companies are releasing “throw-back” versions of their drinks with real sugar,sometimes the tried-and-true operating systems just taste a little sweeter.Best Mobile OSAndroidRunner-up: MeeGoWith the death of Maemo, the abandonment of MeeGo and thediscontinuation of webOS, the obvious winner this year is Androidwith 80% of the vote. MeeGo takes enough of the remainingvote to get our runner-up spot, but it’s a bitter prize, as MeeGo’sfuture looks pretty bleak. Will Android get another open-sourcecompetitor? Will the lack of open competition stifle Androidinnovation? Only time will tell. For current Linux-based handsets, however, Google’s Androidtruly can say, “All your base are belong to us.”WWW.LINUXJOURNAL.COM / DECEMBER 2011 / 51

FEATURE Readers’ Choice Awards 2011Best Desktop EnvironmentGNOMERunner-up: KDELast year it was a tie. This year, our back-and-forth battle falls to GNOME as the best desktopenvironment. Due to the timing of the GNOME 3 release, it’s hard to tell if the victory is becauseof version 3 or in spite of it. Nonetheless, GNOME ekes a victory with a 3% margin over KDE.The next-closest desktop environment is XFCE with less than one-third the votes of either ofthe big two. With such big contenders for first and second, however, that third-place spot issignificant, and XFCE is gaining ground. We think it’s one to keep an eye on next year.Best Web BrowserFirefoxRunner-up: Chrome/ChromiumAfter its huge popularity spike last year, we thought the race for bestbrowser would be neck and neck this year. The Firefox team stepped upits game, however, and this year made several major revisions. Although Chrome/Chromium isa major contender and even gained a few percentage points, Firefox still dominates with morethan twice as many votes. As a Firefox user myself, it doesn’t surprise me to see my favorite fieryfox on top, but it’s hard to argue with Chrome/Chromium’s lightning-fast response time. As moreand more extensions are being ported to Google’s browser, Firefox has some real competition.Hopefully, that competition will inspire greatness from both teams. As users, we can only benefit!Best E-mail ClientThunderbirdRunner-up: Gmail Web ClientLike its foxy-browser sibling, Thunderbird takes top spot again this yearin the e-mail category. Now that Canonical has adopted Thunderbird asits default e-mail client in Ubuntu, we see the popularity rising for our blue-birdie friend. Stillhanging on tightly to second place is Gmail. Is Gmail an app? We’re not sure, but it certainlydoes get votes as the best e-mail client. Because Thunderbird can access Gmail servers easily,it’s possible this category blurs the line a bit, as users simply can use both. When it comes topicking a favorite though, Thunderbird is the clear victor with more than twice as many votes asour runner-up.52 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

Best IM ClientPidginRunner-up: SkypeThe results are similar, but the trend obviously is shifting in our Best IM Clientcategory. Pidgin takes the number one spot with a full half of your votes, whileSkype barely squeaks out a second-place win with 15%. Although its video chat ishard to beat, we think Skype lost some points due to its purchase by Microsoft.What does that new ownership mean for the future of Skype? No one knows forsure, but it has Linux users scrambling to find alternative video chat clients “just in case”.Best IRC ClientPidginRunner-up: X-ChatFor years I’ve been touting the awesome IRC features Pidgin boasts. It’s nice to see Pidgintake first place again as favorite IRC app. As my geek-fu has matured, so has my chattingpreference, however, and I skipped right over our second-place IRC client X-Chat. I’m now usingIRSSI for my textual communication needs. Although my preferences seldom represent thoseof the masses, I’ll be shaking my IRSSI pom-poms next year for the awesome underdog. Creditwhere credit is due, however; it’s hard to beat the flexibility of Pidgin and the huge feature setof X-Chat. It’s clear why they are the Readers’ Choice victors.Best Microblogging ClientGwibberRunner-up: ChoqokOur top two microblogging clients from last year retain theirstatus as class favorites. Gwibber and Choqok, GNOME- andKDE-native, respectively, garnered the most votes again this year.The ever-popular AIR application TweetDeck is right on their heels,however, and it’s throwing in its cross-platform flexibility to make the contest interesting. Nativeclients (with odd names) still hold favor among readers, but those fancy Adobe AIR and HTML5alternatives slowly are gaining ground. Who will win next year? We’ll be sure to tweet theanswer when the time comes.WWW.LINUXJOURNAL.COM / DECEMBER 2011 / 53

FEATURE Readers’ Choice Awards 2011Best Office SuiteLibreOfficeRunner-up: OpenOffice.orgThe king has been dethroned! Well, sorta. Yes, technically the newcomer LibreOfficestomped on the former-champion OpenOffice.org. Because LibreOffice is a fork ofOpenOffice.org, however, it seems like we should have an asterisk in there somewhere.Shortly after Oracle bought Sun Microsystems, OpenOffice.org was forked. The goodnews for users is that LibreOffice has a large dev crew, and updates and featureenhancements are coming out at a really nice rate. The king is dead; long live the king!Best Single Office ProgramOOWriterRunner-up: AbiWordWe’re not entirely clear if OOWriter is referring to the OpenOffice.org version or theLibreOffice version of the word-processing program. Basically, we’re considering thewinner, “the program that saves to .ODT by default”, and we think that covers it. Therunner-up again this year is not a member of the LibreOffice/OpenOffice.org suite, butrather the standalone AbiWord word processor. In fact, AbiWord is what this articleis being typed on as we speak. Will the LibreOffice takeover change the favorite appcategory in the future? Based on voting this year, we guess not. It’s nice to see ourunderdog-favorite AbiWord continue to get votes though.Best Digital PhotoManagement TooldigiKamRunner-up: PicasaThe past few years have been an epicbattle between these two programs. This year, we think the contestants might be a littletired, because although they still are clearly the top two choices, their popularity prettymuch has leveled off. digiKam ekes out a victory by less than two percentage points,which means it’s hard to go wrong when you pick one of these two. Whichever youchoose, it’s bound to be better than my solution: shoebox full of photos.54 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

Best Graphics Design ToolGIMPRunner-up: InkscapeGIMP kicks butt and takes names, as it scores two-thirdsof the total votes this year. Inkscape remains in second place,but it’s a very distant second. There certainly are other options available, but time andtime again, we turn to GIMP for editing those photos. Although the learning curvefor GIMP can be a bit steep, the same often is said for Linux itself. And like Linux, thereward is great.Best Audio ToolAudacityRunner-up: ArdourWhen I saw the Best Audio Tool category, my first instinct was to vote for speakers.I’m clearly in the minority, however, as 85% of you instantly thought of Audacity.Ardour is down another couple percentage points this year, but we still gave it runner-upstatus. We don’t want Audacity to get too prideful after all.Best Audio PlayerAmarokRunner-up: VLCIt’s still clear readers love Amarok.In a surprisingly close second place thisyear is VLC. Although not normallythought of as an audio player, it doesthe job well enough to get just 7%fewer votes than Amarok. And lastyear’s runner-up Rhythmbox? Sadly, it’sfar in the distance behind these twofront-runners.WWW.LINUXJOURNAL.COM / DECEMBER 2011 / 55

FEATURE Readers’ Choice Awards 2011Best Media PlayerVLCRunner-up: MPlayerSeeing VLC take runner-up in the audio player category makesthis victory a no-brainer. VLC plays just about any sort of video youthrow at it. I think if you shove a paper flip book into your floppydrive, VLC will animate it on-screen for you. VLC takes such a huge margin this year,we almost didn’t include MPlayer as a runner-up. VLC is the favorite, without question.(Note: we don’t actually recommend shoving a paper flip book into your floppy drive.)Best Bookmark Sync ToolFirefox SyncRunner-up: Chrome BookmarksOur bookmark sync category completely rewrote history and gaveus two brand-new winners. Firefox Sync, now built in to the browser,takes the victory handily with twice the votes of the runner-up, ChromeBookmarks. This split makes absolute sense, because Firefox beat Chrome in the browser warby the same margin. In fact, if these numbers were different, it would cause our highly scientificvoting process to look suspect. As it is, for Firefox users, we recommend Firefox Sync, and forChrome users, we recommend Chrome Bookmarks. Feel free to call me Captain Obvious.Best On-line Collaboration ToolGoogle DocsRunner-up: WikisWe didn’t title this “Most Popular Collaboration Tool”,because as painful as it is, the majority of on-line collaborationtends to be e-mail messages with subjects like “RE:Fwd:Fwd:Re:This one Re: Final FWD: Final2” and ugly multi-fonted .doc files.Although popular, that’s definitely not ideal. Google Docs takes the spoils of war againthis year with its ever-improving feature set for on-line collaboration. It’s even possibleto watch as someone else edits a document. For everything else, wikis are still popular.Easy to edit and easy to maintain, wikis are a godsend for living documents.56 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

Best Cloud-Based File StorageDropboxRunner-up: Ubuntu OneWhen it comes to cloud storage, it’s hard to beat Dropbox.Although security is an often-touted concern with the cloudstoragebehemoth, ease of use trumps those concerns. Ubuntu One is a distantsecond to the cross-platform, simple-to-use Dropbox. I’d put my Dropbox referralcode here to get some free space, but I suspect our editor would frown on such athing, plus you’d all likely flog me.Best Kid-Friendly ApplicationTux PaintRunner-up: GComprisBill Kendrick’s Tux Paint continues as the crowdfavorite for 2011. Whether you’re a kid of 5 or 50,it’s hard not to smile when creating paintings withBill’s user-friendly application. GCompris is noslouch in the kid-friendly category either and quitenicely takes second place with its educationalfocus and lively graphics.Best GameWorld of GooRunner-up: Battle for WesnothFor the first time in the history of histories, FrozenBubble is not the most popular game! In fact, FrozenBubble didn’t even take second this year, as it lostto Battle for Wesnoth by half of a percentage point.(Normally we’d consider that a tie, but Battle for Wesnoth deserves recognition forbumping off the Bubble.) World of Goo is a game similar in addictiveness to FrozenBubble, but with better graphics and more modern gameplay. If you’re a casualgamer, check out World of Goo, it’s really Goo-ood.WWW.LINUXJOURNAL.COM / DECEMBER 2011 / 57

FEATURE Readers’ Choice Awards 2011Best MonitoringSolutionNagiosRunner-up: OpenNMSI misspelled “monitoring” as I wastyping this section, and AbiWord’srecommendation for correction was“minotaur”. Although I wouldn’t arguea minotaur would be a wonderfulmonitoring solution for manycircumstances, when it comes to computerhardware, Nagios is a little better, and farmore popular. OpenNMS is a newcomerto our victory circle, and although it’s farbehind Nagios, it still scored quite well. “Minotaur”, as it were, got very few votes.RNagios XI is the most powerfulmonitoring solution for yourchallenging IT environment.Features Include:- Dynamic PHP Interface- Advanced Reporting- Hypermap- Data Visualizations- Integrated Performance Graphs- Database Backend- Configuration GUI- Monitoring Wizards- Professional Support20% DiscountFor LinuxJournal Readers58 / DECEMBER 2011 / WWW.LINUXJOURNAL.COMFor more information, visit:http://go.nagios.com/linuxjournal

Best DatabaseMySQLRunner-up: PostgreSQLIt may not be the most-exciting topic around, but databases make the world goround. MySQL with its dolphin mascot takes first place again this year, with more thantwice as many votes as its closest competition, PostgreSQL.Best Backup SolutionrsyncRunner-up: tarWe geeks like our command line, and to back up stuff, nothing can beat rsync and tar.rsync is three times more popular than tar based on reader votes, but nothing else comesclose when it comes to backup. For two years running, we don’t need no stinkin’ GUI!RNagios Certifications Now Available!Official certifications help spotlight your skills and prove you’rean admin who can wield your Nagios knowledge with precision.CertifiedAdministrator TMRCertifiedProfessional TMRCertification Features Include:- Proctored, Web-Based Exams- Professional and Administrator Levels- Online Certification Verification- Discounts on Products and OfferingsFor more information, visit:http://go.nagios.com/linuxjournal

FEATURE Readers’ Choice Awards 2011Best Virtualization SolutionVirtualBoxRunner-up: VMwareAlthough the Oracle purchase certainly affected theOpenOffice.org popularity, VirtualBox’s new ownershipdoesn’t seem to bother anyone. VirtualBox is more than fourtimes as popular as the distant runner-up, VMware. We evenlumped the VMware options together, and they received only a cumulative 15% ofthe vote. VirtualBox beat virtually every other option hands down.Best Revision Control SystemGitRunner-up: SubversionThe Linus-Torvalds-created Git remains in the number one spot this year, as itwidens the gap a bit more from the runner-up, Subversion. Either will do the job, butSubversion is becoming more and more the underdog. Perhaps having Linus on yourside is an advantage in the Open Source world!Best Open-Source ConfigurationManagement ToolPuppetRunner-up: OpenQRMIn another repeat performance from last year, Puppet takestop spot for configuration management. If you administergreater than zero servers, you will benefit from using a toollike Puppet. If you’re managing fewer than zero servers, well,we’re not sure what that means. You probably need Puppet to manage your countingskills. Whatever your reason, configuration management is a hot topic, and Puppet isthe hottest.60 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

Best Programming LanguagePythonRunner-up: C++A three-time winner in our best programmingcategory, Python continues to dominate. Close on itsheels this year, however, is C++. In fact, a scant 6%separated the two. It’s quite obvious, however, thatour readers don’t suffer from ophidiophobia in theleast—hiss.Best Scripting LanguagePythonRunner-up: BashIt hardly seems fair that Python gets both best programming and best scriptinglanguage, but I suppose excellence knows no bounds. A newcomer to our runner-upcircle is Bash, the only language I can program with at all. Hats off to Python though,as it takes both categories again this year.Best IDEEclipseRunner-up: vimAdmit it, you weren’t surprised at allto see Eclipse in the top seat. Seeingvim in the copilot seat, however, was anostalgic treat for me. Eclipse is incrediblyextensible and remarkably quick in mostenvironments. Nothing can beat thesimplicity of vim, however, and it took only 9% fewer votes than Eclipse. There is nodenying it, we’re geeks.WWW.LINUXJOURNAL.COM / DECEMBER 2011 / 61

FEATURE Readers’ Choice Awards 2011Best Platform for Developing Rich InternetApplicationsHTML5Runner-up: Adobe AIRHTML5 is the new kid on the block this year, and it managed to take 80% of thevote! Adobe AIR gets an honorable mention, but only on principle. It was an entireorder of magnitude less popular than HTML5.Best Package Management ApplicationaptRunner-up: SynapticIt’s no surprise that with Ubuntu and Debian in the top spots for distributions,apt would win handily for package management. Synaptic is a far-off second place,with dozens of others taking up the rear. But, our favorite response for this topicwas configure; make; make install.Best ContentManagement SystemWordPressRunner-up: DrupalOur Webmistress, KatherineDruckman, is a die-hard Drupalfan—and for good reason, theentire Linux Journal site uses it andhas for many years. Perhaps justto prove she didn’t rig the voting,WordPress takes top spot againthis year by a fairly narrow marginto Drupal. The great thing aboutopen source is that it’s hard to losewhichever route you take.62 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

Best Linux-Friendly Web-Hosting Company“Other”Runner-up: 1&1 and GoDaddy.com (tie)We’re taking these results as a good sign, in that perhaps so many Web-hostingcompanies are Linux-friendly, it was hard to pick one over the other. So in reality,GoDaddy and 1&1 took more votes than any other single Web-hosting company.Because their single-digit “victories” seemed a bit strange to celebrate, we gavethem runner-up status to all the other options you sent in. Feel free to cry foul; it justseemed like the logical thing to do.Best LinuxLaptop VendorDellRunner-up: ASUSDell still grabs the top spot here,similar to last year. In a second-placeupset, however, ASUS grabs thesilver medal, and Lenovo (last year’srunner-up) didn’t even make thechart. This category is becoming lessand less important every year, onlybecause Linux is working on moreand more laptops out of the box. Wethink that’s a great problem to have.Best Linux Desktop Workstation VendorDellDell is on everyone’s love-letter list this year and took the desktop workstationcategory by storm. In fact, the competition was so lopsided, we can’t even declare arunner-up. Dell gets all the penguin love.WWW.LINUXJOURNAL.COM / DECEMBER 2011 / 63

FEATURE Readers’ Choice Awards 2011Best Linux Server VendorIBMRunner-up: DellNot to be outdone, IBM pulls through with a verynarrow victory over the ever-popular Dell in our servercategory. When it comes to server racks, our readers trust Big Blue over anyone else(but just barely).Best Linux BookLinux in a Nutshell by Ellen Siever et al.Runner-up: Just for Fun: The Story of an AccidentalRevolutionary by Linus Torvalds and David DiamondLinux in a Nutshell from O’Reilly remains yourfavorite book again this year. In fact, it took twiceas many votes as the number two favorite, Just forFun. We mentioned Just for Fun in last year’s Readers’Choice awards, and apparently many of you took thehint and bought it.Best Linux Journal ColumnHack and / by Kyle RankinRunner-up: Work the Shell by Dave TaylorKyle takes the rest of us to task again this year, stealingthe number one spot for his Hack and / column. It’s hardto hate Kyle, because he truly is a helpful, humble, easygoingguy. The second spot goes to an equally awesomeindividual, Dave Taylor. I know I’m biased, but picking afavorite Linux Journal column is like picking a favorite flavorof ice cream—it’s hard to go wrong!64 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

FEATURE Readers’ Choice Awards 2011Best Other Linux-Based GadgetAmazon KindleRunner-up: TomTom Navigation SystemThe Kindle easily takes its place as your favorite Linux gadget thisyear. We may have to change our categories a bit next year, as thetablet/gadget/smartphone categories are starting to blend together.However you slice it, the Kindle wins this year. And if you needdirections to the store in order to buy a Kindle? We recommend theTomTom Navigation System, also running Linux and also one of your favorites.Best New Open-Source Project(released in 2010 or 2011)LibreOfficeAlthough certainly standing on the shoulders of its progenitor, LibreOffice continuesto progress at an impressive rate. Because the fork is technically a new project, yourwrite-in votes were counted, and LibreOffice wins the coveted spot as best new opensourceproject.Product of the YearGNOME 3And the moment you’ve all been waitingfor...the winner is...GNOME 3! Although verycontroversial and barely edging out Android,GNOME 3 takes our product of the year title for2011. GNOME 3 represents a drastic change inthe way we compute on the desktop, and like itsrelative Unity, it has some people shaking theirheads in frustration. You’ve proven, however,that change isn’t always a bad thing, andGNOME 3 wins!66 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

Be sure to follow our Web site this year,as we explore some of these winners, andperhaps watch the runners-up to see ifthey edge out the current incumbents. Asfor me, I’ll be playing World of Goo. I hearit’s rather good.And finally, a big thanks to everyonefor participating in the voting. If you haveideas for new categories you’d like us toinclude for Readers’ Choice 2012, sende-mail to ljeditor@linuxjournal.com.■Shawn Powers is the AssociateEditor for Linux Journal.He’s also the Gadget Guy forLinuxJournal.com, and he hasan interesting collection ofvintage Garfield coffee mugs.Don’t let his silly hairdo foolyou, he’s a pretty ordinary guyand can be reached via e-mailat shawn@linuxjournal.com.Or, swing by the #linuxjournalIRC channel on Freenode.net.GPU Computing SpecialistWorkstations with upto 2048 GPU cores4U Servers with upto 4096 GPU coresPreconfiguredCPU/GPU ClustersWSCA Contract# B27157GSA Schedule Contract# GS35F-0400TUS Dept. of Energy BPA# DE-EM0000349See Ace atBooth #2120www.acecomputers.com1-877-223-2667

INDEPTHComplexity,Uptime and theEnd of the WorldPoorly implemented monitoring systems can drive an administratorcrazy. At best, they are distracting. At worst, they’ll keepwhoever is on pager duty up for nights at a time. This articlediscusses the best practices for designing systems that willkeep your systems up and stay quiet when nothing is wrong.MICHAEL NUGENTAfter being in the computer industry for20-odd years, I’ve come to realize thereis a single thing everyone can agree on:no matter how new, how stable or howawesome any piece of technology is, itwill break.Fortunately, system administrators planfor these things. Whether it’s a redundantserver in the data center or a secondavailability zone in EC2, the first and bestway to ensure uptime is to decrease thenumber of single points of failure acrossthe network. There are drawbacks tothis approach though. Increasing a Webcluster from one to ten boxes decreasesthe chance of hardware failure takingdown the entire site by a factor of ten.Although this increases redundancy, italso dramatically increases the expenseand complexity of the network. Insteadof running a single server, there’s now aseries of boxes with a shared data storeand load balancers. This complexity comeswith drawbacks. It’s ten times as likelythat hardware failure will occur and asystem administrator will wake up, andthat only counts the actual Web servers.Whether you’re in a data center or in thecloud, this kind of layering of servicessignificantly increases the chances that asingle device will go down and alert in themiddle of the night.Preventing this kind of thing is usuallyhigh on a system administrator’s list of68 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

INDEPTHdesires, even if it tends to be pushedlower on the priority list in practice.Waking up in the middle of the night tofix a server or piece of software is badfor productivity and bad for morale. Youcan take two steps to help make sure thisdoesn’t happen. The first is to implementthe necessary amount of redundancywithout increasing the complexity of thesystem past what is required for it torun. The second step is to implement amonitoring system that will allow you tomonitor exactly what you want as opposedto worrying about which individual box isusing how much RAM.The End of the World methodologyis a thought experiment designed tohelp choose the level of redundancy andcomplexity required for the application. Ithelps determine acceptable scenarios fordowntime. Often when you ask peoplewhen it’s acceptable for their sites to bedown, they’ll say that it never is, but that’snot exactly true. If an asteroid strikesEarth and destroys most of the humanrace, is it necessary for the site to stayup? If the application is NORAD, maybeit is necessary, but for Groupon, not somuch. That kind of uptime requires massiveinfrastructure placed in strategic locationsaround the globe and the kind of capitalinvestments and staffing to which onlylarge governments usually have access.Backing off step by step from this kindof over-the-top disaster, you can findwhere the acceptable level is. What if thedisaster is localized to just the continent?Is it acceptable to be down at this time?If the site is focused on those customers,it may be. If the site is an internationaltool, such as Amazon or Google, possiblynot. What if it’s local to the data centeror availability zone where your boxes arekept? Most shops would like to stay upeven if a backhoe cuts the power to theirdata center.When the problem is framed this way, itbecomes obvious that there is an acceptablelevel of downtime. Administrators canWWW.LINUXJOURNAL.COM / DECEMBER 2011 / 69

INDEPTHfind the sweet spot between uptime andcomplexity. Finding the outer boundsof these requirements will uncover therequirements for monitoring the serviceas a whole. Notice that this is a serviceand not a server. Although it’s easy tomonitor whether a network interfaceis available, it’s far more interesting tomonitor the health of an entire cluster.In our ten-server cluster, if www6goes down on a cluster that gets 40%utilization at night, it’s probably notworth getting up for. If the entire Webservice goes down, that’s something thatneeds to be acted upon immediately.A monitoring system is basically ascheduler and data collection tool thatexecutes checks against a service andreports the results back to be presentedon a common dashboard. It seemslike one of those innocuous pieces ofsoftware that just runs in background,like network graphs or log analysis, butit has a hidden ability to hurt an entireengineering department. False positivescan wake people up in the middle of thenight and cause ongoing dread of going onpager duty. This results in people puttingthings in maintenance mode to quiet thefalse positives and can end up with anunnoticed failure of services.Dealing with false positives often isTry Before You Buy!Benchmark Your Code on Our GPU Cluster withAMBER, NAMD, or Custom CUDA CodesConfigure your WhisperStation or Cluster today!www.microway.com/tesla or 508-746-7341NEW Microway MD SimCluster with8 Tesla M2090 GPUs, 8 CPUs and InfiniBand30% Improvement Over Previous TeslasGSA ScheduleContract Number:GS-35F-0431N

INDEPTHmore of a policy problem than a designproblem. Choosing what to monitor is farmore important than choosing how tomonitor it. Many companies have a historyof monitoring things like CPU and RAMusage. They feel that sometimes spikesare precursors to crashes, so alerting onthem is reasonable. The problem here isthings that can cause the computer touse CPU and RAM, and most of them arewithin the normal bounds of an operatingsystem. When the system administratorchecks on the box, the resource is inuse, but the application is functioningwithout a problem. Unless there is a cleardocumented link between RAM overa certain level and a crashing service,skipping on alerts for this kind of resourceuse leads to far fewer false positives.Monitors should be tied to a defined goodor bad value with respect to a particularproduction service.Another path that leads to a largenumber of false positives is usingpercentages in differently equipped boxes.For example, if a system has a 137G drivethat’s 95% full, it has only around 6G free.On sites with heavy traffic or sites with alot of instrumentation in the code, 6G cango pretty quickly. Applying this monitor tothe same Web server with a 2TB disk seemslike less of an emergency. Leaving “only”Microway’s Proven GPU ExpertiseThousands of GPU cluster nodes installed.Thousands of WhisperStations delivered.Award Winning BioStack – LSAward Winning WhisperStation Tesla – PSC with 3D‘11AWARDBESTBest NewTechnologyns/Day (Higher is Better)CPU + GPUCPU Only1.070.332.020.653.541.301 Node2 Nodes 4 NodesNAMD F1-ATP Performance GainVisit Microway at SC11 Booth 2606

INDEPTH100G free on a system overnight is usuallynot a problem. If the average disk use fora day of work for a particular box is 5G,monitoring for 15G left and only allowingalerts for it during business hours will givethree days notice. Alerts this far aheadof time let the system administrator plandowntime for the system if it is required, sothat the server can be maintained withouttaking the supported service down.The two most popular open-sourcemonitoring systems are Zenoss andNagios. Both of these systems offersimilar monitoring capabilities. Zenossprovides more functionality and easeof use, incorporating some basic autodiscoveryof nodes, built-in RRD graphing,syslog management and the ability todeduplicate events. Nagios provides alarger community and lighter install thanZenoss that allows administrators to usetheir own graphing solutions withoutduplicating software. The best part isthat they have a common format formonitoring scripts—the processes that dothe actual checking of services.Although both systems come withbasic templates for monitoring HTTPports with similarly popular services,much of the power of these systemscomes from the ability to write customscripts. This is a great way to check notonly that a Web server is up, but alsothat the application itself is working. Thefollowing is an example of a script thatwill monitor the success of Hudson jobsby calling its JSON API:#!/usr/bin/env ruby# Call as:# check_hudson_job.rb ${jobname} ${hostname}require 'rubygems'require 'json'require 'net/http'jobname = ARGV[0]hostname = ARGV[1]url = URI.parse("http://#{hostname}/job/#{jobname}/➥lastBuild/api/json")res = JSON.parse(Net::HTTP.get_response(url).body)lastResult = res["result"]if lastResult == "SUCCESS"puts "OK|Status=0"exit(0)elsefailurl = URI.parse("http://#{hostname}/job/➥#{jobname}/api/json")failres = JSON.parse(Net::HTTP.get_response(failurl).body)health = failres["healthReport"][0]["description"]puts "Job #{jobname} broke: #{health}"exit(1)endThe monitoring system calls the codewith command-line parameters of thename of the job and the name of the host.The code then looks for the result fromthe Hudson server and checks for success.The return value and exit code are how themonitoring script replies to the monitoringsystem. A nonzero exit code indicates afailure, and the return value is a string72 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

INDEPTHHooking up the EC2 command-line programs tothe monitoring service will allow new boxes tobe launched if some are experiencing problemsdue to resource starvation, load or programscrashing on the box.that the system displays as the reason forthe failure. On Zenoss, this is also used indeduplication. On success, the monitoringscript has an exit code of 0 with a stringreturned in a special form for the system toprocess (see code).Using this structure, system administratorscan work with developers to build customURLs that the monitoring system can accessto determine the health of the applicationwithout worrying about every system inthe set.It may seem hard to swallow thatit’s acceptable to leave a box downovernight. It may be the first in acascading series of failures that causemultiple servers to go down, eventuallyresulting in a downed service, but thiscan be addressed directly from theload balancer or front-end applianceinstead of indirectly looking at the boxesthemselves. Using this method, thealert can be set to go off after a certainnumber of boxes fail at certain times ofday, and there is no need to solve harderproblems, such as requiring each box toknow the state of the entire cluster.So far, the design for the systems hasbeen fairly agnostic as far as geographiesand cloud footprint. For most applications,this doesn’t make a lot of difference.Usually, with multiple geographies,each data center has its own instanceof the monitoring system with each onemonitoring its siblings in the other locations.Operating in the cloud offers greaterflexibility. Although it still is necessary tomonitor the monitoring system, this can bedone easily using Amazon’s great, but farless configurable system to monitor Nagiosor Zenoss EC2 instances.What really stands out about Amazon’scloud is that it’s elastic. Hooking up theEC2 command-line programs to themonitoring service will allow new boxesto be launched if some are experiencingproblems due to resource starvation,load or programs crashing on the box. Ofcourse, this needs to be kept in check, orthe number of instances could spiral outof control, but within reasonable bounds,launching new instances in place ofcrashing or overloaded ones from inside ofa monitoring script is relatively easy.Here is an example of a script thatmonitors the load of a Hadoop cluster andadds more boxes as the number of jobsrunning increases:WWW.LINUXJOURNAL.COM / DECEMBER 2011 / 73

INDEPTH#!/bin/bash# Call as:# increase_amazon_set.sh ${threshold} ${AMI}THRESHOLD=$1AMI=$2NUM_JOBS=`/opt/hadoop/current/bin/hadoop job -list |➥head -1 | awk {'print $1'}`if [[ $NUM_JOBS -gt $THRESHOLD ]] ; thenecho "Warning: $NUM_JOBS running, increasing cluster size by 3"ec2-run-instances $AMI -n 3 --availability-zone us-east-1aexit 1;elsefiecho "OK|Status=0"exit 0;This follows the same format as theprevious script, passing in variables fromthe command line and returning valuesto the monitoring system using the exitcondition and returned strings. The bigdifference here is that you’re not justmonitoring a problem and passing it off to asystem administrator to act on it. This scriptacts as an orchestrator, attempting to fixthe problem it sees. Although care shouldbe taken to place proper bounds on theway this works, and the computer shouldnot be able to run amuck on the network,this kind of intelligent scheduler can be apowerful tool in automating tasks.Although the idea of setting up a newmonitoring system from scratch with greatalerting rules and intelligent orchestrationis a great idea, it’s often just not possible.Most organizations have a monitoringsystem in place already, and often it’s full ofold alerts and boxes that have been placedin maintenance mode because they’remore noisy than broken. If this is the case,it’s time to cut out the cruft. Delete allthe current alerts and take everything outof maintenance mode that isn’t actuallyundergoing maintenance. Take the top tennoisy and badly behaved devices, and eitherstop monitoring the items that are provokingfalse positives or rewrite the scripts so theyprovide more meaningful data. When thesefirst ten are under control, move to the nextgroup. It may take a few iterations over afew days, but in the end, you’ll care moreabout the messages coming from what couldbe a very powerful tool for you.Monitoring systems often are overlookedas a required annoyance, but with a little bitof effort, they can be made to work for you.Monitoring for services, looking at clusteredapplications and alerting only on actual errorsthat can be handled provide real metricsto use for capacity planning and let systemadministrators sleep through the night so thatthey can be more proactive from day to day.■Michael Nugent has spent a good deal of his time designinglarge-scale solutions to fit into a tiny budget and leveraging Linuxto fulfill the roles that typically would be filled by large commercialappliances. Recently, Michael has been working to designmap-reduce clusters and elastic cloud systems for growingstartups in the Silicon Valley area. When not building systems,he likes sailing, cooking and making things out of other things.Michael can be reached at michael@michaelnugent.org.74 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

Security Threats 2012:Secure & Empower Today’s EnterpriseProtection in a Cloud, Collaboration, andConsumerization EnvironmentJanuary 23, 2012 - Pre Conference WorkshopJanuary 24-25, 2012 - ConferenceWashington Plaza Hotel, Washington, DCThe consumerization of IT is in full tilt. The new application paradigm offers tremendouspower – but challenges established security, risk, and compliance practices. Yesterday’ssolutions can’t meet today’s IT reality. Cloud computing, mobile apps, always–onconnectivity, and social media force security professionals to develop new, morecomprehensive solutions. Providing effective, unobtrusive security is the true modernday IT objective. Security Threats 2012 presents the best practices for tomorrow’s securityenvironment.At this forum, leading-edge IT and security experts will discuss how they simultaneouslyprotect and empower their businesses. There are few unbiased IT/security discussions inthe marketplace, however, at this intimate forum you’ll have the opportunity to learn fromthought-leaders making these daily decisions.Sponsorship and Exhibiting OpportunitiesIf you are interested in sponsoring, speaking or exhibiting at this event,please call 212-532-9898 or email info@opalevents.orgRegisterTo register, visit us online at www.opalevents.orgor email us at marketing@opalevents.orgREF CODE: SETEA1203

INDEPTHMariaDB/MySQL,PostgreSQL andSQLite3: ComparingCommand-LineInterfacesDon’t be afraid of using your chosen database’s command-lineclient. DANIEL BARTHOLOMEWI might as well say this upfront: I don’t like using GUI(aka non-command-line orgraphical) tools with mydatabases. This is likely becausewhen I first learned it waswith command-line tools, buteven so, I think command-linedatabase tools often are thebest way to interact with adatabase manually.Two of the most populardatabases in use on Linux areMySQL and PostgreSQL. Eachof them have very useful, ifslightly different, commandlineclients. If you ever needTo Serve...or NotPostgreSQL and MariaDB have what is known asa client/server architecture. Clients connect to theserver, and although client and server often areinstalled together and you may think of them as asingle entity, they actually are not. The client doesnot need to be run on the same machine as theserver. The MariaDB server is called mysqld, and italways is running while the server is up. Likewise,the PostgreSQL server is called postgres.SQLite does not have a client/server architecture. Thereis just the database you are using, which is a localfile, and client programs, which can interact with it.76 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

INDEPTHto move betweenthese two databases,or if you’re new todatabases in general,a comparison of thetwo is helpful.But, because atwo-horse race isn’tas thrilling as a threehorseone, I wantedto include a thirddatabase commandlineclient in mycomparison. I choseSQLite, because it isarguably the mostpopular database inthe world. You probably have several SQLitedatabases on your local computer right now.The command-line client is nice too.Also, I use MariaDB instead of MySQL in myexamples, because that’s what I have installed,and because I like the improvements MariaDBincludes in both the command-line client andin the database server. MariaDB and MySQLare very compatible, and my examples aresimple, so whenever I mention MariaDB, youcan assume it applies to MySQL as well.Figure 1. The MariaDB, PostgreSQL and SQLite3 Clients in Actiondistribution’s documentation for instructions.On Ubuntu, you can install all three withthe following:sudo apt-get install mariadb-server postgresql sqlite3Other Linux distributions are just as easyfor the most part. (You need to have addedthe appropriate MariaDB Ubuntu repositoryfor the above to work. Instructions are onthe MariaDB downloads page.)InstallationI won’t go into how to install MariaDB,MySQL, PostgreSQL or SQLite3 here. Mostdistributions have packages for them, and inthe case of MariaDB, there are packages forDebian, Ubuntu, Red Hat and a generic Linuxbinary available from its download page.See the documentation for each and yourBasic Client CommandsThe client programs for MariaDB,PostgreSQL and SQLite3 are mysql,psql and sqlite3, respectively. I’ve listedseveral useful commands for each clientin Table 1. The first entry shows the basiccommand used to connect to a database;however, each client has several options.WWW.LINUXJOURNAL.COM / DECEMBER 2011 / 77

INDEPTHTable 1. MariaDB/MySQL, PostgreSQL and SQLite Client Cheat SheetTask MariaDB/MySQL PostgreSQL SQLiteConnect to a database mysql psql sqlite3 Client help help contents \? .helpSQL help help contents \h n/aList databases SHOW DATABASES; \l .databasesChange database USE \c n/aList tables SHOW TABLES; \dt .tablesShow table info DESCRIBE ; \d .schema Load data LOAD DATA INFILE '' \i .import Export data SELECT ... INTO OUTFILE '' \o .dump Exit the client quit (or exit) \q .exitThese include (in the case of MariaDB andPostgreSQL) options for specifying the user,password and database host server. Youwill need these often, so refer to the manpages for the clients for what they are andhow to use them. Some of the commandslisted in Table 1 have extended options;refer to the documentation for details.The first time you connect to a newlyinstalled MariaDB or PostgreSQL database,you need to connect as the databasesuperuser because you likely have not setup any other users.To launch a freshly installed MariaDBmysql client, do the following:Creating and Deleting a DatabaseJust installing database clients and/or serversdoes not automatically give you a databaseto work with. For MariaDB and PostgreSQL,a database can be created either with theclient or with an external utility.In MariaDB and PostgreSQL, to create adatabase called library, the command is:CREATE DATABASE library;To connect to this newly created databasein MariaDB, do:USE librarymysql -u root -pIn PostgreSQL, do:You will be prompted for the password youentered during the package install process.To launch a freshly installed PostgreSQLpsql client, do the following:sudo su - postgrespsql\c libraryTo delete the newly created librarydatabase, drop it with:DROP DATABASE library;78 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

INDEPTHI shouldn’t have to say this, but becareful with the previous command. If youjust dropped the library database, create itagain. You’ll need it later to follow alongwith the examples in this article.In SQLite3, there is no database server,and databases are just regular files, oftenwith a .db extension. To create a database,name it on the command line when youlaunch the client, and if it doesn’t exist, theclient will create it, like so:sqlite3 library.dbTo remove an SQLite3 database, justremove it like you would any other file(with rm or via your file manager).Managing Users and PermissionsThere isn’t space to go into the details ofhow to create and manage the permissionsof database users here. Refer to the MariaDBand PostgreSQL documentation for details.I will continue to use the default superuseraccounts for the examples here.There is no internal database user or userpermissions management with SQLite3. Iflocal users have write access to the databasefile, they can do anything they want.Common SQL OperationsThis article is about the command-line clientsfor MariaDB, PostgreSQL and SQLite, but oneof the main things you do when using suchclients is write SQL statements. So let’s look atsome of the basic SQL-related similarities anddifferences between the three.The most common SQL statements areselects, inserts, updates and deletes. Asa computer language, SQL is one of themore popular ones, and there is an officialstandard, ANSI SQL, which has gone throughvarious revisions through the years. Mostrelational database management systems(RDBMSes) use SQL as their query language,but they differ in how closely they adhere toANSI SQL. Of the three I’m exploring here,PostgreSQL sticks closest to the standard.MariaDB drifts from the standard in placesto make it easier to use. SQLite3 doesn’tpretend to support every feature of ANSISQL. Instead, it supports only a subset. Afterall, it’s supposed to be “Lite”.Some people would like to see SQL dieand never be used again. I am not one ofthose people. SQL has issues, but so domost computer languages. I find SQL easyto read, flexible and well worth the timeit takes to learn it. The examples beloware simple, and I gloss over a lot of thecomplexity of SQL. I also don’t explain everypart of every statement. My goal here isto give you a taste of what SQL looks likein practice and to point out some of thesimilarities and differences between the threedatabases. The on-line documentation foreach of these databases (and the in-clienthelp for MariaDB and PostgreSQL) includesextensive information on SQL syntax. I foundthe SQLite syntax diagrams to be especiallyhelpful for that database.SQL statements can be written on a singleline, or they can be broken up across manylines to make it easier to read. In the examplesWWW.LINUXJOURNAL.COM / DECEMBER 2011 / 79

INDEPTHThe SERIAL DatatypeA datatype is how you tell the databasewhat type of data is in a column.Common datatypes include integer, text,varchar and date. The SERIAL datatypeis a special one. In MariaDB, the SERIALdatatype is an alias for the following:have the same value in that column.In PostgreSQL, the SERIAL datatype isan alias for this:INTEGER NOT NULL DEFAULT nextval('tablename_colname_seq')BIGINT UNSIGNED NOT NULL AUTO_INCREMENT UNIQUEThat’s quite a mouthful, but it doesthe job of creating a column suitablefor use as a PRIMARY KEY. BIGINT isa large integer; UNSIGNED means nonegative values; NOT NULL means itcan’t be empty; AUTO_INCREMENTmeans that if a specific value is notspecified when a row is inserted, thevalue should be “the current highestvalue + 1”; and UNIQUE means thatno other row in that table is allowed toThe odd nextval('tablename_colname_seq')bit is referring to an “ALTER SEQUENCE”,specifically:ALTER SEQUENCE tablename_colname_seq OWNED BY tablename.colname;This is just PostgreSQL’s way ofcreating an auto-incrementing column.Thankfully, when you create a columnwith type SERIAL, PostgreSQL createsthe ALTER SEQUENCE for you. Thiscolumn also is suitable for use as aPRIMARY KEY.below, I do the latter. SQL statements usuallyend with a semicolon (;).The CREATE TABLE StatementYou won’t get very far in your databaseadventures without some tables. If you’re notfamiliar with databases, think of databasetables as spreadsheet sheets, without all thefonts and border styles.Returning to our library example, the mostcommon things in a library are books, so let’screate a books table:CREATE TABLE books (bookid serial PRIMARY KEY,title varchar(100) NOT NULL,seriesid integer,authorid integer);The above works for both MariaDB andPostgreSQL, but it doesn’t work for SQLite3,because of the use of the SERIAL datatype,which often is used as the datatype for aPRIMARY KEY. See the “The SERIAL Datatype”80 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

INDEPTHsidebar for more information.A common feature of many databasetables is a PRIMARY KEY. This key uniquelyrefers to a single row of a table. The PRIMARYKEY can be a combination of two or morecolumns in a row (as long as the combinationis guaranteed to be unique in that databasetable), but most commonly, there is a specific,auto-incrementing column that is used as thePRIMARY KEY.Every row in an SQLite3 table automaticallyhas a PRIMARY KEY column (SQLite calls it theRowID) created when you create the table.However, it is hidden unless you specify acolumn with a type of integer PRIMARYKEY. So for SQLite, change the bookid line inthe CREATE TABLE statement above to this:bookid integer PRIMARY KEY,And, SQLite3 will create a table withequivalent settings to MariaDB and PostgreSQL.The INSERT StatementNow that you have a table, it’s time to enter(or INSERT) some information. Insertingdata between the three databases is verysimilar, but there is one important difference.Both MariaDB and PostgreSQL allow you toinsert multiple rows of information in onestatement. SQLite3, on the other hand, letsyou insert only a single row at a time.For example, to insert some data into thebooks table you created earlier, use this SQLstatement for both MariaDB and PostgreSQL:INSERT INTO books (title, seriesid, authorid) VALUES('The Fellowship of the Ring', 1, 1),('The Two Towers', 1, 1),('The Return of the King', 1, 1),('The Sum of All Men', 2, 2),('Brotherhood of the Wolf', 2, 2),('Wizardborn', 2, 2),('The Hobbbit', NULL, 1);You may have noticed a typo in the lastline. I did it on purpose so you would havesomething to fix later.For SQLite3, each row that you are insertingneeds to be done separately, like so:INSERT INTO books (title, seriesid, authorid) VALUES('The Fellowship of the Ring', 1, 1);INSERT INTO books (title, seriesid, authorid) VALUES('The Two Towers', 1, 1);...and so on.In the SQL statements above, I don’t specifythe bookid in the column names section. Ido this because that column is set up as thePRIMARY KEY, and it is filled automatically bythe database with the correct value.The SELECT StatementSELECT is the most common databaseoperation. The only reason I didn’t talkabout this first is because until the table wasCREATE-ed and had data INSERT-ed into it,as you did in the previous sections, there wasnothing to SELECT.On all three of the databases, SELECTstatements work pretty much the same. BasicSELECT statements, such as the following, willwork on all three:WWW.LINUXJOURNAL.COM / DECEMBER 2011 / 81

INDEPTHSELECT * FROM books;to work on all three:SELECT title, authorid FROM books WHERE authorid = 1;SELECT * FROM books ORDER BY authorid;Joins also work very well across allthree. Joins are where you combineinformation from two or more tablestogether. For example, here is a join thatmatches author names to their booksbased on the authorid number:SELECT title AS "Book Title",givenname, surnameFROM books INNER JOIN authorsUSING (authorid)ORDER BY surname;The above SELECT statement presupposesthe creation of an authors table and theinsertion into it of at least a couple rows ofdata, like so:On MariaDB and PostgreSQL:CREATE TABLE authors (authorid serial PRIMARY KEY,surname varchar(100),givenname varchar(100),birthdate date);On SQLite3, change the authorid line to thefollowing, and the CREATE TABLE statementwill work properly:authorid integer PRIMARY KEY,Here is some data for the table, formattedINSERT INTO authors (surname, givenname) VALUES('Tolkien', 'J.R.R.');INSERT INTO authors (surname, givenname) VALUES('Farland', 'David');Now, you can run the SELECT ...JOIN statement.The UPDATE StatementRemember that typo? Well, it’s time to fix it.This UPDATE line works for all three:UPDATE books SET title = 'The Hobbit' WHERE title = 'The Hobbbit';The DELETE StatementDeleting rows also is the same across all three:DELETE FROM books WHERE bookid = 7;The above will delete the row in the bookstable that has a bookid of 8. If you’ve beenfollowing along, there should not be an entrywith that bookid, so nothing will happen.The ALTER StatementSuppose I decide to remove the seriesidcolumn from the books table. In MariaDB andPostgreSQL, the following statement will do it:ALTER TABLE books DROP seriesid;SQLite3, on the other hand, does notsupport the removal of columns from tables.You can add columns to a table, or modifycolumns, but the only way to remove a82 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

INDEPTHSQLite OutputWhen trying the SQL examples, you willnotice the SQLite output is not nearlyas pretty as the output from MariaDB/MySQL or PostgreSQL. By default,SQLite doesn’t print column names ortry to pad columns so that they line upnice and fancy like the others do. Tomake SQLite do so for the SELECT ...JOIN statement, enter the the followingcommands before the statement:.explain ON.mode column.width 30 10 10The .explain command instructsSQLite to display column headers;.mode sets the output to display incolumns, and the .width commandsets the width of the columns. Theonly issue with doing this is that it willmess up the output of future queries(unless they happen to look fine with the.width values you specified). To resetthings back to the default, set the outputmode back to the default “list” with.mode list. Doing this also turns offexplain and resets the column widthsback to their defaults.column is to create a new table without aseriesid column, transfer the data from theold table to the new table, drop the oldtable, and then rename the new table to theoriginal name. It’s not as annoying as youmight think, thanks to some SQL INSERTtrickery (well, I thought it was tricky the firsttime I saw it in action). The basic idea is touse the output of a SELECT statement as theinput to an INSERT statement, like so:CREATE TABLE books2 ();bookid integer PRIMARY KEY NOT NULL,title varchar(100) NOT NULL,authorid integerINSERT INTO books2 (bookid, title, authorid)SELECT bookid, title, authorid FROM books;DROP TABLE books;ALTER TABLE books2 RENAME TO books;The above trick also works as written inMariaDB and PostgreSQL as long as youchange the bookid line of the CREATE TABLEstatement to the following:bookid serial PRIMARY KEY,But, that’s an awful lot of work if you justwant to drop a column from a table.These examples should be enough SQL togive you a picture of how the three compareto each other.ConclusionIt is not hard to interact with databases onWWW.LINUXJOURNAL.COM / DECEMBER 2011 / 83

INDEPTHSingle vs. Double QuotesIn the SQL examples I use single quotes(') for most things and double quotes(") sparingly. MariaDB and SQLite allowyou to use single or double quotesinterchangeably for most quoted textin queries. PostgreSQL is pickier,because it tries to stay closer to theANSI SQL standard, which says singlequotes should be used for values (forexample: title = 'The Hobbbit'),and double quotes should be used forsystem identifiers (field names, tablenames and so on—for example:SELECT title AS "Book Title"...).You can force MariaDB to obey thestandard—and reject double-quotedvalues—with the command SETsql_mode='ANSI_QUOTES'.the command line. In my opinion, doingthe tasks listed above is much easier on thecommand line than through a graphicaldatabase program.Of course, manipulating your database byhand, whether on the command line or with agraphical program, probably should be avoidedin many cases in favor of using an automatedfront end—for example, a PHP contentmanagement front end for the database thatcontains the content for your company Website. However, for those times when you doneed to dive in and tweak something manually,or for small projects that don’t justify the timeor expense of a custom front end, there is noneed to be afraid of using the command-lineclient of your chosen database.■Daniel Bartholomew works for Monty Program(http://montyprogram.com) as a technical writer and systemadministrator. He lives with his wife and children in North Carolina,and he often can be found hanging out in #maria on Freenode IRC(he occasionally pokes his head into #linuxjournal too).ResourcesMariaDB Web Site: http://mariadb.orgMariaDB Documentation: http://kb.askmonty.orgMariaDB Downloads:http://downloads.askmonty.orgPostgreSQL Web Site: http://www.postgresql.orgPostgreSQL Documentation:http://www.postgresql.org/docsPostgreSQL Downloads:http://www.postgresql.org/downloadSQLite Web Site: http://www.sqlite.orgSQLite Documentation:http://www.sqlite.org/docs.htmlSQLite Downloads:http://www.sqlite.org/download.htmlSQLite SQL Syntax Diagrams:http://www.sqlite.org/syntaxdiagrams.htmlWikipedia Article on SQL:http://en.wikipedia.org/wiki/SQLWikibooks Article on Moving between MySQLand PostgreSQL: http://en.wikibooks.org/wiki/Converting_MySQL_to_PostgreSQL84 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

INDEPTHUsing Linuxwith EFIEFI features and how they impact Linux. RODERICK W. SMITHA seismic shift is under way in thecomputer world, and many people areunaware of the trembling beneath theirfeet. Since the IBM PC’s introduction in1981, most x86 and x86-64 computershave used the Basic Input/Output System(BIOS) firmware. The BIOS, however, isantiquated and limiting. The industry is,therefore, shifting from it to a new system,the Extensible Firmware Interface (EFI)and its even newer variant, the Unified EFI(UEFI). Although most computer featuresaren’t affected by this change, it doesgreatly affect how the computer boots andhow you must prepare your hard disk forOS installation. This article is the first in aseries of four that describes these changesand helps you get Linux up and running onthe new EFI-based computers. (I use “EFI”to refer to both the original EFI 1.x and thenew UEFI, which is essentially EFI 2.x.)This article describes the overall featuresand principles of EFI, including why youmight want to use it, how EFI boots andwhat types of bootloaders you might usewith it to enable Linux to boot on an EFIcomputer. The next three parts of thisseries will describe how to prepare to installLinux on an EFI computer, how to performthe installation and how to manage thecomputer once it’s up and running.Why Use EFI?Chances are you’re using BIOS-basedcomputers today. They work fine, and theyboot fine, so why should you switch to EFI?In practice, you won’t have much choice,because manufacturers are phasing BIOSout. Beyond this very pragmatic reason forswitching, EFI has certain advantages overBIOS. To understand these new EFI featuresthough, you first must understand whatfirmware in general does and the ways inwhich BIOS is deficient.Firmware is software that’s built in toa device, stored in nonvolatile memory,such as electrically erasable programmableread-only memory (EEPROM) chips.Motherboards, plug-in cards and manyexternal devices all use firmware. BothBIOS and EFI firmware run on a computer’smotherboard and perform severalimportant tasks. Most important, thefirmware contains the first code that thecomputer runs when it starts up. This codeincludes hardware check functions and86 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

INDEPTHModern hardware makes larger firmware morepractical than it was in 1981, so EFI implementationscan be more complex than older BIOS designs.functions that read and execute programsfrom the hard disk.The IBM PC was introduced in 1981, soits BIOS was simple by today’s standards. Inparticular, to begin the boot process, theBIOS loads the first sector of the boot deviceand executes it. On a hard disk, the firstsector often is called the Master Boot Record(MBR), and it has several limitations thathave been causing problems for 30 years:n The MBR bootloader is tiny. Typically,it chainloads additional code in apartition’s boot sector or in some otherlocation. The methods used to locatethis extra code are usually simple,because neither the BIOS nor the MBRbootloader understands filesystems.n The computer’s boot process is vulnerableto changes caused by writing new codeto the MBR. OS installations sometimesrender other OSes unbootable becausethey overwrite the MBR code, and virusesthat embed themselves in the MBR havewreaked havoc over the years too.n Getting multiple OSes or OS installationsto coexist can be difficult.n Because the BIOS design dates back30 years, it uses ancient 16-bit 8086operating modes. A 32-bit or 64-bitcomputer is unlikely to need theseoperating modes at all except for theboot process, but CPU manufacturersmust continue to build those modes intotheir products just to support the BIOS.EFI aims to overcome some of theseBIOS limitations. Modern hardware makeslarger firmware more practical than it was in1981, so EFI implementations can be morecomplex than older BIOS designs. This addedcomplexity enables EFI to perform tasks thatBIOS implementation’s can’t handle. The keyfeatures of EFI include the following:n EFI can parse partition tables andfilesystems, which enables bootloadercode to be stored in files in a partition.Bootloaders, therefore, can be complex,and you can store as many of them asyou like on your computer.n EFI implementations usually providesome means of selecting whichbootloader to use and, therefore, whichOS to boot, at boot time. In practice,these user interfaces still are usuallypretty limited, so you may want to useanother bootloader as the selector. (TheWWW.LINUXJOURNAL.COM / DECEMBER 2011 / 87

INDEPTHupcoming section, “Choosing anEFI-Capable Bootloader”, describessome EFI bootloader options for Linux.)n If your OSes are well behaved,they won’t overwrite each other’sbootloaders. Unfortunately, bugs canand do cause problems sometimes.Also, one bootloader must bedesignated as the primary one, andan OS might change the primarybootloader when it installs itself.n EFI supports drivers for filesystems andhardware, enabling you to boot fromdevices on plug-in boards even if thosedevices lack firmware of their own.n EFI implementations typically providea simple command-line shell and ascripting language, enabling you towrite boot-time scripts that can performvarious tasks before any OS boots. Youcan use tools, such as text editors andpartitioning utilities, to adjust yoursystem if you run into boot problems.EFI UserInterfacesOne of UEFI’s selling points forthe public is a prettier GUI-basedinterface to the firmware’s setuputility. This “eye candy” can be nice,but it doesn’t fundamentally alter thefirmware’s capabilities, much lesshow any OS boots.In fact, many motherboards withplain text-mode user interfacesto their firmware use UEFI. Incombination with the BIOS emulationmode, this can make the computeract just like a BIOS-based model, soyou may not even realize that you’reusing a UEFI PC!n The EFI specification describes a newpartition table type, the GUID PartitionTable (GPT). The old MBR partitionsystem is limited to 232 sectors, whichworks out to 2 TiB on disks with 512-byte sectors. GPT uses 64-bit pointers,so its limit is 264 sectors, or 8 ZiB(zebibytes). Although you can use GPTon BIOS-based computers, Windowsrefuses to boot from GPT on BIOSbasedcomputers. Because Windowsboots fine from GPT on UEFI-basedcomputers, UEFI is a practical necessityto boot Windows from a GPT disk.n Most modern EFI implementationsinclude a BIOS emulation mode. Thisis a stopgap measure that enables youto install an OS with no or poor EFIsupport even on an EFI-based computer.Intel-based Macintoshes use this featureto boot Windows using Apple’s BootCamp software.n EFI designs can boot a computer more88 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

INDEPTHquickly than can BIOS designs. In mytests, the results typically are about 20or 30 seconds faster when using EFIboot mode rather than BIOS boot mode.EFI has its drawbacks too, of course. Themost important of these is the fact that it’snew. This means that old bootloaders don’twork with it and users are unfamiliar withit. One more significant problem is that theEFI boot process assumes the OS will run inthe same bit depth as the EFI. Because allUEFI-based PCs and most EFI-based Macsuse 64-bit firmware, this means that 64-bitOSes work best with these computers. (Theearliest Intel-based Macs used 32-bit EFIsthough.) Installing a 32-bit version of Linuxon a computer with a 64-bit EFI is possible,but you’ll give up runtime EFI interfaces.This makes bootloader maintenanceharder, since the efibootmgr utility (whichI'll describe later in this series of articles)relies on such interfaces. For this reason, Irecommend installing a 64-bit distributionif your firmware is 64-bit.Overall, EFI’s feature set provides agreat deal of flexibility. In theory, itshould enable easier coexistence betweendifferent OSes on multiboot computers andeasier maintenance of the boot processeven on computers that boot just one OS.In practice, EFI booting still is new enoughthat it’s sometimes awkward simplybecause the tools are new and small innumber. Lack of familiarity also can makeEFI awkward to those who know all theBIOS booting tricks.EFI’s Boot ModelRecall that the BIOS begins the bootprocess by reading a single sector (theMBR) from the hard disk. EFI is morecomplex, however, so it can read abootloader from a file in a filesystem. Todo this though, the EFI requires its ownpartition, just as OSes usually requiretheir own partitions. The EFI’s dedicatedpartition is known as the EFI SystemPartition (ESP). Because the EFI’s main job isto boot the computer, you’re likely to findOS-specific bootloaders on the ESP.The EFI specification states that theESP should use the FAT-32 filesystem, butin practice, any filesystem that the EFIsupports will work. This normally meansFAT-12, FAT-16 and FAT-32. Macintoshesalso can use HFS+. Some versions ofWindows refuse to accept an ESP withanything but FAT-32, so I stronglyrecommend using FAT-32 on your ESP.The ESP is identified by a specific typecode in the partition table. On a GPT disk,you can set this type code in various ways:n If you use GPT fdisk (gdisk, cgdisk orsgdisk) to partition a GPT disk, youshould give your ESP a type code of0xEF00.n If you use a libparted-based utility, suchas parted or GParted, you should set the“boot flag” on the disk. Note that this“boot flag” is not equivalent to a “bootflag” on an MBR disk, and on a GPTdisk, you should set it only on the ESP.WWW.LINUXJOURNAL.COM / DECEMBER 2011 / 89

INDEPTHLinux installations normally mountthe ESP at /boot/efi. The EFI directoryholds subdirectories, each of whichholds bootloaders and their supportfiles. For instance, EFI/Microsoft holdsthe Windows bootloader files, andEFI/ubuntu holds Ubuntu’s bootloader.In Linux, these directories would be/boot/efi/EFI/Microsoft and /boot/efi/EFI/ubuntu. The EFI/BOOT directory holds adefault bootloader file, should no otherbootloader be installed. If you install anEFI bootloader independently of your OSinstallations, you probably will either placeit in the EFI/BOOT directory as the defaultbootloader or create a new subdirectorynamed after the bootloader itself.EFI programs, including bootloaders, have.efi filename extensions. You can use any nameyou like, although the default bootloaderin the EFI/BOOT directory has a specialname: BOOTX64.EFI on x86-64 systems.You can store a startup script inthe startup.nsh file in the ESP’s rootdirectory (that is, /boot/efi/startup.nsh inLinux). You can use this file to launch abootloader or to provide user-selectablepreboot options, but I don’t describe thatin detail in this series.The EFI specification doesn’t providemuch guidance on the size of the ESP.The Microsoft Windows installer creates a100 MiB ESP by default; Mac OS X createsa 200 MiB ESP, and Linux distributionscreate ESPs of various sizes. I recommendcreating an ESP that’s in the 200–300MiB range, particularly if you use ELILO(described shortly).EFI implementations should provide aboot manager that enables you to selectwhich OS to boot. EFI maintains a listof bootloaders in Flash storage on themotherboard, and you normally can entera boot manager utility at system startuptime by pressing a special key, such as F10or F12. Sometimes you can use this bootmanager or the firmware’s more completesetup utility to add or remove items fromthe boot manager’s menu. If you can’tfind such options, you can use Linux’sefibootmgr utility (described later in thisseries of articles) to manage your bootoptions.Because the EFI boot manager userinterface varies so much from oneimplementation to another, you shouldconsult your motherboard’s or computer’sdocumentation to learn more. If thedocumentation is unclear, you may needto experiment with it.Choosing an EFI-Capable BootloaderThe universe of EFI bootloaders is quitelimited compared to the range availablefor BIOS. Nonetheless, several bootloadersfor Linux exist. Table 1 summarizes theirfeatures. The bootloaders include:n ELILO: in my experience, ELILO is themost reliable Linux bootloader onUEFI-based PCs; however, I’ve had littleluck with it on a 32-bit Mac Mini. It canload Linux kernels from the ESP, butnot from other locations. This means90 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

INDEPTHTable 1. Bootloader FeaturesBootloader Load Linux Kernel Location ChainloadELILO Y ESP NGRUB Legacy Y any partition YGRUB 2 Y any partition or LVM YrEFIt N N/A YThe Linux kernel Y ESP Nthat your ESP must be big enough tohold as many Linux kernels and initialRAM disks as you care to install. ELILOcan’t chainload another bootloader,so if you want to multiboot withother OSes, you’ll need to use yourfirmware’s boot manager or anotherbootloader in addition to or insteadof ELILO. It reads its configuration file,elilo.conf, from the same directory inwhich its .efi file resides.n GRUB Legacy: the official versionof GRUB Legacy doesn’t supportEFI booting; however, Fedora hascreated a heavily patched versionthat does support EFI. This versionof GRUB supports booting a Linuxkernel or chainloading to another EFIbootloader. Thus, you may be ableto use GRUB Legacy as your primarybootloader in a multiboot environment.It can read kernels from any commonLinux filesystem, but not from anLVM or RAID configuration. In myexperience, it’s pretty reliable. It readsits configuration file from the samedirectory as its binary .efi file. Theconfiguration file is named after thebinary file—normally grub.conf.n GRUB 2: GRUB 2 officially supports bothBIOS and EFI booting; however, youmust install an EFI-capable package,such as grub2-efi under Debian orUbuntu. GRUB 2 can load a kernel fromany Linux filesystem on a partition,LVM or RAID configuration. It also canchainload to another EFI bootloader.The main problem with GRUB 2 is itscomplexity, which makes its installationsdelicate. Distribution configurationscripts sometimes get details wrong,at least for EFI installations, which canrender your computer unbootable.n rEFIt: unlike ELILO, GRUB Legacy andGRUB 2, rEFIt isn’t capable of directlybooting a Linux kernel. Instead, itpresents a menu of bootloader options.When you select a bootloader, rEFItchainloads it. This makes rEFIt a usefulreplacement for an EFI implementation’sbootloader, if that bootloader is limitedor awkward. By default, rEFIt presentsa graphical menu. The most commonrEFIt binaries use a “fat” 32-/64-bitformat that’s usable only on Macs. IfWWW.LINUXJOURNAL.COM / DECEMBER 2011 / 91

INDEPTHyou have a UEFI-based PC, you musttrack down a pure 64-bit version ofthe program. (Debian and Ubuntuboth ship with such packages; see theResources section for another sourcefor such binaries.)Roderick W. Smith is a Linux consultant, writer and open-sourceprogrammer living in Woonsocket, Rhode Island. He is theauthor of more than 20 books on Linux and other open-sourcetechnologies, as well as of the GPT fdisk (gdisk, cgdisk andsgdisk) family of partitioning software.n The Linux kernel: work is under way toembed an EFI bootloader in the Linuxkernel itself. When this work is done,you will be able to launch Linux directly,without using ELILO, GRUB Legacy orGRUB 2. You’ll have to store the kerneland its initial RAM disk on the ESPor some other partition that the EFIcan read though. This code is not yetavailable in any publicly released kernel,as of the 3.1-rc7 kernel, but see theResources section for a set of patches.Overall, my preference for an EFI-capableLinux bootloader is either ELILO or Fedora’spatched GRUB Legacy. When multibootingwith a non-Linux OS, ELILO works bestwhen paired with rEFIt. GRUB 2 is just toofinicky and unreliable. It might eventuallybe possible to boot the Linux kerneldirectly using no other bootloader, butthis support is still extremely new and isnot yet available in released kernels.Next TimePart two of this series covers preparatory stepsfor installing Linux on an EFI computer—disk partitioning and understanding thefeatures and limitations of some commonLinux distributions with respect to EFI.■ResourcesOfficial UEFI documentation can be obtainedat the UEFI home page: http://www.uefi.org.ELILO is based at http://elilo.sourceforge.net.GRUB is headquartered athttp://www.gnu.org/software/grub.This page is mostly dedicated toGRUB 2, although older GRUB Legacydocumentation is still available.You can learn more about rEFIt athttp://refit.sourceforge.net.Pure 32- and 64-bit builds of rEFIt thatinclude patches to eliminate some videoglitches on UEFI systems are availablefrom http://www.rodsbooks.com/efi-bootloaders/refit.html.The patches to turn the Linux kernel intoits own EFI bootloader can be found athttps://groups.google.com/group/linux.kernel/browse_thread/thread/9aac8bf3b646bf62/f0963b50a956f3d9?lnk=gst&q=x86+EFI+boot+stub#f0963b50a956f3d9. Be awarethat only those familiar with softwarepatching and kernel compilation shouldattempt to use this feature at the moment.92 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

2012 CONFERENCESIn Cooperation: 19th Annual Network &Distributed System Security Symposium(NDSS 2012)SPONSORED BY THE INTERNET SOCIETY IN COOPERATION WITHUSENIXFebruary 5–8, 2012, San Diego, CA, USAhttp://www.isoc.org/isoc/conferences/ndss/1210th USENIX Conference on File and StorageTechnologies (FAST ’12)SPONSORED BY USENIX IN COOPERATION WITH ACM SIGOPSFebruary 14–17, 2012, San Jose, CA, USAhttp://www.usenix.org/fast12In Cooperation: EuroSys 2012SPONSORED BY ACM SIGOPS IN COOPERATION WITH USENIXApril 10–13, 2012, Bern, Switzerlandhttp://eurosys2012.unibe.ch9th USENIX Symposium on Networked SystemsDesign and Implementation (NSDI ’12)SPONSORED BY USENIX IN COOPERATION WITH ACM SIGCOMM ANDACM SIGOPSApril 25–27, 2012, San Jose, CA, USAhttp://www.usenix.org/nsdi12Workshops co-located with NSDI ’12 include:2nd Workshop on Hot Topics in Management ofInternet, Cloud, and Enterprise Networks andServices (Hot-ICE ’12)April 24, 2012, San Jose, CA, USAhttp://www.usenix.org/hotice12Paper registration due: January 6, 20125th USENIX Workshop on Large-Scale Exploitsand Emergent Threats (LEET ’12)April 24, 2012, San Jose, CA, USAhttp://www.usenix.org/leet12Submissions due: February 13, 2012In Cooperation: 5th Annual InternationalSystems and Storage Conference (SYSTOR 2012)SPONSORED BY IBM IN COOPERATION WITH USENIXJune 4–6, 2012, Haifa, Israelhttp://www.research.ibm.com/haifa/conferences/systor20124th USENIX Workshop on Hot Topics inParallelism (HotPar ’12)SPONSORED BY USENIX IN COOPERATION WITH ACM SIGMETRICS,ACM SIGSOFT, ACM SIGOPS, ACM SIGARCH, AND ACM SIGPLANJune 7–8, 2012, Berkeley, CA, USAhttp://www.usenix.org/hotpar12Paper registration due: January 24, 20122012 USENIX Federated Conferences WeekJune 12–15, 2012, Boston, MA, USA2012 USENIX Annual Technical Conference(USENIX ATC ’12)June 13–15, 2012http://www.usenix.org/atc12Paper titles and abstracts due: January 10, 20123rd USENIX Conference on Web ApplicationDevelopment (WebApps ’12)June 13–14, 2012http://www.usenix.org/webapps12Submissions due: January 23, 20124th USENIX Workshop on Hot Topics in CloudComputing (HotCloud ’12)4th USENIX Workshop on Hot Topics in Storageand File Systems (HotStorage ’12)21st USENIX Security Symposium(USENIX Security ’12)August 6–10, 2012, Bellevue, WA, USA10th USENIX Symposium on Operating SystemsDesign and Implementation (OSDI ’12)October 8–10, 2012, Hollywood, CA, USAhttp://www.usenix.org/osdi12Submissions due: May 3, 201226th Large Installation System AdministrationConference (LISA ’12)December 9–14, 2012, San Diego, CA, USAStay Connected... http://www.usenix.org/facebook http://twitter.com/usenixUSENIX: The Advanced Computing Systems AssociationFOR A COMPLETE LIST OF USENIX AND USENIX CO-SPONSORED EVENTS, SEE HTTP://WWW.USENIX.ORG/EVENTS

INDEPTHMercurial—Revision ControlApproximatedMercurial provides a Git-like repository with the flexibility ofa plugin architecture. JOEY BERNARDA short while ago, an article appeared inLinux Journal implying Git was the be-alland end-all of source code revision controlsystems (“Git—Revision Control Perfected”by Henry Van Styn, August 2011). I wouldlike to challenge that assumption anddeclare to the world that the real perfectversion control system is here, and itsname is Mercurial.In case you didn’t notice it, my tongue wasfirmly in my cheek in that last paragraph. Ithink version control systems are like editors.They are all different and fit people andtheir work habits differently. There is no oneperfect system to rule them all. Git may bethe perfect fit for some people, and RCS mayfit someone else better. This article describesanother option to add to the mix. Mercurialprovides some of the features of systems likeGit, and some of the features of systems likeCVS or Subversion. Hopefully, after readingthis article, you’ll have enough informationto make a rational choice as to what is bestfor you.The main Mercurial site contains lotsof documentation for end users anddevelopers alike. Several tutorials areavailable, and they even include a seriesof work flows that cover how end userscan use Mercurial for their developmentprojects. Using those, you can see how youcould use Mercurial as a solo developer oras one of a group of developers, or howto work with a central repository like CVS.These work flows are great starting pointsfor you to create your own.First, let’s look at what makes upMercurial. A Mercurial repository consistsof a working directory, which is paired witha store. The store contains the history ofthe repository. Every working directory ispaired with its own copy of the store. Thismeans that Mercurial has a distributedsystem, much like Git. When you commit aseries of file changes, a single changeset iscreated, encapsulating these changes. Eachchangeset gets a sequential number, calledthe revision number. But, remember that94 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

INDEPTHFigure 1. Here you see that Mercurial repositoriesare tagged for easy finding.Figure 2. Right-clicking a file and pulling up theproperties gives you lots of Mercurial information.each working directory gets its own copyof the store, so these revision numbersmay not actually match up. For thisreason, each revision also gets a 40-digithexadecimal globally unique ID.So, what happens when two users aredoing parallel development? Assumingthey are starting with equal repositories,any committed changes by user onecreates a new branch, and any committedchanges by user two also creates a newbranch. User one then pulls in any changesfrom user two’s repository. This createstwo branches in user one’s repository: onebranch for user one’s changes and onebranch for user two’s changes. User onethen needs to merge these two branchestogether in order to incorporate all thechanges since the last synchronization ofrepositories. User two would need to dothe same thing (pull and merge) in order tosynchronize the repositories. Changes alsocan be pushed to another repository.One of Mercurial’s strengths is its use ofextensions. Several extensions are availablefrom the project, and you always can goahead and write your own. Extensions arewritten in Python, so hone your scriptingskills. You can enable these extensionsby adding them to the [extensions]section of your configuration file.So, how do you actually use Mercurial?You probably will want to set some basicconfiguration options in the .hgrc file.Mercurial needs a user name for recordingcommits. You can set this option in theconfiguration file with:[ui]username = John Doe The first thing to do is to create yourlocal repository. If you are working off acopy from someone else, you would make aclone. The format of the clone command is:hg clone [OPTIONS...] SOURCE [DEST]The source option can take severaldifferent forms. If the repository you arecloning is on the same machine, you simplycan provide the filesystem path to the sourceWWW.LINUXJOURNAL.COM / DECEMBER 2011 / 95

INDEPTHrepository. Mercurial includes a Web serverthat can be used to provide access to arepository over HTTP. If you are cloning sucha repository, the command simply would be:hg clone http://[user[:pass]@]somemachine.com[:port]/[path][#revision]You also can do this over HTTPS. At mywork, we keep backup copies of repositorieson a machine that is accessible only overSSH. And, that’s fine, because Mercurial isperfectly happy cloning over SSH. You canuse the following to do so:hg clone ssh://user@host[:port]/[path][#revision]You need to have a valid login on theremote machine, of course. The path isrelative to your home directory, so if youwant to use a full path, you need to startit with two forward slashes:hg clone ssh://user@host//full/path/to/repoCreating a new repository is even easier.All you need to do is create a directoryto house all of the files going into yourrepository. Then, you can cd to thatdirectory and execute the following:hg initThis command creates a subdirectorynamed .hg, containing all of the store filesfor your new repository.Changing your repository’s contents is donethrough the add and remove commands.There also is a rename command you canuse to change the name of a file within yourrepository. You can use that command tomove files around within your repository aswell. Let’s say you want to move a file tosubdirectory dir1. You would execute this:hg rename file1.c dir1You can get the current state of a filewith the status command. This will tellyou whether a file has been modified,added, removed and so on. The diffcommand shows the differences in a filefrom the current version and the lastcommitted version. If you decide to tossaway all of these changes, you can usethe revert command to reset the file tothe last committed version. Once you arehappy with your edits, you can commit anychanges with the commit command.At the level of the repository as a whole,a lot of commands are available. When youhave done a lot of editing and committedall your changes to your local copy ofthe repository, you can send the changesout to another repository with the pushcommand. The destination for the pushcommand can have any of the forms shownabove in the clone command examples.If the changes of interest were made byanother user at a remote repository, you canuse the pull command to grab them andput them into your local repository.You may want to check what is goingto happen before merging these changes.Before pushing changes out, you can use the96 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

INDEPTHoutgoing command to see what changesetswould have been sent had you actually issueda push command. For pulls, you can use theincoming command to see what changesetswould be brought in had you issued a pullcommand. Once this is done, these changessit in a separate branch. You then need tomerge this branch back in to the main one inorder to incorporate the changes.But, what if you don’t really have anykind of direct access over the network?You can use the bundle command togenerate a compressed file containing thechangeset. This can then be transferred,either by e-mail or SneakerNet, to theremote repository. Once it is there, you canuse the unbundle command to importthe changeset into the remote repository.Again, you can use the incomingand outgoing commands, with the--bundle filename option, to check outthe changesets and see what they will dobefore actually running the real commands.As I mentioned earlier, Mercurialincludes a Web server that can provideaccess to your repository over HTTP. It isnot appropriate to provide public full-timeaccess to a repository, because it doesn’tprovide any type of user authentication.In those cases, you would use a realWeb server, like Apache, to serve up therepository. But, if you simply want to throwup the server for quick temporary access, orif you are just offering up access internallyon a local network and don’t need to worrytoo much about security, this gives youreally quick access. You simply need to run:hg serve [OPTIONS...]Some of the more common optionsinclude -d or --daemon. This drops theMercurial Web server into the background.You may want to set the port that it islistening on with the option -p or --port.The default port is 8000. You can push andpull from such a Web server. If you want toserve over HTTPS rather than HTTP, you canuse the option --certificate to set theSSL certificate file to use.Several clients are available for workingwith Mercurial repositories. For GNOMEusers, there is a handy one called tortoise.The really great part of this client is that itintegrates nicely with Nautilus. This meansyou can interact with your repository,commit changes, clone it, synchronize itwith a remote repository and much more.You also get informational icons withinNautilus, letting you see immediately whichfiles are outdated, changed or whatevertheir status may be. All of the tools aresimply a right-click away. Some greatstandalone clients also are available, so lookaround and see what you like.Hopefully, this introduction gives yousome ideas on what you can get done withMercurial. Now you don’t have any excusesfor not putting your source code underversion control.■Joey Bernard spends his days helping university researchersdo scientific computing. But by night, he is a masked crusaderfighting crime—at least, once he gets the kids to sleep andcan hit the sack himself.WWW.LINUXJOURNAL.COM / DECEMBER 2011 / 97

INDEPTHThe OpenRISCProcessor:Open Hardwareand LinuxHow you can use open-source hardware in your next embeddedsystem project. JAMES TANDONLinux has become a very matureoperating system with support for awide variety of devices and processors.There have been several bumps on theway though—a common problem forkernel developers is that manufacturerssometimes want to keep hardwareproprietary by releasing binary-only devicedrivers. This issue has caused considerableconsternation to both the manufacturersand the Open Source Softwarecommunity. Open-source hardwareresulted partially from this friction.A sizable community developsopen hardware so that hobbyists andprofessional FPGA and ASIC developerscan implement advanced hardwarefunctions in their systems. The fruitsof their labor have matured into theOpenRISC soft processor core. The GNU Ccross-compiler project team for OpenRISCworked in tandem with the processorproject team, and now it runs our favoriteoperating system, Linux! My aim inthis article is to explain how embeddedsystem engineers now use open hardwarefor system design, and how Linux can runon an OpenRISC 1200 chip. The articleexplores what people have done so farwith OpenRISC, then introduces you toseveral options for developing with Linuxand OpenRISC yourself.What Is Open-Source Hardware?The principle behind open-sourcehardware is the same as with software,except that they generally use differentdesign languages. The common languagesused today for hardware are Verilog andVHDL. Just as you can download theC language source code to Linux, youcan download the Verilog code to the98 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

INDEPTHOpenRISC 1200 processor. Do you wantto implement a multicore OpenRISC?No problem! You can instantiate asmany cores as you want. Do you wantto implement a network processor?Just download an Ethernet MAC coreand connect it. Do you want hardwareacceleration for your MPEG codec inyour embedded Linux system? You canimplement this in Verilog and access itdirectly from your processor with customprocessor instructions. This is the beautyof open hardware. You can mix andmatch hardware components (commonlycalled cores) to create your own uniqueprocessor. Sound, graphics, networking,robotic control—anything is possible.The tricky part is implementation. Veryfew people have a budget to pay a chipfoundry like IBM or TSMC to manufacturechips using the fanciest chip technology.For a small manufacturing run, youcan expect to pay tens of thousands ofdollars to build a test chip. Also, thereis no guarantee it will work the firsttime. Because of this, only companieswith millions of dollars, or governmentorganizations, can afford it. However,hobbyists are not left out completely. Thefield programmable gate array (FPGA)is a special kind of chip that can run“synthesized” Verilog code. This is verysimilar to running compiled C code. Thepenalty for using an FPGA is that thecircuit runs about two to three timesslower while using more power. However,you can purchase a prebuilt FPGA boardwith an Ethernet PHY, RS232, VGA orsimilar devices for as little as $200. It is amuch more reasonable way to implementcomplex hardware.What Is the OpenRISC?The OpenRISC architecture is a 32-bitinstruction, 32-/64-bit data processor.It is a specification that allows chipdevelopers to implement the processorso that it is optimized for high speed,reduced power or minimized cost. Itincludes an optional cache and memorymanagement unit (MMU) as well, whichmakes porting Linux possible with onlyminimal changes to the primary codebase.Floating-point instructions also are anoption. The processor uses a specialon-chip bus architecture called Wishboneto connect to other on-chip devices likean Ethernet, VGA or SDRAM controller.Also, because the processor is opensourced,it is possible for you to extendthe processor with your own instructionsand registers. This is very useful forhardware acceleration. And, one keypoint in its favor: OpenRISC is licensedunder the Lesser GNU Public License(LGLP), so it has very wide appeal.The OpenRISC 1200 is animplementation of the OpenRISCarchitecture in Verilog that is known as a“soft core”. Several soft-core processorsare available for chip designers topurchase today. One you may know is theARM architecture. Apple licensed an ARMsoft core for the iPhone, then customizedWWW.LINUXJOURNAL.COM / DECEMBER 2011 / 99

INDEPTHTime, money and red tape prevent individuals fromdownloading and improving on proprietary cores,whereas OpenRISC is freely available to anybody.the processor for implementation inmobile applications. Nintendo used twoseparate ARM processors in the NintendoDS handheld video game system.The OpenRISC uses a very differentinstruction set from ARM. However, manyof the processing capabilities are thesame. If you want to use the ARM softcore for integrating your project, youneed the backing of a large companyor research institution to license thecore. Time, money and red tape preventindividuals from downloading andimproving on proprietary cores, whereasOpenRISC is freely available to anybody.You can download the source andsimulate it now in an open-source Verilogsimulator like Icarus Verilog or Verilator.An open-source processor is verybeneficial to the community, but thisraises a question: what can it do? Ratherthan talk about possibilities, let’s lookat a sample of what the community hasaccomplished so far with the OpenRISC:n A full processor architecturespecification with an extendibleinstruction set.n Implementation in the Verilog HDL(OpenRISC 1200).n Simulation in Icarus Verilog, an opensourcesimulator.n VGA and PS2 keyboard interfaceimplemented for usage like atraditional desktop.n Verified OR1200 to run at 50MHz orbetter in FPGA.n ASIC implementation of OR1200 runsat 150MHz in 0.18um technology(possible to run much faster in thelatest 28nm technology).n Implemented as control processor forrobotic control.n Full integration with Ethernet toimplement an embedded Web server.n Play a digital music file.n Other devices that you can implementinclude USB, UART, I2C, SPI, SDRAM,SD and many more.n GNU C Compiler 4.5.1 cross-compilerworks.n Working implementations of both100 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

INDEPTHuClibc and newlib: two standard Clibrary implementations for embeddedsystems.n OpenRISC support included in theLinux 3.1 code base!n A software simulator so programmerscan write software without purchasinghardware.If you decide to use the OpenRISCprocessor for your project, you canproceed with the knowledge that yourbase system has been proven in hardwaremultiple times.Getting Started with OpenRISCNow that you know something aboutOpenRISC, you might be ready to trydownloading the soft core and associateddevelopment tools yourself. Severalsmaller projects compose the OpenRISCProject that you should look at:n OpenRISC 1000/2000 specifications—provides a full listing of the processorarchitecture and instruction opcodes.n Wishbone bus specification—standardinterface for connecting devices toOpenRISC.n OpenRISC 1200 processor source code(Verilog)—synthesizable soft-coreimplementation.n ORPSoC—synthesizable embeddedprocessor with Ethernet, SPI, SDRAM,VGA and other peripherals.n OR1KSim—the software simulatorfor people developing software forOpenRISC processors.n GNU C cross compiler for OpenRISC(version 4.5.1).n GNU binutils for OpenRISC (version2.20.1).n uClibC for OpenRISC (version 0.9.0).n Linux for OpenRISC (version 2.6modified or version 3.1 integrated).The OpenRISC 1200 is a bare core thatdoes not include any peripherals—not evenRAM. It is just a processor with a bare,addressable bus interface. If you want amore-complete processor with a bootloaderand basic I/O interfaces, such as a serialport, VGA controller or keyboard interface,you need to start with the more-completeORPSoC package. You can check out thelatest version of the OpenRISC sourcetree using Subversion. Visit the Web sitehttp://opencores.org/or1k/Main_Pageto get started. Follow the links describinghow to check out a copy of theOpenRISC repository with Subversion.This will give the the latest version ofthe processor with all relevant bug fixes.If you prefer to download tarballs, visitWWW.LINUXJOURNAL.COM / DECEMBER 2011 / 101

INDEPTHhttp://opencores.org/download,or1k instead.The OpenRISC trunk line does not includeall the software, however. You probably willwant to download the latest versions of Linuxand GCC that are patched for OpenRISC aswell. These are stored in git repositories andno registration is required. Just follow theinstructions at http://www.openrisc.net/toolchain-build.html to get started.These development versions may beunstable, so obtaining a release copyof the Linux 3.1 kernel or later fromftp://ftp.kernel.org may be a safer approach.This will get you the latest versionsof the OpenRISC software developmenttools. With these packages, you now haveeverything you need to perform softwareonlyand hardware-software simulation onOpenRISC with Linux. A lot of informationis available for installing and running thesetools, but if compiling and simulatingwith these packages seems intimidating,don’t worry. All software is built usingfamiliar configure scripts and make files.For the hardware hackers, the Web sitehttp://www.opencores.org has a largecommunity dedicated to OpenRISC and anumber of peripherals that can connectdirectly to it.Chances are, if the extensive on-linehelp section does not answer yourquestions, somebody in the forums willhave encountered your problem before.If you are more of a software person witha knack for programming embedded systems,take a look at http://www.openrisc.netand the mailing lists listed there instead.Experimenting with OpenRISCNow that you have the software andhardware source code, your designs arelimited only by your creativity. You can takethree separate development paths whendeveloping with the current version ofOpenRISC: embedded software development,custom digital circuit implementation inFPGA-based systems or creation of yourown custom processor (ASIC).If you do not have the time orinclination to understand and developan FPGA or ASIC but want to developsoftware, you will want to use theemulator, or1ksim. This high-levelsoftware emulator allows you to testyour programs for correctness withouthaving to purchase a prebuilt system. Forinstance, say you created a Web servercontrol panel, but you want to test it onOpenRISC. The simulator has an Ethernetdevice option that your software canaccess and control. If you wished tosimulate a handheld game system, thereis emulation for general-purpose I/O(for push-buttons) and a VGA display.Here’s another possibility: you alreadyhave an open-source project, but wouldlike to test that it compiles and runson OpenRISC/Linux. The simulator canhelp you see if your software packagecompiles and runs properly. The projectis still young, so do not expect real-time3-D rendering (yet). However, if you wantto test basic functionality of your opensourceproject to OpenRISC, or1ksim isthe way to go.102 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

INDEPTHProgramming Embedded ProcessorsSo, you are ready to bootstrap yourembedded Linux system on a brandnewOpenRISC implementation. Howdo you do this? With a cross compiler,of course. The GNU C compiler projectcan generate binary executables for32-bit i386 processors or x86-6464-bit processors on your basic GNU/Linux installation. However, if you wantto develop for OpenRISC, you have torecompile the GNU C compiler so it cantarget OpenRISC. This is a rather involvedprocess, which may require a specialimplementation of the standard C library—this means you need to write code sothat printf()and scanf() know how to readand write characters! The instructions athttp://www.openrisc.net are conciseand quick. If you follow them, you canget a full working copy of the GNUtoolchain for OpenRISC very quickly.If you want to understand how tobuild the OpenRISC toolchain fromscratch, or make modifications for yourcustom implementation, I recommendreading http://www.openrisc.net/toolchain-build.html by Gene Sally.He does an excellent job of introducingthe details of cross compilers.The hardware hackers reading thisarticle mostly will develop with FPGAs.Altera and Xilinx produce excellentFPGA options for running the OpenRISCprocessor. Both companies haveembedded processor cores; however,neither is open-sourced. If you developfor the Xilinx Microblaze processor, forinstance, you are locked in to using Xilinxexclusively. OpenRISC gives you freedomto choose the best FPGA for speed,power or area optimization.The fastest tested processor speed Ihave observed in FPGA tests is 50MHz,although I suspect that the fastest maygo above 150MHz if the source Verilogand implementation scripts are fine-tunedproperly for one of the newest FPGAslike the Stratix V or the Virtex-6. Alteraclaims that digital circuits can reach550MHz on its latest FPGA architecture,although this is a best-case scenario.The OpenRISC processor is fully capableof controlling a robotic arm, and thisapplication has been demonstrated(video at http://www.youtube.com/watch?v=Lv1Gow7WZxM).The most extreme development pathyou might take is integrated circuit(ASIC) development. This work tends tobe left to professionals, but open-sourcetools exist for layout and simulationof integrated circuits. Compiling, alsoknown as synthesizing, Verilog and VHDLWWW.LINUXJOURNAL.COM / DECEMBER 2011 / 103

INDEPTHto a net list consisting of only simplelogic gates is possible, though veryrudimentary without proprietary tools.Also, open-source tools for placementand routing of logic gates in integratedcircuits are virtually nonexistent. Thebest place to look for source code thatcompiles a net list, does place and route,or extracts parasitics is to search the Webpages of university research projects.If you have the budget for proprietarychip-design software (think millionsof US dollars), it is possible to designfor the latest manufacturing processavailable. Apple, Inc., has licensedthe ARM processor and developed theA6 processor in a 28nm technologynode for the iPhone, and at thetime of this writing, it is currently intesting. OpenMoko and Android areboth Linux-based distributions forsmartphones. A company developing asmartphone processor that is targetedfor use with these distributions mightconsider using the OpenRISC target.General chip design is beyond the scope ofthis article, but if you are really interested,http://opencircuitdesign.com or my personalWeb site http://www.jamestandon.comwill get you started.One group of people who havebenefited tremendously from theOpenRISC are university researcherswho explore new technologies indigital and mixed-signal developmentof integrated circuits. If a graduatestudent needs to research how customhardware interacts with softwarequickly, it is possible to simulate andfabricate a processor without spendingyears developing aprocessor core with acorresponding C compiler.A researcher can test thesystem in FPGA, then submita custom chip for fabricationwithin six months.Figure 1. Layout of an OpenRISC 1200 processor in 0.18umtechnology implemented at the University of Tokyo. It hasa serial port, Ethernet, I2C, 128kB of on-chip memory and acustom serial communication interface.The FutureOpenRISC has a largefollowing in the integratedcircuit research communityand the hardware hackercommunity. The OpenRISC/Linux development teamrecently submitted theirpatch for inclusion in the104 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

INDEPTHmain Linux distribution. If all goes well,you will soon see the OpenRISC buildtarget when you download version3.1 of the Linux kernel. Work on thelatest 3.1 OpenRISC kernel patchcontinues. If you feel inclined to help,a lot of testing is needed for the latestdevelopment version.Will it be possible to purchase anOpenRISC ASIC in the future? Quitepossibly. The team at http://opencores.orgis soliciting donations for a custom ASICimplementation. Hardware hackers whodecide to commercialize their FPGAprojects may want a faster, lower-powerversion of OpenRISC when selling theirapplications. Another possibility is thatopen-source integrated circuit designwill become available as well, makingit possible for small groups of people,or even individuals, without millions ofdollars, to submit custom chip designs toa manufacturer. The fabless semiconductorcompany has become a much more viablebusiness model in the past 15 years.The place where OpenRISC truly shinesright now is in FPGA design. Anybody canpurchase a single FPGA for as little as a fewdollars or a prebuilt FPGA board for as littleas $200 USD. This means that anybodywith a little extra cash can build a customembedded system with the OpenRISC.While processor cores that manufacturersprovide confine you to their architecture,the OpenRISC allows you to choose thebest FPGA for your project. Because Linuxcan run effectively on OpenRISC, youcan include any number of open-sourceprojects on your custom hardware. If thisarticle has whet your interest for more,check the list of Resources for this article.Also, if you implement a project with Linuxand OpenRISC, send me an e-mail to letme know!■James Tandon is a post-doctoral researcher at the VLSI Design& Education Center of Tokyo University, Japan. He has fabricatednumerous digital and mixed-signal integrated circuits intechnology nodes from 0.35um down to 65nm. His personalWeb address is http://www.jamestandon.com.ResourcesHome Page for the Linux and GCCDevelopment Projects:http://www.openrisc.netLarge Repository of Open-Source HardwareCores: http://www.opencores.orgHome Page of the OpenRISC 1000 Project:http://opencores.org/or1k/Main_PageOpen-Source Verilog Simulator (good forsimulating before implementation and verystable): http://iverilog.icarus.comVerilog Programming Tutorial:http://www.asic-world.com/verilog/veritut.htmlDemonstration of OpenRISC Controllinga Robot: http://www.youtube.com/watch?v=Lv1Gow7WZxMWWW.LINUXJOURNAL.COM / DECEMBER 2011 / 105

INDEPTHFixing BrokenProtocols withNF_QUEUEBroken protocols can be fixed on the fly with Netfilter’s abilityto direct packets to userspace programs. PAUL AMARANTHRecently, one of my clients wasexperiencing problems with remote printservers. These print servers were on aninternally NATted network connected to acentral records system located across thestate through the Internet. The print serverswould not stay connected, dropping theconnection after a few minutes. Investigationfinally tracked this problem down to thekeep-alive protocol used between the centralsystem and the remote print servers. Thekeep-alive protocol employed a UDP packetwith the source and destination IP addressescontained within the data. Normally, thiswould just mirror the addresses in the UDPheader and would seem to be redundant.In this case, the server ignored the UDPheader addresses and used only the internaladdresses. When the packets went throughNAT translation, the internal addresseswere sent through unchanged, and thecentral server was attempting to reply to anonroutable 10.xxx.xxx.xxx address.Working with my client, we identified theproblem and located documentation thatspecifically stated the protocol would notwork with print servers behind NAT. For anumber of reasons, my client was unableto move the print servers to a non-NATtedenvironment, which left the problem offixing the protocol.Because all the keep-alive packets alreadywere passing through the Linux firewall,that seemed to be the logical place to fixthem. The NF_QUEUE facility in Netfilterturned out to be the perfect solution.Netfilter and NF_QUEUENetfilter is part of the packet filteringframework within the Linux 2.4.x and 2.6.xkernels. As stated on the Netfilter homepage: “Netfilter provides a set of hooksinside the Linux kernel that allows kernelmodules to register callback functions withthe network stack. A registered callbackfunction is then called back for every packetthat traverses the respective hook within thenetwork stack.”106 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

INDEPTHThe NF_QUEUE facility extends this abilityto userspace, allowing packets to be directedusing iptables rules to a userspace program.The program then can look at the packet andtake action based on the packet content. Theprogram might decide to accept or reject thepacket, for example, allowing the firewall tofilter packets based on content. The programalso might decide to modify the packet andreturn it to Netfilter for further processing.It is this latter ability that allows brokenprotocols to be fixed on the fly.The QUEUE facility initially was introducedinto the 2.3 kernel and allowed for a singlequeue. This was changed to NF_QUEUEin the 2.6.14 and later kernels to allowfor multiple queues with a 16-bit queueidentifier, so it is possible to have up to65,535 queues. If the queue number is leftoff the iptables rule, it will default to queue0 and the behavior is equivalent to the olderQUEUE behavior.An important point to remember is this isa queue. The kernel queues up packets forprocessing in userspace, and there is finitespace to buffer the packets. If it takes toolong to process the packet, the queue willfill up and packets will be lost.Although my situation and this exampleuse IPv4, the NF_QUEUE facility is alsoavailable in the IPv6 Netfilter code and theip6tables command. The details of manglingthe packet change to reflect the protocoland headers that are involved are different,and there are slight differences in theip6tables chain traversal, but the overallprocess remains substantially the same.Because the packet processing takes placein userspace, you are not limited to writingthe program in C. You can use any languageyou want, as long as there is a binding tothe NF_QUEUE facility. At the time of thiswriting, in addition to C and C++, you canuse Perl and Python to write your packethandler (see Resources).In my case, I chose to write my routinesin C. Because the firewall in question alsoserves as a gigabit router between twointernal networks, supports a VPN gatewayas well as handling almost a thousandiptables rules, I was interested in keepingoverhead low. C was the natural choice.Before using NF_QUEUE, it must be enabledin the kernel. If your kernel supports theconfig.gz option, you can use the following:gzcat /proc/config.gz|grep -E "NETLINK|NFQUEUE"and see if the configuration options listedbelow are set. If you do not have gzcat, it’sjust a hard link to gzip; many distributionsseem to leave that out.If NF_QUEUE is not configured, you’llhave to configure and rebuild the kernel.The configuration parameters that you willneed to set are the following:CONFIG_NETFILTER_NETLINK=yCONFIG_NETFILTER_NETLINK_QUEUE=yCONFIG_NETFILTER_NETLINK_LOG=yCONFIG_NETFILTER_XT_TARGET_NFQUEUE=yUse your favorite kernel configurationtool, rebuild, install and reboot.WWW.LINUXJOURNAL.COM / DECEMBER 2011 / 107

INDEPTHIf not present, you also need to build andinstall libnfnetlink and libnetfilter_queue(see Resources).Listing 1. NF_QUEUE Boilerplate Code in Cstruct nfnl_handle *nh;struct nfq_handle *h;int ec, fd, rv;char buf[1500];// 1) Open library handle. For space reasons, the// error checking is not shown.h = nfq_open();// 2) Unbind existing nf_queue handler for AF_INET.// Ignore return code for 2.6.23 kernel.// See Resources for link.ec = nfq_unbind_pf(h, AF_INET);// 3) Bind the queue connection handle.ec = nfq_bind_pf(h, AF_INET);// 4) Create queue, bind to queue 0 since that is// what the default QUEUE target in iptables// expects, specify callback function.qh = nfq_create_queue(h, 0, &nfqueue_cb, NULL);// 5) Set the amount of data to be copied to// userspace for each packet sent to the queue.nfq_set_mode(qh, NFQNL_COPY_PACKET, 0xffff);// Get the netlink handle associated with the queue// connection handle.nh = nfq_nfnlh(h);// Get a file descriptor for the netlink handle.fd = nfnl_fd(nh);// Packet loop.while ((rv = recv(fd, buf, sizeof(buf), 0)) &&rv >= 0) {nfq_handle_packet(h, buf, rv);}Listing 1 shows the standard boilerplatecode for an NF_QUEUE packet handler.Steps 1–3 are basic setup. Step 4 creates aspecified queue and binds the program to it.The queue identifier must match the queuenumber used in the iptables rules. This callalso registers the callback function, whichis where the packet actually is processed.Step 5 tells NF_QUEUE how much datais to be sent to the userspace program.The choice is none, all or just the packetmetadata (information from NF_QUEUE,but no packet data).This is pretty standard, except that under2.6.23 kernels, step 2 will return an error,which may be safely ignored. The packet isnot actually read using the recv() function;it is accessed by a callback function invokedby the nfq_handle_packet() function. Inaddition to the packet data, this allowsaccess to additional NF_QUEUE metadataand permits re-injection of the packet aswell as ACCEPT and DROP decisions to bemade. The return code from the recv() callmay be used to determine if the queue hasfilled and packets are being dropped.Before the program exits, it should closegracefully by unbinding the queue with acall to nfq_destroy_queue(), followed by acall to nfq_close(). In my implementation,I elected to include a signal handler thatclosed any log files and unbound the queueon receipt of a SIGINT or SIGHUP signal.Packet ManglingThe callback function is where the real actionis. Here you have access to the entire packet,108 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

INDEPTHListing 2. Sample Callback Function That Will Dump a Packet in Hex// Sample NF_QUEUE callback function.static int nfqueue_cb(struct nfq_q_handle *qh,struct nfgenmsg *nfmsg,struct nfq_data *nfa,void *data) {struct nfqnl_msg_packet_hdr *ph;int id = 0;int size = 0;int i;unsigned char *full_packet;unsigned char * c;struct in_addr ipa;char src_ip_str[20];char dst_ip_str[20];ph = nfq_get_msg_packet_hdr(nfa);if (ph) {// Print out metatdata.id = ntohl(ph->packet_id);fprintf(stdout,"hw_protocol = 0x%04x hook = %u id = %u\n",ntohs(ph->hw_protocol), ph->hook, id);// Retrieve packet payload.size = nfq_get_payload(nfa, &full_packet);// Get IP addresses in char form.ip = (struct iphdr *) full_packet;ipa.s_addr=ip->saddrstrcpy (src_ip_str, inet_ntoa(ipa));ipa.s_addr=ip->daddr;strcpy (dst_ip_str, inet_ntoa(ipa));fprintf(stdout,"Source IP: %s Destination IP: %s\n",src_ip_str, dst_ip_str);// Print out packet in hex.c = (unsigned char *)payload;for (i=0; i

INDEPTHwill be unchangedunless you modify thesource or destinationIP addresses.However, anychange to the packetcontents will changethe checksum forUDP or TCP packets.When the kernel NATsthe packet, it doesnot recompute thechecksums. Instead, itoptimizes the processby modifying only thecurrent checksumswith the difference between the old and newaddresses. If the kernel does not use NAT,the checksum is not even inspected. Thismeans the checksums must be correct whenreturning the packet to the kernel. Actually,with UDP packets, you can cheat by using a0 checksum. The UDP protocol specificationstates the checksum is optional, and a 0 valueindicates that you have not calculated it. Thatis not a recommended practice, particularlywhen traversing external networks, but youcan get away with it. With TCP packets,this is not an option; the TCP header mustcontain a correct checksum. Any packet withan incorrect checksum will be dropped by thenext network device it hits.There are a number of ways to determineif your checksum is correct. The easiest isto look at the packet using a sniffer likeWireshark (Figure 1). Unlike tcpdump, whichwill print only the packet contents, WiresharkFigure 1. Not only does Wireshark tell you the checksum is wrong, it alsotells you what it should be.will verify the packet checksums and even tellyou what they should be if they're not correct.The UDP checksum is fairly easyto calculate, although it does involveconstructing a pseudo-header containingthe source IP, destination IP, UDP data lengthand the UDP protocol number. I was in ahurry and grabbed two different routinesoff the Net and found that both calculatedincorrect checksums on a 64-bit platform. Ifinally had to rewrite one of the routines togenerate correct checksums (see Resourcesfor a download link). During development,you can use the checksum calculator URL inResources to paste in a hex dump of yourpacket to verify your results.NF_QUEUE HooksWhen NF_QUEUE activates a callback,along with the requested data is aparameter containing the hook that110 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

INDEPTHinvoked it. The possible values of the hookare defined in netfilter_ipv4.h as:NF_IP_PRE_ROUTING 0NF_IP_LOCAL_IN 1NF_IP_FORWARD 2NF_IP_LOCAL_OUT 3NF_IP_POST_ROUTING 4The value of the hook tells you whichiptables chain was employed to direct thepacket to the callback. You can use differentiptables commands with different chainsto change the behavior of your program bychecking the value of the hook parameter.You might direct packets from your internalNet to the PREROUTING chain and packetsfrom the external Net to the POSTROUTINGchain, for example.Understanding howiptables behave is essentialin picking the right hook.The URL for faqs.org listedin Resources has one ofthe clearest explanationsI have found. Figure 2,adapted from this reference,illustrates the packet paththrough iptables. The toplabel in the ovals is the tablename while the lower labelidentifies the chain.In this case of manglingpackets transiting thefirewall, the LOCAL_IN andLOCAL_OUT hooks will notbe used (they apply onlyto packets originating from or destined tothe local host). That leaves three choices:PRE_ROUTING, FORWARD or POST_ROUTING(note that older kernels had more limitedchoices for the mangle table).In this case where the IP header and actualsource and destination addresses are notchanged, any of the three choices wouldwork. This might not be the case if youmodify the source or destination addresses,which might affect subsequent routingdecisions. If the destination address werechanged to the local system, for example, youwould be limited to the PREROUTING chain. Ifyou want to modify the packet after all filteringhas been done and intercept any locallygenerated packets as well, you would use thePOSTROUTING chain. The FORWARD chainis useful for packets transiting the system.Figure 2. Packet flow through iptables tables and chains. Tablenames are in lowercase, chain names in uppercase.WWW.LINUXJOURNAL.COM / DECEMBER 2011 / 111

INDEPTHiptablesAt this point, you have a kernel withNF_QUEUE enabled, the nfqueue andnfnetlink libraries are installed, you havea packet sniffer ready to go, and yourprogram is compiled and ready to test.How do you connect it to iptables?The iptables target QUEUE will send anymatching packets to a program that hasregistered itself for that queue, which defaultsto Queue 0. An optional --queue-numparameter may be used to specify anondefault queue. It is also possible touse a --queue-balance parameterwith recent kernels that specifies a rangeof queues. This allows multiple instancesof a userspace program on multicorearchitectures to improve throughput. If noprogram has registered itself for the queue,the QUEUE target is equivalent to DROP.iptables has four built-in tables: filter, nat,mangle and raw. Each table supports differentchains (Figure 2). The filter table, which isthe default for the iptables command, isuseful if your program is making an acceptor deny decision on the packet, thus allowingfirewall filtering based on content. The nattable is used for address translation, and theraw table, a recent addition, is used only forsetting marks on packets that should not behandled by the conntrack system. The tableto use when altering packets is, as the nameimplies, the mangle table. Listing 3 illustratesa few iptables commands that will set upNF_QUEUE forwarding for UDP packetsdestined for port 1331. In practice, thiscan become more complicated if you limitListing 3. iptables Commands to RouteUDP Packets on Port 1331 to NFQUEUE// Set up a new chain in the mangle table.iptables -t mangle -N PktMangle// In the mangle FORWARD chain, route UDP packets// to the new chain.iptables -t mangle -A FORWARD -p udp -m udp \--dport 1331 -j PktMangle// Log the packet and invoke the queue facility.iptables -t mangle -A PktMangle -j LOG \--log-level info --log-prefix "PktMangle rule"iptables -t mangle -A PktMangle -j QUEUEthe source and destination addresses usingadditional iptables commands or includeother selection criteria.TestingThe NF_QUEUE application must run asroot, at least when setting up the queue.Otherwise, you will get a -1 return fromthe unbind or bind calls.In my case, the print server generateda ready supply of keep-alive packets, so Ihad no need of a packet generator. In thegeneral case, you will need some way ofgenerating test packets to verify that yoursystem is operating correctly. A plethoraof packet generators are available. Oneexample is PackETH, which is a GUI-basedpacket generator that is quite easy to use,although still a little unfinished.Another necessary requirement is theability to capture packets both before andafter processing to verify the output packetis correct. This can be easily done usingtcpdump or Wireshark to view packets onthe input and output interfaces of the test112 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

INDEPTHsystem. Wireshark may be used directly ifyou have X libraries available on the testsystem. In my case, since I was runningthe packets through a production system, Iused tcpdump and then viewed the packetdump files off-line with Wireshark.ConclusionsThe project turned out to be very successful.Fixing the keep-alive packets as theytraversed the firewall resolved the problemwithout requiring any configurationchanges on either endpoint. It was acompletely transparent solution and myclient was very happy with the result.After being in place for a while, a crisisensued when the remote system IP waschanged. The printers stopped, because theremote IP was hard-configured into the printservers, and local personnel were unavailableto reconfigure them. With the addition of aDNAT firewall rule and a tweak to the protocoldæmon to fix the server address within thepacket (all done remotely), the printers cameback on-line in time to run payroll.TCP protocols also can be fixed using thisapproach. Because TCP is a connectionorientedprotocol rather than a datagram,the program will need to keep some stateinformation as it processes the data stream.This is a little more complicated than a UDPprotocol, but not unreasonable in practice.I’ve posted a small sample NF_QUEUEpacket sniffer on my Web site containingcomplete build and execution directions.It’s fairly basic, but it allows you to geta hex dump of UDP or TCP streamsdetermined by iptables rules and can serveas a basic framework if you’re buildingan NF_QUEUE handler.■Paul Amaranth (paul@auroragrp.com) is a Principal Consultantat Aurora Group, Inc., where he builds secure systems (includingLinux firewalls), does software development and handles the oddbit of system administration. He’s been involved with computersmuch longer than he cares to remember.ResourcesNetfilter Hacking HOWTO: http://netfilter.org/documentation/HOWTO/netfilter-hacking-HOWTO.htmlNetfilter Error Return in 2.6.23 Kernels:http://www.spinics.net/lists/netfilter/msg42063.htmllibnfnetlink: http://www.netfilter.org/projects/libnfnetlink/index.htmllibnetfilter_queue: http://www.netfilter.org/projects/libnetfilter_queue/index.htmllibnetfilter_queue man Page: http://code.google.com/p/nattt/wiki/libnetfilter_queueNF_QUEUE Language Bindings: http://www.nufw.org/projects/nfqueue-bindings/wikiCorrect UDP Checksum Routine:http://www.auroragrp.com/downloadsSample nfq_sniff Application to Dump Packets:http://www.auroragrp.com/downloadstcpdump: http://www.tcpdump.orgWireshark: http://www.wireshark.orgPackETH: http://packeth.sourceforge.netOn-line Checksum Generator:http://moat.nlanr.net/Software/HEC/hexhec.htmliptables Packet Flow: http://www.faqs.org/docs/iptables/traversingoftables.htmlWWW.LINUXJOURNAL.COM / DECEMBER 2011 / 113

TALES FROMTHE SERVER ROOMKYLE RANKINZoning OutThis month, Bill describes that incrediblesinking feeling.BILL CHILDERSSometimes events and equipmentconspire against you and your team tocause a problem. Occasionally, however,it’s lack of understanding or foresightthat can turn around and bite you.Unfortunately, this is a tale of where wefailed to spot all the possible things thatmight go wrong.Flashback...It was 2006, and we were just gettingour feet wet with piloting a new serverarchitecture for our company. We’d justreceived our first fully populated Hewlett-Packard blade chassis (a P-Class chassiswith eight dual-core blades, for those ofyou who’re savvy with that type of gear),a new EMC Storage Area Network (SAN)and three VMware ESX licenses. We hadjust finished converting a fair amountof the development network over to theVMware environment using a Physical-to-Virtual (P2V) migration, and things weregoing quite well. Matter of fact, many ofthe people in the company didn’t quiteunderstand exactly the improvements wewere making to the systems, but they didnotice the performance boost of goingfrom machines that were something likesingle-processor Pentium 4-class serverswith IDE disks to a dual-core Opteronwhere the storage was backed by thespeed of the Fibre Channel SAN. In all,things were going quite well, and thefeedback we’d received to date fueleda rather rapid switch from the agingphysical architecture to a much fastervirtual machine architecture.BackgroundBefore we dive into the story, a couple bitsof background information will becomevery important later on. As I said, we’dreceived eight dual-core blades, but onlythree of them at that time were set asidefor VMware hosts. The rest were slatedto become powerful physical machines—Oracle servers and the like. All these newblades were configured identically: theyeach had 16GB of RAM, two dual-core114 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

TALES FROM THE SERVER ROOMOpteron processors, two 300GB disksand Fibre Channel cards connected to theshiny new EMC SAN. With respect to theSAN, since we were devoting this SANstrictly to the blade servers, the decisionwas made not to add the complexity ofzoning the SAN switch. (Zoning a SANswitch means that it is set up to allowonly certain hosts to access certain disks.)The last tidbit relates to kickstart.Both Kyle and I have written a fewarticles on the topic of kickstarting andautomated installation, so by now you’reprobably aware that we’re fans of that.However, this was 2006, and we bothwere getting our feet wet with thattechnology. We’d inherited a half-setupkickstart server from the previousIT administration, and we slowly weremaking adjustments to it as we grewmore knowledgeable about the tech andwhat we wanted it to do.[Kyle: Yes, the kickstart environmenttechnically worked, but it required thatyou physically walk up to each machinewith a Red Hat install CD, boot from it,and manually type in the full HTTP pathto the kickstart file. I liked the idea ofkicking a machine without getting upfrom our desks, so the environmentquickly changed to PXE booting among anumber of other improvements. That wasconvenient, because those blades didn’thave a CD-ROM drive.]Getting back to the story...we’d moveda fair amount of the development andcorporate infrastructure over to theVMware environment, but we still hada demand for high-powered physicalmachines. We’d gotten a request for anew Oracle database machine, and sincethey were the most powerful boxes in thecompany at the time, with connections tothe Storage Area Network, we elected tomake one of the new blades an Oracle box.As my imperfect memory recalls, Kylefired up the lights-out management on whatwas to be the new Oracle machine andstarted the kickstart process, while I wasdoing something else—it could have beenanything from surfing Slashdot to filling outsome stupid management paperwork. I don’tremember, and it’s not critical to the story, asabout 20 minutes after Kyle kickstarted thenew Oracle blade, both of our BlackBerriesstarted beeping incessantly.[Kyle: Those of you who worked (orlived) with us during that period mightsay, “Weren’t your BlackBerries alwaysbeeping incessantly?” Yes, that’s true,but this time it was different: one, wewere awake, and two, we actually werein the office.]Trouble in ParadiseWe both looked at our BlackBerries as westarted getting “host down” alerts frommost of the machines in the developmentenvironment. About that time, mutteringcould be heard from other cubicles too:“Is the network down? Hey, I can’t getanywhere.” I started getting that sinkingfeeling in the pit of my stomach as Kyleand I started digging into the issue.WWW.LINUXJOURNAL.COM / DECEMBER 2011 / 115

TALES FROM THE SERVER ROOMSure enough, as we started looking,we realized just about everything wasdown. Kyle fired up the VMware consoleand tried restarting a couple virtualmachines, but his efforts were met with“file not found” errors from the consoleupon restart. File not found? That sinkingfeeling just accelerated into free-fall.I started looking along with Kyle andrealized that all the LUNs (disks wherethe virtual machines reside) just flat-outstopped being available to each VM host.[Kyle: It’s hard to describe the sinkingfeeling. I was relatively new to SAN atthe time and was just realizing howbroad a subject it is in its own right.SAN troubleshooting at a deep level wasnot something I felt ready for so soon,yet it looked like unless we could figuresomething out, we had a large number ofservers that were gone for good.]I jumped on the phone and called VMwarewhile Kyle continued troubleshooting.After a few minutes on the line, theproblem was apparent. The LUNscontaining the virtual machines had theirpartition tables wiped out. We luckilycould re-create them, and after a quickreboot of each VM host, we were back inbusiness, although we were very worriedand confused about the issue.[Kyle: So that’s why that sinking feelingfelt familiar. It was the same one I hadthe first time I accidentally nuked thepartition table on my own computer witha bad dd command.]Our worry and concern jumped tonear-panic when the issue reared its heada second time, however, under similarcircumstances. A physical machine kickstartwound up nuking the partition table onthe SAN LUNs that carried the virtualmachine files. We placed another call toVMware, and after some log mining, theydetermined that it wasn’t a bug in theirsoftware, but something on our end thatwas erasing the partition table.A Light DawnsKyle and I started to piece together thechain of events and realized that eachtime this occurred, it was preceded by akickstart of a blade server. That led us tolook at the actual kickstart control filewe were using, and it turned out therewas one line in there that caused thewhole problem. The directive clearpart--all --initlabel would erase thepartition table on all disks attached to aparticular host, which made sense if theserver in question had local disks, butthese blades were attached to the SAN,and the SAN didn’t have any zoning inplace to protect against this. As it turnsout, the system did exactly what it wasset up to do. If we had placed the LUNsin zones, this wouldn’t have happened, orif we’d have audited the kickstart controlfile and thought about it in advance, theproblem wouldn’t have happened either.[Kyle: Who would have thought thatkickstart would become yet another oneof those UNIX genie-like commands likedd that do exactly what you say. We not116 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

TALES FROM THE SERVER ROOMonly placed the LUNs in zones, but we alsomade sure that the clearpart directive wasvery specific to clear out only the diskswe wanted—lucky for us, those HP RAIDcontrollers show up as /dev/cciss/ devices,so it was easy to write the restriction.]Lessons LearnedWe learned a couple things that day.First was the importance of zoning yourSAN correctly. The assumption we wereoperating under—that these boxeswould all want to access the SAN and,therefore, zones were unnecessary—was flat-out wrong. Second, was theimportance of auditing and understandingwork that other sysadmins had doneprior and understanding how that workwould affect the new stuff we wereimplementing. Needless to say, our SANalways was zoned properly after that.■Kyle Rankin is a Sr. Systems Administrator in the San FranciscoBay Area and the author of a number of books, including TheOfficial Ubuntu Server Book, Knoppix Hacks and Ubuntu Hacks.He is currently the president of the North Bay Linux Users’ Group.Bill Childers is an IT Manager in Silicon Valley, where he liveswith his wife and two children. He enjoys Linux far too much,and he probably should get more sun from time to time. In hisspare time, he does work with the Gilroy Garlic Festival, but hedoes not smell like garlic.Linux JournaLnow availablefor the iPad andiPhone at theApp Storelinuxjournal.com/iosFor more information about advertising opportunities within Linux Journal iPhone, iPad andAndroid apps, contact Rebecca Cassity +1-713-344-1956 x2 or ads@linuxjournal.com.

EOFRealityFidelity FieldDOC SEARLSWe have to die. Code doesn’t.I’m writing this the day after SteveJobs died, and the man’s famous“reality distortion field” has surelysurvived him. It will be months or yearsbefore anybody can get a good-enoughhandle on what the guy did, and meant.Meanwhile his death looms, larger thanlife. As it should, because death and lifeboth matter, and both need each other.Life, far as we can tell, exists onlyon the surface of one planet of onestar among a hundred billion in ourgalaxy, which is one of a hundredbillion other galaxies. The chance of lifehappening elsewhere exceeds zero, butnot the chance of finding out for surein our lifetimes. Our deathtimes areanother matter. We are all dead formost of eternity.But if life exists elsewhere, it dependson death no less than does our own.That’s because death is more thanthe end or the absence of life. It is acondition required by life. The living eatthe dead. The living also heat, forge,manufacture, build and destroy withthe dead. Except for wind, water, sunand radioactive elements, we produceall our electricity with products and byproductsof death. Oil, coal, wood andgas come straight from death. So doasphalt, concrete, plastics and all naturaland artificial fabrics. Without death, wewould not have handy forms of geologyknown only on Earth: limestone, marble,travertine, chert, diatomite. None ofthe world’s most beautiful caves wouldhave formed, and there would be nostalactites or stalagmites. Without death,no great pyramids, no Notre Dame, noPantheon, no Parthenon.Death uses us to make more of itself.It wants us out of the way. Sooner orlater, we are obliged to cease living,and to burn or flush our remains intodeath’s system. Steve Jobs knew that,years before he departed. Here’s whathe said in a commencement address tograduating students at Stanford in 2005:No one wants to die. Even peoplewho want to go to heaven don’t118 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

want to die to get there. And yetdeath is the destination we allshare. No one has ever escaped it.And that is as it should be, becauseDeath is very likely the single bestinvention of Life. It is Life’s changeagent. It clears out the old to makeway for the new. Right now thenew is you, but someday not toolong from now, you will graduallybecome the old and be clearedaway. Sorry to be so dramatic, but itis quite true.Your time is limited, so don’t wasteit living someone else’s life. Don’tbe trapped by dogma—which isliving with the results of otherpeople’s thinking. Don’t let thenoise of others’ opinions drownout your own inner voice. Andmost important, have the courageto follow your heart and intuition.They somehow already knowwhat you truly want to become.Everything else is secondary.Advertiser IndexThank you as always for supporting ouradvertisers by buying their products!ADVERTISER URL PAGE #AceComputerEmac, Inc.iXsystemsJC Technologyhttp://www.acecomputers.com67http://www.emacinc.com23http://www.ixsystems.com7http://www.jc-technology.com31Logic Supply http://www.logicsupply.com 35, 69Lullabothttp://store.lullabot.com2Microway http://www.microway.com 3, 70, 71Nagios http://www.nagios.com 58, 59Opal EventsRackMountProSCALEhttp://www.opalevents.org75http://www.rackmountpro.com11http://www.socallinuxexpo.org/scale10x85SharePoint Technology Conference http://www.sptechcon.com 121Silicon MechanicsUSENIX Lisahttp://www.siliconmechanics.com21http://www.usenix.org/lisa11/lj93As I write this, much is being saidabout how the tech world will missSteve’s creative muse, his leadership, histaste and the rest of it. Which it will.But “what would Steve do” is exactlythe kind of dogma trap the man warnedus about. So is being like Steve, or likeanybody other than ourselves.Like lots of other people, I appreciateda lot of what Steve Jobs created, even asATTENTION ADVERTISERSThe Linux Journal brand’s following hasgrown to a monthly readership nearlyone million strong. Encompassing themagazine, Web site, newsletters andmuch more, Linux Journal offers theideal content environment to help youreach your marketing objectives. Formore information, please visithttp://www.linuxjournal.com/advertising.WWW.LINUXJOURNAL.COM / DECEMBER 2011 / 119

EOFMost of us in the Linux community have long madea point of working as far as possible outside ofSteve Jobs’ shadow, as well as those of Bill Gatesand other industrial giants of the computing andnetworking worlds.his control freakiness also drove me nuts.But he did what only he could do, andnow he’s gone. His shadow remains long.Yet we have to step out of it.Most of us in the Linux communityhave long made a point of working asfar as possible outside of Steve Jobs’shadow, as well as those of Bill Gatesand other industrial giants of thecomputing and networking worlds. WhatI’m wondering, now that Steve’s deadand Bill has left the building, is whatmore we can do with Linux, free softwareand open source, than we would if theywere still around.There are huge opportunities,especially with mobile devices. It’s beenfun to see Samsung rolling out kernelcode (http://www.androidpolice.com/2011/09/21/att-samsung-galaxy-s-iikernel-source-code-released)for itsAndroid devices, even as Google closesdoors on Android 3.0 “Honeycomb”(http://www.zdnet.com/blog/google/google-android-30-honeycomb-opensource-no-more/2845).I even take heart in the weirdnessof Nokia once again moving towardLinux, this time for its low-end phones,through something called Meltremi(http://thenextweb.com/mobile/2011/09/29/nokias-meltemi-project-tipped-to-bringnew-low-end-linux-os-to-the-next-billion).(What was wrong with Maemo?)But a strength of Linux has alwaysbeen its non-corporate nature. Androidmight belong to Google, but Linuxdoesn’t belong to anybody. Linux’s solepurpose is to be useful. What makes uskeep improving it is a deeply felt need tomaximize that usefulness. That usefulnessoutlives us, but not because it’s dead.See, free and open code is a kind ofliving building material. It’s like woodthat’s still quick. And nothing needs todie for it to keep improving. It’s a productof life that lives to support more life. Allit needs is to be used, patched and reused.As creators, the roles are reversed:code’s gods are mortal, but code doesn’thave to be.■Doc Searls is Senior Editor of Linux Journal. He is also a fellowwith the Berkman Center for Internet and Society at HarvardUniversity and the Center for Information Technology and Societyat UC Santa Barbara.120 / DECEMBER 2011 / WWW.LINUXJOURNAL.COM

Register Earlyand SAVE!Choose from overClasses90 & Workshops!“Great content and speakers.”—Dan Stolts, IT Pro Evangelist, MicrosoftDowntownLocation!San FranciscoHiltonNEW!Check out more than55 exhibiting companies!“I really enjoyed SPTechCon. It was intense, likebeing in a parallel universe. It’s a great place tofind answers to problems, issues and concerns;everyone really wants to help! ”—Bisi Adebesin, Business Analys, Actionet“Great place to get a lot of knowledge in a shortperiod of time.”—Lola Flippo, Sr. Business Solutions Architect, MedseekFollow us at twitter.com/SPTechConA BZ Media Eventwww.sptechcon.com

Linux Journal | December 2011 | Issue 212 - ACM Digital Library

Create successful ePaper yourself

Delete template?

Save as template?