http://www.secuobs.com Observatoire de la securite Internet fr webmaster@secuobs.com 2009-08-11 10:52:22 - unethical blogger : It seems every time jasonrubenstein, ggoss3, cablelounger and I sit down to have lunch together, we invariably sway back and forth between generic venting about work stuff and best practices for doing aforementioned work stuff better The topic of reusable code came up over Mac 'n Cheese and beers this afternoon, and I felt it warranted wider distribution so to speak yet-another-lame-Slide-inside-joke We, Slide, are approaching our fourth year in existence as a startup which means all sorts of interesting things from an investor standpoint, employees options are starting to become fully-vested and other mundane and boring financial terms Being an engineer, I don't care too much about the stocks and such, but rather about development four years is a lot from a code-investment standpoint my bias towards code instead of financial planning will surely bite me eventually Projects can experience bitrot, bloating read Vista'ing and a myriad other illnesses endemic to software that's starting to grow long in the tooth At Slide, we have a number of projects on slightly different trajectories and timelines, meaning we have an intriguing cross-section of development histories representing themselves We are no doubt experiencing a similar phenomenon to Facebook, MySpace, Yelp and a number of other startups who match this same age group of 4-7 years Just like our bretheren in the startup community, we have portions of code that fit all the major possible categories That which was written extremely fast, without an afterthought to what would happen when it serve tens of millions of users That which was written slowly, trying to cater to every possible variation, ultimately to go over-budget and over-schedule That which has been rewritten And rewritten And rewritten Then the exceptionally rare, that which has been written in such a fashion that it has been elegantly extended to support more than it was originally conceived to support In all four cases, we whereas we refers to an engineering department have invested differently in our code portfolio depending on a number of factors and information given at the time For example, it's been a year since Component X was written Component X is currently used by every single product The Company owns, but over the past year it's been refactored and partially rewritten each time a new product starts to use Component X In its current state, Component X's code reads more like an embarrasing submission to The Daily WTF with its hodge-podge of code, passed from team to team, developer to developer, like some expensive game of Telephone for software engineers After the fact, it's difficult and not altogether helpful to try to lay blame with the mighty sword of hindsight, but it is feasible to identify the reasons for the N number of developer hours lost fiddling, extending, and refactoring Component X Was the developer responsible for implementing Component X originally aware of the potentially far reaching scope of their work Was the developer given an adequate time frame to implement a proper solution, or this should have shipped yesterday Did somebody pass the project off to an intern or somebody who was on their way out the door Were other developers in similar realms of responsibility asked questions or for their opinions Is was the culture proliferated by Engineering Leads and Managers encouraging of best practices that lead to extensible code I've found, watching Slide Engineering culture evolve, that the majority of libraries or components that go through multiple time resource-expensive iterations tend to have experienced shortcomings in one of the five sections above More often than not, a developer was given the task to implement Some Thing Simple enough, Some Thing is developed with the specific use-case in mind, and the developer moves on with their life Three months later however, somebody else asks another developer, to add Some Thing to another product Product X has Some Thing, and it works great for them, let's incorporate Some Thing into Product Y by the end of the week Invariably this leads to heavy developer drinking And then perhaps some copy-paste, with a dash of re-jiggering, and quite possibly multiple forks of the same code That is, if Some Thing was not properly planned and designed in the first place Working as a developer on products that move at a fast pace, but will be around for longer than three months is an exercise in investment strategy ie managing technical debt What makes great Engineering Managers great is their ability to determine when and where to invest the time to do things right, and where to write some Perl-style write-only code zing What makes a startup environment a more difficult one to work on your code portfolio is that you don't usually know what may or may not be a success, and in a lot of cases getting your product out there now is of paramount importance Unfortunately there isn't any simple guideline or silver bullet, and there is no bailout, if you invest your time poorly up front, there will be nobody to save you further down the line when you're staring an resource-devouring refactor in its ugly face Where do you invest the time in any given project What will happen if you shave a few days by deciding not to write any tests, or documentation Will it cost you a week further down the road if you take shortcuts now I wish I knew IMAGE IMAGE IMAGE http://www.secuobs.com/revue/news/129798.shtmlhttp://www.secuobs.com/revue/news/129798.shtml Secuobs.com : 2009-08-11 10:52:22 - unethical blogger - disregard the mismatched bedding, I'm told we're going to fix that IMAGE IMAGE IMAGE http://www.secuobs.com/revue/news/129797.shtmlhttp://www.secuobs.com/revue/news/129797.shtml Secuobs.com : 2009-07-29 10:22:35 - unethical blogger - Since moving to the infrastructure team at Slide I've found the rate at which my software gets deployed has plummeted, while the quantity of the code that I am deploying to the live site has sky-rocketed When on an applications-team within Slide, code is typically pushed in small incrememnts a few days a week, if not daily This allows for really exciting compact milestones that make more fine-grained analysis achievable, post-push for product management and metrics purposes On the infrastructure team however, the requirements are wholly different, the fail-fast, ship-now mentality that prevails when doing user-facing web application development just does not work in infrastructure The most important aspects of building out infrastructure components are stability and usability, our customers are the rest of engineering, and that has a definite effect on your workflow Code Review ----------- One of the things that jasonrubenstein and I always did when we worked together, was occasional code review In the majority of cases, our code review sessions were more or less rubber duck debugging, but occasionally it would escalate into more complex discussions about the right way to do something When you're writing infrastructure software for services that are handling tens of millions of users the notion of code review goes from being optional to being absolutely required Discussions are had on the correctness or performance characteristics of database indexes, the necessity of some objects instantiating default values of attributes or having them lazily load, or debating garbage collection of objects while meticulously watching memory consumption For one of my most recent projects, I was working on something in C, a rarity at Slide since we work with managed code in Python the majority of the time As the project neared completion, I counted roughly two or three hours of code review time dedicated by our Chief Architect The attention to detail paid to this code was extremely high, as the service was going to be handling millions of requests from other levels of the Slide infrastructure, before getting cycled or restarted A particularly frustrating aspect of code review by your peers is that a second set of eyes not only will find problems with your code, but will likely mean refactoring or bug fixes, more work In my case, whenever a bug or stability issue was discovered, a test needed to be written for it to make sure the bug did not present itself again, the workload would be larger than if I had just fixed the bug and moved on with my life Testing, oh the testing ----------------------- If you expect to write an API, have it stablize, and then be used, you must write test cases for it I'm not a TDD nut , I actually hate writing test cases, I absolutely abhor it Writing test cases is responsible and the adult thing to do In my experience, it can also be tedious and usually comes as a result of finding flaws in my own software The majority of tests that I find myself writing are admissions of defeat, admitting that I don't crap roses and by george, my code isn't perfect either On the flipside however, I hate debugging even more Stepping through a call stack is on par with waterboarding in my book, torture Which means I'm more than willing to tolerate writing tests so long as it means I can be certain I will be cutting down on the time spent being tortured with either pdb or gdb In almost every situation where I've written tests properly, like the responsible developer that I am, I find them saving me at some point It might be getting late, or I'm just feeling a little cavalier, but tests failing almost always indicates that I've screwed something up I shouldn't have Additionally, now that the majority of my projects are infrastructure-level projects, the tests I write serve a second undocumented purpose, they provide ready-made examples for other developers on how to use my code Bonus The more and more code I write, the more amazed I am at the pushback against testing in general, there exists decent libraries for every language imaginable well, perhaps BrainfuckUnit doesn't exist , and its sole purpose in my opinion is to save develpoment time, particularly when coupled with a good continuous integration server Further to that effect, if you're building services for other developers to use, and you're not writing tests for it, you're not only wasting your time and your employer's money, but the time of your users as well read stop being a jerk Sure there are a lot of articles books etc about writing stable code, but in my opinion, solid code review and testing will stablize your code far more than any design pattern ever will IMAGE IMAGE IMAGE http://www.secuobs.com/revue/news/126093.shtmlhttp://www.secuobs.com/revue/news/126093.shtml Secuobs.com : 2009-07-26 06:58:24 - unethical blogger - A coworker of mine, teepark and I recently fell in love with tiling window managers, Awesome in particular The project has been interesting to follow, to say the least When I first installed Awesome, from the openSUSE package directory, I had version 2, it was fairly basic, relatively easy to configure and enough to hook me on the idea of a tiling window manager After conferring with teepark, I discovered that he had version 3 which was much better, had some new fancy features, and an incremented version number, therefore I required it In general, I'm a fairly competent open-source contributor and user Autoconf and Automake, while I despise them, aren't mean and scary to me and I'm able to work with them to fit my needs I run Linux on two laptops, and a few workstations, not to mention the myriad of servers I'm either directly or peripherally responsible for I grok open sources Thusly, I was not put off by the idea of grabbing the latest stable tarball of Awesome to build and install it That began my slow and painful journey to get this software built, and installed Oh, it needs Lua, I'll install that from the repositories Hm, what's this xcb I need, and isn't in the repositories I guess I'll have to build that myself, oh but wait, there's different subsets of xcb xcb-util, xcb-proto, libxcb-xlib, xcb-kitchensink, etc Well, I need xproto as well, which isn't in the repositories either CMake Really guys Fine ImLib2, I've never even heard of that libstartup-notification huh Fine, i'll build this too After compiling what felt like an eternity of subpackages, I discovered a number of interesting things about the varying versions of Awesome v3 The configuration file format has changed a few times, even between one release candidate to another I ran across issues that other people had that effectively require recompilling X11's libraries to link against the newly built xcb libraries in order to work usr lib libxcb-xlibso0 undefined reference to _xcb_unlock_io Nothing I seemed to try worked as I might expect, if I couldn't recompile the majority of my system to be bleeding edge I was screwed The entire affair was absolutely infuriating There were a few major things that I think the team behind Awesome failed miserably at accomplishing, that every open source developer should consider when releasing software If you depend on a hodge-podge of libraries, don't make your dependency on the bleeding edge of each package Maintain an open dialogue with those that package your software, don't try to make their job hell When a user cannot build your packages with the latest stable versions of their distribution without almost rebuilding their entire system, perhaps you're doin' it wrong Changing file formats, or anything major between two release candidates is idiocy If you don't actually care about your users, be sure to state it clearly, so then we don't bother using or trying to improve your poor quality software In the end, I decided that Haskell isn't scary enough not to install XMonad, so I've started replacing machines that run Awesome, with XMonad, and I'm not looking back Ever IMAGE IMAGE IMAGE http://www.secuobs.com/revue/news/124789.shtmlhttp://www.secuobs.com/revue/news/124789.shtml Secuobs.com : 2009-07-22 09:25:39 - unethical blogger - At the Hudson Bay Area Meetup Hackathon that Slide, Inc hosted last weekend, I worked on the Jython plugin and released it just days after releasing a strikingly similar plugin, the Python plugin I felt that an explanation might be warranted as to why I would do such a thing For those that don't know, Hudson is a Java-based continuous integration server, one of the best CI servers developed in my humblest of opinions What makes Hudson so great is a very solid plugin architecture allowing developers to extend Hudson to support a wide variety of scripting languages as well as notifiers, source control systems, and so on related post on the growth of Hudson's plugin ecosystem Additionally, Hudson supports slaves on any operating system that Java supports, allowing you to have a central manager the master Hudson server node and a vast network of different machines performing tasks and executing jobs Now that you're up to speed, back to the topic at hand Jython versus Python plugin Why bother with either, as gboissinot pointed out in this tweet The interesting thing about the Jython plugin, particularly when you use a large number of slaves is that with the installation of the Jython plugin, suddenly you have the ability to execute Python script on every single slave, regardless of whether or not they actually have Python installed The more third party that can be moved into Hudson by way of the plugin system means reduced dependencies and difficulty setting up slaves to help handle load Take the git versus the git2 plugin, the git plugin was recently criticized on the hudson channel because of it's use of the JGit library, versus git2 which invokes git 1 on the command line The latter approach is flawed for a number of reasons, particularly the reliance on the git command line executables and scripts to return consistent formatting is specious at best even if you aren't relying on porcelain git community terminology for front-end-ish script and code sitting on top of the plumbing , the breakdown is detailed here The command-line approach also means you now have to ensure every one of your slaves that are likely to be executing builds have the appropriate packages installed One the flipside however, with the JGit-based approach, the Hudson slave agent can transfer the appropriate bytecode to the machine in question and execute that without relying on system-dependencies The Hudson Subversion plugin takes a similar approach, being based on SVNKit Being a Python developer by trade, I am certainly not in the Java Fanboy camp, but the efficiencies gained by incorporating Java-based libraries in Hudson plugins and extensions is a no brainer, the reduction of dependencies on the systems incorporated in your build farm will save you plenty of time in maintenance and version woes alone In my opinion, the benefits of JGit, Jython, SVNKit, and the other Java-based libraries that are running some of the most highly used plugins in the Hudson ecosystem continue to outweigh the costs, especially as we find ourselves bringing more and more slaves online IMAGE IMAGE IMAGE http://www.secuobs.com/revue/news/123407.shtmlhttp://www.secuobs.com/revue/news/123407.shtml Secuobs.com : 2009-07-14 20:43:20 - unethical blogger - For about a month now I've been sending weekly Protip emails about Git to the rest of engineering here at Slide I've been using them to slowly and casually introduce some of the more interesting features Git has to offer as we move away from Subversion entirely Below is the first Protip I sent around, I'll be sure to send the rest in good time --------------------------------------------------------------------- Given the nature of how Git is structured, in that your working copy is also your repository you might find yourself switching branches relatively often While you can actually switch branches with uncommited changes outstanding in your Git directory, it's not advised as you might forget you're commiting changes in the wrong branch, etc You have two options if you are halfway through some work, one is to commit a checkpoint revision, but the other is to make use of the git stash command A scenario where this becomes especially useful would be Bill is working in his local branch wip-funtime on replacing large swaths of unnecessary useless code and Ted accidentally pushes some of Bill's other changes from another branch live and things break Bill could commit his code and write a fairly uninformative log message like checkpoint which cheapens the value of the revision history of his changes or Bill can use git stash to snapshot his currently working state and context switch In this case Bill would execute the following commands git stash git checkout master-media perform hotfixes git checkout wip-funtime git stash pop After performing the git stash pop command, Bill's Git repository will be in the exact same state, all his uncommitted changes in tact, as it was when he originally stashed and context-switched For specific usage of git stash refer to the git stash man page Example usage of git stash Stashing changes away 1 tyler starfruit source git main bt git stash 2 Saved working directory and index state WIP on master-topfriends 7b1ce9e TOS copy fix 3 To restore them type git stash apply 4 HEAD is now at 7b1ce9e TOS copy fix 5 tyler starfruit source git main bt Looking at the stash 1 tyler starfruit source git main bt git stash list 2 stash 0 WIP on master-topfriends 7b1ce9e TOS copy fix 3 stash 1 On master-topfriends starfruit complete patchset 4 stash 2 On wip-classmethod starfruit patches 5 tyler starfruit source git main bt Grabbing the latest from the stash 1 tyler starfruit source git main bt git stash pop 2 Dropped refs stash 0 94b9722b5a999c32c4361d795ee8f368d8412f9a 3 tyler starfruit source git main bt Grabbing a specific stash 1 tyler starfruit source git main bt git stash list 2 stash 0 WIP on master-topfriends 7b1ce9e TOS copy fix 3 stash 1 On master-topfriends starfruit complete patchset 4 stash 2 On wip-classmethod starfruit patches 5 tyler starfruit source git main bt git stash apply 2 6 On branch master-topfriends 7 Changed but not updated 8 use git add to update what will be committed 9 10 modified db dbrootpy 11 modified gogreen coropy 12 modified py bin makepyreleasepy 13 modified py initpkgpy 14 modified py misc distpy 15 modified py misc testing test_initpkgpy 16 modified py path local localpy 17 modified py test terminal terminalpy 18 tyler starfruit source git main bt --------------------------------------------------------------------- Did you know Slide is hiring Looking for talented engineers to write some good Python and or JavaScript, feel free to contact me at tyler at slide http://www.secuobs.com/revue/news/120460.shtmlhttp://www.secuobs.com/revue/news/120460.shtml Secuobs.com : 2009-07-14 20:43:20 - unethical blogger - I've been sending weekly Protip emails about Git to the rest of engineering here at Slide for a while now, using the Protips as a means of introducing more interesting and complex features Git offers Below is the second Protip written to date --------------------------------------------------------------------- One of the major benefits to using Git is the entirety of the repository being entirely local and easily searched queried For this, Git has a very useful command called git log which allows you to inspect revision histories in numerous different ways between file paths, branches, etc There are a couple basic scenarios where git log has become invaluable, for me at least, in order to properly review code but also to track changes effectively from point A to point B What's Dave been working on lately with diffs git log -p --no-merges --author dave The --no-merges option will prevent git log from displaying merge commits which are automatically generated whenever you pull from one Git branch to another Before I merge this branch down to my team master, I want to know what files have been changed and what revisions git log --name-status master-topfriendsproj-topfriends-thing Git supports the ability with git log and with git diff to provide unidirectional branch lookups or bidirectional branch lookups For example, say the left branch has commits A, B and the right branch has commits A, C The syntax will output C , whereas will output B, C I just got back from a vacation, I wonder what's changed git log --since 2 weeks ago --name-status -- templates At the tail end of a git log command you can specify particular paths to look up the histories for with the -- operator, in this case, I will be looking at the changes that have occured in the templates directory over the past two weeks Most recent X number of commits with diffs git log -n 10 --no-merges -p All git log commands automatically filter into less 1 so you can page through the output like you would normally if you executed a svn log less Because git log is simply reading from the locally stored revision history you can quickly grep the history by any number of different search criteria to gain a better understanding of how the code base is changing and where For more specific usage of git log refer to the git log man page --------------------------------------------------------------------- Did you know Slide is hiring Looking for talented engineers to write some good Python and or JavaScript, feel free to contact me at tyler at slide http://www.secuobs.com/revue/news/120459.shtmlhttp://www.secuobs.com/revue/news/120459.shtml Secuobs.com : 2009-07-14 20:43:20 - unethical blogger - I've been sending weekly Protip emails about Git to the rest of engineering here at Slide for a while now, using the Protips as a means of introducing more interesting and complex features Git offers Below is the third Protip written to date --------------------------------------------------------------------- The concept of revert in Git versus Subversion is an interesting, albeit subtle, change, mostly in part due to the subtle differences between Git and Subversion's means of tracking commits Just as with Subversion, you can only revert a committed change, unlike Subversion there is a 1-to-1 mapping of a commit to a revert The basic syntax of revert is quite easy git revert 0xdeadbeef, and just like a regular commit, you will need to push your changes after you revert if you want others to receive the revert as well In the following example of a revert of a commit, I also use the -s argument on the command line to denote that I'm signing-off on this revert ie I've properly reviewed it xdev3pourcents git revert -s c20054ea390046bd3a54693f2927192b2a7097c2 ---------------- Vim ---------------- Revert merge-to-release unhide 10000 coin habitat This reverts commit c20054ea390046bd3a54693f2927192b2a7097c2 Signed-off-by R Tyler Ballance Please enter the commit message for your changes Lines starting with ' ' will be ignored, and an empty message aborts the commit On branch wip-protips Changes to be committed use git reset HEAD to unstage modified bt apps pet datapy python bt qa git post-commitpy -m svn slidecom Sending a commit mail to svn slidecom Created commit a6e93b8 Revert merge-to-release unhide 10000 coin habitat 1 files changed, 4 insertions , 3 deletions - Reverting multiple commits -------------------------- Since git revert will generate a new commit for you every time you revert a previous commit, reverting multiple commits is not as obvious side note I'm aware of the ability to squash commits, or --no-commit for git-revert 1 , I dislike compressing revision history when I don't believe there shouldn't be compression If you want to revert a specific merge from one branch into the other, you can revert the merge commit provided one was generated when the changes were merged Take the following example commit 81a94bb976dfaaaae42ae2600b7e9e88645ebd81 Merge 8134d17 d227dd8 Author R Tyler Ballance Date Thu Nov 20 10 15 16 2008 -0800 Merge branch 'master' into wip-protips I want to revert this merge since it refreshed my wip-protips branch from master, and brought in a lot changes tat have destablized my branch In the case of reverting a merge commit, you need to specify -m and a number to denote where the mainline branch for Git to pivot off of is, -m 1 usually suffices So the revert of the commit above will look something like this git revert 81a94bb976dfaaaae42ae2600b7e9e88645ebd81 -m 1 Then my revert commit will be committed after I review the change in Vim commit 8cae4924c4c05dadaaeccb3851cfc9ec1b8efd0f Author R Tyler Ballance Date Thu Nov 20 10 20 44 2008 -0800 Revert Merge branch 'master' into wip-protips This reverts commit 81a94bb976dfaaaae42ae2600b7e9e88645ebd81 Let's take the extreme case where I don't have a merge commit to pivot off of, or I have a particular set of bare revisions that I need to revert in one pass, you can start to tie Git subcommands together like git-rev-list 1 to accomplish this This hypothetical situation might occur if some swath of changes have been applied to a team-master that need to be backed out Without a merge commit to key off of, you have to revert the commits one by one, but that doesn't mean you have to revert each one by hand for r in git rev-list mastermaster-fubar --since 8 00 --before 12 00 --no-merges do git revert --no-edit -s r done In the above example, I can use git-rev-list 1 to give me a list of revisions that have occurred on master-fubar that have not occurred on master between the times of 8 am and 12 pm, excluding merge commits Since git-rev-list 1 will return a list of commit hashes by default, I can loop through those commit hashes in chronological order and revert each one The inner part of the loop signs-off on the revert -s and then tells git-revert 1 to auto-commit it without opening the commit message in Vim --no-edit What this then outputs is the following xdevpourcents for r in git rev-list mastermaster-fubar --since 8 00 --before 12 00 --no-merges do git revert --no-edit -s r done Finished one revert Created commit b6810d7 Revert a test, for you 1 files changed, 1 insertions , 1 deletions - Finished one revert Created commit 83156bd Revert These are not the droids you are looking for 1 files changed, 2 insertions , 0 deletions - Finished one revert Created commit 782f328 Revert commented out stuff 1 files changed, 0 insertions , 3 deletions - Finished one revert Created commit 2b8d664 Revert back on again 1 files changed, 1 insertions , 1 deletions - xdevpourcents For specific usage of git-revert or git-rev-list refer to the git-revert 1 man page or the git-rev-list 1 man page --------------------------------------------------------------------- Did you know Slide is hiring Looking for talented engineers to write some good Python and or JavaScript, feel free to contact me at tyler at slide http://www.secuobs.com/revue/news/120458.shtmlhttp://www.secuobs.com/revue/news/120458.shtml Secuobs.com : 2009-07-14 20:43:20 - unethical blogger - Rod reminded me with his comment in one of my other posts that I've not yet mentioned github I've got a bunch of my nonsense thrown up on githubcom rtyler, it's awesome no really, github rocks my socks, those guys are good people http://www.secuobs.com/revue/news/120457.shtmlhttp://www.secuobs.com/revue/news/120457.shtml Secuobs.com : 2009-07-14 20:43:20 - unethical blogger - I've spent the better part of my weekend messing around with mail clients, and once again Evolution comes out on top and once again, I'm not happy about it I tried Claws, Thunderbird, Alpine formerly Pine , Mutt, Balsa, KMail and TkRat None of them worked as well as I wanted, is it too much to ask for to have a mail client that doesn't puke and die on large 2GB of IMAP mail Supports proper jwz mail threading And caches IMAP mail locally so I can actually access it while disconnect Turns out it actually is too much to ask That's not what this is about though While hunting around, I started to look at my Slide IMAP mail account, and see something interesting it looks suspiciously like a filesystem The general layout I have right now is something like this INBOX Sent Drafts Development Commits Pushes External Git Hudson Metrics QA Exceptions Trac Clearly, it's a very filesystem-esque looking tree of mail and a couple gigabytes of it When you start to really dig into e-mail technology, you really get a feeling for how royally screwed up the whole ecosystem is Between Exchange, IMAP and POP3 and their SSL counterparts , mbox and Maildir, and of course the venerable SMTP e-mail technology is a clusterfuck No wonder barely anybody can implement an e-mail client that doesn't suck At a basic level, mail is organized into messages and folders Messages map very easily to actual files on the filesystem, and folders naturually map into actual directories on the filesystem Imagine that you could chose any program you wanted to read and write your email The only pre-requisite can it read from the filesystem You could have any program register to receive filesystem events to notify you when mail appears in specific directories, and you could move mail around with a simple drag-and-drop in Nautilus Thunar Finder What about writing mail though Easy enough, you create a new file in the Drafts folder, writes would naturally be propogated to the Drafts folder on the IMAP server, and when you were done with the message, you could copy or move it into the Sent folder, which would have a hook to recognize the new file, and send it The IMAP tree from above, starts to look something like this Imuse Settings Accounts Slide INBOX Sent Drafts Development Commits Pushes External Git Hudson Metrics QA Exceptions Trac Accounts and Settings would likely need to be special insofar that Imuse would just create them out of thin air, Accounts would need to be a virtual directory to actually contain the appropriate account listings, and in Settings I'd likely want to have a couple of flat configuration files that you could edit in order to actually configure Imuse appropriately If there are simply lists of files in each of the Accounts' folders, each representing a particular email, then the problem of dealing with all my e-mail becomes a much easier one to handle, then it's just a matter of picking my filesystem browser of choice Even then it's not really limited to filesystem browsers like Nautilus, the scope of programs that I can use to access my mail is opened up to EDITOR as well Most editors like Notepad , Vim, Emacs, Gedit, and TextMate all support the ability to view a directory, and open it's contents up for reading editing I'm a big fan using Vim, so Imuse coupled with vtreeexplorer would be phenomonal to say the least I've started toying around with building FUSE filesystems and I've pushed my experimenting up to GitHub in my imuse repository It's currently in C, since I either cannot get either of the two FUSE Python bindings to work properly This presents a certain level of difficulty, since the standard means of accessing IMAP data from C seems to be c-client, which is reasonably well documented, but lacks sample code On the other hand, if I can get the Python bindings to cooperate, then I have access to the wonderful Twisted Mail library or even the basic imaplib Given my obvious time restrictions, I wanted to open the idea up to more eyes and ears to see what others thought and maybe even find somebody else willing to pitch in For the time being however, Evolution is still sifting through my mail, and I'm still not enjoying it http://www.secuobs.com/revue/news/120456.shtmlhttp://www.secuobs.com/revue/news/120456.shtml Secuobs.com : 2009-07-14 20:43:20 - unethical blogger - In addition to frothing at the mouth about Git, I've been really getting into the concept of automated unit tests lately thus my interest in Hudson Just like code comments however, tests are good, no tests is bad, wrong tests is worse That means once you give in to the almighty power of unit testing, you are saddled with the curse of knowing that you will have to update them, forever Taking up Test-driven Development is like having a child, if you are at a point in your life where you're ready to accept that kind of responsibility, it can be wonderful, a lot of work, but ultimately you will feel satisfied with your new role as a Responsible Developer tm If you're not prepared to take on the burden that TDD will present you with, you will likely regret it or neglect your tests Deadbeat Developer, I like this metaphor In the Top Friends Team at Slide, we practice the more loose definition of TDD tests are not written before functionality is written, but rather functionality is written, and then as part of the QA and release process, the appropriate and accompanying tests are written Our basic workflow is usually as follows Tickets are written and assigned to milestones and developers in Trac Branch is created in central Git repository General plan-of-action is discussed between developers Hack-hack-hack Code complete is reached, QA starts to test milestone Developers write tests if needed for functionality Once QA signs off, and tests look solid, code is shipped live There are two primary flaws with this workflow, the first and most obvious one is that it is far to easy to forget to write the tests That is, the project scheduled to start development tends to flow forward into the allotted test-writing time As important as test coverage is, at the end of the day Slide did not raise funding on having solid test coverage, and our priorities lie in shipping software, first and foremost Solving the flow-forward of scheduled projects into any available space is something that can be worked on, but never solved, it really comes down to discipline between those in charge of setting up any given project's particular roadmap The second, more subtle flaw in this workflow, and I think all Test-driven Devleopment workflows, revolves around the writer of the tests The fundamental nature of almost all bugs in software is human error, our natural tendency to make mistakes means that nothing we do will ever be perfect, including our tests If Developer A is writing a couple new methods to handle data validation prior to that data going into the database Chances are that Developer A's life is going to be made far easier by writing some test cases to run through some predefined user-input, and pass his validation code over it Therein lies the problem, if the developer doesn't think of a particular edge case when he's writing the code to handle the data validation, the chances he'll remember and account for that particular edge case while he's working on the unit tests is nil How do you really ensure that tests are of high enough quality to actually catch errors and regressions I think a certain extent of intra-team test writing and code review, depending on the level of communication between developers, can really help In this case less developer communication is better If Developer A tells Developer B how his code works, Developer B is now going to have an unnecessary expectation when he starts to write tests for Developer A's code If Developer B reviews the code for what it actually is, instead of what Developer A thinks it is, the tests that will ultimately be written will be more thorough than if Developer A had written the whole suite himself This still isn't sufficiently fool-proof to where I feel all that confident in test coverage, the tests being written are subject to the availability, thoroughness and understanding that Developer B brings to the table Inside a small team like this one, one of those is almost always in short supply usually availability One approach I'm anxious to try is the more active involvement of QA engineers in the test writing process, both in the pre-fail and post-fail scenarios The pre-fail scenario being one like that which I detailed above, where new code is being written In this case a QA engineer's experience can help guide the developer on what sets of user-input have typically caused issues in the past The second case, post-fail, is actually already occuring at Slide a live issue, data validity bug, or regression is caught by QA engineers who detail the reproduction case in Trac and as a result a regression test can be written for that specific issue This still is subject to the three things I cited above availability, thoroughness and understanding of those involved I still have a lot of unanswered questions about the ideal QA and Dev workflow however, how does this scale to a team of tens or hudnreds Who writes the tests for large teams What about a team of 1 Dev and a 1 QA, what about the lone-hacker How do you write quality code, without getting bogged down in the mush of writing thousands of tests for everything you can imagine could go wrong Who writes the tests http://www.secuobs.com/revue/news/120455.shtmlhttp://www.secuobs.com/revue/news/120455.shtml Secuobs.com : 2009-07-14 20:43:20 - unethical blogger - I've been sending weekly Protip emails about Git to the rest of engineering here at Slide for a while now, using the Protips as a means of introducing more interesting and complex features Git offers Below is the fourth Protip written to date --------------------------------------------------------------------- While the concept of tagging or labeling code is not a new, or original idea that was introduced with Git, our use of tags in a regular workflow does not predate the migration to Git however At it's most basic level, a tag in any version control system is to take a picture of how the tree looks at a certain point in time such that it can be re-created later This can be extremely helpful for both local and team development, take the following scenario for local development using tags Tim is extremely busy, most of his days working at an exciting, fast-paced start-up seem to fly by With one particular project Tim is working on, a lot of code is changing at a very fast pace and the branch he's currently working in is stable one minute and destabilized the next Tim has two basic options for leaving himself bread-crumbs to step back in time to a stable or an unstable state The first, complicated option, is to mark his commit messages with something like STABLE , etc so he can git diff or git reset --hard from the current HEAD to the last stable point of the branch The second option is to make use of tags Whenever Tim reaches a stable point in his turmultuous development, he can simply run git tag wip-protips_ date pourcentss or something similar, date added to ensure the tag is unique If Tim finds himself too far down the wrong path, he can rollback his branch to the latest tag git reset --hard protiptag , create a new stable branch based on that tag git checkout -b wip-protip-2 protiptag , or diff his current HEAD to the tag to see what all he's changed since his branch was stable git diff protiptagHEAD This local development scenario can become a team development scenario involving tags, if for example, Tim needed QA to start testing portions of his branch his changes are just that important Since the current HEAD of Tim's branch is incredibly unstable, he can push his tag to the central repository so QA can push a stage using the tag to the last stable point in the branch's history with the command git push origin tag protiptag Tags are similar to most other refs in Git insofar that they are distributable, if I execute git fetch your-repo --tags, I can pull the tags you've set in your-repo and apply them locally aid development The distributed nature is primarily how tags differ in Git from Subversion, nearly the rest of the concept is the exact same Currently at Slide, tag usage is dominated by the post-receive hook in the central repository, where every push into the central repository origin in the branch release branch is tagged This allows us to quickly revert bad live pushes temporarily, by simply pushing the last good tagged release, to ensure minimal site destabilization while we correct live issues outside of the release branch For more specific usage of git tag refer to the git-tag 1 man page --------------------------------------------------------------------- Did you know Slide is hiring Looking for talented engineers to write some good Python and or JavaScript, feel free to contact me at tyler at slide http://www.secuobs.com/revue/news/120454.shtmlhttp://www.secuobs.com/revue/news/120454.shtml Secuobs.com : 2009-07-14 20:43:20 - unethical blogger - I've been sending Protip emails about Git to the rest of engineering here at Slide for a while now, using the Protips as a means of introducing more interesting and complex features Git offers --------------------------------------------------------------------- There are those among us who can look at a reproduction case for a bug and just know what the bug is For the rest of us mere mortals, finding out what change or set of changes actually introduced a bug is extremely useful for figuring out why a particular bug exists This is even more true for the more elusive bugs or the cases where code looks correct and you're stumped as to why the bug exists now, when it didn't yesterday last week last month The options in most classical version control systems you have available to you are to sift through diffs or wade through log message after log message trying to spot the particular change that introduced the regression you're now tasked with resolving Fortunately of course Git offers a handy feature to assist you in tracking down regressions as they're introduced, git bisect Take the following scenario Roger has been working on some lower level changes in a project branch lately When he left work last night, he ran his unit tests everything passed , committed his code and went home for the day When he came in the next morning, per his typical routine, he synchronized his project branch with the master branch to ensure his code wasn't stomping on released changes For some reason however, after synchronizing his branch, his unit tests started to fail indicating that a bug was introduced in one of the changes that was integrated into Roger's project branch Before switching to Git, Roger might have spent an hour looking over changes trying to pinpoint what went wrong, but now Roger can use git bisect to figure out exactly where the issue is Taking the commit hash from his last good commit, Roger can walk through changes and pinpoint the issue as follows Format for use is git bisect start -- xdev4pourcents git bisect start HEAD 324d2f2235c93769dd97680d80173388dc5c8253 Bisecting 10 revisions left to test after this 064443d3164112554600f6da39a36ffb639787d7 Changed the name of an a b test xdev4pourcents This will start the bisect process, which is interactive, and start you halfway between the two revisions specified above see the image below Following the scenario above, Roger would then run his unit tests Upon their success, he'd execute git bisect good which would move the tree halfway between that good revision and the bad revision Roger will continue doing this until he lands on the commit that is responsible for the regression Knowing this, Roger can either revert that change, or make a subsequent revision that corrects the regression introduced IMAGE A sample of what this sort of transcript might look like is below xdev4pourcents git bisect good Bisecting -1 revisions left to test after this bcf020a6c4ac7cc5df064c66b182b2500470000a Merge branch 'cjssp' into master xdev4pourcents git bisect bad bcf020a6c4ac7cc5df064c66b182b2500470000a is first bad commit xdev4pourcents git show bcf020a6c4ac7cc5df064c66b182b2500470000a commit bcf020a6c4ac7cc5df064c66b182b2500470000a Merge 62153e2 064443d Author Chris Date Tue Jan 27 12 57 45 2009 -0800 Merge branch 'cjssp' into master xdev4pourcents git bisect log bad 7a5d4f3c90b022cb66fd8ea1635c5de6768882d7 Merge branch 'foo' into master good d1014fd52bebd3c56db37362548e588165b7f299 Merge branch 'bar' git bisect start 'HEAD' 'd1014fd52bebd3c56db37362548e588165b7f299' '--' 'apps' good 064443d3164112554600f6da39a36ffb639787d7 Changed the name of an a b test PLEASE PICK ME UP WITH NEXT PUSH thx git bisect good 064443d3164112554600f6da39a36ffb639787d7 bad bcf020a6c4ac7cc5df064c66b182b2500470000a Merge branch 'cjssp' into master git bisect bad bcf020a6c4ac7cc5df064c66b182b2500470000a xdev4pourcents git bisect reset xdev4pourcents Instead of spending an hour looking at changes, Roger was able to quickly walk a few revisions and run the unit tests he has to figure out which commit was the one causing trouble, and then get back to work squashing those bugs Roger is, like most developers, inherently lazy, and running through a series of revisions running unit tests sounds like work that doesn't need to be done Fortunately for Roger, git-bisect 1 supports the subcommand run which goes hand in hand with unit tests or other tests In the example above, let's pretend that Roger had a test case exhibiting the bug he was noticing What he could actually do is let git bisect run automatically run a test script to run his unit tests to find the offending revision ie xdev4pourcents git bisect start HEAD 324d2f2235c93769dd97680d80173388dc5c8253 Bisecting 10 revisions left to test after this 064443d3164112554600f6da39a36ffb639787d7 Changed the name of an a b test xdev4pourcents git bisect run mytestsh After executing the run command, git-bisect 1 will binary search the revisions between GOOD and BAD testing whether or not mytestsh returns a zero success or non-zero failure return code until it finds the commit that causes the test to fail The end result should be the exact commit the regression was introduced into the tree, after finding this Roger can either grab his rubber chicken and go slap his fellow developer around or fix the issue and get back to playing Nethack All in all git-bisect 1 is extraordinarily useful for pinning down bugs and diagnosing issues as they're introduced into the code base For more specific usage of git bisect refer to it's man page here git-bisect 1 man page --------------------------------------------------------------------- Did you know Slide is hiring Looking for talented engineers to write some good Python and or JavaScript, feel free to contact me at tyler at slide http://www.secuobs.com/revue/news/120453.shtmlhttp://www.secuobs.com/revue/news/120453.shtml Secuobs.com : 2009-06-18 11:06:23 - unethical blogger - For one of the projects I've been working on lately, I've been makinguse of Glib's Testing functionality to write unit tests for a C-basedproject I'm a fairly big fan of the Hudson Continuous IntegrationServer and I wanted to run tests for my C-based projectUnfortunately, the gtester application generates XML in a customformat that Hudson cannot understand ie JUnit formatted XML Inorder to come up with some JUnit XML, I spent about an hour and a halftoying with XSLT for transforming the XML gtester generates see thesnippet belowI added a shell script build step to the end of the build process thatruns:gtester test_app --keep-going -o=Testsxmlxsltproc -o testsjunitxml gtesterxsl TestsxmlThen I just specified "testsjunitxml" in the Publish JUnit testresult report section of the Post-build Actions and then Hudson wouldproperly process and post the test results when the job was finishedIMAGEIMAGEIMAGEhttp://www.secuobs.com/revue/news/111110.shtmlhttp://www.secuobs.com/revue/news/111110.shtml Secuobs.com : 2009-05-16 23:28:53 - unethical blogger - Since becoming the de-facto maintainer of the Cheetah project I'vebeen thinking more and more about what a templating engine should doand where the boundary between template engine and language are drawnAt their most basic level, template engines are means ofprogrammatically generating large strings or otherwise massagingchunks of text What tends to separate template engines from oneanother are: the language they're written in and what level of"host-language" access they offer the author of the templateCheetah is special in that for all intents and purposes Cheetah isPython which blurs the line between the controller layer and the viewlayer, as Cheetah is compiled into literal Python code In fact, oneof the noted strengths of Cheetah is that Cheetah templates cansubclass from regular Python objects defined in normal Python modules,and vice versa That being the case, how do you organize your code,and where should particular portions physically reside in the sourcetree What qualifies code to be entered into a py file versus a tmplfile If you zoom out from this particular problem, to a larger scope,I believe there is a much larger question to be answered here: as alanguage, what should Cheetah provideSince Cheetah compiles down to Python, does it merit introducing allthe Python constructs that one has at their disposal within Cheetah,including:* Properties* Decorated methods* Full/multiple inheritance* Metaclasses/class factoriesAttacked from the other end, what Cheetah-specific language constructsare acceptable to be introduced into Cheetah as a Python-based hybridlanguage Currently some of the language constructs that exist inCheetah that are distinct to Cheetah itself are:* #include* #filter* #stop* #shBang* #block* #indent* #transform* #silent* #slurp* #encodingSome of the examples of unique Cheetah directives are necessary inorder to manipulate template output in ways that aren't applicable tonormal Python take #slurp, #indent, #filter for example, but wheredoes one draw the lineToo add yet another layer of complexity into the problem, Cheetah isnot only used in the traditional Model-View-Controller set up egDjango + Cheetah templates but it's also used to generate other code,ie Cheetah is sometimes used as a means of generating source code bash,C, etcIn My Humble Opinion--------------------Cheetah, at least to me, is not a lump of text files that you canperform loops and use variables in, it is a fully functional,object-oriented, Pythonic text-aware programming language Whether ornot it compiles to Python or is fully interoperable with Python islargely irrelevant that is not to say that we don't make use of thisfeature As far as "what should Cheetah provide" I think the bestway to answer the question is to not think about Cheetah as Python, oras a "strict" template engine Mako, Genshi, etc but rather as adomain specific language for complex text generation and templatingWhen deciding on what Python features to expose as directives inCheetah the language the litmus test that should be evaluatedagainst is: does this make generating text easierCheetah need not have hash-directives for every feature available inPython, the idea of requiring meta-classes in Cheetah is ridiculous atbest, a feature like decorators however could prove quite useful intext processing/generation eg function output filters, along withproper full inheritanceMy goals ultimately with Cheetah, are to make our lives easierdeveloping rich interfaces for our various web properties, but also tomake "things" faster Whereas "things" can fall under a few differentbuckets: development time, execution time, maintenance timeCheetah will likely look largely the same a year from now, and if wethe developers of Cheetah have done our jobs correctly, it should bejust as simple to pick up and learn, but even more powerful andexpressive than beforeIMAGEIMAGEIMAGEhttp://www.secuobs.com/revue/news/97313.shtmlhttp://www.secuobs.com/revue/news/97313.shtml Secuobs.com : 2009-04-29 18:37:07 - unethical blogger - In my last post on Git, Experimenting with Git at Slide, I discussedmost of the technical hurdles that stood in our way with evaluatingGit for a Subversion tree that has 90k+ revisions and over 2GB of dataheld within As I've learned from any project that involves more thanjust myself, technology is only half the battle, the other half is thehuman element One of the most difficult things to "migrate" whenswitching to something as critical to a developer's workflow as a VCS,is habits, good ones and bad onesThe Bad HabitsWhen moving my team over to Git, I was able to identify some habitsthat I view as "bad" that could either be blamed on how we have usedSubversion here at Slide, or the development workflow that Subversionencourages For the sake of avoiding flamewars, I'll say it's 51% us,49% the system* The Occasional CommitterChances are that if you're working on "something super important" youfall into this bad habit Because of the nature of trunk inSubversion, if you commit half-finished work into a team-branch ortrunk itself, you could cause plenty of pain for your fellowdevelopers As a result, you tend to commit at the end of a long dayworking on something, or only after something has been completed The9 hours of sweat and frustration you've spent pounding away on 300lines of code is now summed up in one commit message:Turns out there was a race-condition here, re #52516Now three months from now when you return to the same 300 lines ofcode and try to figure out what the hell led to this mess, you're leftwith the commit message above, and nothing more* The Less-than-attentive DeveloperI've worked on a Mac for the majority of my time at Slide, as do mostof my compatriots, and sooner or later one of two things will happen:svnadd some/directory/ and/or svn commit This usually results in asecond commit to come into the tree with a commit message like:Whoops, accidentally checked in resource forksThis isn't that large of a problem, except for the implication of thesecond command there, svn commit will commit all outstanding changesin your working copy starting in the current working directory, andrecursing through children directories I'm probably moreanal-retentive about my commits than most, but I usually do a diffbefore I commit to make sure I'm aware of what I'm about to commit,but I've seen plenty of developers skip this step* The Over-Confident MergerI've fallen into this trap numerous times when merging "old" branchesback into trunk, especially with binary files that may have beenchanged in trunk, or in my branch hell if I know One thing I canspeak to anecdotally from our work at Slide, is that the probabilityof nonsensical conflicts rises with a branch's age The rate of ourrepository progresses at about 50 commits to trunk per day ~150commits across the board, if there is a branch cut from trunk,usually within two weeks it can become extremely difficult to mergeback into trunk without constant "refreshes" or merges from trunk intothe branchIf you're not careful when folding that branch back down into trunk,you can inadvertantly revert old binary files or even text files toprevious states which will usually cause other individuals in theengineering organization gripe at you and your QA department to peltyou with rocks For bonus points, you could as I have done beforeaccidentally commit conflicting files earning a gold star and a duncehat for the day This merging pain led me to originally write mymerge-safepy script so long agoThe Slide Way to GitFortunately for us, I think the decentralized nature of Git has helpedus enforce some best practices when it comes to the bad habits above"The Occassional Committer" is all but done away with thanks to theability to atomically commit and revert revisions at a whim and havethose changes not propogated to other developers until there has beenan explicit push or pullUnfortunately however, "The Less-than-attentive Developer" isn'tsolved so easily To date I've sat next to two engineers that were newto Git, and watched them both execute the same fateful command: gitadd Not realizing their mistake, they accidentally commit a truckload ofbuild and temporary files so, swp, pyc, etc interspersed withtheir regular work that they meant to commit Git cannot prevent adeveloper from shooting themselves in the foot, but it does preventthem from shooting everybody else in the foot along with it unlessthey commit, and then push their changes upwards"The Over-confident Merger" grows more and more confident in theGit-based workflow Since Git handles changesets atomically, itbecomes trivial to merge branch histories together or cherry-pick onerevision and apply to an entirely separate branch I've not yet seen aGit conflict that wasn't a true conflict insofar that it was quiteliterally one line of code changing in two different ways betweenbranch histories As an aside, when using git-svn, be prepared for allthe merging "fun" that Subversion has to offer when propogatingchanges between the two systemsBasic Team WorkflowIMAGEThe manner in which we use Git is more like acentralized-decentralized version control system We still have a"master" repository, which provides a central synchronization pointwhen pushing stage servers, or when bringing code into Subversion tobe pushed to live servers For any particular project one of thedevelopers will create a branch that will serve as the primary projectbranch, take the "superpoke-proj" branch as an example That developerwill push this branch to "origin" or the master repository such thatother developers can "track" that branch and contribute code For thepurposes of this example, let's say Paul and Peter are working in"superpoke-proj", while Paul is working he will incrementally commithis work, but once he has resolved/fixed a ticket, he will perform agit push and then mark the ticket appropriately such that a QAengineer can verify the fix If Paul and Peter are working onsomething that "breaks the build" but they need to collaborate on ittogether, Paul can perform a git pull from Peter and vice versa, andagain, once they're done those changes will be pushed to origin Thismodel allows for developers to work in relative isolation so they'renot inadvertantly stepping on each others' toes, but also close enoughthat they can collaborate in explicit terms, ie when they are readyfor changes to be propogated to each other or the rest of the teamConclusionOur workflow, like most things at companies under 500 employees isstill a "work in progress™" I think we've found the right balancethus far for the team between freedom and process the allow forsufficient mucking around in the codebase in a way that provides themost amount of time actually writing code with as little possible timespent dealing with the overhead of anything else merging, etcThere's nothing inherently special in the way we use Git, but we'vefound that it works for the way we work, which is to say in a verytight release schedule that's requires multiple branches per week andplenty of merging from branch-to-branch whether it be from anotherteam or another part of the same teamOf course, your mileage may vary---------------------------------------------------------------------Did you know Slide is hiring Looking for talented engineers to writesome good Python and/or JavaScript, feel free to contact me attyleratslideIMAGEIMAGEIMAGEhttp://www.secuobs.com/revue/news/89995.shtmlhttp://www.secuobs.com/revue/news/89995.shtml Secuobs.com : 2009-04-29 18:37:07 - unethical blogger - During the usual friday-frenzy I sat down and wrote a quick 10 minutelittle bookmarklet to start a Hudson job Unlike most bookmarkletsthat "do things" this one actually "does things" but doesn't take youaway from your current page Using the Hudson Remote Access API youcan query information from Hudson programmatically, but you can alsokick off builds remotely with nothing more than a simple HTTP requestto the properly formed URLBy dragging the link below to your bookmark bar, and updating the URLwithin "http://hudson/" to the URL of your Hudson instance, you canqueue a Hudson build from any page at any time without leaving thepageThe BookmarkletBuild Hudson JobThe CodeBuild Hudson JobHow it actually worksAfter talking the concept of making cross-domain HTTP requests overwith Sergio, he suggested just using an "IMG" tag or "IFRAME" toaccomplish the task The bookmarklet doesn't actually have to send anyform parameters or receive any data, Hudson just needs to receive anyHTTP request to the right URL By creating the IMG object inJavaScript, and appending it to the body of the current page the useris on, it'll effectively con the browser into making the HTTP requestwithout needing to pull off any XmlHttpRequest hacks One of the moreinteresting things that we found out when playing with the end of thebookmarklet, was that if we returned "false" or tried to wrap thewhole thing in a closure, the link would still execute and the browserwould change pages However, if we stuck an "alert" call into thetail end of the bookmarklet JavaScript, execution would stop and thelink wouldn't change the page in the browser tested this in Firefox3Happy Hudsoning :IMAGEIMAGEIMAGEhttp://www.secuobs.com/revue/news/89994.shtmlhttp://www.secuobs.com/revue/news/89994.shtml Secuobs.com : 2009-04-29 18:37:07 - unethical blogger - I saw this referenced by an op-ed piece I read in a restaurant a fewweeks back and had to share, as it's become more and more depressingThis is a piece run by The Onion on January 17, 2001 shortly afterGeorge W Bush came into office netitlted: Bush: 'Our Long NationalNightmare Of Peace And Prosperity Is Finally OverIMAGEI'm going to go cry in a corner nowIMAGEIMAGEIMAGEhttp://www.secuobs.com/revue/news/89993.shtmlhttp://www.secuobs.com/revue/news/89993.shtml Secuobs.com : 2009-04-29 18:37:07 - unethical blogger - There has been some amount of discussion on the Hudson user's listrecently about the status of the "Locks and Latches" plugin Theplugin allows for one to create "locks" for Jobs in a similar mannerto how "locks" work in a multithreaded programming environment Theneed for such a plugin becomes very clear once you start to runmultiple jobs that depend on some set of shared resources, take thefollowing example:* Jobs A,B,C must run unit tests that fetch data from a test site* Slave #1 can only run one instance of Apache at a timeHow one would accomplish this with the Locks and Latches plugin wouldbe to create a lock like "Site Lock" in the Hudson configuration, andthen bind Jobs A, B, C to that Lock Making the large assumptionthat the plugin works correctly and locks properly in order to preventA and B from running concurrently, this would be enough to satisfy therequirements we have for the scenario above Unfortunately it seemsthe plugin is largely unmaintained and buggy; in the past couple weeksof experimenting with such a set up on a variety of different slaveswe've noticed that the locks aren't always respected, causing somelocked jobs to execute in parallel spewing bad test results and buildfailures the crux of this issue seems ot have been reported by SergioFernandes in #2450The Loopback SlaveThe easiest way I found to work around the short-comings of the Locksand Latches plugin was to "break up" the Locks Locks are only reallyuseful if you have more than one "executor" on a Hudson node, in orderto allow Hudson to execute jobs simultaneously In essence, if youonly have one executor, the Hudson queueing system will technicallyperform your "lock" for you by default And thus the "loopback slave"was born When explaining this to a co-worker, I likened my workaroundto the fork2 call, whereas the Locks and Latches plugin is much moreof a pthread_mutex_lock2 call According to the "Distributed Builds"page on the Hudson wiki, you can start slave agent headlessly on anymachine, so why not the master nodeIMAGEAbove is the configuration of one such "loopback slave" that took theplace of one of the executors on the master nodeIMAGEAfter settingup the loopback slave, it's just a matter of tying the Job to thatnode for buildingIn short our set up was before: Jobs A, B, C all use the Lock "SiteJob" in order to queue properly With this change, now there is nolock, and Jobs A, B, C are all bound to the loopback slave in place ofthe lock on the master node While certainly not ideal, given thefrustrations of the Locks and Latches plugin going unmaintained thisis the best short-term solution I've come up with thus far---------------------------------------------------------------------Did you know Slide is hiring Looking for talented engineers to writesome good Python and/or JavaScript, feel free to contact me attyleratslideIMAGEIMAGEIMAGEhttp://www.secuobs.com/revue/news/89992.shtmlhttp://www.secuobs.com/revue/news/89992.shtml Secuobs.com : 2009-04-29 18:37:07 - unethical blogger - A very long time ago I mentioned on Twitter that I was looking at Gitas a replacement for Subversion and Perforce with my personalprojects, but lamented moving to Git at Slide would not be feasibleIMAGELikemost disagreements I've had with people on technology in the past,immediately after I said it, I actively tried to prove myself wrongBack in April when I made the statement above, Subversion 14 was"good enough" just barely for what we wanted to do as far as sourcecontrol, but I became more and more curious about whether or not wecould move to Git IMAGEBack in April, after spending a week with projects like Tailor andgit-svn1 I started to look at the potential of moving just my teamover to Git for evaluation purposes By the end of May I had requestedGit to be installed on the machines that we use for development on aday-to-day basis and we moved the team over to Git by the second weekof JuneWhat followed were six months of sloshing uphill, some of the mostnotable milestones that we had to figure out in this time frame were:* Whereas in the Subversion architecture with a central repositorythere is a very clear development focal point for sharing codebetween developers, what is this in the Git workflow* How do you ensure developers don't forget code was committed "inthat one branch, in that one repository" and keep track of code* How will Git integrate with Hudson, Trac and our other pieces ofdevelopment infrastructure answered hereI'll be answering these questions and share some of the scripts,hooks, and documentation we've written internally to make moving toGit throughout the company a reality I wish I could say I wasresponsible for it all, but there were a number of other engineersthat were extremely important in defining best practices, and whatthis shiny new world without Subversion would look likeAt the end of the day, I'm pleased as punch with the transition Idon't hate Subversion, I just love Git; call me "spoiled" but I thinkwe deserve something more than a system that strives to be "a betterCVS"Update: I've posted an addendum: Why we chose Git, a rebuttal---------------------------------------------------------------------Did you know Slide is hiring Looking for talented engineers to writesome good Python and/or JavaScript, feel free to contact me attyleratslideIMAGEIMAGEIMAGEhttp://www.secuobs.com/revue/news/89991.shtmlhttp://www.secuobs.com/revue/news/89991.shtml Secuobs.com : 2009-04-29 18:37:07 - unethical blogger - As I mentioned in my previous post about Git at Slide, I wanted toanswer some questions that we had to answer to migrate to Git for ourdevelopment workflow One of the major questions that had to beanswered, especially for our QA department to sign off on the ideawas:How will Git integrate with Hudson, Trac and our other pieces ofdevelopment infrastructureFor us to use any version control system, centralized ordecentralized, there had to be a "central" point for changes tointegrate into in order for us to properly test releases and then shipthem to the live site With this requirement, we oriented our use ofGit around a centralized repository which developers pull from, andpush to on a regular basisIn order for Git to integrate into Trac and Hudson, we opted forbaking the functionality we needed into the post-receive hook on thecentralized repository instead of relying on GitTrac, or the HudsonGit plugin to do what we needed it do toYou can find the script below, or in this GitHub repository Thescript requires the Trac XML-RPC plugin to be installed in order toproperly annotate tickets when changes are pushed into the centralrepository The notation syntaxes that the post-receivepy scriptsupports in commit messages are:re #12345qa #12345attn bbum,fspeirsAs one might expect, the first notation: "re #12345" will simplyannotate a ticket with the commit message and the branch in which thecommit was pushed into The "qa #12345" notation part of an internalnotation of marking tickets in Trac as "Ready for QA", which let's ourQA engineers know when tickets are ready to be verified; a "qa" notein a commit message will reference the commit and change the status ofthe ticket in question The final notation that the script supports: "attnbbum,fspeirs" is purely for calling attention to a code change, or toask for a code review When a commit is pushed to the centralrepository with "attn" in the commit message, an email with the commitmessage and diff will be emailed to the specified recipientsIn addition to updating Trac tickets, pushes into any branch that havea Hudson job affiliated will use the Hudson External API to queue abuild for that branch In effect, it you "git push origin master", thepost-receivepy script will ping Hudson and ask it to queue a build ofthe "master" jobI have included the script inline below for those weary of clickinglinks like this one to the GitHub repository containing the scriptEnjoy :1 '' '2 Copyright c 2008 Slide, Inc3  4 Permission is hereby granted, free of charge, to any person obtaining a copy5 of this software and associated documentation files the "Software", to deal6 in the Software without restriction, including without limitation the rights7 to use, copy, modify, merge, publish, distribute, sublicense, and/or sell8 copies of the Software, and to permit persons to whom the Software is9 furnished to do so, subject to the following conditions:10  11 The above copyright notice and this permission notice shall be included in12 all copies or substantial portions of the Software13  14 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR15 IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,16 FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT IN NO EVENT SHALL THE17 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER18 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,19 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN20 THE SOFTWARE21 ' ''22  23 '' '24 For questions, patches, etc contact R Tyler Ballance 25 ' ''26 import getpass27 import os28 import re29 import socket30 import smtplib31 import sys32 import time33 import xmlrpclib34  35 from optparse import OptionParser36  37 MAIL_SERVER = 'your_mail_servercom'38 MAIL_SUFFIX = '@mycompanycom'39 BUILD_HUDSON = True40 HUDSON_URL = 'http://hudson'41 TRAC_XMLRPC_URL = 'URL_TO_TRAC/projects/MYPROJECT/login/xmlrpc'42  43 def rpcProxy user = 'qatracbot' , password= None :44 password = password or os getenv 'TRAC_PASS' 45 return xmlrpclib ServerProxy 'https://%s:%s@%s' % user , password, TRAC_XMLRPC_URL 46  47 def _send_commit_mail user , address, subject, branch, commits, files, diff :48 print 'Sending a GITRECEIVE mail to %s' % address49 message = 'Commits pushed to %s: -------------------------------------- %s -------------------------------------- %s -------------------------------------- %s' % branch, commits, files, diff 50 _send_mail user , address, subject, message 51 def _send_attn_mail user , destuser, diff :52 print 'Sending a "please review" mail to %s' % destuser53 message = '' 'Good day my most generous colleague I would hold you in the highest esteem and toast you over my finest wines if you would kindly review this for me - %users Diff: ------------------------------------------------ %diffs' '' % { 'diff' : diff, 'user' : user }54 addresses = 55 for d in destuser split ',' :56 addresses append '%s%s' % d, EMAIL_SUFFIX 57 _send_mail user , addresses, 'Please review this change' , message 58  59 def _send_mail user , address, subject, contents :60 try :61 if not isinstance address, list :62 address = address 63 s = smtplib SMTP MAIL_SERVER 64 message = 'From: %s%s To: %s Subject: %s %s ' % user , MAIL_SUFFIX, ', ' join address , subject, contents 65 s sendmail '%s%s' % user , MAIL_SUFFIX , address, message 66 s quit 67 except :68 print 'Failed to send the email :'69  70 def _update_ticket ticket, message, options= { } :71 rpc = rpcProxy 72 rpc ticket update ticket, message, options 73 return rpc74  75 def find_re commit :76 return map int , re findall r 'i s +re s *#0-9+' , commit 77 def handle_re branch, commit, ticket :78 print 'Annotating ticket #%s' % ticket79 message = '' 'The following was committed in "%branchs":80 {{{ 81 %commits }}}82 ' '' % { 'branch' : branch, 'commit' : commit }83 _update_ticket ticket, message 84  85 def find_qa commit :86 return map int , re findall r 'i s +qa s *#0-9+' , commit 87 def handle_qa branch, commit, ticket :88 print 'Marking ticket #%s as "ready for QA"' % ticket89 message = '' 'The following was committed in "%branchs":90 {{{ 91 %commits }}}92 ' '' % { 'branch' : branch, 'commit' : commit }93 rpc = _update_ticket ticket, message, options= { 'status' : 'qa' } 94  95 def find_attn commit :96 return re findall r 'i s +attn s *A-Za-z,+' , commit 97 def handle_attn branch, commit, attn :98 # Unpack commit from this: "commit 5f4c31f3c31347c62d68ecb5f2c9afa3333f4ad0Author: R Tyler Ballance Date: Wed Nov 12 16:57:32 2008 -0800 Merge commit 'git-svn' "99 try :100 commit_hash = commit split ' ' 0 split ' ' 1 101 except :102 return # fuk it103 diff = os popen 'git show --no-color %s --pretty=format:"Author: %%cn %%n%%s%%n%%n%%b%%n%%n%%H"' % commit_hash read rstrip 104 _send_attn_mail getpass getuser , attn, diff 105  106 def mail_push address, oldrev, newrev, refname :107 user = getpass getuser 108 machine = socket gethostname 109 base_git_diff = 'git diff %s %s' % oldrev, newrev 110 files_diffed = os popen '%s --name-status' % base_git_diff read rstrip 111 full_diff = os popen '%s -p --no-color' % base_git_diff read rstrip 112 '' ' git rev-parse --not --branches | grep -v "$new" | git rev-list "$old""$new" --stdin ' ''113 commits = os popen 'git rev-parse --not --branches | grep -v "%s" | git rev-list %s%s --stdin --pretty=format:"Author: %%cn %%nDate: %%cd %%n%%n %%s %%n%%n %%b %%n %%n-------post-receive marker------%%n" --first-parent ' % newrev, oldrev, newrev read rstrip 114 branch = refname split '/' -1 115 mail_subject = 'GITRECEIVE %s/%s %s files changed' % machine, branch, len files_diffed split ' ' 116  117 if branch == 'master-release' :118 print 'Tagging release branch'119 tagname = 'livepush_%s' % time strftime '%Y%m%d%H%M%S' , time localtime 120 sys stderr write 'Creating a tag named: %s ' % tagname 121 os system 'git tag %s' % tagname 122 mail_subject = '%s tagged: %s' % mail_subject, tagname 123  124 if BUILD_HUDSON_JOB:125 print 'Queuing the Hudson job for "%s"' % branch126 os system '/usr/bin/env wget -q -O /dev/null http://%s/job/%s/build' % HUDSON_URL, branch 127  128 _send_commit_mail user , address, mail_subject, branch, commits, files_diffed, full_diff 129  130 if branch == 'master' :131 return # we don't want to update tickets and such for master/merges132  133 commits = filter lambda c: len c , commits split '-------post-receive marker------' 134 commits reverse 135 for c in commits:136 if c find 'Squashed commit' = 0 :137 continue # Skip bullshit squashed commit138  139 for attn in find_attn c :140 handle_attn branch, c, attn 141  142 for ticket in find_re c :143 handle_re branch, c, ticket 144  145 for ticket in find_qa c :146 handle_qa branch, c, ticket 147  148  149 if __name__ == '__main__' :150 op = OptionParser 151 op add_option '-m' , '--mail' , dest= 'address' , help = 'Email address to mail git push messages to' 152 op add_option '-o' , '--oldrev' , dest= 'oldrev' , help = 'Old revision we \' re pushing from' 153 op add_option '-n' , '--newrev' , dest= 'newrev' , help = 'New revision we \' re pushing to' 154 op add_option '-r' , '--ref' , dest= 'ref' , help = 'Refname that we \' re pushing' 155 opts, args = op parse_args 156  157 if not opts address or not opts oldrev or not opts newrev or not opts ref :158 print '*** You left out some needed parameters ***'159 exit160  161 mail_push opts address , opts oldrev , opts newrev , opts ref ---------------------------------------------------------------------Did you know Slide is hiring Looking for talented engineers to writesome good Python and/or JavaScript, feel free to contact me attyleratslideIMAGEIMAGEIMAGEhttp://www.secuobs.com/revue/news/89990.shtmlhttp://www.secuobs.com/revue/news/89990.shtml Secuobs.com : 2009-04-29 18:37:07 - unethical blogger - One thing I learned early on in the internet, when is was more of acobbling instead of a series, of tubes, was not to feed trollsIMAGEThat said, I found that my post "Delightfully Wrong About Git"had found it's way into such silly news aggregation machines as DZone,Reddit and Hacker News Some of the points raised in the comments werevalid and warrant a response, while the majority of them were thestandard responses to any discussion about version control "psh, dumbshould have used Bazaar/Mercurial/Darcs/Subversion/Team FoundationSystem"Why not another DVCSOne of the most resounding criticisms/questions was this one, why notBazaar Why not Mercurial My favorite, albeit childish, retort is"why" But I can say that I have tried a variety of other versioncontrol systems, Git, Bazaar, CVS, Subversion, Perforce and some otherproprietary VCSes at previous employers While both Darcs andMercurial seem to be very solid DVCSes, they suffer from a problem ofmomentum, Darcs in particular They both appear to be victims of Git'ssuccess, while there is inherently nothing wrong with either of them,they are competing with Linus' love-child, Git When chosing to moveto a new VCS in a company that is well over 50+ employees, the stayingpower of the technology you chose is important I feel confident thatGit will not only be supported, but actively developed and improvedfor years to comeMore importantly than that though, I like Git Is that not enoughright there Slide makes excessive use of branches, tags and other"complex" VCS concepts that centralized systems like CVS andSubversion have trouble withGit Branch Madness With Subversioncreating branches in the volume in which we create branches spiralledout of control with branches becoming "stale" quickly, meaning that ifwe didn't refresh the branch regularly with updates from trunk itwould be nearly impossible to cleanly merge back down into trunk Withmy current Git clone of our primary repository, I have 23 branchesroughly 6 personal local branches, 5 old branches, and 12 activebranches Our primary Git repository has been online for about 6months and currently has 68 branches in it, rougly 55 are activeWhy all the love for Git, but nobody every talks about Bazaar,Mercurial, Darcs, etc Sure Git is faster, but unless you've got aenormous code base like the linux kernel, it seems like Bazaaror Mercurial would be a better choice than GitOne of the better known selling points of Git is that it's fast Mycloned repository of the primary Slide Git repository weighs in at ahefty 71GB The latest revision number in our Subversion repositoryis in the 103,000 range, tacked onto that our tree is just over 2GB insize, and you've got a lot of history to keep track of Git handlesthis without a sweat despite hitting the disk extremely hard whenswitching to a very out of date branch With this fix from Nico, thelast of the mmap2 allocation issues we were experiencing vanished aswellStop re-inventing the wheelOne of the more interesting sentiments I noticed perusing the variouscomments made regarding my previous post were that we are"re-inventing the wheel" by writing scripts, hooks and other wrappersto use a product like Git The notion that having scripts and hooksfor something you use in daily development is re-inventing the wheel,or gratuituous strikes me as laughable at best We're developers Wewrite scripts Why didn't I ever write a myriad of scripts when I wasan avid Subversion user I did There's an enormous different betweenwriting scripts to compensate for a poorly performing product, andwriting scripts to further enhance you or your colleagues' workflows,Git's hook support falls into the latter categoryThe "religion" aspect of the whole version control debate was neverconsidered in our transition to Git, nor was the buzz I'm far moreinterested in what makes other VCSes better or worse than Git, so thatGit can be improved instead of a justification to ditch Git foryet-another-dvcs I like to think of the various tools like versioncontrol that we developers use as something more relatable: workpants A good pair of work pants should be flexible enough to allowyou to get your work done, modest enough to stay out of the way andmost importantly, a good pair of work pants should keep your junk safe;I'm still happy to answer more specific questions about Git andhow/why it works for us as well as it has, but I think most of thequestions I've seen thus far have been answered above---------------------------------------------------------------------Did you know Slide is hiring Looking for talented engineers to writesome good Python and/or JavaScript, feel free to contact me attyleratslideIMAGEIMAGEIMAGEhttp://www.secuobs.com/revue/news/89989.shtmlhttp://www.secuobs.com/revue/news/89989.shtml Secuobs.com : 2009-04-29 18:37:07 - unethical blogger - For about a month now I've been sending weekly "Protip" emails aboutGit to the rest of engineering here at Slide I've been using them toslowly and casually introduce some of the more "interesting" featuresGit has to offer as we move away from Subversion entirely Below isthe first Protip I sent around, I'll be sure to send the rest in goodtime---------------------------------------------------------------------Given the nature of how Git is structured, in that your "working copy"is also your "repository" you might find yourself switching branchesrelatively often While you can actually switch branches withuncommited changes outstanding in your Git directory, it's not advisedas you might forget you're commiting changes in the wrong branch,etc You have two options if you are halfway through some work, oneis to commit a checkpoint revision, but the other is to make use ofthe git stash commandA scenario where this becomes especially useful would be: Bill isworking in his local branch "wip-funtime" on replacing large swaths ofunnecessary useless code and Ted accidentally pushes some of Bill'sother changes from another branch live and things break Bill couldcommit his code and write a fairly uninformative log message like"checkpoint" which cheapens the value of the revision history of hischanges or Bill can use git stash to snapshot his currently workingstate and context switch In this case Bill would execute thefollowing commands:* git stash* git checkout master-media* perform hotfixes* git checkout wip-funtime* git stash popAfter performing the git stash pop command, Bill's Git repository willbe in the exact same state, all his uncommitted changes in tact, as itwas when he originally stashed and context-switchedFor specific usage of `git stash` refer to the git stash man pageExample usage of `git stash`Stashing changes away1 tyler @ starfruit:~ / source / git / main / bt git stash2 Saved working directory and index state "WIP on master-topfriends: 7b1ce9e TOS copy fix"3 To restore them type "git stash apply" 4 HEAD is now at 7b1ce9e TOS copy fix5 tyler @ starfruit:~ / source / git / main / bt Looking at the stash1 tyler @ starfruit:~ / source / git / main / bt git stash list2 stash @ { 0 } : WIP on master-topfriends: 7b1ce9e TOS copy fix3 stash @ { 1 } : On master-topfriends: starfruit complete patchset4 stash @ { 2 } : On wip-classmethod: starfruit patches5 tyler @ starfruit:~ / source / git / main / bt Grabbing the latest from the stash1 tyler @ starfruit:~ / source / git / main / bt git stash pop2 Dropped refs / stash @ { 0 } 94b9722b5a999c32c4361d795ee8f368d8412f9a 3 tyler @ starfruit:~ / source / git / main / bt Grabbing a specific stash1 tyler @ starfruit:~ / source / git / main / bt git stash list2 stash @ { 0 } : WIP on master-topfriends: 7b1ce9e TOS copy fix3 stash @ { 1 } : On master-topfriends: starfruit complete patchset4 stash @ { 2 } : On wip-classmethod: starfruit patches5 tyler @ starfruit:~ / source / git / main / bt git stash apply 26 # On branch master-topfriends7 # Changed but not updated:8 # use "git add " to update what will be committed9 #10 # modified: db/dbrootpy11 # modified: gogreen/coropy12 # modified: py/bin/_makepyreleasepy13 # modified: py/initpkgpy14 # modified: py/misc/_distpy15 # modified: py/misc/testing/test_initpkgpy16 # modified: py/path/local/localpy17 # modified: py/test/terminal/terminalpy18 tyler @ starfruit:~ / source / git / main / bt ---------------------------------------------------------------------Did you know Slide is hiring Looking for talented engineers to writesome good Python and/or JavaScript, feel free to contact me attyleratslideIMAGEIMAGEIMAGEhttp://www.secuobs.com/revue/news/89988.shtmlhttp://www.secuobs.com/revue/news/89988.shtml Secuobs.com : 2009-04-29 18:37:07 - unethical blogger - I've been sending weekly "Protip" emails about Git to the rest ofengineering here at Slide for a while now, using the "Protips" as ameans of introducing more interesting and complex features Git offersBelow is the second Protip written to date---------------------------------------------------------------------One of the major benefits to using Git is the entirety of therepository being entirely local and easily searched/queried For this,Git has a very useful command called git log which allows you toinspect revision histories in numerous different ways between filepaths, branches, etc There are a couple basic scenarios where git loghas become invaluable, for me at least, in order to properly reviewcode but also to track changes effectively from point A to point B* What's Dave been working on lately with diffs* git log -p --no-merges --author=daveThe --no-merges option will prevent git log from displaying mergecommits which are automatically generated whenever you pull fromone Git branch to another* Before I merge this branch down to my team master, I want to knowwhat files have been changed and what revisions* git log --name-status master-topfriendsproj-topfriends-thingGit supports the ability with git log and with git diff to provideunidirectional branch lookups or bidirectional branch lookups Forexample, say the left branch has commits "A, B" and the rightbranch has commits "A, C" The syntax will output "C", whereas will output "B, C"* I just got back from a vacation, I wonder what's changed* git log --since="2 weeks ago" --name-status -- templatesAt the tail end of a git log command you can specify particularpaths to look up the histories for with the -- operator, in thiscase, I will be looking at the changes that have occured in thetemplates directory over the past two weeks* Most recent X number of commits with diffs* git log -n 10 --no-merges -pAll git log commands automatically filter into less1 so you can pagethrough the output like you would normally if you executed a svn log |less Because git log is simply reading from the locally storedrevision history you can quickly grep the history by any number ofdifferent search criteria to gain a better understanding of how thecode base is changing and whereFor more specific usage of `git log` refer to the git log man page---------------------------------------------------------------------Did you know Slide is hiring Looking for talented engineers to writesome good Python and/or JavaScript, feel free to contact me attyleratslideIMAGEIMAGEIMAGEhttp://www.secuobs.com/revue/news/89987.shtmlhttp://www.secuobs.com/revue/news/89987.shtml Secuobs.com : 2009-04-29 18:37:07 - unethical blogger - I've been sending weekly "Protip" emails about Git to the rest ofengineering here at Slide for a while now, using the "Protips" as ameans of introducing more interesting and complex features Git offersBelow is the third Protip written to date---------------------------------------------------------------------The concept of "revert" in Git versus Subversion is an interesting,albeit subtle, change, mostly in part due to the subtle differencesbetween Git and Subversion's means of tracking commits Just as withSubversion, you can only revert a committed change, unlike Subversionthere is a 1-to-1 mapping of a "commit" to a "revert" The basicsyntax of revert is quite easy: git revert 0xdeadbeef, and just like aregular commit, you will need to push your changes after you revert ifyou want others to receive the revert as wellIn the following example of a revert of a commit, I also use the "-s"argument on the command line to denote that I'm signing-off on thisrevert ie I've properly reviewed itxdev3% git revert -s c20054ea390046bd3a54693f2927192b2a7097c2----------------Vim----------------Revert "merge-to-release unhide 10000 coin habitat"This reverts commit c20054ea390046bd3a54693f2927192b2a7097c2Signed-off-by: R Tyler Ballance # Please enter the commit message for your changes Lines starting# with '#' will be ignored, and an empty message aborts the commit# On branch wip-protips# Changes to be committed:# use "git reset HEAD " to unstage## modified: bt/apps/pet/datapy+ python bt/qa/git/post-commitpy -m svn@slidecomSending a commit mail to svn@slidecomCreated commit a6e93b8: Revert "merge-to-release unhide 10000 coin habitat"1 files changed, 4 insertions+, 3 deletions-Reverting multiple commits--------------------------Since git revert will generate a new commit for you every time yourevert a previous commit, reverting multiple commits is not as obviousside note: I'm aware of the ability to squash commits, or --no-commitfor git-revert1, I dislike compressing revision history when I don'tbelieve there shouldn't be compression If you want to revert aspecific merge from one branch into the other, you can revert themerge commit provided one was generated when the changes weremerged Take the following example:commit 81a94bb976dfaaaae42ae2600b7e9e88645ebd81Merge: 8134d17 d227dd8Author: R Tyler Ballance Date: Thu Nov 20 10:15:16 2008 -0800Merge branch 'master' into wip-protipsI want to revert this merge since it refreshed my wip-protips branchfrom master, and brought in a lot changes tat have destablized mybranch In the case of reverting a merge commit, you need to specify-m and a number to denote where the mainline branch for Git to pivotoff of is, -m 1 usually suffices So the revert of the commit abovewill look something like this:git revert 81a94bb976dfaaaae42ae2600b7e9e88645ebd81 -m 1Then my revert commit will be committed after I review the change inVim:commit 8cae4924c4c05dadaaeccb3851cfc9ec1b8efd0fAuthor: R Tyler Ballance Date: Thu Nov 20 10:20:44 2008 -0800Revert "Merge branch 'master' into wip-protips"This reverts commit 81a94bb976dfaaaae42ae2600b7e9e88645ebd81Let's take the extreme case where I don't have a merge commit to pivotoff of, or I have a particular set of bare revisions that I need torevert in one pass, you can start to tie Git subcommands together likegit-rev-list1 to accomplish this This hypothetical situation mightoccur if some swath of changes have been applied to a team-master thatneed to be backed out Without a merge commit to key off of, you haveto revert the commits one by one, but that doesn't mean you have torevert each one by hand:for r in `git rev-list mastermaster-fubar --since="8:00" --before="12:00"--no-merges`; do git revert --no-edit -s $r; doneIn the above example, I can use git-rev-list1 to give me a list ofrevisions that have occurred on "master-fubar" that have not occurredon "master" between the times of 8 am and 12 pm, excluding mergecommits Since git-rev-list1 will return a list of commit hashes bydefault, I can loop through those commit hashes in chronological orderand revert each one The inner part of the loop signs-off on therevert -s and then tells git-revert1 to auto-commit it withoutopening the commit message in Vim --no-edit What this then outputsis the following:xdev% for r in `git rev-list mastermaster-fubar --since="8:00" --before="12:00" --no-merges`; do git revert --no-edit -s $r; doneFinished one revertCreated commit b6810d7: Revert "a test, for you"1 files changed, 1 insertions+, 1 deletions-Finished one revertCreated commit 83156bd: Revert "These are not the droids you are looking for1 files changed, 2 insertions+, 0 deletions-Finished one revertCreated commit 782f328: Revert "commented out stuff"1 files changed, 0 insertions+, 3 deletions-Finished one revertCreated commit 2b8d664: Revert "back on again"1 files changed, 1 insertions+, 1 deletions-xdev% For specific usage of "git-revert" or "git-rev-list" refer to thegit-revert1 man page or the git-rev-list1 man page---------------------------------------------------------------------Did you know Slide is hiring Looking for talented engineers to writesome good Python and/or JavaScript, feel free to contact me attyleratslideIMAGEIMAGEIMAGEhttp://www.secuobs.com/revue/news/89986.shtmlhttp://www.secuobs.com/revue/news/89986.shtml Secuobs.com : 2009-04-29 18:37:07 - unethical blogger - Let's be honest for a second, anybody who knows me knows that I'mclearly an insecure person; I spend the majority of my time trying mybest to appear cool I've owned a lot of Macs in my life, not becausethey're solid machines with a fantastic operating system, but becauseI felt so damn smug and cool whenever I was doing anything on my MacsI also developed Mac software for a while, not because it was mypassion or Objective-C and Cocoa are practically God's gift tosoftware, but because Mac developers are so cool, what with theblack-rimmed glasses and fancy coffees Hell, I remember when Ifinally traded my MacBook Pro for a Thinkpad running Linux; it hadnothing to do with an ideological stance against Apple's treatment ofdevelopers or frustrations with Leopard, it was all about the newgeek-chic that was Linux Thus far, my life has basically been one bigquest for more leet-pointsThen came GitWhen I started out in the software world, I was using CVS, which was anotch less cool than a slim IBM salesman's tie The constant moaningand groaning of fellow developers using CVS, combined with the shamethat I felt when I finally told my parents about my use of CVS was toomuch to bear I had to switchI remember the first time I tried Subversion, I remember talking toDave and saying "Meh, I'll stick with CVS" Soon enough, just like theMacarena, Subversion swept the nation up Subversion was the newest,coolest thing ever, developers rushed into the streets exclaiming "itsucks less than CVS It sucks less than CVS" I switched over toSubversion and all of a sudden I was cool again One by one, opensource projects I knew about switched over to Subversion, then SourceForge switched over to Subversion and in an instant, Subversionreplaced CVS and became the mainstream version control systemSubversion had grown up, gotten married, a 401k and health insurance,how uncoolAfter joining Slide, which used Subversion, I found myself burning upinside Here I was at this hip start-up, really feeling cool, butstill using the same version control system that uncool companieslike, Yahoo and Sun use I would not stand for this As 2007 became2008 the writing was on the wall, Git was our new bicycle It had beenblessed by Saint Torvalds and clearly we needed to get in on theground floor of the new cool before it became mainstreamWe needed to switch to Git immediately Who cares if Git is extremelyfast, it's not like time is money or something ridiculous like thatWhat do I care if Git handles branches and merge histories unlike CVSor Subversion With its immense coolness-factor, I didn't evenconsider that Git will allow us to work in a decentralized workflow ora centralized workflow, nope, didn't even cross my mind If one wereto make a list of Pros and Cons of Git versus whichever other versioncontrol system, you could just put "Pro: Cool" at the top of the list,underlined, in bold, and the rest would be moot as far as I'mconcernedUnlike Subversion or Perforce, Git doesn't have corporate backing, Gitis distributed, like a guerilla-force sweeping through the jungleready to pownce on an unsuspecting platoon; that's freakin' cool Gitrides a motorcycle, wears a leather jacket, makes women swoon andkicks ass and/or jukeboxesGit is the Fonz CoolDon't make any false assumptions about my feelings towards Git, it'snot like it's a clearly superior version control system or anything,I'm using it only because I want to be cool tooIMAGEIMAGEIMAGEhttp://www.secuobs.com/revue/news/89985.shtmlhttp://www.secuobs.com/revue/news/89985.shtml Secuobs.com : 2009-04-29 18:37:07 - unethical blogger - A coworker of mine was kind enough to let me borrow his Nokia n810 fora couple days to try it out as he know I was considering purchasingone for myself I'm very glad I tried it before buying it, since I'mnot going to buy it now sorry Nokia The princess is in anothercastleThe thought of a handheld, wireless capable, Linux device is veryintriguing for me That said, I'm not sure what I would even do withit As I mentioned in my previous post, I like to feel cool, and theprospect of answering the question "is that Linux in your pocket orare you just happy to see me" is far to enticing to pass upRegardless, I think the n810 suffers from some critical hardware, andsoftware, deficienciesHardwareThe n810 is powered by a 400Mhz ARM processor, and comes equipped witheither 128MB or 256MB or RAM from what I can tell, I'm not entirelycertain which is to blame for the sluggishness of the experience, butmy guess is on the RAM Particularly when running the browserGecko-based I would experience "hiccups" where the device spent afew seconds registering input, before actually following a clickedlink This may be more at fault of the software, but for an internettablet, the sluggishness of the browser in both user interaction andrendering time was absolutely infuritatingThe built-in keyboard is smooth, a little too smooth for my taste; Ifound myself constantly struggling to hit the right keys with myfingers my thumb is the width of 25 columns of keys Unlike most USkeyboard layouts, the n810 keyboard has a lot of keys in "weird"places that I could not get a hang of over the course of a weekend Ieventually gave up on trying to chat or use SSH on the device becauseI found it so painful to try to type on the deviceThe battery life was nothing to write home about, closer to a laptopbattery life, instead of a phone's battery lifeThe SoftwareDespite being Linux-based, the device doesn't feel like Linux at all,which I think is a good thing for the mass-market The "Home" screenwas pretty slick, with the ability to add applets to the "desktop" toreport things like weather, time, VPN status, etc A cross-betweensystray and Dashboard, the Home screen was where I felt mostcomfortable in the device the "home" screen in my Smartphone is setup with similar informational panels Once leaving "Home" I was soonfrustrated again, I still haven't figured out whether or not the"Accounts" preference in the Control Panel for IM accounts and theinstallation of Pidgin are the same thing or not Email and IM, thetwo other foundations of what I would expect from an "internet tablet"were weak Neither of them cooperated with any of the IMAP/SSL orJabber/SSL servers I use, and they both seemed to be targeted atwebmail and chat services like GMail and GTalkMaemo does use deb packages for installation, so I could prettyeasily find some of my favorite open source applications in the Maemorepositories, unfortunately the GUI frontend for apt-get on Maemoallows for only one operation at a time no checking multiple boxesand then clicking "Install" so adding new software was literally a 30minute operationConclusionI don't think I'm being too negative in saying that I'm disappointedin Nokia for releasing what I think is such a substandard productWith the ubiquity of wireless in San Francisco, having a nice solidultra-portable machine that I can actually fit into my pocket isexciting, The Nokia n810 is certainly not that machineThis week I'm shipping my ASUS Eee PC off for my little sister, so I'mstarting to look more and more for something even more portable tofill the void, right now the leader is the OQO model 02 which is about2 times the price of the n810, and ships with Vista by default, butwith Ubuntu and close to 6 hours of battery life I think it could bethe ultra-portable that I've been looking forIMAGEIMAGEIMAGEhttp://www.secuobs.com/revue/news/89984.shtmlhttp://www.secuobs.com/revue/news/89984.shtml Secuobs.com : 2009-04-29 18:37:07 - unethical blogger - Rod reminded me with his comment in one of my other posts that I'venot yet mentioned githubI've got a bunch of my nonsense thrown up on githubcom/rtyler, it'sawesome no really, github rocks my socks, those guys are goodpeopleIMAGEIMAGEIMAGEhttp://www.secuobs.com/revue/news/89983.shtmlhttp://www.secuobs.com/revue/news/89983.shtml Secuobs.com : 2009-04-29 18:37:07 - unethical blogger - I've spent the better part of my weekend messing around with mailclients, and once again Evolution comes out on top and once again, I'mnot happy about it I tried: Claws, Thunderbird, Alpine formerlyPine, Mutt, Balsa, KMail and TkRat None of them worked as well as Iwanted, is it too much to ask for to have a mail client that doesn'tpuke and die on large 2GB of IMAP mail Supports proper jwz mailthreading And caches IMAP mail locally so I can actually access itwhile disconnect Turns out it actually is too much to askThat's not what this is about though While hunting around, I startedto look at my Slide IMAP mail account, and see something interesting:it looks suspiciously like a filesystem The general layout I haveright now is something like this:* / * INBOX* Sent* Drafts* Development/* Commits* Pushes* External/* Git* Hudson* Metrics* QA/* Exceptions* TracClearly, it's a very filesystem-esque looking tree of mail and acouple gigabytes of it When you start to really dig into e-mailtechnology, you really get a feeling for how royally screwed up thewhole ecosystem is Between Exchange, IMAP and POP3 and their SSLcounterparts, mbox and Maildir, and of course the venerable SMTP;e-mail technology is a clusterfuck No wonder barely anybody canimplement an e-mail client that doesn't suckAt a basic level, mail is organized into messages and foldersMessages map very easily to actual files on the filesystem, andfolders naturually map into actual directories on the filesystemImagine that you could chose any program you wanted to read and writeyour email The only pre-requisite: can it read from the filesystemYou could have any program register to receive filesystem events tonotify you when mail "appears" in specific directories, and you couldmove mail around with a simple drag-and-drop inNautilus/Thunar/Finder What about writing mail though Easy enough,you create a new file in the "Drafts" folder, writes would naturallybe propogated to the "Drafts" folder on the IMAP server, and when youwere done with the message, you could copy or move it into the "Sent"folder, which would have a hook to recognize the new file, and sendit The IMAP tree from above, starts to look something like this:* ~/Imuse* Settings* Accounts/* Slide/* INBOX* Sent* Drafts* Development/* Commits* Pushes* External/* Git* Hudson* Metrics* QA/* Exceptions* Trac"Accounts" and "Settings" would likely need to be "special" insofarthat Imuse would just create them out of thin air, Accounts would needto be a virtual directory to actually contain the appropriate accountlistings, and in Settings I'd likely want to have a couple of flatconfiguration "files" that you could edit in order to actuallyconfigure Imuse appropriatelyIf there are simply lists of files in each of the Accounts' folders,each representing a particular email, then the problem of dealing withall my e-mail becomes a much easier one to handle, then it's just amatter of picking my filesystem browser of choice Even then it's notreally limited to filesystem browsers like Nautilus, the scope ofprograms that I can use to access my mail is opened up to $EDITOR aswell Most editors like Notepad++, Vim, Emacs, Gedit, and TextMate allsupport the ability to view a directory, and open it's contents up forreading/editing I'm a big fan using Vim, so Imuse coupled withvtreeexplorer would be phenomonal to say the leastI've started toying around with building FUSE filesystems and I'vepushed my experimenting up to GitHub in my imuse repository It'scurrently in C, since I either cannot get either of the two FUSEPython bindings to work properly This presents a certain level ofdifficulty, since the standard means of accessing IMAP data from Cseems to be c-client, which is reasonably well documented, but lackssample code On the other hand, if I can get the Python bindings tocooperate, then I have access to the wonderful Twisted Mail libraryor even the basic imaplibGiven my obvious time restrictions, I wanted to open the idea up tomore eyes and ears to see what others thought and maybe even findsomebody else willing to pitch in For the time being however,Evolution is still sifting through my mail, and I'm still not enjoyingit :IMAGEIMAGEIMAGEhttp://www.secuobs.com/revue/news/89982.shtmlhttp://www.secuobs.com/revue/news/89982.shtml Secuobs.com : 2009-04-29 18:37:07 - unethical blogger - In addition to frothing at the mouth about Git, I've been reallygetting into the concept of automated unit tests lately thus myinterest in Hudson Just like code comments however, tests are good,no tests is bad, wrong tests is worse That means once you give in tothe almighty power of unit testing, you are saddled with the curse ofknowing that you will have to update them, foreverTaking up Test-driven Development is like having a child, if you areat a point in your life where you're ready to accept that kind ofresponsibility, it can be wonderful, a lot of work, but ultimately youwill feel satisfied with your new role as a Responsible Developertm If you're not prepared to take on the burden that TDD willpresent you with, you will likely regret it or neglect your testsDeadbeat Developer, I like this metaphorIn the Top Friends Team at Slide, we practice the more "loose"definition of TDD; tests are not written before functionality iswritten, but rather functionality is written, and then as part of theQA and release process, the appropriate and accompanying tests arewritten Our basic workflow is usually as follows:* Tickets are written and assigned to milestones and developers inTrac* Branch is created in central Git repository* General plan-of-action is discussed between developers* Hack-hack-hack* Code complete is reached, QA starts to test milestone* Developers write tests if needed for functionality* Once QA signs off, and tests look solid, code is shipped liveThere are two primary flaws with this workflow, the first and mostobvious one is that it is far to easy to "forget to write the tests"That is, the project scheduled to start development tends to "flowforward" into the allotted test-writing time As important as testcoverage is, at the end of the day Slide did not raise funding onhaving solid test coverage, and our priorities lie in shippingsoftware, first and foremost Solving the flow-forward of scheduledprojects into any available space is something that can be worked on,but never solved, it really comes down to discipline between those incharge of setting up any given project's particular roadmapThe second, more subtle flaw in this workflow, and I think allTest-driven Devleopment workflows, revolves around the writer of thetests The fundamental nature of almost all bugs in software is humanerror, our natural tendency to make mistakes means that nothing we dowill ever be perfect, including our tests If Developer A is writing acouple new methods to handle data validation prior to that data goinginto the database Chances are that Developer A's life is going to bemade far easier by writing some test cases to run through somepredefined user-input, and pass his validation code over it Thereinlies the problem, if the developer doesn't think of a particular edgecase when he's writing the code to handle the data validation, thechances he'll remember and account for that particular edge case whilehe's working on the unit tests is nilHow do you really ensure that tests are of high enough quality toactually catch errors and regressionsI think a certain extent of intra-team test writing and code review,depending on the level of communication between developers, can reallyhelp In this case less developer communication is better IfDeveloper A tells Developer B how his code works, Developer B is nowgoing to have an unnecessary expectation when he starts to write testsfor Developer A's code If Developer B reviews the code for what itactually is, instead of what Developer A thinks it is, the tests thatwill ultimately be written will be more thorough than if Developer Ahad written the whole suite himselfThis still isn't sufficiently fool-proof to where I feel all thatconfident in test coverage, the tests being written are subject to theavailability, thoroughness and understanding that Developer B bringsto the table Inside a small team like this one, one of those isalmost always in short supply usually availabilityOne approach I'm anxious to try is the more active involvement of QAengineers in the test writing process, both in the pre-fail andpost-fail scenarios The pre-fail scenario being one like that which Idetailed above, where new code is being written In this case a QAengineer's experience can help guide the developer on what sets ofuser-input have typically caused issues in the past The second case,post-fail, is actually already occuring at Slide; a live issue, datavalidity bug, or regression is caught by QA engineers who detail thereproduction case in Trac and as a result a regression test can bewritten for that specific issueThis still is subject to the three things I cited above: availability,thoroughness and understanding of those involved I still have a lotof unanswered questions about the ideal QA and Dev workflow however,how does this scale to a team of tens or hudnreds Who writes thetests for large teams What about a team of 1 Dev and a 1 QA, whatabout the lone-hacker How do you write quality code, without gettingbogged down in the mush of writing thousands of tests for everythingyou can imagine could go wrongWho writes the testsIMAGEIMAGEIMAGEhttp://www.secuobs.com/revue/news/89981.shtmlhttp://www.secuobs.com/revue/news/89981.shtml Secuobs.com : 2009-04-29 18:37:07 - unethical blogger - I've been sending weekly "Protip" emails about Git to the rest ofengineering here at Slide for a while now, using the "Protips" as ameans of introducing more interesting and complex features Git offersBelow is the fourth Protip written to date---------------------------------------------------------------------While the concept of "tagging" or "labeling" code is not a new, ororiginal idea that was introduced with Git, our use of tags in aregular workflow does not predate the migration to Git however Atit's most basic level, a "tag" in any version control system is totake a "picture" of how the tree looks at a certain point in time suchthat it can be re-created later This can be extremely helpful forboth local and team development, take the following scenario for localdevelopment using tags:Tim is extremely busy, most of his days working at an exciting,fast-paced start-up seem to fly by With one particular project Tim isworking on, a lot of code is changing at a very fast pace and thebranch he's currently working in is stable one minute and destabilizedthe next Tim has two basic options for leaving himself "bread-crumbs"to step back in time to a stable or an unstable state The first,complicated option, is to mark his commit messages with something like"STABLE", etc so he can git diff or git reset --hard from the currentHEAD to the last stable point of the branchThe second option is to make use of tags Whenever Tim reaches astable point in his turmultuous development, he can simply run:git tag wip-protips_`date "+%s"or something similar, `date` added to ensure the tag is unique IfTim finds himself too far down the wrong path, he can rollback hisbranch to the latest tag git reset --hard protiptag, create a newstable branch based on that tag git checkout -b wip-protip-2protiptag, or diff his current HEAD to the tag to see what all he'schanged since his branch was stable git diff protiptagHEADThis local development scenario can become a team development scenarioinvolving tags, if for example, Tim needed QA to start testingportions of his branch his changes are just that important Sincethe current HEAD of Tim's branch is incredibly unstable, he can pushhis tag to the central repository so QA can push a stage using the tagto the last stable point in the branch's history with the command: gitpush origin tag protiptagTags are similar to most other "refs" in Git insofar that they aredistributable, if I execute git fetch your-repo --tags, I can pull thetags you've set in "your-repo" and apply them locally aid developmentThe distributed nature is primarily how tags differ in Git fromSubversion, nearly the rest of the concept is the exact sameCurrently at Slide, tag usage is dominated by the post-receive hook inthe central repository, where every push into the central repository"origin" in the branch release branch is tagged This allows us toquickly "revert" bad live pushes temporarily, by simply pushing thelast "good" tagged release, to ensure minimal site destabilizationwhile we correct live issues outside of the release branchFor more specific usage of `git tag` refer to the git-tag1 man page---------------------------------------------------------------------Did you know Slide is hiring Looking for talented engineers to writesome good Python and/or JavaScript, feel free to contact me attyleratslideIMAGEIMAGEIMAGEhttp://www.secuobs.com/revue/news/89980.shtmlhttp://www.secuobs.com/revue/news/89980.shtml Secuobs.com : 2009-04-29 18:37:07 - unethical blogger - I'm going to go ahead and admit something, something that's difficultfor most men to admit in my situation I shop at Old Navy I'm sorry,I like their collared shirts Sue meThis past weekend I decided to use an oldnavycom gift card that I wasgiven to buy some new jeans as my favorite pair now has a hole in theknee A "cute" side effect of redeeming an oldnavycom gift card wasthat I needed to create an oldnavycom account "Cute"After I created my account, with a site-specific password I generatethrow-away passwords for sites that abuse the privilege of mybusiness, I received the following email:IMAGELike I said, "cute" Damn idiotsIMAGEIMAGEIMAGEhttp://www.secuobs.com/revue/news/89979.shtmlhttp://www.secuobs.com/revue/news/89979.shtml Secuobs.com : 2009-04-29 18:37:07 - unethical blogger - On the topic of online shopping "sucking", I have been sitting on thisbeautiful screenshot for a whileA couple of months ago I bought a watch on Amazon Not a spectacularwatch, a very basic Seiko analog watch that I had previously owned buthad lost I went on to Amazon to buy "my watch", and after finding it,I happily ordered the watchShortly after the watch arrived, I noticed a huge influx of quitetopical SPAMIMAGEI'm pleased to say that I've not purchased anything from Amazon sinceI discovered that Amazon, or somebody that Amazon deals with sold myinformation to everybodyThis still makes my blood boil Rat bastardsIMAGEIMAGEIMAGEhttp://www.secuobs.com/revue/news/89978.shtmlhttp://www.secuobs.com/revue/news/89978.shtml Secuobs.com : 2009-04-29 18:37:07 - unethical blogger - I've spent the entire day thinking about "cloud computing", which isquite a twist for me Seeing "impressive" conferences centered around"cloud computing" I've ridiculed the concept mercilessly, it has aphenomenally high buzzword/usefulness ratio, which makes it difficultto take seriously It tends to have an air of idiocy attached to it ofthe same style that the re-invention of thin-clients did a few yearsback That said, I think the concept is sound, and useful for a numberof companies and uses once distilled of the buzzTake Slide for example, we have a solid amount of hardware, hundredsof powerful machines constantly churning away on a number of tasks:serving web pages, providing back-end services, processing databaserequests, recording metrics, etc If I start the work-week needing anew pool of machines either set up or allocated for a particular task,I can usually have hardware provisioned and live by the end of theweek depending on my Scotch offering to the Operations team, I canget it as early as the next day If I can have the real thing Iclearly have no need for cloud computing or virtualizationThat's what I thought, at least, until I started to think more aboutwhat would be required to get Slide closer to the lofty goal ofcontinuous deployment As I was involved in pushing for and setting upour Hudson CI server, I constantly check on the performance of thesystem and help make sure jobs are chugging along as they should be,I've become the defacto Hudson janitorOur current continuous integration setup involves one four-coremachine running three separate instances of our server software asdifferent users, processing jobs throughout the day One "job"typically consists of a full restart of the server software Pythonand running literally every test case in the suite we walk the entiretree aggregating tests On average the completion of one job takesclose to 15 minutes, and executes around 400+ test cases andgrowing Fortunately, and unfortunately, our Hudson machine is nolonger able to service this capacity during development peak in themiddle of the day, this is where the "cloud" comes inWe have a few options at this point:* Setup another one or more machines* Rethink how we provision hardware for continuous integrationThe fundamental problem with provisioning resources for continuousintegration, at least at Slide, is that the requirements are bursty atbest We typically queue a job for a particular branch when adeveloper executes a git push via the Hudson API and a post-receivehook From around 9 pm until 9 am we don't need but maybe twoactual "executors" inside Hudson to handle the workload the night-owldevelopers tend to place on Hudson, from 12 pm until 7 pm howeverour needs fluctuate rapidly between needing 4 executors, and 10executors To exacerbate things further, due to "natural trafficpatterns" in how we work, mid-afternoon on Wednesday and Thursdayrequire even more resources as teams are preparing releases andfinishing up milestonesThe only two possible solutions to solve the problem are to: build acontinuous integration farm with full knowledge capacity will remainunused for large amounts of time, or look into "cloud computing" withservice provides like Amazon EC2 which will allow for Hudson slaves tobe provisioned on demand The maintainer of Hudson, Kohsuke Kawaguchihas already started work on "cloud support" for Hudson via the EC2plugin which makes this a real possibility Note: using EC2 for thisat Slide was Dave's idea, not mine :Using Amazon EC2 isn't the only way to solve this "bursty" problemhowever, we could just as easily solve the problem in house withprovisioning of Xen guests across a few machines The downside ofdoing it yourself is amount of time between when you know you needmore capacity and when you can actually add that capacity to your ownlittle "cloud" Considering Amazon has an API for not only runninginstances but terminating them, it certainly provides a compellingreason to "outsource" the problem to Amazon's cloudI recommend following Kohsuke's development of the EC2 plugin forHudson closely, as continuous integration and "the cloud" seem like amatch made in heaven alright, that pun was unnecessary, it sort ofslipped out At the end of the day the decision comes down to a veryfundamental business decision: which is more cost effective, buildingmy own farm of machines, or using somebody else'sfootnote: I'll post a summary of how and what we eventually do tosolve this problemIMAGEIMAGEIMAGEhttp://www.secuobs.com/revue/news/89977.shtmlhttp://www.secuobs.com/revue/news/89977.shtml Secuobs.com : 2009-04-29 18:37:07 - unethical blogger - I've been sending "Protip" emails about Git to the rest of engineeringhere at Slide for a while now, using the "Protips" as a means ofintroducing more interesting and complex features Git offers---------------------------------------------------------------------There are those among us who can look at a reproduction case for a bugand just know what the bug is For the rest of us mere mortals,finding out what change or set of changes actually introduced a bug isextremely useful for figuring out why a particular bug exists This iseven more true for the more elusive bugs or the cases where code"looks" correct and you're stumped as to why the bug exists now, whenit didn't yesterday/last week/last month The options in mostclassical version control systems you have available to you are tosift through diffs or wade through log message after log messagetrying to spot the particular change that introduced the regressionyou're now tasked with resolvingFortunately of course Git offers a handy feature to assist you intracking down regressions as they're introduced, git bisect Take thefollowing scenario:Roger has been working on some lower level changes in a projectbranch lately When he left work last night, he ran his unit testseverything passed, committed his code and went home for the dayWhen he came in the next morning, per his typical routine, hesynchronized his project branch with the master branch to ensurehis code wasn't stomping on released changes For some reasonhowever, after synchronizing his branch, his unit tests started tofail indicating that a bug was introduced in one of the changesthat was integrated into Roger's project branchBefore switching to Git, Roger might have spent an hour looking overchanges trying to pinpoint what went wrong, but now Roger can use gitbisect to figure out exactly where the issue is Taking the commithash from his last good commit, Roger can walk through changes andpinpoint the issue as follows:## Format for use is: git bisect start -- xdev4% git bisect start HEAD 324d2f2235c93769dd97680d80173388dc5c8253Bisecting: 10 revisions left to test after this064443d3164112554600f6da39a36ffb639787d7 Changed the name of an a/b testxdev4%This will start the bisect process, which is interactive, and startyou halfway between the two revisions specified above see the imagebelow Following the scenario above, Roger would then run his unittests Upon their success, he'd execute "git bisect good" which wouldmove the tree halfway between that "good" revision and the "bad"revision Roger will continue doing this until he lands on the committhat is responsible for the regression Knowing this, Roger can eitherrevert that change, or make a subsequent revision that corrects theregression introducedIMAGEA sample of what this sort of transcript might look like is below:xdev4% git bisect good Bisecting: -1 revisions left to test after thisbcf020a6c4ac7cc5df064c66b182b2500470000a Merge branch 'cjssp' into masterxdev4% git bisect badbcf020a6c4ac7cc5df064c66b182b2500470000a is first bad commitxdev4% git show bcf020a6c4ac7cc5df064c66b182b2500470000acommit bcf020a6c4ac7cc5df064c66b182b2500470000aMerge: 62153e2 064443dAuthor: Chris Date: Tue Jan 27 12:57:45 2009 -0800Merge branch 'cjssp' into masterxdev4% git bisect log# bad: 7a5d4f3c90b022cb66fd8ea1635c5de6768882d7 Merge branch 'foo' into master# good: d1014fd52bebd3c56db37362548e588165b7f299 Merge branch 'bar'git bisect start 'HEAD' 'd1014fd52bebd3c56db37362548e588165b7f299' '--' 'apps'# good: 064443d3164112554600f6da39a36ffb639787d7 Changed the name of an a/b test PLEASE PICK ME UP WITH NEXT PUSH thxgit bisect good 064443d3164112554600f6da39a36ffb639787d7# bad: bcf020a6c4ac7cc5df064c66b182b2500470000a Merge branch 'cjssp' into mastergit bisect bad bcf020a6c4ac7cc5df064c66b182b2500470000axdev4% git bisect reset xdev4%Instead of spending an hour looking at changes, Roger was able toquickly walk a few revisions and run the unit tests he has to figureout which commit was the one causing trouble, and then get back towork squashing those bugsRoger is, like most developers, inherently lazy, and running through aseries of revisions running unit tests sounds like "work" that doesn'tneed to be done Fortunately for Roger, git-bisect1 supports thesubcommand "run" which goes hand in hand with unit tests or othertests In the example above, let's pretend that Roger had a test caseexhibiting the bug he was noticing What he could actually do is letgit bisect run automatically run a test script to run his unit teststo find the offending revision ie:xdev4% git bisect start HEAD 324d2f2235c93769dd97680d80173388dc5c8253Bisecting: 10 revisions left to test after this064443d3164112554600f6da39a36ffb639787d7 Changed the name of an a/b testxdev4% git bisect run /mytestshAfter executing the run command, git-bisect1 will binary search therevisions between GOOD and BAD testing whether or not "mytestsh"returns a zero success or non-zero failure return code until itfinds the commit that causes the test to fail The end result shouldbe the exact commit the regression was introduced into the tree, afterfinding this Roger can either grab his rubber chicken and go slap hisfellow developer around or fix the issue and get back to playingNethackAll in all git-bisect1 is extraordinarily useful for pinning downbugs and diagnosing issues as they're introduced into the code baseFor more specific usage of `git bisect` refer to it's man page here:git-bisect1 man page---------------------------------------------------------------------Did you know Slide is hiring Looking for talented engineers to writesome good Python and/or JavaScript, feel free to contact me attyleratslideIMAGEIMAGEIMAGEhttp://www.secuobs.com/revue/news/89976.shtmlhttp://www.secuobs.com/revue/news/89976.shtml Secuobs.com : 2009-04-29 18:37:07 - unethical blogger - Over the past couple years I've talked a lot of trash about JavaScriptreally, a lot but I've slowly started to come around to a moreneutral stance, I actually hate browsers, I like JavaScript just fineby itself While the prototype-based object system is a little weirdat first coming from a more classical object-oriented background, theconcept grows on you the more you use itSince I hate browsers so much I really do, I was pleased as punch tohear that Google's V8 JavaScript Engine was embeddable While WebKit'sJavaScriptCore is quite a nice JavaScript engine, it doesn't lenditself to being embedded the same way that V8 does The only immediatedownside to V8 is that it's written entirely in C++, which doesprovide some hurdles to embedding for example, I'm likely never goingto be able to embed it into a Mono application, but for the majorityof cases embedding the engine into a project shouldn't be all thatdifficultA few weekends ago I started exploring the possibility of runningserver-side JavaScript courtesy of V8, after reading about mod_v8 Ifelt more confident to try my project: FastJSIn a nutshell, FastJS is a FastCGI server to process server-sideJavaScript, this means FastJS can hook up to Lighttpd, Nginx, or evenApache via mod_fcgi Currently FastJS is in a state of "extremelyunstable and downright difficult", there's not a lot there as I'mexploring what should be provided by the FastJS server-side software,and what should be provided by JavaScript libraries As it stands now,FastJS preloads the environment with jQuery 132 and a "fastjs"object which contains some important callbacks like:fastjswrite // write to the output streamfastjslog // write to the FastCGI errorlgofastjssource // Include and execute other JavaScript filesOn the server side, a typical request looks something like this fornow:2009-03-09 05:04:06: responsec114 Response-Header:HTTP/11 200 OKTransfer-Encoding: chunkedContent-type: text/htmlX-FastJS-Request: 1X-FastJS-Process: 11515X-FastJS-Engine: V8Date: Mon, 09 Mar 2009 09:04:06 GMTServer: lighttpd/1418Below is an example of the current test page "indexfjs":1 var index = new Object ;2  3 index header = function {4 fastjs write "FastJS" ;5 fastjs write "" ;6 fastjs write "FastJS Test Page" ;7 } ;8  9 index footer = function { 10 fastjs write "" ;11 } ;12  13 index dump_attributes = function title, obj {14 fastjs write "" ;15 fastjs write title ;16 fastjs write "" ;17  18 for var k in obj {19 fastjs write k + " = " ;20  21 if typeof obj k = "string" 22 fastjs write typeof obj k ;23 else24 fastjs write obj k ;25  26 fastjs write " " ;27 }28 } ;29  30 function {31 index header ;32  33 fastjs source "pages/testfjs" ;34  35 index dump_attributes "window" , window ;36 index dump_attributes 'location' , location ;37 index dump_attributes "fastjsenv" , fastjs env ;38 index dump_attributes "fastjsfcgi_env" , fastjs fcgi_env ;39  40  41 index footer ;42  43 fastjs log "This should go into the errorlog" ;44 } ;The code above generates a page that looks pretty basic, butinformative nonetheless click to enlarge:IMAGEPretty fun in general to play with, I think I'm near on the pointwhere I can stop writing more of my terrible C/C++ code and get backinto the wonderful land of JavaScript As it stands now, here's whatstill needs to be done:* Proper handling of erroring scripts via an informative 500 pagethat reports on the error* Templating Lots of fastjswrite calls are likely to drive youmad* Performance concerns As of now, the whole stack jQuery + fjsare evaluated every page request* Tests I should really get around to writing some level ofintegration tests to make sure that FastJS is returning expectedresults for particular chunks of fjs scriptsThe project is hosted on GitHub right now, here and is under a2-clause BSD licenseIMAGEIMAGEIMAGEhttp://www.secuobs.com/revue/news/89975.shtmlhttp://www.secuobs.com/revue/news/89975.shtml Secuobs.com : 2009-04-29 18:37:07 - unethical blogger - One of the nice things about living in Silicon Valley is that you haverelatively easy access to a number of the developers you may work withthrough open source projects, mailing lists, IRC, etc Today KohsukeKawaguchi of Sun Microsystems, the founder of the Hudson project,stopped by the Slide offices to discuss Hudson and the "cloud",continuous deployment and our workflow with Hudson here at SlideContinuous deployment being the most interesting topic for me, and themost relevant in terms of the importance of Hudson in our currentinfrastructureSince reading Timothy Fitz's post on the setup for "continuousdeployment" at IMVU, I've become obsessed to a certain degree withpushing Slide in that direction as an engineering organizationCurrently we push a number of times a day as necessary, and it'salmost as if we have manual-continuous-deployment as it is right now,there's just a lot of room for optimizations and automation to cutdown on the tedium and allow for more beer drinking@agentdero continuous deployment = when build is green, autoshipsounds terrifying@tlipconAs a concept, continuous deployment can be quite scary "wait, somerobot is going to deploy code to my production site, wha" It'simportant to remember that the concept of continuous deploymentdoesn't necessarily mean that no QA is involved in the releaseprocess, it is however ideal to have enough good test cases such thatyou can do a fully automated unit/integration/system test run Thebiggest difficultly with the entire concept of "continuous deployment"however is not writing tests or actually implementing a system todeploy, it forces you to understand your releases and productionenvironment; it's about eliminating the guess work from your processand reducing the amount of human error or potential for human errorinvolved in deploymentsIn my opinion, continuous deployment isn't about making a hard switch,firing your QA and writing boat-loads of tests to ensure that you canpush the production site straight from "trunk" as much as humanlypossible Continuous deployment is far more about solidifying yourunderstanding of your entire stack, evolving your code base to whereit is both more testable and better covered by your tests, thenputting your money where your mouth is and relying on those tests Ifyour codebase moves rapidly, unit/integration/system tests are onlygoing to be up to date and valuable if you actually rely on them Ifbreaking a single unit test pre-deployment becomes a Big Deal™, thenthe developer responsible for the code being deployed will make surethat: a the test is valid and up to date and b the code that thetest is covering does not contain any actual regressionsTake the typical repository layout for most companies which is, as faras I've seen, made up of a volatile trunk, stable release branch andthen a number of project branches In an engineering department QAwould be responsible for ensuring that projects are properly vettedbefore merging from project branches also called "topic branches" inthe Git community into the more volatile trunk branch Once the CIserver ie Hudson picks up on changes in trunk, the testing processwould begin at that particular revision Provided the test suitespassed with flying colors Hudson would start to kick up the process todo a slow/sampled deploy as Timothy describes in his post If thetests failed however, alarms would start beeping, sirens would wailand there would be much gnashing of teeth, somebody has now brokentrunk and is blocking any other deployments coming down the pipe Inthis "disaster scenario" the QA involved in the process would bethoroughly shamed obviously but then given the choice to blockfuture pushes while the developers create a fix or revert theirchanges out of trunk and take them back to a project branch to correctthe deficiencies This attention to detail will have an larger benefitin that developers won't become numb to test failures to where they'reno longer importantWhat good is writing tests if there aren't have real-consequences forthem failing Releases shouldn't be a scary time of theday/week/month, you should certainly be nervous keeps you sharp, butif you fear releases then it is probably an error in your releaseprocess that allows for too much uncertainty: inadequate testcoverage, insufficient blackbox testing, poor release practices, etcContinuous deployment might not be the magic solution to your woe ofshipping software but the practice of moving towards continuousdeployment will greatly improve your release process whether or notyou ever actually make the switch over to a fairly automateddeployment process as the engineers at IMVU haveHow confident are you in your test coverageIMAGEIMAGEIMAGEhttp://www.secuobs.com/revue/news/89974.shtmlhttp://www.secuobs.com/revue/news/89974.shtml Secuobs.com : 2009-04-29 18:37:07 - unethical blogger - Over the past couple years that I have been working at Slide, IncI've had a love/hate relationship with the Cheetah templating engineLike almost every templating engine, it allows for abuse by its users,which can result in some templating code that looks quite horrendous,contributing significantly to some negative opinions of the templatingengine At one point, I figured an upgrade of Cheetah would helpcorrect some of these abuses and I distinctly remember pushing toupgrade to the 2xx series of Cheetah I then found out that I hadunintentionally volunteered myself to oversee the migration and alsoto update any ancient code that was lying around that depended on"features" see: bugs in Cheetah prior to the 2xx series Weupgraded to Cheetah 2xx and life was good, but Cheetah waspractically deadThe last official release of Cheetah was in November of 2007, this isnot something altogether uncommon in the world of open sourcedevelopment Projects come and go, some reach a point in their growthand development where they're abandoned, or their communitydissipates, etc As time wore on, I found myself coming up with apatch here and there that corrected some deficiency in Cheetah, but Ialso noticed that many others were doing the same There was veryclearly a need for the project to continue moving forward, and with myintroduction to both Git and GitHub as a way of distributingdevelopment, I did what any weekend hacker is prone to do, I forkeditMeet Community Cheetah----------------------On January 5th, 2009 I started to commit to my local fork of theCheetah code base taken from Cheetah CVS tree, making sure mypatches were committed but also taking the patches from a number ofothers on the mailing list By mid-March I had collected enoughpatches to properly announce Cheetah Community Edition v210 to themailing list I was entirely unprepared for the responseWhereas the previous 6 months of posts to the mailing list averagedabout 4 messages a month, March exploded to 88 messages, 20 of them inthe thread announcing Cheetah CE now deemed Community Cheetah it hada better ring to it, and an available domain name to boot All of asudden the slumbering community is awake and the patches have startedto trickle inWe've fixed some issues with running Cheetah on Python 26, Cheetahnow supports compiling templates in parallel, issues with importbehavior have been fixed and added a number of smaller features In2008 there were six commits to the Cheetah codebase, thus far in 2009there have been over seventy I'm still waiting on a few patches fromcolleagues at other startups in Silicon Valley as wellI'm not going to throw up a "Mission Accomplished" banner just yet,Cheetah still needs a large amount of improvement It was writtenduring a much different era of Python, the changes in Python 26 andmoving forward to Python 30 present new challenges in modernizing atemplate engine that was introduced in 2001Being a maintainer------------------Starting your own open source project is tremendously easy, especiallywith the advent of hosts like Google Code or GitHub What's terrifyingand difficult, is when other people depend on your work By steppingup and becoming the de-facto maintainer of Community Cheetah, I'veopened myself up to a larger collection of expectations than Ioriginally anticipated I feel as if I have zero credibility with thecommunity at this point, which means I painstakingly check the changesthat are committed and review as much code as possible before tagginga release I'm scared to death of releasing a bad release of CommunityCheetah and driving people away from the project, the nightmarescenario I play over in my head when tagging a release in Git issomebody going "this crap doesn't work at all, I'm going to stick withCheetah v201 for now" such that I cannot get them to upgrade tosubsequent releases of Community Cheetah I think creators of aproject have a lot of "builtin street cred" with their users andcommunity of developers, whereas I still have to establish my streetcred through introduction of bug fixes/features, knowledge of the codebase and generally being available through the mailing list or IRCMoving Forward--------------Currently I'm preparing the third Community Cheetah release which Itagged today v211 which comes almost a month after the previous oneand introduces a number of fixes but also some newer features like the#transform directive, markdown support, and 100% Python 26compatibilityThanks to an intrepid contributor, Jean-Baptiste Quenot, we have av22 release lined up for the near future which fixes a large numberof Unicode specific faults that Cheetah currently has the code cancurrently be found in the unicode branch and moves the internalrepresentation of code within the Cheetah compiler/parser to a unicodestring object in PythonI eagerly look forward to more and more usage of Cheetah, with othertemplating engines out there for Python like Mako and Genshi I stillfeel Cheetah sits far and above the others in its power andversatility but has just been neglected for far too longIf you're interested in contributing to Cheetah, you can fork it onGitHub, join the mailing list or find us on IRC #cheetah onFreenodeThis experiment on restarting an open source project is far from over,but we're off to a promising startIMAGEIMAGEIMAGEhttp://www.secuobs.com/revue/news/89973.shtmlhttp://www.secuobs.com/revue/news/89973.shtml Secuobs.com : 2009-04-29 18:37:07 - unethical blogger - It's not been a secret that I'm a big fan of open source software, Iwould likely not be where I am today had I not started with projectslike OpenBSD or FreeBSD way back when While my passion for opensource software and the "bazaar" method of developing software mightnot be shared by everybody at Slide, Inc, everybody can certainly geton board with the value of incorporating open source into our stack,which is almost entirely comprised of Python and an assortment ofother web technologiesAlong those lines, there's been some amount of discussion about whatwe can or should open source from what we've developed at Slide, butwe've not really pushed anything out into the ether as of yet Todayhowever, I think we finally put our foot in the door as far ascontributing back to the open source community as a whole, we're nowon GitHub as "slideinc", yay coincidentally we have a slideinctwitter account tooCurrently the only project that's come directly out of Slide, andshared via the slideinc GitHub account is PyVE, a Python Virtual Earthclient that I hacked together recently to tinker with some Geocodingreleased under a 3-clause BSD license In the hopefully nearfuture we'll continue to open source some other components we'veeither created or extended internallyIf you're not a GitHub user, you should definitely check GitHub out,it's a pretty impressive site If you are a GitHub user, or a Pythondeveloper, you should "follow" the slideinc user on GitHub to catchthe cool stuff that we may or may not ever actually release ;IMAGEIMAGEIMAGEhttp://www.secuobs.com/revue/news/89972.shtmlhttp://www.secuobs.com/revue/news/89972.shtml