Janitor Programmer

webrat needs to be bundled as well

2011-07-08T14:19:00.001-07:00

In addition to getting rspec installed correctly, webrat is required to test the page content with response.should have_selector("title"....

Here are the instructions. I had to include the proper version (0.7.0) in the Gemfile, then do a bundle install.

getting rspec installed

2011-07-08T13:13:00.001-07:00

The tutorial I'm using calls for using rspec for testinhttp://www.blogger.com/img/blank.gifg. In order to get it so I could install it and add it I had to follow these instructions.

Finally really working on Rails

2011-07-07T13:20:00.001-07:00

After several false starts we have a project at work where we are looking at using Rails. So far so good. I'm using http://ruby.railstutorial.org/ruby-on-railshttp://www.blogger.com/img/blank.gif-tutorial-book as my tutorial and so far it is my kind of book. I'm using the bitnami rubystack virtual machine using Rails 3 in ubuntu. There have been a couple of hiccups along the way. 1) is remembering to use sudo on the VM for most installs and updates and even running the server to test the appliction. Not a HUGE deal, but a minor annoyance. Also, when doing the push the first time to deploy to heroku, I got an error saying that heroku does not appear to be a git repository. The fix for that is here.

Math and web design

2011-04-17T21:08:00.001-07:00

An interesting article about using prime numbers and tiling backgrounds for creating interesting effects with a minimal cost.

http://designfestival.com/the-cicada-principle-and-why-it-matters-to-web-designers/?utm_medium=email&utm_campaign=Design+View+83+-+Cicadas+Pseudo-random+Organic+Tiling+and+Top+Design+Mistakes+HTML&utm_content=Design+View+83+-+Cicadas+Pseudo-random+Organic+Tiling+and+Top+Design+Mistakes+HTML+CID_068c8ba45a4520e40c5d7eb008dc7279&utm_source=Newsletter&utm_term=The+Cicada+Principle+and+Why+It+Matters+to+Web+Designers

useful erlang posts

2011-01-11T07:50:00.000-08:00

I'm starting to learn about erlang. here are some useful posts I found.

http://yarivsblog.blogspot.com/2008/05/erlang-vs-scala.html

http://ferd.ca/an-open-letter-to-the-erlang-beginner-or-onlooker.html

There Are No Internal Users

2011-01-03T19:25:00.000-08:00

Anybody who has worked for a small company or in an IT department where most of your users are 20 feet down the hall has heard the following; "Don't worry about creating an admin interface, this application is just for internal users."

The problem with internal users is that they don't really exist in the sense that "decision makers" would like them to. There are people that use your application and people that do not use your application. That said people are separated by a hallway instead of an interstate does not make them particularly special. When you are asked to design for internal users you are usually being asked to skimp on these features:

Administrative interfaces.

Ease of use enhancements.

Documentation.

Please, please, PLEASE avoid the temptation to cut drastically on the above features. They don't have to have the spit and polish you would give to a client facing application (a misspelled word here or there won't kill anybody) but these features should exist, they should make sense, and they should be useable without people having to email you every 3 months asking if you can make one little change here or there.

Without an administrative interface, you will need to remember what all the tables and files in your application are responsible for. You will need to completely memorize the internal structure of the application, or you will need to wade through piles of code every time somebody has a request. Neither situation is good. Those brain cells are better spent remembering good places to eat lunch, and the time you save by not designing a proper administrative interface will be spent every time somebody has a question about making a configuration change. Either spend a little time and effort now, or a lot later. Multiple times. Until you either get fed up and kill somebody, quit, or write the administrative interface anyway.

Anything you put in front of users should be intuitive. Granted in most cases internal users, or users in any niche market, can be expected to have a level of sophisitication that is above average, but if interfaces don't make sense, the application isn't serving its purpose. Computers and applications are supposed to be tools that make people's lives easier, not puzzles. At least, not in this case.

Documentation is critical. Even if it is nothing more than an email briefly explaining the application's feature set and how to use the main features. You don't need a complete manual detailing why you chose the lovely magenta headers, but you should have something that users, and you, can refer to that explains what the application does and how it does it. For this sort of documenation I prefer a wiki. It can be a pain to go update the documentation every time something changes, but you will be kicking yourself when, 8 months after deployment, somebody comes down the hall with a question that you don't remember the answer to. Instead of taking 15 minutes to skim through some docs, you will be taking hours going through an old code base you barely remember.

I've found this particularly true for instances where I am writing small one-off applications, excel macros, python scripts, etc. that are written to solve a particular issue. There are stretches where I crank out 3 or 4 of these a week, and then many months later somebody asks me for a program that performs some function I've already taken care of, or they ask about using something they haven't touched in a while and they've deleted the email that contains the directions. If you write many small utilities, at some point you will start losing track of what they all do. Simple to use interfaces with clear documentation will save your users from some angst and will save you from wasting time either going through old code or, worse, reinventing your own wheel.

The problem with Javascript

2010-12-16T07:09:00.000-08:00

Let me open this by saying that I like JavaScript. The syntax is comfortable for a lot of people and it lets you do all kinds of funky stuff because functions are first class objects and prototype based classes. It also enjoys an extremely wide install base because it is in almost every browser people use. I believe that it has a bad reputation mostly because it has suffered from years of poor and differing implementations. I also think it bears the brunt of having a variety of DOMs (These days IE vs. everybody else) associated with it; much the same way that I think C++ has a bad reputation because a lot of people's experience with C++ began and ended with MFC.

But there is one problem JavaScript has that was clarified to me after reading this critique of Google Closure. The problem, in a nutshell, is that the developer has to act as an optimizing compiler in many instances. Looking at most of the critiques, they revolve around optimizations that most people in the 21st century are either accustomed to having taken care of for them, or their computing power is so great compared to what they do that they don't care. At this point an interpreted language in a browser is handicapped in both of these areas.

Looking at the list of critiques, slow loops because you access a property on every loop check, slow case statements and dealing with multiple types of strings are all things that most developers don't have to worry about anymore. The problems with the code in the library and more efficient ways to do the same thing cover a large part of the post. That you can write many paragraphs on this sort of thing and that these optimizations are necessary is indicative that implementations of the language still has plenty of room for improvement. The concatenation of "" to a value to make it a string more efficiently than using String() is, to me, the most egregious example of this. (somevalue + "").replace() may be fast, but it sure is ugly to look at.

The author at one point even takes a jab at Google, saying their time may have been better spent just writing proper JavaScript instead of making the investment in making Chrome's JavaScript performance better. I'm guessing this is tongue in cheek, wry humor. Otherwise the implication is that JavaScript is fine on the performance front. As long as the programmer has to perform the sorts of optimization hacks described in the above post, that is simply not the case.

random subsets of data ii

2010-12-02T13:31:00.000-08:00

In doing more work to get random subsets, the previous solution fails miserably if you are going to be using a view. You can run the EXEC statement in a stored procedure but cannot use it in a view. And furthermore, calling a stored procedure from a view is somewhere between extremely problematic and impossible, depending on who you talk to. Using the RANK function and partitioning the data you can get a similar result.

Let us use the same idea as the previous example and assume we want at least one employee from every department. The only requirement for the following query is that you are asking for more employees than you have departments.


select top 100 T1.* from

(select 

RANK() over 
(PARTITION by Department order by newid()) as r,  

* 
from Employees
where Active = 1

) as T1
order by t1.r, t1.Department

Note, as before, that this is a sql server query. Modifications may need to be made for different flavors of SQL. What this is doing is for every department the employees are getting randomly ranked thanks to order by NEWID(). By selecting the top 100 and ordering by rank and then department, you'll get all of the 1's from every department, then all of the 2's and so on until you get 100.

The order by department is strictly unnecessary. I did it to make viewing and verifying the results easier.

random subsets of data

2010-11-19T12:42:00.000-08:00

I recently had a request to return a random subset of contacts from a database, but to be sure that there were some contacts from each group represented. Let us say that you have a database with employees and you want to get a list of some random people from each department. Here is how I went about doing it:



declare @s1 varchar(max)

select @s1 = coalesce(@s1 + ' union all ', '') +
e from (

select distinct
'select * from (select top 11 percent * from 
Employees where Department = '''
+ Department + ''' order by newid()) 
as [t' + Department + ']'  as e
from Employees) T1
exec (@s1)

What we are doing here is generating the text for a query that will select the top N contacts from the Employee list for each department. The coalesce function will put all of these queries together into one query that will union the results into a single table. The exec function will execute the query.

Note that this is a TSQL example to run on SQL Server. You may have to translate this, depending on your database system.

If you want a fixed number then what I would do is tweak the percent to get a number than is slightly more than the number desired then just select the top N from that. This runs the risk of not getting somebody from every department. Another method would be to select a number slightly less than your target, then select some random number of records to add to the result. This is more complex, but may be what you need.

Returning data tables from Web Services in .NET

2010-10-29T09:02:00.000-07:00

This is just a small reminder, and I don't know why it gets me every time, but it does. I guess I return data tables from web services infrequently enough where I don't think about it. Maybe writing it down somewhere will help me remember. When creating a data table to return from a web service, give the table a name.


DataTable dt = new DataTable("nameMyTable");

If you don't, the first time you hit the service when testing you will get an exception about the table not having a name. And then you will curse to yourself and then give it a name and move on with your life.

scripts for processing stuff

2010-10-22T08:51:00.000-07:00

Often I need to write something up quick to process a file, or process files in a directory and I'll want a GUI for these things so I can pass them along to other people to use. Here are two python scripts that I use as templates for picking either a file or a directory and then doing something with it. The object here isn't to have everything "correct" according to the latest design fashion or methodology. With this script I can import a python module that takes a file or directory name and does the processing I want, plug in the function name, pass the parameter and then move on with my life.

Pick a file and do something with it:


from tkinter import *
from tkinter.messagebox import *
from tkinter.filedialog import *

class FileProcessor(Frame):

# Object constructor
    def __init__(self, parent=None):
        Frame.__init__(self, parent)

        self.txtFile = Entry(parent)
        self.txtFile.place(x=6,y=64,width=375,height=24)
        self.txtFile.insert(0,'')

        self.btnPickFile = Button(parent,text='Pick File', command=self.btnPickFileClick)
        self.btnPickFile.place(x=6,y=20,width=96,height=24)

        self.btnRun = Button(parent,text='RUN', command=self.btnRunClick)
        self.btnRun.place(x=6,y=114,width=96,height=30)


# Methods (event handlers) of object
    def btnPickFileClick(self):
        print (self.txtFile.get())
        self.txtFile.delete(0,END)
        self.txtFile.insert(0,askopenfilename())

    def btnRunClick(self):
        fileName = self.txtFile.get()
        self.runFile(fileName)

    def runFile(self, fileName):
        print('code goes here to run when click: ' + fileName)
        showinfo('The file is',fileName)

# Method called if script is run directly
# instead of imported or used as a class
if __name__ == '__main__':
    root = Tk()
    root.title('Process File')
    myForm = FileProcessor(root)
    myForm.pack()
    root.geometry("423x156")
    root.minsize(423,156)
    root.maxsize(423,156)
    root.mainloop()

Pick a directory and do something wit it:


from tkinter import *
from tkinter.messagebox import *
from tkinter.filedialog import *

class DirectoryProcessor(Frame):

# Object constructor
    def __init__(self, parent=None):
        Frame.__init__(self, parent)

        self.txtDir = Entry(parent)
        self.txtDir.place(x=6,y=64,width=375,height=24)
        self.txtDir.insert(0,'')

        self.btnPickFile = Button(parent,text='Pick Directory', command=self.btnPickFileClick)
        self.btnPickFile.place(x=6,y=20,width=96,height=24)

        self.btnRun = Button(parent,text='RUN', command=self.btnRunClick)
        self.btnRun.place(x=6,y=114,width=96,height=30)


# Methods (event handlers) of object
    def btnPickFileClick(self):
        print (self.txtDir.get())
        self.txtDir.delete(0,END)
        self.txtDir.insert(0,askdirectory())

    def btnRunClick(self):
        dirName = self.txtDir.get()
        self.runDir(dirName)

    def runDir(self, directoryName):
        print('code goes here to run when click: ' + directoryName)
        showinfo('The directory is',directoryName)

# Method called if script is run directly
# instead of imported or used as a class
if __name__ == '__main__':
    root = Tk()
    root.title('Process Directory')
    myForm = DirectoryProcessor(root)
    myForm.pack()
    root.geometry("423x156")
    root.minsize(423,156)
    root.maxsize(423,156)
    root.mainloop()

Putting a string on a single line

2010-09-30T10:54:00.000-07:00

Often times text is written on multiple lines for the sake of clarity. Sometimes these lines need to be consolidated to a single line. This happens to me a lot with SQL queries that end up needing to be in a config file somewhere. I'll write them out all pretty so they make sense (as much sense as some SQL can make...) and then I'll need to consolidate that down to one line. Here is a python one liner that does just that, given a string named mystr

" ".join([x.strip() for x in mystr.split("\n")])

Don't make me slap you, Beavis

2010-09-08T09:54:00.000-07:00

Every single time somebody asks "how hard can it be?" or "why is the estimate so big?", I want to smack them. Hard. Some folks at 37signals were kind enough to offer an example that answers these sorts of questions when asked about a seemingly simple feature.

I applaud the fine folks at 37signals for publishing things like this.

splitting csv files

2010-08-09T08:24:00.001-07:00

I'm sure I've ranted about this before. I'll rant about this again. I just hate seeing examples from people in C# where they say that you just split a CSV file by using mystring.Split(',') and you will get an array where each item is the value for each field. One would assume, since they mention C#, that at some point in their lives they have worked with excel style CSV files that have data like

"Bob","Hello, Dear","""Dude, where's your car"""

The string split method obviously will not correctly handle this case at all. Here is a page I found with a nice little function that works for me in most cases.

http://www.tedspence.com/index.php?entry=entry070604-124237

Hopefully this will work for you, too.

I leave it as an exercise to the reader...

2010-07-30T14:01:00.000-07:00

When did this become a euphemism for "this is hard"?

I say this because I'm doing research on Visual Studio 2010 and one of the selling points is the Parallel Extensions for .NET. In a great many of the articles and tutorials I've found, the author goes through some totally brain dead examples that have no relevance to real life to show the basics. This is then followed by a discussion of some of the more interesting parts of the framework and then a paragraph or two about things you might try and finally something like "I leave it as an exercise for the reader to...." followed by something that might actually be interesting.

This is one of the reasons, by the way, that I prefer actual books to cobbling together a manual from various web pages. I know a lot of people say they will never by another programming book again because you can find everything you need on the web. I have not found this to be the case. And even when it is the case in a lot of instances one has to go through a lot of effort to cull the good nuggets from the pile of information you find. It's almost like panning for gold. Most good books do this for you already and the really good ones even include some decent examples of using libraries and techniques that have some relevance to real life.

I just needed to vent. Back to my research.

Working with outlook

2010-07-23T12:42:00.000-07:00

I found a nice piece of software called Outlook Redemption. I had to process some msg files that I had exported and this thing came in handy and was super simple to use. For what I needed, this was enough to get going.

From time to time I'm still amazed at the software that is freely available to just get things done and move on with your life.

link with good info on posting to page

2010-06-19T11:20:00.001-07:00

here is some information on posting to pages in facebook using the API

http://forum.developers.facebook.com/viewtopic.php?id=45796

NULL is a four letter word

2010-06-18T10:11:00.000-07:00

I used to be a big fan of NULL. It represents something that isn't really there and in many, many cases it makes sense logically and mathematically. I even had a good way of explaining it to lay-people that didn't really get it. think of it like bank accounts and account numbers for those bank accounts. An existing account with a balance of zero is different than not having an account at all, right? This worked for most people to help explain the difference, especially in text fields where the value of some fields would be the empty string and others would be NULL.

Lately though, I've come to the realization that this doesn't really matter. You can explain this in a multitude of ways that make sense, but when Judy in marketing wants to send an email to all of the people that are not in the Accounting department, she doesn't care about NULL. As the more technically or mathematically inclined among us know, if the department field for somebody is NULL, in most databases it would not be comparable to a string because it has no value to compare.

Select * from Employees where dept != 'Accounting'

will not return the people where dept is NULL in most cases. And this is usually very bad in the real world.

I'm done with NULL if anything close to a reasonable default exists. It just eliminates so many real world problems. I know from time to time I've come across people on discussion boards that have expressed this view and for the longest time I've disagreed with that notion on principle. I know many others that have as well. After much wailing and gnashing of teeth, I finally get it.

Consider it the 8th dirty word, after fuck, shit, piss, cunt, cocksucker, motherfucker and tits.

Fun with HTML 5

2010-06-10T09:51:00.000-07:00

I found a sample that had a simple particle system in html 5. I made one small improvement (pass in the type of particle to render) and changed it from raindrops to what could be a simple campfire. I like fire :-)

I put this up on a page here

test facebook like button

2010-05-17T11:06:00.000-07:00

The cyclical nature of programming

2010-02-19T07:29:00.000-08:00

Anybody who has been in this business for a long time knows that there are a lot of cycles in technology. Boom/bust cycles. Cycles where we process a lot of stuff on servers, then distribute it to nodes on the network, then move back to processing a lot stuff on servers. I'm sure there are others, but these are the two big ones that come to mind.

One thing I've noticed lately is that this is happening over on hacker news. Over the past couple of weeks there appear to have been a lot of links to items that I read many years ago. Some of them weren't even new then. Lots of Peter Norvig stuff. Some links to old Dykstra papers. That sort of thing. Things that should be part of the collective programmer consciousness, I guess.

I think this sort of thing is why we spend a lot of time rebuilding the wheel, as it were. You don't have to look to hard to find a large group of software folks that are convinced the credentials and formal schooling are waaaay overrated. But then you run into these periods where a large group seems to stumble upon some piece of writing or some paper and has to share it with the world, as if they've discovered something new and exciting and awesome. While I respect and love the enthusiasm of these sorts of things, I'm wondering how much overlap there is in these two groups. I would guess a lot. Maybe that's just my natural pessimism and cynicism kicking in.

I know that several well known people have lists of required reading for programmers. It just seems that these aren't very well known in the programming community at large. I've passed some of these around to various people. Some of them really bug me, however, because they are ephemeral. I know a lot of people, for example, that used to recommend one or more Extreme Programming books. Very enthusiastically. Now, they wouldn't touch them with a ten foot pole. They've moved on to Scrum. And soon I imagine they will move to some yet unnamed flavor of the month. Mostly in an effort to collect more consulting revenue.

I think I'm going to start collecting some of these. I'd like to break this particular cycle. Here are a few that come to mind. They mostly contain basic fundamentals or things that have stood the test of time. They are the sorts of things that are, in some cases, part of the collective hacker unconsciousness. Eric Raymond has written about this hacker culture stuff and has done a much better job than I could hope to do.

Eric can be polarizing, but he has some great writings. No matter what side of the ESR fence you sit on, at least he makes you think.
How to become a hacker
Cathedral and the bazarr
The Art of Unix Programming

norvig.com - Just an all around smart guy with lots of good things to say.

Joel on Software - The forums on this site used to be GREAT. The business of software board is still pretty good. The older essays are definitely worth a read.

The Mythical Man-month. Fred Brooks. Classic book. Short, too, which is nice :-)

I'll add others as they come to mind.

Writing more code isn't always the answer

2010-02-10T09:41:00.000-08:00

This is one of my very favorite stories of writing good code and how metrics can go horribly wrong.

http://www.folklore.org/StoryView.py?project=Macintosh&story=Negative_2000_Lines_Of_Code.txt

"Bill Atkinson, the author of Quickdraw and the main user interface designer, who was by far the most important Lisa implementor, thought that lines of code was a silly measure of software productivity. He thought his goal was to write as small and fast a program as possible, and that the lines of code metric only encouraged writing sloppy, bloated, broken code. "

Generally speaking, I'm with Bill.

Finding potential problems in CSV files

2009-12-16T12:10:00.000-08:00

I have to periodically work with CSV files from a variety of sources and the problems with them are pretty well known. Here is a little python script I use that helps me find values that contain double quotes, which are often not properly escaped in the files I receive


flines = open(fn,"r").readlines()
currline = flines[1]

for l in flines:
    vals = l.split(",")
    for v in vals:
        if '"' in v.strip()[1:-1]:
            print v

Javascript, sleep and closures

2009-12-06T14:54:00.000-08:00

My wife is working on a web site and she needs a different div to show up every few seconds, ad nauseum. I hack in javascript enough to be dangerous, so every time I try something new or something I haven't done in a long time, I turn to Google. My idea for this was to simply have a loop that goes through the elements, makes one visible, then sleeps for a few seconds, show the next one, etc. So I thought I would google javascript sleep and see what came back.

What I got was, to me at least, mostly a horror show.

Lots of examples with busy waiting and splitting functions up and a lot of code that worked, in so much as it did what somebody wanted. But the code was hard to read, hard to parse, hard to update and in some cases would burn up all the cpu you let it. Knowing all of the features available in javascript I KNEW there had to be a better way, but I wasn't finding anything I thought was "good", at least in the first page of google results. Maybe I just didn't look hard enough. But at this point I figured if I thought about this for a few minutes I would come up with a better solution in less time than I could find one that pleased me. Here, after a few minutes of thought, is what I came up with


function divrotate(interval,divarray)
{
 currI = 0
 currLen = divarray.length
 currE = document.getElementById(divarray[currI]);
 currE.style.display = '';
 
 
 
 showNext = function()
 {
  currE.style.display = 'none';
  currI++;
  if (currI == currLen) currI = 0;
  currE = document.getElementById(divarray[currI]);
  currE.style.display = '';
 }
 
 setInterval("this.showNext()", interval);
}

Here is what the call would look like from your HTML page


      <script src="/divrotate.js" type="text/javascript">
   </script>
      
      <script type="text/javascript">
   divrotate(8000, new Array("section1","section2","section3"));
   </script>

what you are passing in is an interval, in milliseconds, and the array of elements (in this case DIV) that you want to iterate through. The divrotate function gets the current element and shows it, then a closure is created that will hide the current element, get the next one and show it. setInterval is used to call the closure at the interval specified by the caller.

Simple code, and it doesn't peg the CPU.

I am sure that there has got to be some other examples that are straightforward and allow the equivalent of sleep. This seems like too common of a thing to not be elegantly handled somewhere, but quick searches in scriptaculous and mootools don't show my anything promising. I see lots of forum posts with the usual "Use setInterval or setTimeout, you'll see lots of examples if you Google for it!"

I know this is far from a generic sleep example, but I hope this helps somebody that wants to do something similar. I also hope that it serves as a decent example of closures. Closures are a powerful language construct that, judiciously used, can be of great benefit when they are available. A good discussion of closers in javascript can be found here, and upon reading it there is an example that comes close to what I have done above, but the example is generic enough that it does not show the use of variables that are in scope.

Simple utility for getting data from CSV files

2009-11-02T12:21:00.000-08:00

Most programmers at some point spend a lot of time hacking apart CSV files. I recently got to yank apart yet another csv and while Excel is handy for these sorts of tasks, it can't do everything quickly, so I hacked up a little utility using php5 and sqlite to help query data and get results fast.

Here is csvsql

You just upload the file, first row is headers and you can type in the filter part of a where clause to narrow the data. click a button and put out a CSV representation of the data that should be excel friendly.

A Post After My Own Heart

2009-09-24T04:51:00.000-07:00

Joel writes about what he calls Duct Tape Programmers. Best thing he's written in a while, if you ask me. Not that you did, but it is definitely worth a read. I'm also planning on picking up Coders At Work, which he mentions in the article. Sounds like a great read.

"Anonymous" data

2009-09-08T14:46:00.001-07:00

Just one more reason to be very careful about what information you put on the internet. Most people think gender, zip code and birthdate won't give much away. Most people are wrong.

This is just plain cool

2009-08-11T09:38:00.000-07:00

When I was at U of M my software engineering professor would talk about how small storage and computers were becoming and how we won't be needing these big boxes anymore.

I saw this on hacker news and it just looks cool. With a bunch of SheevaPlugs you can turn a powerstrip or 2 into a server farm :-)

Rails Stuff

2009-08-09T22:18:00.000-07:00

Finally been hacking on some code this weekend for my rails project. I want to have people login with user credentials and a couple of resources on using restful_authentication that I found EXTREMELY useful are here:

A great demo
http://media.railscasts.com/videos/067_restful_authentication.mov

and

if you get an error when trying to login

http://blog.anthonychaves.net/tag/rails

Also the controller for my default class had this added to it so we don't get into an infinte loop when trying to login for the first time

skip_before_filter :login_required

I also added this on the user controller so we can go create a new user without having to login. Duh!

I'm using Rails for .NET Developers from The Pragmattic Programmers to guide me along with lots of online docs. I'm also learning Ruby as I go. It seems to share with python the same notion of 'what you expect will work'. Pretty much I've been typing things and they are working, without too many tweaks. It's kind of nice. I'm doing super simple run of the mill stuff, so I'm not expecting too many things to blow up as of yet.

Best Quote Talking about the Apple App store ever

2009-08-04T21:00:00.000-07:00

"Apple requires you to be 17 years or older to purchase a censored dictionary that omits half the words Steve Jobs uses every day."

This little gem is from what appears to be an increasingly common scenario of rejections leading to rants against the App Store. I'm sure the Apple Faithful don't mind (I'm sure the faithful would be happy to watch Steve give their spouses/partners a rim job and then split them open, bathe in their entrails, cook them and then eat them), but for the average developer, it would seem that a lot of Apple's coolness is wearing very, very thin.

Here's the full post about Ninjawords and their debacle.

Sofware Engineering and Metrics

2009-07-17T10:36:00.000-07:00

Tom DeMarco (Yes, THAT Tom DeMarco) wrote an interesting piece on his current view of Software Engineering. Definitely worth the read.

Clouds can be Dangerous

2009-07-15T08:47:00.000-07:00

Agree or not with TechCrunch's decision to publish some data from some Twitter documents they received, I think the most important thing to note is what they say about using gmail and other cloud services

"It’s not our fault that Google has a ridiculously easy way to get access to accounts via their password recovery question. It’s not our fault that Twitter stored all of these documents and sensitive information in the cloud and had easy-to-guess passwords and recovery questions. We’ve been sitting in the office for eight hours now debating what the right thing to do is in this situation. We’ve spoken with our lawyers. We’ve spoken with Twitter. And we’ve heard what our readers have to say. All of that factors in to our decision on what to post or not to post."

I have been wondering how many people will need to get burned and to what degree before they start taking this sort of thing seriously. Given that Google's entire business model is selling targeted advertising, they have an incentive to collect as much data about you as possible. You would think for this reason alone people would be wary of dumping too much stuff into Google's hands. The annoyance factor would get to be outrageous, I would think. That's not even considering that Google has to be a HUGE target for any sort of cracker that wants to track down any kind of information. I'm sure they do their very best to keep everything as locked down as possible, but it's really hard to compromise the information if it isn't there to be compromised in the first place.

The current and future state of programming

2009-07-10T13:18:00.001-07:00

An interesting read from Philip Greenspun. Apparently some students were able to find their admission status by modifying a URL. This, apparently, qualifies as "hacking". Ugh.

I am particularly fond of the sentence "As progressively dumber programmers build progressively more complex systems we will see more of this kind of attempt to paper over coding mistakes with lawyers, sanctions, policies, and laws." I know people that have been lamenting this sort of thing for years. To hear it put so clearly is refreshing. Not that anybody will really care. Until their credit cards and bank accounts get hacked, that is. But by then it will be too late. And they'll care for about 20 minutes and then get back on with their lives. *sigh*

NIH Syndrome

2009-07-09T06:28:00.000-07:00

Most people in software are at least aware of Not Invented Here syndrome. We're in the middle of fighting with it right now, on the other side where we have some dependencies that are causing issues because we rely on some libraries for a very important piece of our software and the library has upgraded and if we use the upgrade we are, for all practical purposes, going to have to do a rewrite. So I was going to write this big long article in defense of NIH and why it makes sense to roll your own sometimes, but then I remembered that Joel on Software had tackled this already.

I reread the piece and it is basically saying the exact same thing I was going to say, so here it is. I can't say that I would go to Joel's extreme of "If it's a core business function -- do it yourself, no matter what.", but you need to seriously think about the cost of upgrading dependencies if the dependency is a core piece of what you are doing. It's not a matter of IF it will bite you in the ass, but WHEN and how hard.

Reviewing files from Subversion

2009-06-19T14:28:00.000-07:00

Recently I had to deploy some updates to a website and wanted to get a list of all the files that had been updated to make sure I got everything. With recent versions of subversion and using python this turned out to be rather simple:

First, get a list of all the files that have been checked in and stick them in a text file.

C:\Utils\svn-win32-1.4.3\bin>svn diff --summarize -r902:966 svn://myServer/myRepository/myProject/trunk > c:\diff.txt

Then use the following lines of python to get the file names and sort them to make tracking them down easy

mylines = open("c:\\diff.txt" , "r").readlines()
myarr = [x.split(" ")[-1].strip() for x in mylines]
myarr.sort()

Now print out myarr, put it in its own file or do whatever you want with it to deploy or review your updates.

False Productivity

2009-06-18T07:34:00.001-07:00

I'm finishing up a project for a buddy of mine that is in php. It's an old school simple CRUD app and as such I didn't go too overboard with the functionality. It does what the client asked and the code is reasonably clean but not something that you could refer to as having much of an architecture.

Since it is all raw HTML and PHP I had to code everything up by hand. At my regular job we do almost everything in ASP.NET. This got me to thinking as I was working on these two things side by side that, after roughly clocking the hours of similar projects, that a lot of the productivity I've been feeling I've been getting from ASP.NET is false. As one of my co-workers is fond of saying, the complexity has to exist somewhere, all we really do is move it around. This struck me keenly because my layout and design skills are not great, so even coding up a simple interface that does what it is supposed to takes a little bit of effort and since I'm not using any frameworks at all for the PHP app, I got to code it all by hand. In the APS.NET world you would drag and drop a couple controls and be done. Poof!

But, then, the data access layer coding began. And the DAL for the PHP app was dead simple to code. And it wsa doing some reasonably interesting things. Nothing crazy, but a couple joins here and there, you're usual stuff. In ASP.NET, once you go outside their little box, things become somewhat painful. I've had instance where putting together the interface was drag and drop a few things and then the prototype of the data access was a few minutes of drag and drop but then things start happening. The production data has millions of rows and your test data only has a few thousand, so the paging and access for the paging needs to be recoded. Oh, wait, there's a fancy new datagrid widget doo-hickey that your manager read about somewhere. Can we use that? And on and on with the little things that make a seemingly quick and simple job time consuming. Just like that the productivity you thought you had gained went up in smoke. In some cases it actually gets worse because you either have to start over or make what your additions fit into what you've written already AND make it fit into the ASP.NET universe.

I guess the point here is that things aren't always as time saving as they appear. I know people say a lot of things. My language is X more productive than yours. You get demos from vendors showing how easy something is. Three drags, a double click and a pinch of salt and you have a fully functioning app! Just be suspicious of such claims. Chances are the three drag demo has little resemblance to the actual work you do.

Well, I'm drinking the Kool Aid

2009-05-29T10:18:00.000-07:00

My brother in law and I had an idea for a project and from what he and many others have said, it would be a perfect Rails project, so I downloaded Rails and got a copy of Rails for .NET Developers and I started reading last night. It looks like it should do the trick. This is a pretty simple application, so it seems like a great fit for what Rails is supposed to be good for. Having seen some of the things people put together for Rails Rumble, I'm hoping I can bang this thing out fairly quickly.

Anyway, given I've done my share of python, C++, C# and VB with a smattering of Javascript and PHP and vague memories of lisp and a bitter taste in my mouth for perl, I'll see how this works out.

Thank you Microsoft

2009-05-10T09:09:00.000-07:00

Look, a lot of us in this industry bash Microsoft and in many cases rightfully so. But this weekend I was doing some work for a friend who has a client that has a site running IIS, SQL Server and PHP. I did the prototype of what I'm working on in PHP and mySQL on my laptop and then had to port the database layer over to SQL Server. This app is far from rocket science, so I wasn't worried in the least about doing the first version against mySQL. So I downloaded SQL Server, installed it on my laptop after hitting many Next> buttons, updated my code to use the standard mssql module that comes with PHP and then started testing. That's when the problems started. I did the usual googling and came up with somethings I had forgotten to do on the install, like enable TCP/IP in the Configuration Manager and other assorted bookkeeping stuff. But after that and getting other clients to connect successfully, the PHP app kept having issues connecting, never mind doing any actual work. I had done a ton of research, tried everything I found and then some, and finally gave up trying to use the standard PHP module.

Since in my regular job I work almost exclusively with Microsoft technology, I am mostly up to date on what they do. I know as of late they've been doing some work to help support a lot of web and open source technology out there. So I was wondering if maybe, just maybe, then had a PHP client. They have a database client for java, after all. So I figured what the hell and found the SQL Server driver for PHP 1.0. After downloading the file, I think it was just over an hour from extracting the files to the PHP extensions directory until I had a working version on SQL Server. I've had somewhat similar experiences lately with flickr and facebook tools found on codeplex.

The one thing that made this very painless is everything I needed to get this installed and configured was all in one place. The help file was actually helpful and told me what I needed to know to get things going. I didn't have ot hit half a dozen websites in order to figure out I needed to have the SQL Server 2005 native client driver installed. It just told me. Granted it was annoying to have to uninstall the 2008 driver and download and install the 2005 driver, but given a co-workers recent experiences with setting up ubercart, this was completely painless. He spent HOURS going from site to site, getting modules, downloading updates, installing dependencies when the modules he installed didn't come with everything he needed, etc. I'm glad it wasn't me. I probably would have given up. Personally I find this the most frustrating part about the open source community. I don't understand how projects like this gain the traction they do. They're hard to set up and hard to keep updated because in many ways these things are like a house of cards. I use some open source tools (python being by far my favorite) but, by and large, I find the lack of good documentation (by good I mean useful. I've found a LOT of documentation on most things, but generally it isn't worth the electrons spent rendering them on my screen) to be a HUGE hurdle to overcome. There is no reason I should have to spend hours and hours searching for documentation that still doesn't fix the problem. In general the open source community doesn't understand why people don't use their products more. How can they keep flocking to "Micro$oft Windoze" and their other schlock when there are cheaper and superior alternatives available???? Well, I think documentation is about 70% of the answer to that question. And the usual response is that open source projects are, in a lot of cases, done by people for free in their spare time and, let's face it, I do not know a single developer that likes writing documentation that is for end users. Well, if you don't like doing that, people aren't going to be able to use your product, no matter how superior it might be in a technological sense.

So I wanted to take a minute to say thank you to the people at Microsoft for putting these types of tools together and for the community that surrounds CodePlex and other similar sites. As a company that is generally viewed as closed and being competitive to a fault, it is nice to see that this is pretty much a caricature of the organization. They obviously have their issues, but I think they are really beginning to realize that if they are ultimately going to not only survive, but thrive, that they need to embrace and support a lot of the other good work going on out there. There are, of course, plenty of selfish reasons to do this, but there are just as many reasons not to do this and it is probably easier not to.

In general I think this is starting to be a return to what made Microsoft the dominant software company on the planet. When office and Windows were first born, they had a LOT of competition and, as others have pointed out, one of the things that made Microsoft software good back in the day is that they went to great lengths to interoperate with other software out there. Excel worked well with Lotus files and Word was able to read and write WordPerfect files, for example. Not to mention DOS and Windows being able to run on a wide assortment of machines. Maybe not well all the time, but well enough.

Case insensitive validators in .NET

2009-04-24T12:12:00.000-07:00

I have just done some work on a control that needed to have some validation for dynamically generated text boxes. I did some poking around for adding validators dynamically that would be client side and would check for words by being case insensitive. I found a few solutions to this that involved server side validation and I wanted to keep it on the client. The ASP.NET validators use javascript on the client side and there isn't anything I could find that would easily let me do a case insensitive compare on the client side using the .NET controls. For example, I couldn't pass /sometext/i to the validator and have the 'i' attribute recognized.

A solution hit me on the way home. I was checking for the common occurences of 'true' and 'false', 'True|False|true|false'. Which was probably going to be fine but every now and then you hold the shift a second too long or you have caps lock down and you enter TRUE or FAlse and those wouldn't work out and while they would evaluate to a boolean once they hit the server, they would never get there because of the validation.

So now my expression looks like '[Tt][Rr][Uu][Ee]|[Ff][Aa][Ll][Ss][Ee]'. that does the trick very nicely. It isn't great to read, but this is a very specific case and I've certainly seen worse. And I don't need to do a mix of server and client side validation now, which makes me happy. I wouldn't do this for something that was involved, but for this particular case it is simple to implement, not totally unreadable and prevents a useless round trip to the server.

11, 58, 29, 16, 33, 21, 24, 29, ...

2009-03-11T12:30:00.000-07:00

These are the minutes remaining on a large file copy (6+ GB) in Windows. Why bother giving the estimate?

That's today's rant. Back to waiting for the file copy to finish. 8 minutes to go. Maybe...

8 steps to millions?

2009-02-23T13:45:00.000-08:00

Doesn't this sound easy?

http://www.guardian.co.uk/technology/gamesblog/2009/feb/10/gameculture-apple

I wonder how many people started cranking out yet another game after reading this. I'm half tempted.

Agile

2009-01-04T19:24:00.000-08:00

Generally speaking, I'm not a big fan of Agile Development for two reasons.

1. All the hype.

2. A lot of contradictory information and advice, sometimes from the same person. One time you hear that you have to adopt all the steps of a given methodology. Another time you hear that you can't be brain dead about adopting the steps, but need to pick and choose what works for you. A big selling point of Agile is little documentation, given that as a rule developers hate writing documentation. But then this gets taken to its illogical conclusion and many of the Agile work I've seen has zero design and planning. People just start coding and think the design will magically assert itself at some point. In fact, for a trade that generally prides itself on being logical, there is a lot, in practice, that I see illogical about adoption of Agile methodologies.

That being said, I agree with the position of most items on the Agile Manifesto. It's just in practice I haven't seen it work too well often. Even the reported successes are usually qualified and, depending on the stakeholder you ask, the success really isn't a success.

To that end, I think this latest entry on Artima is a good read and crystallizes a few of the issues that lead to this state of things. http://www.artima.com/weblogs/viewpost.jsp?thread=246513

Marketing to developers

2008-12-11T11:40:00.000-08:00

I just got a piece of marketing schlock in my inbox today from Salesforce.com

Code Warriors: Have We Got a Challenge for You!

By now you've heard the buzz and excitement about Force.com Sites, and may even have attended our recent Sites Developer Preview Webinar. Now be among the first to get hands-on with this ground-breaking technology:

Enter the Force.com Sites Developer Challenge—and broadcast your stellar coding skills to the world! We'll reward you with hero status on Force.com developer Web sites and blogs, plus a terrific prize.

<and it goes on and on and on....>

Stuff like this drives me, and most good developers I know, crazy.

Look, the majority of developers I know are not warriors. They couldn't fight their way out of a wet paper bag. Like King Arthur's nights, a vicious rabbit could decimate most of them.

There is no buzz and excitement from anybody I personally know about Force.com Sites. First I heard of anything resembling buzz and excitement is in the email that came from Force.com.

Most people with stellar coding skills are working on operating system kernels, 3d rendering images, games, bio informatics, genetic algorithms, language compilers or interpreters, etc. They most certainly aren't working on web apps that mainly consist of gluing API calls together. Product catalogs and surveys don't exactly stretch the grey matter.

I'm guessing the people that buy into this stuff are probably the development managers that usually receive this stuff and, well, since salesforce.com might be part of some strategic direction, the developers get this email forwarded to them and they are "encouraged" to enter. I got this in my inbox because I work at a small company and, for good or ill, I get to deal with most of this stuff directly. The only reason I even opened it was because I thought it might make good blog fodder. Most of this kind of stuff is insulting to the intelligence.

Code Warrior. Sheesh. Now where did I leave my swords...

working with utf16 in python

2008-12-09T12:57:00.000-08:00

I got a file from a client that was exported from SQL Server and was encoded as utf16. I needed to do some work on it. I had to google around a bit to find some help on handling the gobble-de-gook that I was seeing

>>> f = open("f:\\contact.txt","r")
>>> l = f.readline()
>>> l
'\xff\xfe6\x003\x003\x00D\x003\x00A\x008\.....\n'

Here is how to do it

>>> import codecs
>>> f = codecs.open("f:\\contact.txt", "r", "utf16")
>>> l = f.readline()
>>> l
u'633D3A84-3870-4A93-9755-000215260850,8568,NULL,Scooby,Shaggy,NULL,,NULL,mymail@address.com,1902-06-01 00:00:00.000,NULL\r\n'
>>>

eXtreme Annoyance

2008-11-20T12:10:00.000-08:00

It is hard for me to get eXcited about eXtreme spellings. Most of these ideas aren't eXceptional or eXquisite in any eXplainable way. A lot of these eXtreme methods lack eXactness, clear eXplanations and fall apart under close eXamination. Even the Success Stories for eXtreme Programming are less than eXceptional.

If this is the new trend, I quit!

2008-11-18T08:40:00.001-08:00

http://discuss.joelonsoftware.com/default.asp?joel.3.700535.8

Roof over your head for a part time job. Seems creepy.

FINALLY

2008-10-24T10:31:00.000-07:00

Some new installments on regularexpressions. That has been one of my favorite reads on the internet for actual practical information on using scripting languages and things outside the usual enterprise application java/.NET/LAMP world that so many of us inhabit.

Glad it's back with new material!!

Opening explorer shells

2008-10-17T14:24:00.000-07:00

I'm often in the command line on a windows box and need to open an explorer shell for a variety of reasons. You can simply type explorer . (don't forget the dot) at the prompt and a window will open that points at the current directory.

I got tired of typing that much and created a file called ex.bat that I dropped in my windows directory. You can put it anywhere in your path. Here is the contents of the batch file

@echo off
explorer .

Just copy the above, stick it in a text file and save it to c:\windows\ex.bat. Then typing ex at the command prompt will do the same as explorer .

New spot for quick updates

2008-10-17T08:55:00.000-07:00

Having spent time writing full length articles on janitor programmer, I figured I needed a spot to just put quick updates of things I've stumbled across and figured out. Rather than write my own or install wordpress or one of a hundred other things, I figured I may as well take advantage of blogger.