Finally Starting To Pick Up And Go

My current employer offers Professional Development as one of the benefits it offers its employees. It's not quite the old Google 20% time, but it's pretty close. So I finally have a little time to spend on the Go language, which I've been wanting to pick up for a while now. I'll blog here from time to time related to what I find. The first thing I'll mention is that I've always worked at MS shops, and this job is no different. All of the Go tutorials and introductions worth a damn are on MacOS or Linux, so here are a few things to be aware of when first picking up the language and running the hello world program

Getting Go

Grab the MSI for the latest stable version here https://golang.org/dl/

The Official Getting Started Guide

https://golang.org/doc/install

Setting GOPATH

The guide uses the export command. This doesn't exist on windows. On windows, open up a command prompt and create a directory for your Go workspaces. I called mine c:\mine\golang\workspace. Then at the command prompt run

c:\mine\golang\workspace>set GOPATH=c:\mine\golang\workspace

Note that just setting this in the command prompt will mean this variable is only available in the current instance. If you were to open up another command prompt and use the echo command to view the variable (echo %GOPATH%) you would not see the value you just set. If you are going to make this your permanent workspace directory you should set the GOPATH in your environment variables under System Properties.

How To Write Go Code

https://golang.org/doc/code.html

Under your newly created workspace directory add your pkg, bin and hello directories. The Writing Code guide and the linked screencast talk about adding a path underneath linked to your github account. I didn't bother with that. For the hello world program I simply created a hello directory under src. So after writing the hello world program in src\hello\hello.go, I was able to run the following

c:\mine\golang\workspace>go install hello

This writes hello.exe to c:\mine\golang\workspace\bin\hello.exe, which you can run as instructed in the tutorial

After doing this I have the following structure in my folder.
C:.
├───bin
│       hello.exe
│
├───pkg
└───src
    └───hello
            hello.go

Hopefully this is enough, combined with the existing online material, to get you writing programs in Go on windows. These may seem like trivial little things, but the idea of workspaces and using directory structures like this will seem foreign to people using nothing but Visual Studio for their development work, which hides a lot of this underlying work from you.

Saving and Sharing Web Page Performance Information from Chrome

I was perusing a web site during lunch and once again lamenting its performance. I fired up the devtools for chrome and the Network tab does a good job of showing you the sticking points in the app and I was wondering if there is a way to share this sort of information. It turns out there is. The data from the Network tab can be saved in a format called HAR. HAR is a JSON format that contains all of the data you see in the network tab in devtools.

There appear to be quite a few tools available for allowing you to view HAR data. The most accessible one I found was here http://ericduran.github.io/chromeHAR/

A good summary of HAR files is here

http://www.speedawarenessmonth.com/what-is-a-har-file-and-what-do-i-use-it-for/

If you want to save data from the network tab in a HAR file in Chrome devtools, right click in the log of network traffic and select Save As HAR With Content. This file can then be dropped onto the above HAR viewer page and you can see the output, including the timeline.

Cleaning up Disk Space on Your Machine

One recurring annoyance with working on lots of different projects is that over time your disk just gets full. Working in many MS SQL Server databases over the years I've noticed that there is a pretty simple way I can usually reclaim a lot of diskspace from SQL Server on my machine. I currently have over 100 development databases on my machine and a lot of those were one time uses. I'm sure I could remove a bunch of them, but in a pinch, this allowed me to very quickly reclaim over 30GB of disk space

Dynamically generate the dbcc shrink database syntax for all of the databases on my box

select 'dbcc shrinkdatabase([' + name + '], TRUNCATEONLY);' from sys.databases where name <> 'master'

copy the output into a sql server query window and run it

I had issues with one database because it was restoring. After removing that database from the list of queries, it ran rather quickly and I got a gobs of disk space back. If you have databases on your machine that you do not want to shrink, you can remove those from the list

For more information on SHRINKDATABASE

http://msdn.microsoft.com/en-us/library/ms190488.aspx

Update

You can use the following to generate a dynamic sql statement and execute it using EXEC

DECLARE @SQL VARCHAR(MAX)

select @SQL = COALESCE(@SQL,'') + '
dbcc shrinkdatabase([' + name + '], TRUNCATEONLY);' from sys.databases
where name not in ( 'master', 'model', 'tempdb', 'msdb')

print cast(@SQL as ntext)

EXEC(@SQL)

Fun and insightful read

The Codeless Code is a fun read. Also it looks like it has been waaaaay too long since I've posted anything. I've been at a new job since Oct 2011, and am working mainly in javascript these days. Quite a change from C# in a lot of ways, and most of them good as far as I'm concerned, with the exception of issues I seem to continually have with supporting libraries we use. That and my CSS knowledge is somewhere between none and enough to be dangerous. I need to work on that.

webrat needs to be bundled as well

In addition to getting rspec installed correctly, webrat is required to test the page content with response.should have_selector("title"....

Here are the instructions. I had to include the proper version (0.7.0) in the Gemfile, then do a bundle install.

getting rspec installed

The tutorial I'm using calls for using rspec for testinhttp://www.blogger.com/img/blank.gifg. In order to get it so I could install it and add it I had to follow these instructions.

Finally really working on Rails

After several false starts we have a project at work where we are looking at using Rails. So far so good. I'm using http://ruby.railstutorial.org/ruby-on-railshttp://www.blogger.com/img/blank.gif-tutorial-book as my tutorial and so far it is my kind of book. I'm using the bitnami rubystack virtual machine using Rails 3 in ubuntu. There have been a couple of hiccups along the way. 1) is remembering to use sudo on the VM for most installs and updates and even running the server to test the appliction. Not a HUGE deal, but a minor annoyance. Also, when doing the push the first time to deploy to heroku, I got an error saying that heroku does not appear to be a git repository. The fix for that is here.

Math and web design

An interesting article about using prime numbers and tiling backgrounds for creating interesting effects with a minimal cost.

http://designfestival.com/the-cicada-principle-and-why-it-matters-to-web-designers/?utm_medium=email&utm_campaign=Design+View+83+-+Cicadas+Pseudo-random+Organic+Tiling+and+Top+Design+Mistakes+HTML&utm_content=Design+View+83+-+Cicadas+Pseudo-random+Organic+Tiling+and+Top+Design+Mistakes+HTML+CID_068c8ba45a4520e40c5d7eb008dc7279&utm_source=Newsletter&utm_term=The+Cicada+Principle+and+Why+It+Matters+to+Web+Designers

useful erlang posts

I'm starting to learn about erlang. here are some useful posts I found.

http://yarivsblog.blogspot.com/2008/05/erlang-vs-scala.html

http://ferd.ca/an-open-letter-to-the-erlang-beginner-or-onlooker.html

There Are No Internal Users

Anybody who has worked for a small company or in an IT department where most of your users are 20 feet down the hall has heard the following; "Don't worry about creating an admin interface, this application is just for internal users."

The problem with internal users is that they don't really exist in the sense that "decision makers" would like them to. There are people that use your application and people that do not use your application. That said people are separated by a hallway instead of an interstate does not make them particularly special. When you are asked to design for internal users you are usually being asked to skimp on these features:

• Administrative interfaces.


• Ease of use enhancements.


• Documentation.

Please, please, PLEASE avoid the temptation to cut drastically on the above features. They don't have to have the spit and polish you would give to a client facing application (a misspelled word here or there won't kill anybody) but these features should exist, they should make sense, and they should be useable without people having to email you every 3 months asking if you can make one little change here or there.

Without an administrative interface, you will need to remember what all the tables and files in your application are responsible for. You will need to completely memorize the internal structure of the application, or you will need to wade through piles of code every time somebody has a request. Neither situation is good. Those brain cells are better spent remembering good places to eat lunch, and the time you save by not designing a proper administrative interface will be spent every time somebody has a question about making a configuration change. Either spend a little time and effort now, or a lot later. Multiple times. Until you either get fed up and kill somebody, quit, or write the administrative interface anyway.

Anything you put in front of users should be intuitive. Granted in most cases internal users, or users in any niche market, can be expected to have a level of sophisitication that is above average, but if interfaces don't make sense, the application isn't serving its purpose. Computers and applications are supposed to be tools that make people's lives easier, not puzzles. At least, not in this case.

Documentation is critical. Even if it is nothing more than an email briefly explaining the application's feature set and how to use the main features. You don't need a complete manual detailing why you chose the lovely magenta headers, but you should have something that users, and you, can refer to that explains what the application does and how it does it. For this sort of documenation I prefer a wiki. It can be a pain to go update the documentation every time something changes, but you will be kicking yourself when, 8 months after deployment, somebody comes down the hall with a question that you don't remember the answer to. Instead of taking 15 minutes to skim through some docs, you will be taking hours going through an old code base you barely remember.

I've found this particularly true for instances where I am writing small one-off applications, excel macros, python scripts, etc. that are written to solve a particular issue. There are stretches where I crank out 3 or 4 of these a week, and then many months later somebody asks me for a program that performs some function I've already taken care of, or they ask about using something they haven't touched in a while and they've deleted the email that contains the directions. If you write many small utilities, at some point you will start losing track of what they all do. Simple to use interfaces with clear documentation will save your users from some angst and will save you from wasting time either going through old code or, worse, reinventing your own wheel.

The problem with Javascript

Let me open this by saying that I like JavaScript. The syntax is comfortable for a lot of people and it lets you do all kinds of funky stuff because functions are first class objects and prototype based classes. It also enjoys an extremely wide install base because it is in almost every browser people use. I believe that it has a bad reputation mostly because it has suffered from years of poor and differing implementations. I also think it bears the brunt of having a variety of DOMs (These days IE vs. everybody else) associated with it; much the same way that I think C++ has a bad reputation because a lot of people's experience with C++ began and ended with MFC.

But there is one problem JavaScript has that was clarified to me after reading this critique of Google Closure. The problem, in a nutshell, is that the developer has to act as an optimizing compiler in many instances. Looking at most of the critiques, they revolve around optimizations that most people in the 21st century are either accustomed to having taken care of for them, or their computing power is so great compared to what they do that they don't care. At this point an interpreted language in a browser is handicapped in both of these areas.

Looking at the list of critiques, slow loops because you access a property on every loop check, slow case statements and dealing with multiple types of strings are all things that most developers don't have to worry about anymore. The problems with the code in the library and more efficient ways to do the same thing cover a large part of the post. That you can write many paragraphs on this sort of thing and that these optimizations are necessary is indicative that implementations of the language still has plenty of room for improvement. The concatenation of "" to a value to make it a string more efficiently than using String() is, to me, the most egregious example of this. (somevalue + "").replace() may be fast, but it sure is ugly to look at.

The author at one point even takes a jab at Google, saying their time may have been better spent just writing proper JavaScript instead of making the investment in making Chrome's JavaScript performance better. I'm guessing this is tongue in cheek, wry humor. Otherwise the implication is that JavaScript is fine on the performance front. As long as the programmer has to perform the sorts of optimization hacks described in the above post, that is simply not the case.

random subsets of data ii

In doing more work to get random subsets, the previous solution fails miserably if you are going to be using a view. You can run the EXEC statement in a stored procedure but cannot use it in a view. And furthermore, calling a stored procedure from a view is somewhere between extremely problematic and impossible, depending on who you talk to. Using the RANK function and partitioning the data you can get a similar result.

Let us use the same idea as the previous example and assume we want at least one employee from every department. The only requirement for the following query is that you are asking for more employees than you have departments.

select top 100 T1.* from

(select 

RANK() over 
(PARTITION by Department order by newid()) as r,  

* 
from Employees
where Active = 1

) as T1
order by t1.r, t1.Department

Note, as before, that this is a sql server query. Modifications may need to be made for different flavors of SQL. What this is doing is for every department the employees are getting randomly ranked thanks to order by NEWID(). By selecting the top 100 and ordering by rank and then department, you'll get all of the 1's from every department, then all of the 2's and so on until you get 100.

The order by department is strictly unnecessary. I did it to make viewing and verifying the results easier.

random subsets of data

I recently had a request to return a random subset of contacts from a database, but to be sure that there were some contacts from each group represented. Let us say that you have a database with employees and you want to get a list of some random people from each department. Here is how I went about doing it:

declare @s1 varchar(max)

select @s1 = coalesce(@s1 + ' union all ', '') +
e from (

select distinct
'select * from (select top 11 percent * from 
Employees where Department = '''
+ Department + ''' order by newid()) 
as [t' + Department + ']'  as e
from Employees) T1
exec (@s1)

What we are doing here is generating the text for a query that will select the top N contacts from the Employee list for each department. The coalesce function will put all of these queries together into one query that will union the results into a single table. The exec function will execute the query.

Note that this is a TSQL example to run on SQL Server. You may have to translate this, depending on your database system.

If you want a fixed number then what I would do is tweak the percent to get a number than is slightly more than the number desired then just select the top N from that. This runs the risk of not getting somebody from every department. Another method would be to select a number slightly less than your target, then select some random number of records to add to the result. This is more complex, but may be what you need.

Returning data tables from Web Services in .NET

This is just a small reminder, and I don't know why it gets me every time, but it does. I guess I return data tables from web services infrequently enough where I don't think about it. Maybe writing it down somewhere will help me remember. When creating a data table to return from a web service, give the table a name.


DataTable dt = new DataTable("nameMyTable");


If you don't, the first time you hit the service when testing you will get an exception about the table not having a name. And then you will curse to yourself and then give it a name and move on with your life.

scripts for processing stuff

Often I need to write something up quick to process a file, or process files in a directory and I'll want a GUI for these things so I can pass them along to other people to use. Here are two python scripts that I use as templates for picking either a file or a directory and then doing something with it. The object here isn't to have everything "correct" according to the latest design fashion or methodology. With this script I can import a python module that takes a file or directory name and does the processing I want, plug in the function name, pass the parameter and then move on with my life.

Pick a file and do something with it:

from tkinter import *
from tkinter.messagebox import *
from tkinter.filedialog import *

class FileProcessor(Frame):

# Object constructor
    def __init__(self, parent=None):
        Frame.__init__(self, parent)

        self.txtFile = Entry(parent)
        self.txtFile.place(x=6,y=64,width=375,height=24)
        self.txtFile.insert(0,'')

        self.btnPickFile = Button(parent,text='Pick File', command=self.btnPickFileClick)
        self.btnPickFile.place(x=6,y=20,width=96,height=24)

        self.btnRun = Button(parent,text='RUN', command=self.btnRunClick)
        self.btnRun.place(x=6,y=114,width=96,height=30)


# Methods (event handlers) of object
    def btnPickFileClick(self):
        print (self.txtFile.get())
        self.txtFile.delete(0,END)
        self.txtFile.insert(0,askopenfilename())

    def btnRunClick(self):
        fileName = self.txtFile.get()
        self.runFile(fileName)

    def runFile(self, fileName):
        print('code goes here to run when click: ' + fileName)
        showinfo('The file is',fileName)

# Method called if script is run directly
# instead of imported or used as a class
if __name__ == '__main__':
    root = Tk()
    root.title('Process File')
    myForm = FileProcessor(root)
    myForm.pack()
    root.geometry("423x156")
    root.minsize(423,156)
    root.maxsize(423,156)
    root.mainloop()

Pick a directory and do something wit it:

from tkinter import *
from tkinter.messagebox import *
from tkinter.filedialog import *

class DirectoryProcessor(Frame):

# Object constructor
    def __init__(self, parent=None):
        Frame.__init__(self, parent)

        self.txtDir = Entry(parent)
        self.txtDir.place(x=6,y=64,width=375,height=24)
        self.txtDir.insert(0,'')

        self.btnPickFile = Button(parent,text='Pick Directory', command=self.btnPickFileClick)
        self.btnPickFile.place(x=6,y=20,width=96,height=24)

        self.btnRun = Button(parent,text='RUN', command=self.btnRunClick)
        self.btnRun.place(x=6,y=114,width=96,height=30)


# Methods (event handlers) of object
    def btnPickFileClick(self):
        print (self.txtDir.get())
        self.txtDir.delete(0,END)
        self.txtDir.insert(0,askdirectory())

    def btnRunClick(self):
        dirName = self.txtDir.get()
        self.runDir(dirName)

    def runDir(self, directoryName):
        print('code goes here to run when click: ' + directoryName)
        showinfo('The directory is',directoryName)

# Method called if script is run directly
# instead of imported or used as a class
if __name__ == '__main__':
    root = Tk()
    root.title('Process Directory')
    myForm = DirectoryProcessor(root)
    myForm.pack()
    root.geometry("423x156")
    root.minsize(423,156)
    root.maxsize(423,156)
    root.mainloop()

Putting a string on a single line

Often times text is written on multiple lines for the sake of clarity. Sometimes these lines need to be consolidated to a single line. This happens to me a lot with SQL queries that end up needing to be in a config file somewhere. I'll write them out all pretty so they make sense (as much sense as some SQL can make...) and then I'll need to consolidate that down to one line. Here is a python one liner that does just that, given a string named mystr

" ".join([x.strip() for x in mystr.split("\n")])

Don't make me slap you, Beavis

Every single time somebody asks "how hard can it be?" or "why is the estimate so big?", I want to smack them. Hard. Some folks at 37signals were kind enough to offer an example that answers these sorts of questions when asked about a seemingly simple feature.

I applaud the fine folks at 37signals for publishing things like this.

splitting csv files

I'm sure I've ranted about this before. I'll rant about this again. I just hate seeing examples from people in C# where they say that you just split a CSV file by using mystring.Split(',') and you will get an array where each item is the value for each field. One would assume, since they mention C#, that at some point in their lives they have worked with excel style CSV files that have data like

"Bob","Hello, Dear","""Dude, where's your car"""

The string split method obviously will not correctly handle this case at all. Here is a page I found with a nice little function that works for me in most cases.

http://www.tedspence.com/index.php?entry=entry070604-124237

Hopefully this will work for you, too.

I leave it as an exercise to the reader...

When did this become a euphemism for "this is hard"?

I say this because I'm doing research on Visual Studio 2010 and one of the selling points is the Parallel Extensions for .NET. In a great many of the articles and tutorials I've found, the author goes through some totally brain dead examples that have no relevance to real life to show the basics. This is then followed by a discussion of some of the more interesting parts of the framework and then a paragraph or two about things you might try and finally something like "I leave it as an exercise for the reader to...." followed by something that might actually be interesting.

This is one of the reasons, by the way, that I prefer actual books to cobbling together a manual from various web pages. I know a lot of people say they will never by another programming book again because you can find everything you need on the web. I have not found this to be the case. And even when it is the case in a lot of instances one has to go through a lot of effort to cull the good nuggets from the pile of information you find. It's almost like panning for gold. Most good books do this for you already and the really good ones even include some decent examples of using libraries and techniques that have some relevance to real life.

I just needed to vent. Back to my research.

Working with outlook

I found a nice piece of software called Outlook Redemption. I had to process some msg files that I had exported and this thing came in handy and was super simple to use. For what I needed, this was enough to get going.

From time to time I'm still amazed at the software that is freely available to just get things done and move on with your life.