Almost two years ago, I decided I was going to sit down and make some changes to my life. I had just finished a four year stint at microsoft, felt burnt out on tech, gotten out of shape due to working regular sixty hour weeks in front of a computer and smoking like a chimney. I sat down and thought about what it was that I wanted to achieve and came up with a list of stuff that I wanted to do. I decided to focus on accomplishing three -smoking, weight loss, and starting a company. I did several things to accomplish these goals, but one of the best, and easiest, was just changing my home page.
Nowadays, when I open up my browser in the morning, I’m greated with three pages. They are as follows: (my company) entrenza.com’s beta page, gmail, and foremost – a google docs spreadsheet entitled yyyy-mm goals and actions. The beta page is related to one of these goals, but lets talk about the spreadsheet. It’s really simple and looks something like this:

It’s been a pretty successful system. I don’t worry about huge strides, I just worry about accomplishing, one thing towards each goal each day. So far, ‘ve registered a LLC and and have regularly worked part time on a startup for over a year. We’re right about to go into beta. As for getting in shape, when I left seattle two years ago and got my California license, it listed my weight at 205lbs. I renewed last week and had 185lbs on it. Prior to leaving Seattle, you could add another 20 lbs to that. I no longer smoke.

I don’t know if this will be useful to anyone else, but I’ve had some success with it, and who knows, maybe it will work for you. If you try it out, I’d love to hear from you and see how it goes.

I’ve been working on a startup project on the side for almost a year now – focusing on pattern recognition, natural language processing stuff, and predictive statistical modeling… it’s been fun. At the core, we’ve put together a language analysis engine which looks at a chunk of text and figures out if it’s positive or negative. In researching this as a problem, we’ve determined that if you take three individuals, and then have them categorize the same random text (blog, article, website, tweet, etc) they will agree 63% of the time. There’s a little bit of variance depending on what’s shown, but plus or minus a couple percentage points, is about how accurate a human is. We’ve gone through several different models in doing the predictions, and tweaked the algorithm quite a bit over multiple different versions, but we recently hit a pretty major milestone – we’re now rating articles, or our engine is, with a 70+% accuracy rate. In other words, if we rate something as positive (meaning the author felt positive about whatever they were writing) 70% of the time, the human will agree with how we rated it. ^_^

We’re better at determining human opinion than the average human is.

We’re going to be going into beta soon, on a service that will allow you to track how positive or negative your brand is, by tracking the mentions on the internet – effectively doing sentiment analysis and tracking; if you’re interested, you can sign up here. You can read more about the project in general at www.entrenza.com.

Thanks to Steve & Jesse & Ben, my co-collaborators on the project for making this happen!

I’m thinking, along w/ some people I’m working on a start-up with, of running a contest to help up “train” the back end “artificial intelligence engine” which is used in our software.

Here’s the gist: you would log-in to a website, and be presented w/ an “article” – this would be a blog, website, etc. you would then rate it as positive, negative, & so forth. Anyone who rated 1000 articles in a month (each takes about 1-2 seconds) would be eligible to win a prize, which would either be an xbox 360, or a playstation 3.

So: Three questions:

For the last year or so I’ve been working on a tech startup with some friends. In doing so, I’ve gotten to work with some pretty cool stuff, and I thought I’d make a list of some of them. Basically, I wanted to extole the virtues of working on a startup as a great way to get real-life practice projects to work on – I have every expectation that we will have at least some success, but even if it ends up being a failure, here are some of the projects that I’ve gotten to work on:

• wrote an spidering application
• setup a mysql cluster
• setup zimbra
• researched several virtualization options
• setup apt-proxies
• wrote a custom smtp daemon / parser
• learned a boatload about bayesian analysis and other pattern recognition and predictive tools
• setup joomla
• setup drupal
• setup linux natting/routing firewall
• wrote project plans
• managed & motivated
• learned how to incorporate
• setup zimbra
• setup dnsmask for dhcp/dns masqerading
• setup bind dns & replication
• learned how to motivate people and lead weekly conference calls
• worked on project management
• marketing
• sales
• setup ldap athentication
• setup openNAS

Now – I’ve done a bunch of these things before in previous jobs, but its still good practice and there were several I hadn’t played with before. It’s a great opportunity to learn, and even if it doesn’t end up succeeding, the time I’ve put in will not have been wasted. I’ve become a better programmer, a better leader, and I’ve had a lot of fun doing it.

If you are interested in keeping track of the morale at your company, project, or keeping track of how positive people are about your brand, or a search term, we’re looking for beta customers. Feel free to ping me @nickbernstein on twitter if you think you might be interested.

So my 2001 VW jetta is getting a bit up there in the miles – it’s about 95k at the moment, and while this isn’t too much for a VW, I’ve been wanting to get a new car and I figure I should get rid of it while I can still feel good about selling it to someone else. Besides, I want to get a convertible – I live in southern California, and if it’s not the appropriate climate for one, I don’t know where is.

Initially I went to carmax and they offered me $2000. Yeeps! I was shocked. Could it really be worth that little? Kelly Blue book said it was at least worth $4k. So I thought I would test the open market and write a quick perl script to give me the average price for an item on craigslist. Here’s how it works and the code is below. It should be really easy to modify for anyone who could use something like this:


./cl_get_prices.pl 2001+jetta

lowest:         1200
highest:        9999
Average:        6134.48214285714


another example where I search for porsche boxster:


nick:~$ ./cl_get_prices.pl porsche+boxster

lowest:         7600
highest:        33100
Average:        16865.6341463415


Anyway the code is below, and I’ll put a link to the actual perl script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

#!/usr/bin/perl
$wget="http://losangeles.craigslist.org/search/cta?query=";
$wget .= $ARGV[0];
print $wget . "\n";
$html = `wget -q -O - $wget`;
        @words = split(' ', $html);
        foreach $word (@words)
                {
                 if ( $word =~ m/^\$/)
                        {
                                $word =~ s/(\$|,)//g ;
                                if ( $word =~ m/^\d+$/ )
                                {
                                        if ( $lowest eq '') {
                                                $lowest = $word ;
                                        } elsif ( $word < $lowest ) {
                                                $lowest = $word ;
                                        }
                                        if ( $highest eq '') {
                                                $highest = $word ;
                                        } elsif ( $word > $highest ) {
                                                $highest = $word ;
                                        }
                                        $amt += $word ;
                                        $count++;
                                }
                        }
                }
$average = ( $amt / $count ) ;
print "lowest:\t\t$lowest\nhighest:\t$highest\n";
print "Average:\t" . $average ."\n";

Here’s a link to the actual script you can download;. If you find it useful, let me know.

For the last day or so I’ve been playing with moving over my simple word-count-analysis of blogs to actually creating a database with manually ranked training data and extrapolating from that. There were some hiccups and I’ve still got to go back and replace a lot of code, but it’s effectively categorizing new blog entries based on previous rankings. YAY! I’ve been using the perl Algorithm::NativeBayes cpan module, and it’s pretty great – although the the documentation is really poor. The main thing to get is that it returns a hash reference, which means you end up referring to your result as something like:

<pre>print “Sport’s ranking: \t ${$result}{sports} \n” ; </pre>

Which, lets face it, is kinda ugly, but it’s really the only good way to do it, really. It should really be better documented, though. Aside from that, as long as you get the back-end math, you’re pretty OK. Just because you’re doing AI stuff doesn’t mean that you’re automatically familiar with how perl handles references to hashes, though. One of my to-do items is to go back and update the perldoc on it. Anyway, with that in effect I’ve gone and updated the database and I’m now able to get positivity over time. This means I’m actually getting closer to building an internet happiness index, and prediciting how “happy” the internet is as a whole. The next steps are:

• incorporate new bayes functions into existing codebase
• add more sources to the rss feedlist scripts
• optimize the blogparser
• put a nice (fusioncharts?) front end together
• add more hardware for doing the catagorization
• get more people to do more training data
• ???
• profit!

Actually, the “???” is pretty well defined, but honestly, this project will have been fun even it it doesn’t make a dime. Anyway, one step closer.

overview

Figure 1

Netapp’s snap restore product is a fantastic tool in a storage admin’s arsenal. It works well. It’s fast, and it doesn’t need to “restore” data, it just makes a previous snapshot the active file system. That said, I keep seeing the same scenario put forth by netapp and various other folks as a use case for snaprestore, and it’s one where snaprestore would be my second choice. The scenario is this: We’re taking hourly snapshots over the course of a day. A virus breaks out. The files on our cifs shares have been compromised. We’re sure that our current data is infected, and we know that at least some of the data was infected an hour ago. Two hours ago is uncertain, but we are sure that the virus hadn’t broken out three hours ago, so we know the file system at this point is clean. The recommendation you always hear is to use snap restore to roll back the file system to three hours ago, and you’re now clean and not infected. Unfortunately, as a side effect, we’ve lost all the data after that point – which, I should mention, is the intended result: if we hadn’t our files would still be infected, right? My solution is to instead clone the volume using a really neat tool called flexclone.

some snapshot basics

I’d like to suggest an alternate method, but before I get into it, it probably makes sense to talk about how snapshots work. The idea is very simple. Each file on a file system has an inode. An inode contains information about a file – the owner, when it was created, etc. If you’re on a windows box, and you view the “properties” of a file, you’re seeing the information contained in an inode. A hard drive is made up of blocks of data, and another thing that an inode does is point to the blocks that make up the data for a given file. If a file is big, the inode won’t actually point to the data blocks themselves, it will point to indirect blocks which then point to data blocks ( inode -> indirect level 1 -> data ). If a file is really big we can have multiple levels of indirection, like in figure 1. A snapshot is, for all intents and purposes, a copy of just that top level inode block. This block then contains pointers to the indirect blocks, and those indirect blocks to the data blocks. ( Is this fun yet? ) Once we’ve made a copy of the inode (parent) the indirect and data blocks (children) are effectively frozen, since a block cannot be changed unless it’s parent, or parents (original inode and now the snapshot) agree. The way we handle modifications to files in the netapp world is to write those changes to a special reserved section of the volume called the “snapshot reserve” (apt) and to update the original inode to point to those blocks. It’s pretty slick, really.

flexclone

In order to understand how this alternative solition works, we need to understand how flexclone works. Flexclone – to over simplify things – is a writeable snapshot. Actually, it’s really not that much of an over simplification, really. If we go back to our snapshot basics where we talked about how each file has an inode which points to data blocks, I should mention that under the hood, everything on a filesystem is actually just, well, a file… even directories. A directory is just a file that contains a list of file names and where the inodes for those files are, that’s the data in that “file’s” data blocks. The top of any volume (think of this as a drive) is a directory, and that directory in turn, has an inode. This one we call the “root inode”. When we take a snapshot, of volume, this is the inode we’re basing our snapshot of. So, what we end up with is: [ root inode ] [ snapshot copy of root inode ]. What we can do is add one more copy to make a “clone” of the volume. this leaves us with: [root inode ] [ snapshot copy of inode] [clone copy of inode]. Now we can still write to our normal volume by writing to that snapshot reserve that we talked about, and what we do when we create a flex clone is associate a *new* snapshot reserve with the clone. The cool thing is initially the clone takes up no space. Ok, that’s a lie, it takes up a coupe kilobytes, but we’re talking storage systems here, a couple of kilobytes is nothing. This is really useful for things like backing up a database, or doing QA -theres a whole bunch of use cases, such as, oh, avoiding the unneccessary dataloss of using snap restore in the event of a virus outbreak.

the actual solution

Ok, now that we’ve brushed up on our netapp basics, lets review our problem:

• we’ve taken hourly snapshots
• our active file system is infected with a virus, we’ll assume the volume name is vol1
• so is our previous hourly snapshot and probably the one before that
• the normal recommeded solution would be to snap restore to hourly.2 (three hours previous) before the virus broke out

Using this method would mean that we would lose three hours worth of data. That’s not good. Preventing data loss is the whole job of a storage admin, as far as I’m concerned. So, an alternate method:

1. take cifs offline, using the cifs terminate command
2. on your admin host copy the file /vol/vol0/etc/cifsconfig_share.cfg -> cifsconfig_share.cfg.YYMMDD.pre-virus.bak
3. make a clone of vol1 called “vol1_clone” with the command: vol clone create vol1_clone -b vol1 hourly.3
4. open /vol/vol0/etc/cifsconfig_share.cfg and do a find and replace, changing every instance of vol1 to vol1_clone
5. run the command cifs restart
6. run the command cifs shares -add oldvol1 /vol/vol1
7. lock it down with cifs access <your user> “Full Control”

So what did this do?

We quarantined the file system by stopping cifs and removing client access. Then we created a new clone of the volume that had been infected off of the snapshot we new was clean. We updated all of our shares using a quick find and replace giving our users access to their data, so they can get back to work. We also exposed our old volume via a new cifs share, which we’ve restricted only to our user, who theoretically knows better than to muck with infected files. So why did we do all this work? Later, when new virus definitions come out, we can scan that volume over night and have our anti-virus program go clean up our mucky infected files. Once we do, we can open up the share, send out an email, and give our users the ability to recover any important documents that had been created during those three hours, saving extra work, and potentially saving compliance headaches.

There is a final part of this, in which we split off the clone using the vol split command. I would note – you will (at least temporarally) need to have enough free space on your aggregate to contain both of the volumes, w/o any space savings. Once your split is finished, you can do a vol offline vol1 ; vol destroy vol1 to get rid of the old volume and free up that space.

I think this is a better solution to the problem, and one that’s much more elegant than using a snap restore. I’d love to hear any feedback or improvments to this process, so if you found this useful, or can think of a better way, please let me know.

Intro

A while back I decided I wanted to put down on paper some of the lessons I learned working as a systems admin and having worked in an enterprise environment with thousands of servers. Some lessons I learned in the small shops, some I learned at bigger shops. This is going to be the fist in my “Ops 101″ series, and a more broad series of essays about lessons I’ve learned.

The Processes

Process. Procedures. We all hate doing them. Without them, however, all the other stuff never gets done. Sure ? you?ll start off documenting the changes you make, and checking in that script, but unless there?s a process in place and accountability to back that process up, a couple of months down the line you?re going to be asking, ?wait, where does that script run again?? and ?that was upgraded? Since when?!? — without good process, you?ll lose all of those good habits that you?ll thank god you have when things start to break.

First and foremost, MMS should have been a standard feature in te original iphone release. This is a given. The lack of it is all the worse because it’s such a basic feature, and it requires responding to the inevetable question, “did you get that” with “well, yes, but I have to log into the site and I don’t want to do it on the phone, cause I can’t cut and paste and…” this reminds us that we don’t have cut and paste and how damn annoying that is.

If you haven’t seen it before, when someone sends you a picture message on the iphone you get a message saying, “Someone has sent you a picture, go to www.viewmymessage.com and login with the username and password: blah and blah” The usernames they pick are randomly generated strings of characters and not easy to remember, and really, this is an unacceptable solution.

Fix: Change the message. Instead of a username and password, give a direct link: http://viewmymessage.com?u=xxxxxx?&p=yyyyyyyy same information, slightly different formatting – problem solved.

Dear ATT, this is such a simple simple problem to solve, and one with a lot of visibility. Please fix it.

How a delayed flight can make a customer for life:

I’m sitting at JFK airport in New York as I write this, waiting on a flight that was supposed to leave at nine thirty pm. It, instead, is leaving at 12:30am. This is a crappy situation, but it’s not how a company deals with a customer when things are going great that counts, it’s how a company deals with it’s customers when things go awry. Air travel is unpredictable. They need to deal with weather, and lets face it: no matter what we have waiting for us on the other end, the most important thing is to get there safe. We all, as customers, understand that bad weather happens, and it’s not American Airlines fault.

That said, the girl working at the ticket booth has been making announcments about the time being pushed back. If, instead, she had walked over to the seating areas and addressed us in small groups, it would have been much more personal and it would have come across less as a company, and more as a sympathetic face. Even better would be if they had a small budget set aside for food and drinks. Lets compare and contrast:

Announcement #1 via the intercom:

“Ladies and Gentlemen, The aircraft for flight 4750 has not yet left Boston, and is now expected to arrive at 11:20, pushing back our departure time to midnight. Flight 4721 should be arriving at 11:35 and is expected to depart at 12:30. If there are any further changes, we’ll make another announcment.”

Announcement #2, in person, walking over:

“Ladies and Gentlemen, I’m really sorry, but we’ve got some more delays. I know it’s been a long wait, so we’re going to put out a table with coffee, and some snacks. The aircraft for flight 4750 has not yet left Boston, and is now expected to arrive at 11:20, pushing back our departure time to midnight. Flight 4721 should be arriving at 11:35 and is expected to depart at 12:30. I know it’s been a long wait, and we’ll let you know as soon as we know anything more.”

Use what you’ve got:

The worst part of the whole situation is all it would take is a little initiative. Airlines already buy coffee in bulk. Airlines already have snacks. The only requirement is the industrial coffee machines and some of those magic creamers that don’t go bad at room temprature (how do those things work?!). I understand that most people choose air travel on discount websites based on the lowest fair, but if it was well known that if you missed your flight you’d at least get a cup of coffee and some snacks.