Almost two years ago, I decided I was going to sit down and make some changes to my life. I had just finished a four year stint at microsoft, felt burnt out on tech, gotten out of shape due to working regular sixty hour weeks in front of a computer and smoking like a chimney. I sat down and thought about what it was that I wanted to achieve and came up with a list of stuff that I wanted to do. I decided to focus on accomplishing three -smoking, weight loss, and starting a company. I did several things to accomplish these goals, but one of the best, and easiest, was just changing my home page.
Nowadays, when I open up my browser in the morning, I’m greated with three pages. They are as follows: (my company) entrenza.com’s beta page, gmail, and foremost – a google docs spreadsheet entitled yyyy-mm goals and actions. The beta page is related to one of these goals, but lets talk about the spreadsheet. It’s really simple and looks something like this:

It’s been a pretty successful system. I don’t worry about huge strides, I just worry about accomplishing, one thing towards each goal each day. So far, ‘ve registered a LLC and and have regularly worked part time on a startup for over a year. We’re right about to go into beta. As for getting in shape, when I left seattle two years ago and got my California license, it listed my weight at 205lbs. I renewed last week and had 185lbs on it. Prior to leaving Seattle, you could add another 20 lbs to that. I no longer smoke.

I don’t know if this will be useful to anyone else, but I’ve had some success with it, and who knows, maybe it will work for you. If you try it out, I’d love to hear from you and see how it goes.

I’ve been working on a startup project on the side for almost a year now – focusing on pattern recognition, natural language processing stuff, and predictive statistical modeling… it’s been fun. At the core, we’ve put together a language analysis engine which looks at a chunk of text and figures out if it’s positive or negative. In researching this as a problem, we’ve determined that if you take three individuals, and then have them categorize the same random text (blog, article, website, tweet, etc) they will agree 63% of the time. There’s a little bit of variance depending on what’s shown, but plus or minus a couple percentage points, is about how accurate a human is. We’ve gone through several different models in doing the predictions, and tweaked the algorithm quite a bit over multiple different versions, but we recently hit a pretty major milestone – we’re now rating articles, or our engine is, with a 70+% accuracy rate. In other words, if we rate something as positive (meaning the author felt positive about whatever they were writing) 70% of the time, the human will agree with how we rated it. ^_^

We’re better at determining human opinion than the average human is.

We’re going to be going into beta soon, on a service that will allow you to track how positive or negative your brand is, by tracking the mentions on the internet – effectively doing sentiment analysis and tracking; if you’re interested, you can sign up here. You can read more about the project in general at www.entrenza.com.

Thanks to Steve & Jesse & Ben, my co-collaborators on the project for making this happen!

I’m thinking, along w/ some people I’m working on a start-up with, of running a contest to help up “train” the back end “artificial intelligence engine” which is used in our software.

Here’s the gist: you would log-in to a website, and be presented w/ an “article” – this would be a blog, website, etc. you would then rate it as positive, negative, & so forth. Anyone who rated 1000 articles in a month (each takes about 1-2 seconds) would be eligible to win a prize, which would either be an xbox 360, or a playstation 3.

So: Three questions:

I wrote a little while back about some of the things my startup has gotten me to work on, but it occurred to me today, after I being grumpy and frustrated for a good chunk of the day – the firewall died (hardware), the parser’s got a memory leak and is crashing the parse server – about how working on this project has made me a better programmer, technologist, and possibly even a better person.

Tomorrow, after going out to brunch with some friends, and then cleaning the apartment, I’m going to come back and start doing some code profiling to look for memory leaks. I’ve done standard debugging stuff before, and learned the basics of code optimization and such in school back in the day, but the fact of the matter is, until I started working on the startup project, I mainly wrote smaller programs – utility scripts, small apps, stuff like that. I didn’t really ever get into situations where I had to think about memory leaks, or applications which would basically run constantly. It’s other stuff too – if you write a script to automate the creation of this or that, or a .net app to generate foundry server iron configs, you don’t need to put it in an architectural perspective. Now I have to think about that kind of stuff all the time. “what happens when this breaks?” – “How can I write this so I can add another server and scale out horizontally?” – I actually think about this kind of stuff when I’m coding now. I’ll scratch whole bits of things that worked because they’ll cause grief down the line.

I think it’s also made me more disciplined. There’s a big difference between doing your job because you know, eventually, if you slack, you’re going to get grief about it, and if you do it well you’ll get rewarded for it – all by someone else, and setting goals and following through on your own. My home page, over the years has been slashdot, popurls, rootprompt.org and a miriad of other websites… this would be on my work browser. Now, it’s (thank you firefox and chrome for having multiple tabs) a google doc spreadsheet of my goals and columns representing dates and actions. Each day I list what I did to achieve those goals. Another tab contains our ticketing system. Another tab contains our intranet site, in which there are a bunch of daily actions that I try to go through. I would never have approached a job like this if I were being paid by someone else. It took doing this on myself to realize the type of mindset and tools I would have to give myself to accomplish these things.

Another thing that I think has made me a better person is our twice weekly conference call – we have a very loose structure for the company – there’s no office, and we use email, ticketing, IM to communicate and conf. calls to go over progress and complete goals. We basically cover what we’ve done, and what’s next. It also is a good opportunity for me to talk on the phone with friends, who all, at this point, live in different parts of the world. Yes, it’s about a shared goal, and a project, but it’s also about keeping in touch with friends, and doing so regularly. I have traditionally been terrible at keeping in touch with people, and I think that this help me in that regard.

Tomorrow is going to be frustrating as hell. Don’t get me wrong, I’ll have a nice lunch, and put the work out of my mind during that part, and I’ll enjoy the call – tomorrow’s sunday, one of the days we do it – but when I start getting into phase II of the code profiling stuff, and looking for circular references and objects that aren’t being collected, I’m going to get seriously frustrated and stressed out. I’m going to hate it. But I’ll make strides towards getting it fixed. And the idea that taking on a project of this scope, and how hard it is, is making me a better coder will give me some solace. The idea that I’ve had to change my thinking in regards to where it fits in the architecture, I think has made me a better technologist, and the discipline and keeping in touch with friends has made me a better person, I hope. Yes, I’ll definitely be incredibly frustrated when half the stuff I’m trying to do ends up breaking things temporarily, but I think, in the end, it’ll be worth it.

For the last year or so I’ve been working on a tech startup with some friends. In doing so, I’ve gotten to work with some pretty cool stuff, and I thought I’d make a list of some of them. Basically, I wanted to extole the virtues of working on a startup as a great way to get real-life practice projects to work on – I have every expectation that we will have at least some success, but even if it ends up being a failure, here are some of the projects that I’ve gotten to work on:

• wrote an spidering application
• setup a mysql cluster
• setup zimbra
• researched several virtualization options
• setup apt-proxies
• wrote a custom smtp daemon / parser
• learned a boatload about bayesian analysis and other pattern recognition and predictive tools
• setup joomla
• setup drupal
• setup linux natting/routing firewall
• wrote project plans
• managed & motivated
• learned how to incorporate
• setup zimbra
• setup dnsmask for dhcp/dns masqerading
• setup bind dns & replication
• learned how to motivate people and lead weekly conference calls
• worked on project management
• marketing
• sales
• setup ldap athentication
• setup openNAS

Now – I’ve done a bunch of these things before in previous jobs, but its still good practice and there were several I hadn’t played with before. It’s a great opportunity to learn, and even if it doesn’t end up succeeding, the time I’ve put in will not have been wasted. I’ve become a better programmer, a better leader, and I’ve had a lot of fun doing it.

If you are interested in keeping track of the morale at your company, project, or keeping track of how positive people are about your brand, or a search term, we’re looking for beta customers. Feel free to ping me @nickbernstein on twitter if you think you might be interested.

For the last day or so I’ve been playing with moving over my simple word-count-analysis of blogs to actually creating a database with manually ranked training data and extrapolating from that. There were some hiccups and I’ve still got to go back and replace a lot of code, but it’s effectively categorizing new blog entries based on previous rankings. YAY! I’ve been using the perl Algorithm::NativeBayes cpan module, and it’s pretty great – although the the documentation is really poor. The main thing to get is that it returns a hash reference, which means you end up referring to your result as something like:

<pre>print “Sport’s ranking: \t ${$result}{sports} \n” ; </pre>

Which, lets face it, is kinda ugly, but it’s really the only good way to do it, really. It should really be better documented, though. Aside from that, as long as you get the back-end math, you’re pretty OK. Just because you’re doing AI stuff doesn’t mean that you’re automatically familiar with how perl handles references to hashes, though. One of my to-do items is to go back and update the perldoc on it. Anyway, with that in effect I’ve gone and updated the database and I’m now able to get positivity over time. This means I’m actually getting closer to building an internet happiness index, and prediciting how “happy” the internet is as a whole. The next steps are:

• incorporate new bayes functions into existing codebase
• add more sources to the rss feedlist scripts
• optimize the blogparser
• put a nice (fusioncharts?) front end together
• add more hardware for doing the catagorization
• get more people to do more training data
• ???
• profit!

Actually, the “???” is pretty well defined, but honestly, this project will have been fun even it it doesn’t make a dime. Anyway, one step closer.