I like to set myself different challenges every 30 days. In October 2013, I tried to eat better and exercise more. I did alright on that, but without a specific daily goal, I had a hard time deciding how well I did. I mostly got back into the habit of exercising daily, so that was helpful.
For November 2013, I tried to do a “no work November.” I had enough vacation days built up that I was hitting the upper limit for work, so I took a bunch of vacation in November. My in-laws visited one week, then it was a family member’s birthday, so we took some time off at a resort in Arizona. Then it was back home for a week before spending the week before Thanksgiving in Kentucky with my family.
I learned a few things in my month off:
- I still enjoy reading tech and Google news for fun. It’s amazing (or problematic?) how much time you can spend just surfing the web each day and reading what other people are writing.
- My initial goal was to not read work email at all, but I had to give up on that. There were a few urgent things I genuinely had to weigh in on. I eventually settled for reading work email but trying really hard not to reply unless it was an emergency. I probably ended up writing 20-30 replies over the month, along with passing on spam reports that people emailed to me.
- I realized that I’d gotten in the bad habit of giving friends my work email address, as well as forwarding my personal email address to my work email. Takeaway: keep your work email separate from your personal email. Seems like common sense, but after almost 14 years at Google, things had gotten tangled together.
- A couple good pieces of advice that I failed to heed: 1) remove your work account from your phone, so you can’t check work email or docs on your phone. 2) if you have an “email tab” that you keep pinned on your browser, unpin and close that tab. I didn’t take either of those steps, but I should have.
- I didn’t feel the need to start any big projects, or write any Android apps, or blog a lot. I have a newer Linux computer that has configuration issues; I didn’t tackle that. Mostly I enjoyed reading a few books.
- I’m incredibly proud of the whole webspam team at Google. Things ran like clockwork while I was gone. I’m really grateful to the phenomenal people that fight spam for Google’s users every day.
Which brings us to December 2013. Back in September, I threw my back out. I can still move around fine, but it sometimes hurts if I bend in various ways. So my goal for December 2013 is to do 15-20 minutes of stretching–things like cat and camel–each day to help my back recuperate.
How about you? Are you doing any 30 day challenges?
At this point, our webmaster console will alert you to manual webspam actions that will directly affect your site. We’ve recently rolled out better visibility on website security issues, including radically improved resources for hacked site help. We’ve also improved the backlinks that we show to publishers and site owners. Along the way, we’ve also created a website that explains how search works, and Google has done dozens of “office hours” hangouts for websites. And we’re just about to hit 15 million views on ~500 different webmaster videos.
So here’s my question: what would you like to see from Webmaster Tools (or the larger team) in 2014? I’ll throw out a few ideas below, but please leave suggestions in the comments. Bear in mind that I’m not promising we’ll do any of these–this is just to get your mental juices going.
Some things that I could imagine people wanting:
- Make it easier/faster to claim authorship or do authorship markup.
- Improved reporting of spam, bugs, errors, or issues. Maybe people who do very good spam reports could be “deputized” so their future spam reports would be fast-tracked. Or perhaps a karma, cred, or peer-based system could bubble up the most important issues, bad search results, etc.
- Option to download the web pages that Google has seen from your site, in case a catastrophe like a hard drive failure or a virus takes down your entire website.
- Checklists or help for new businesses that are just starting out.
- Periodic reports with advice on improving areas like mobile or page speed.
- Send Google “fat pings” of content before publishing it on the web, to make it easier for Google to tell where content appeared first on the web.
- Better tools for detecting or reporting duplicate content or scrapers.
- Show pages that don’t validate.
- Show the source pages that link to your 404 pages, so you can contact other sites and ask if they want to fix their broken links.
- Or almost as nice: tell the pages on your website that lead to 404s or broken links, so that site owners can fix their own broken links.
- Better or faster bulk url removal (maybe pages that match a specific phrase?).
- Refreshing the existing data in Webmaster Tools faster or better.
- Improve robots.txt checker to handle even longer files.
- Ways for site owners to tell us more about their site: anything from country-level data to language to authorship to what content management system (CMS) you use on different parts of the site. That might help Google improve how it crawls different parts of a domain.
To be clear, this is just some personal brainstorming–I’m not saying that the Webmaster Tools team will work on any of these. What I’d really like to hear is what you would like to see in 2014, either in Webmaster Tools or from the larger team that works with webmasters and site owners.
For the folks that don’t know, I’ve been out for a couple weeks and I’ll be on vacation the rest of November. If you’ve tried to contact me recently and haven’t heard back, that’s probably the reason.
Added: if you enjoy watching our webmaster videos, you can follow @googlewmc to hear as soon as we publish new Webmaster Central videos. It looks like @googlewmc is just about to hit 100,000 followers on Twitter!
On April 21st, 2014, I’m going to run the Boston Marathon. If you want to show your support, please donate to a good cause for cancer research. Anyone who wants to give is welcome.
So many people have been affected by cancer, including members of my own family. The Dana-Farber Cancer Institute funds basic and innovative cancer research. That’s why I’m trying to raise $9,000 for the Dana-Farber Marathon Challenge.
I’ve been running for a few years now (that’s me at the San Francisco marathon), but this is the first time I’m trying to run to raise money for a cause, and I would really appreciate donations. It won’t give you more PageRank or a higher rank on Google, but Dana-Farber is a great institution and I’d love to raise as much money for them as I can before I run in Boston. If you can, please consider donating to kick cancer’s butt. Thank you!
For Halloween 2013 I decided to be the Dread Pirate Roberts from the movie The Princess Bride:
I even grew a slight moustache to help make the character believable:
But to be clear, I wasn’t just any old Dread Pirate Roberts. I decided to be the Silk Road Dread Pirate Roberts. So if you want to buy some Iocaine powder, it’ll cost you five bitcoins:
By the way, I did have a couple Penguin masks and a tuxedo. I thought about doing a Penguin costume, but decided not to. Maybe next year?
Someone recently asked me how I manage my to-do list, so I thought I’d write up the software that I use. Fundamentally I use Google Tasks as the backend, but with extensions and apps that improve on the basic functionality in Google Tasks.
I use a couple different extensions for Chrome:
- Better Google Tasks is a great Chrome extension. Just click a button in Chrome and you have instant access to all your todo items. I like the extension so much that I donated some money to the author, Chris Wiegman. You can get the Better Google Tasks extension from the Chrome Store.
- I also noticed that on the New Tab page of Chrome, seeing thumbnails of my most visited sites (Techmeme, Hacker News, Nuzzel, Google News, etc.) every time I opened a new tab inevitably led me to click over to those sites. The result? I was wasting more time surfing than I wanted. The solution is a great Chrome extension called New Tab to Tasks. It changes Chrome’s new tab page to be your todo list. That way, I get a nice little signal every time I open a tab: “Hey, remember that you’re supposed to be working on stuff, not goofing off.” Thanks to Scott Graham for writing this Chrome extension.
Oh, and one last Chrome recommendation: if you don’t want *any* distractions on Chrome’s new tab page, consider installing Empty New Tab Page, which makes the Chrome new tab page completely blank.
For Android, I use an app called Tasks. It costs $0.99, but there’s also a free version that starts showing ads after 10 days. I like the Tasks app for Android because it syncs with Google Tasks, has nice widgets, you can easily move tasks up and down, and you can indent tasks underneath each other. I only keep a few todo lists (Home, Work, Grocery, etc.), and to switch between lists you just swipe left or right. Tasks works great for me, but if you have tons of different todo lists then swiping between those lists might get old.
I can already imagine someone asking “Okay, but what about Google Keep?” I’m not opposed to Google Keep, but at this point I’ve found various third-party solutions that interoperate with Google Tasks and work well for me on Chrome and Android. Plus I already have my data in Google Tasks, so for the time being I like these solutions for Google Tasks.
Some relatives were visiting this past week, so my inbox has a triple digit backlog. That’s after aggressive pruning of mailing lists and so on. Nearly all of those emails mention me in a “to:” or “cc:” line and request a response. Some observations:
- roughly 40% of those emails are from the outside world (that is, not from colleagues at Google).
- only 5% of my emails are from people who are actually on my team.
- 3% of my current emails are about internal legal matters.
- 1% are from public relations folks.
- about 10-12% of those emails are about a couple recent internal projects that aren’t related to webspam but that I’m helping with.
My 30 day goal this month is to get to a better place with email. Heck, I might make “better email habits” an ongoing 30 day challenge until things are in a better place. Could I get to a healthier place in three months? Four months? I have no idea how long it will take, but email represents my largest source of work stress. When I’ve tracked my time in the past, it takes me about three hours a day to keep from falling behind on email. If my whole day is full of meetings, then I’m spending several hours at night to keep my head above water. Does anybody else tackle email on their vacation so it’s not as bad when they get back? Some of you do, right?
At 40% of my overall load, it’s clear to me that I have to do something different for emails from the outside world. For years I tried to answer everyone who emailed me. I’m going to have to go “lossy” and just let some of those emails drop.
I need to think about whether it makes sense to write a blog post like Chris Sacca did (which
Rick Klau recently surfaced) that tries to address the common things that people email about. Then again, Rand Fishkin did something like that at http://moz.com/rand/making-email-more-scalable/ and he reported that he ended up with “a bunch of very angry people” when he pointed them to a blog post.
So I’m not sure whether it’s better not to reply, or to write up a canned response or maybe a blog post or a flowchart that I can point people to. If you have tips that have worked for you to make email more manageable, let me know in the comments below.
Added, 9/25/2013: This has been a tough challenge. One tactic that has worked well for me is to put email away from Friday evening until Sunday evening. Then (since I’m a workaholic), I ask myself “If someone else were trying to relax this weekend, what would I recommend for them to do?” and I try to do that. As a result, I’ve read more books this month, which has been nice.
The other tactic is to allow myself to go lossy, which means not answering every email. A lot of emails require 5-15 minutes at a minimum to respond, so email becomes a todo list in which anyone can keep adding to the list. Treating any non-trivial email as if it’s a request for 10-15 minutes of my time has helped me figure out which emails I should respond to vs. not replying.
A couple friends have recently had security scares with their Gmail account where they were worried that their accounts might have been hacked. I was emailing one of them about how to make sure that your account is safe, and I realized it might be handy to post this on my blog as well.
Here’s the email that I just wrote to a friend:
Here’s what I’d do:
- change your password (make sure you’re on google.com when you change your password)
- check for any strange activity. In Gmail, go to the bottom right and look for a message that looks like “Last account activity: 30 minutes ago. Open in 1 other location” and click on the “Details” link and look for any unusual logins, for example log ins from countries that you haven’t been in recently.
- Also check for weird forwarding rules. If hackers get into your Gmail, sometimes they’ll create a rule that forwards all your email to them. To check your filtering rules, in Gmail click on the gear icon in the top right, then select Settings from the drop down. Click on the link for “Filters” and just check whether there’s any rules that look suspicious to you.
In an ideal world, you’d turn on two-factor authentication like is described at https://support.google.com/accounts/answer/180744?hl=en . It’s more hassle to use two-factor authentication, but it makes your account much more secure against being hacked.
I’m a big fan of two-factor authentication, but I realize that casual users might not want to turn it on. My take is that it’s a lot better to set up two-factor authentication than worry about a hacked account.
For June 2013, my 30 day challenge was to record a second of video every day. I was inspired by Cesar Kuriyama’s wonderful TED talk about how he records a second of video every day. There’s a couple things Cesar said in his talk that really resonated with me:
- “[A]s the days and weeks and months go by, time just seems to start blurring and blending into each other and, you know, I hated that“. Totally agree. One of the reasons I started doing 30 day challenges was that I was alarmed at how quickly time was passing and I wanted to make my time more memorable.
- “This has really invigorated me day-to-day, when I wake up, to try and do something interesting with my day“. Recording a second of video a day has definitely made me keep my eyes peeled for noticeable sights. That also happened when I took a picture every day for a different 30 day challenge.
Okay, enough talk. Why don’t I show you my video montage for June 2013? (I missed three days, so I added three seconds from May to make it a full 30 days.) Here’s my video:
To make this video, I used Cesar’s 1 Second EveryDay app. The app is available for iPhone and iOS devices now, and Cesar let me beta test the Android app. The Android version of the app just went live, so you can give it a try.
I really enjoyed this challenge. I definitely did more interesting things, and the video is like a diary of travel and events from June 2013. Even on boring days, there’s probably at least one fun second you can save. The video makes my life look more exciting than it actually is, but maybe that’s not such a bad thing?
If you’re new to 30 day challenges, recording a second of video every day is a great way to start.
For May 2013 I decided to try making a Chromebook Pixel my primary laptop. So how did it go? Well, the short version is that I’m still a happy Pixel user, almost three months after my one month challenge started.
Previously, I was using a Thinkpad 420s running Goobuntu. In fact, I’ve been using Thinkpads since 1998, when I got my first one in grad school. Before I talk about the Pixel, here’s what I like about Thinkpads:
- Thinkpad keyboards have been the best in the industry. Great depth and just-right resistance. I especially liked the Thinkpad’s dedicated back/forward buttons for web browsing, located right near the arrow keys. Unfortunately, Lenovo has moved to “chiclet” style keyboards and dropped the back/forward buttons in newer Thinkpads.
- Thinkpads have a red rubber TrackPoint in the middle of the keyboard. TrackPoint pointers are faster and more precise than trackpads or even mice for me, since you don’t have to take your hands off the keyboard. I didn’t think I could use a laptop without a TrackPoint.
- Thinkpads have a consistent power connector that doesn’t change very often. Most Google conference rooms have Thinkpad and Mac power connectors, so you don’t have to haul a power cord around with you.
- The Thinkpad 420s has a black magnesium case that’s not as cold on your lap when it wakes up as aluminum.
- The Thinkpad 420s has a 1600 x 900 widescreen. Until retina-type displays came out, that was one of the highest-resolution laptops you could get.
Okay, Thinkpads are great machines. But what’s not to like?
- Battery life. When I kept my screen pretty bright, I only got 2-3 hours of battery life.
- Heavy. I didn’t notice until I started using the Pixel, but my Thinkpad 420s was 5.2 pounds. That’s pretty darn hefty for a laptop these days.
Overall though, I was very pleased with my Thinkpad and expected to return to it after the 30 day challenge was up. After all, I’ve been using Thinkpads for 15 years.
Then the Chromebook Pixel surprised me. The main thing you need to know about the Chromebook Pixel is that the screen is phenomenal. The resolution is 2560 x 1700 and 239 pixels per inch (ppi), compared to 227 ppi for a Macbook Pro with retina display. To demonstrate the screen, the Pixel comes with an app called TimeScapes which is drop dead gorgeous. The screen is also a 3:2 aspect ratio, which seems weird for a few days but is actually much better for web browsing than a widescreen display because more of a web page fits on the screen.
The Pixel also comes with a terabyte of Google Drive storage for three years and 12 free Gogo wifi passes for airplane trips. And if you’re worried about the Chrome-only, cloud-only aspect of a Pixel, you can install Linux on it. Even Linus Torvalds likes it.
Okay, but how did it work over 30 days? Better than I expected. I was 12 days in when I realized I’d probably keep using the Pixel after the challenge was over. Let’s run down what’s good and bad:
- The screen. So nice. Although I don’t understand why they made it so glossy. Screens should be matte, in my opinion.
- Incredibly easy to set up. I use Chrome Sync to sign into Chrome, so basically I just logged in and all my settings, bookmarks, and extensions showed up like magic.
- No configuration. I spent most of this past January reconfiguring several new computers, so “no muss, no fuss” is a big plus.
- The battery life is better. More like five hours, so I’m not constantly looking for a power adapter. If Google puts a Haswell chip in the Pixel, the sucker should go practically all day.
- The trackpad works great. The physical texture of it is silky-smooth, and I never saw any of the glitches that affected the CR-48. Sometimes I do accidentally click when I’m touching the trackpad just to move the cursor, but that’s hard to get right.
There were a few times I missed a regular laptop though:
- John Dvorak’s blog got hacked, and I wanted to send Dvorak a snippet of code that I fetched from his server, but the Pixel doesn’t have wget installed of course. You can do SSH, so I could have SSH’ed into another computer to fetch the page, but I didn’t bother.
- At one point I was trying to download a list of my books from Good Reads in comma separated value (CSV) form so I could upload the file to My Library on Google Books. The Pixel didn’t know what to do with a .csv file, which surprised me since Gmail and Google Drive seem to handle them fine. I suspect that this is a temporary “slip between the cracks” sort of thing, since it looks like Google is working on editing Office docs on Chrome OS. There have been a few times that I’ve downloaded a file and just wanted a simple text editor to tweak 2-3 characters in the file though.
- When you have a ton of tabs open using a lot of memory, clicking back on a tab that had been unused for a long time could cause the tab to reload. If you had unsaved work in the tab, you might lose it. This got better (but not perfect) over time. I’m not sure if Chrome OS got better, or I just got more careful with my tab management though.
One thing that annoyed me (selecting large blocks of text was slow when you had to scroll) was fixed when Chrome OS updated to a newer version. I have faith that other tiny annoyances–scrolling a page with two fingers doesn’t work for some reason when your cursor is over a tab instead of a web page, for example–will also be fixed. At the same time, I haven’t fully adapted to the touch screen and dedicated search button and don’t use either as much I could. But in general, the Pixel seems like it will just continue to get faster and better over time, not slower and cruftier like most machines.
So is the Pixel perfect? Not completely, but most of that (glossy screen vs. matte, trackpad vs. TrackPoint, chiclet vs. regular laptop keyboard) is a matter of personal preference. If you’re in the Apple ecosystem, I understand the decision to get a MacBook Pro with Retina display so you can run native apps. But I’m not in the Apple ecosystem, and I actually like a machine that discourages me from keeping too much data locally.
A lot of people poke fun at Chromebooks saying that they’re not much use without a WiFi connection. Personally, I believe that practically any computer is not much use without a network connection. Chromebooks are getting better at working well offline, but I have to say: the Pixel I’m using has an LTE option for when WiFi isn’t available, and I didn’t need to use the LTE connection on the Pixel any time in the last three months. Especially when you take the Google Drive storage and 12 internet passes into account, the Pixel is quite a good deal for a premium laptop. I’m going to keep using it.
This is getting long, so I’ll close with an anecdote. My Dad visited earlier this month. I gave him a Samsung Chromebook as a loaner for his visit and he’s been using it happily. He logged into his Gmail account in Chrome and his bookmarks and other Chrome settings just showed up. Dad’s laptop back home is about 5 years old, so we stopped by the Apple store. I was going to outfit him with a top-of-the-line MacBook; since he goes for years between upgrades, I wanted him to have a laptop that would last as long as possible. But after noticing the price, he balked. “Matt, we can buy ten Chromebooks for that much money,” he told me. We’re still discussing it, but the $250 Samsung Chromebook does everything he needs. I think more and more people will discover that’s true for them as well. I’ve been surprised how well the Chromebook Pixel works for me.
I realized that I didn’t mention this widely: my current 30 day challenge (July 2013) is not to read any news or social media. So no Twitter, Google News, Techmeme, Google+, Hacker News, Reddit, Imgur, etc. So if you’re wondering why I haven’t replied to a question, that’s the reason. I might still share a link if I run across something interesting, but I’m trying not to read any social media or news.
Why am I doing this? I find it to be a useful challenge. I’m crunching on a bunch of stuff and really wanted to get my head down and focus on some projects. I did this challenge in January 2013 and got a ton of stuff done. After the no-news challenge in January, I started surfing Twitter less often. I’d still check in every 3-4 days to read the tweets that people were sending my way, but otherwise I’d hit Nuzzel to skim just the most important tweets.
Longer-term, I’m trying to find a healthier approach to news and social media. On the spectrum of books to magazines to newspapers to social media with its second-by-second focus, I’d like to shift my media consumption more toward books and research. I’d also like to spend less time consuming media and more time doing things. See Clay Johnson’s Information Diet book for more about those kinds of ideas.
Back in the 90s, the heart of a computer was the CPU. The faster the CPU, the better the computer was–you could do more, and the speed of the CPU directly affected your productivity. People upgraded their computers or bought new ones whenever they could to take advantage of faster CPU speeds.
I remember the point when computers got “fast enough” though. Around 1997 or 1998, computers started hitting 166 MHz or 200 MHz and you could feel the returns diminishing. At some point, the heart of a computer switched from being a CPU to the hard drive. What mattered wasn’t the speed of your Intel or AMD chip, but the data that you had stored on your computer.
The era of the hard drive lasted for a decade or so. Now I think we’re shifting away from the hard drive to the network connection. Or at least the heart of a computer has shifted for me. In 2006 I contemplated a future where “documents sat in a magic Writely [note: now Google Docs] cloud where I could get to them from anywhere.” Sure enough, I keep all my important files in Google Docs now. At this point, if I have a file that sits only on a local hard drive, I get really nervous. I’ve had local hard drives fail. By 2008, I was spending 98% of my time in a web browser.
Don’t get me wrong. Local hard drives are great for caching things. Plus sometimes you want to run apps locally. But for most people, the heart of a computer will soon be its network connection. Ask yourself: could you get by with a minimal hard drive? Sure. Plenty of people store their files on Dropbox, Box, Google Drive, iCloud, or SkyDrive. Or they back up their data with CrashPlan, SpiderOak, Carbonite, or Mozy. But would you want a computer that couldn’t browse the web, do email, or watch YouTube videos? Not likely.
Folks at Google get cold-call emails out of the blue just like everybody else. Here’s an email that a colleague of mine got recently:
I was on your website www.google.com and wanted to shoot you a quick note. I think I can make a few changes (aesthetically and/or SEO – wise) to make your site convert more visitors into leads and to get it placed higher in the organic search results, for a few of the select terms.
This is NOT like one of those foreign emails you probably get in your inbox every day. Just to be upfront I have 3 agents that work with me for development /SEO.
I would just need to know which (if not both) services you’re open to checking out information about, either web design or SEO. Would you be open to seeing more brief info / quote for what I would like to accomplish?
So this person is offering help to convert Google visitors into leads. Or, you know, to improve Google’s rankings in organic search results. Sigh.
Earlier this week, I got a different email that said
I would like to extend our knowledge to your audience in the form of a uest post [sic]. This post will be written by a college educated writer fluent in English.
To recap we will provide-
- 100% original guest post with statical [sic] data and studies from professional writers.
Here’s my rule of thumb: if someone sends you an email with an SEO offer out of the blue, be skeptical. For example, check out some other fun SEO emails that I’ve gotten in the past.
We started rolling out the next generation of the Penguin webspam algorithm this afternoon (May 22, 2013), and the rollout is now complete. About 2.3% of English-US queries are affected to the degree that a regular user might notice. The change has also finished rolling out for other languages world-wide. The scope of Penguin varies by language, e.g. languages with more webspam will see more impact.
This is the fourth Penguin-related launch Google has done, but because this is an updated algorithm (not just a data refresh), we’ve been referring to this change as Penguin 2.0 internally. For more information on what SEOs should expect in the coming months, see the video that we recently released.
Added: If there are spam sites that you’d like to report after Penguin, we made a special spam report form at http://bit.ly/penguinspamreport . Tell us about spam sites you see and we’ll check it out.
We just recently taped a new round of webmaster videos, and I thought this video deserved a full-fledged blog post. This is my rough estimate (as of early May 2013) of what search engine optimizers (SEOs) and webmasters should expect in the next few months:
Bear in mind that this is a very rough estimate, because priorities, projects, and timing can change based on a lot of different factors. But I hope this gives folks a ballpark idea of what to expect in the coming months as far as what my team is working on.
This is a “hairball” post you can ignore. However, this post does trace my thinking about how to scale webmaster communication. Part of me wants to start answering questions I get via email by stripping out the identifying information and then replying with a blog post. Instead of one person getting a single reply, everybody could see what the answer is.
I spent most of the past week tackling my horrendous email backlog. At the start of the weekend, I was just touching 500 unread emails. I got it down to 218 unread emails and 264 total emails in my inbox. Of course, the ones that are left are the harder messages. And out of those 264 emails, 167 are from outside Google.
A few weeks ago, I flew up to the Kirkland office for a couple days to catch up with the Webmaster Central team. At some point, we were talking about doing videos for webmasters. Someone said “Why don’t we just grab a video camera and see how many videos we can shoot in an hour?” So we did. We managed to tape three pretty informative videos in about an hour, and that includes set-up/breakdown time.
So now I’m looking at these 150+ emails from outside Google, and I’m pondering about how much time I should spend on email compared to other things. Email is a 1:1 communication, so I could answer 10 emails and help roughly 10 people. Or in the same amount of time, I could comment on a forum, start on a blog post, or plan out another video that could benefit a lot more people. I did a series of about 15 videos last year when my wife was out of town, and the videos have been watched over 300K times and downloaded over 100K times.
So to make a long story short, I’m trying to figure out how I should handle email going forward. I’ve been thinking about this for a while, but don’t be offended if I don’t reply to email as much going forward.
A quick “hairball” post about how sometimes it’s better to just go with the flow. I like how xkcd made this point with a chart of whether it’s worth the time to fix something that’s bugging you.
I have a friend who is mechanical engineer. A few years ago he took me for a tour of his workplace that ended in his office. As I looked at his workstation, something leapt out at me. My friend had never changed his background screen. Whatever the computer came with by default, that’s what he was using. With this simple act of indifference, my friend taught me an important productivity lesson.
It’s great to be productive. It really is. But sometimes we chase productivity so much that it makes us, well, unproductive. It’s easy to read a lot about how to be more productive, but don’t forget that you have make that time up.
This is a harmless “hairball” post I had as a draft.
Me: Hey, they added me to popurls.com!
My wife: Never heard of it. (pause) Had you heard of it before?
Wife: (with an extra helping of sarcasm) Really?
Wife: (dripping with condescension) You’re a very important man.
This is the last half-finished “hairball” blog post about USB devices on Linux. I actually did manage to get a working program that controlled a USB foam missile launcher. Unfortunately, I didn’t document all the steps, so this blog post just sort of stops at some point.
I got a USB Missile Launcher for Christmas. The manufacturer, Dream Cheeky, provides software–but only for Windows XP. And I thought to myself, “wouldn’t it be fun to practice some USB reverse engineering skills?” Because another Christmas present was a USB protocol analyzer from Total Phase. I should note that plenty of other people have apparently already written drivers/software for USB missile launcher toys, but I wanted to poke around myself.
Total Phase makes a high-speed USB 2.0 protocol analyzer for $1200, or a regular-speed USB protocol analyzer for $400. Here’s a trick someone mentioned: if you get the cheaper protocol analyzer and need to work with a high-speed USB device, you may be able to plug the high-speed device into a low-speed USB hub to slow the device down.
I decided to start with ladyada’s excellent guide to hacking a Kinect by reverse engineering USB packets. So here’s what I did.
Step 1. Make sure the device works. It would suck to attempt to reverse engineer a broken device. I keep a Windows XP computer lying around, so I downloaded the software for it, installed the program, and plugged in the USB rocket launcher. After the install, XP wanted to restart, so I restarted the XP computer (unplugging my USB rocket launcher after the computer was off), then started the rocket launcher software back up, then plugged in the USB device. Sure enough, everything worked fine. The controls are: pan left/right, tilt up/down, and fire. Tip: the rocket launcher uses bursts of air, so don’t jam the foam rockets down hard on the launcher.
Step 2. Probe the device. I plugged the USB rocket launcher into a Linux machine running Ubuntu 10.04 (Lucid Lynx). I ran the command sudo lsusb -vv and the relevant info from the list of USB devices on my system was this:
Bus 002 Device 045: ID 0a81:0701 Chesen Electronics Corp. USB Missile Launcher Device Descriptor: bLength 18 bDescriptorType 1 bcdUSB 1.10 bDeviceClass 0 (Defined at Interface level) bDeviceSubClass 0 bDeviceProtocol 0 bMaxPacketSize0 8 idVendor 0x0a81 Chesen Electronics Corp. idProduct 0x0701 USB Missile Launcher bcdDevice 0.01 iManufacturer 1 Dream Link iProduct 2 USB Missile Launcher v1.0 iSerial 0 bNumConfigurations 1 Configuration Descriptor: bLength 9 bDescriptorType 2 wTotalLength 34 bNumInterfaces 1 bConfigurationValue 1 iConfiguration 0 bmAttributes 0xa0 (Bus Powered) Remote Wakeup MaxPower 100mA Interface Descriptor: bLength 9 bDescriptorType 4 bInterfaceNumber 0 bAlternateSetting 0 bNumEndpoints 1 bInterfaceClass 3 Human Interface Device bInterfaceSubClass 0 No Subclass bInterfaceProtocol 0 None iInterface 0 HID Device Descriptor: bLength 9 bDescriptorType 33 bcdHID 1.00 bCountryCode 0 Not supported bNumDescriptors 1 bDescriptorType 34 Report wDescriptorLength 52 Report Descriptors: ** UNAVAILABLE ** Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x81 EP 1 IN bmAttributes 3 Transfer Type Interrupt Synch Type None Usage Type Data wMaxPacketSize 0x0001 1x 1 bytes bInterval 20 Device Status: 0x0000 (Bus Powered)
Note that my Vendor ID = 0x0a81 and my Product ID = 0×0701. Also note that bNumEndpoints = 1. An endpoint is a channel for USB data communication. Then we get the Endpoint info:
Endpoint Descriptor: bLength 7 bDescriptorType 5 bEndpointAddress 0x81 EP 1 IN bmAttributes 3 Transfer Type Interrupt Synch Type None Usage Type Data wMaxPacketSize 0x0001 1x 1 bytes bInterval 20
According to ladyada’s write-up, the “IN” means that data goes IN to the computer from the device, and the “Interrupt” transfer type is good for sending large amounts of small data quickly (e.g. a USB mouse).
Step 3. Prepare your Linux system to talk to the device. First, let’s review ladyada’s steps, which is for Windows. She installs libusb-win32 and then runs a program called inf-wizard to make a driver shell. Then plugging the device into Windows will attach the LibUSB-win32 device driver. Next, she installed Python and PyUSB.
I wanted to stick with Linux. I didn’t need libusb-win32 or inf-wizard.exe, and I already had Python installed. So my next step was to download PyUSB, extract the zip into a directory, change into that directory, then run
sudo python setup.py install in that directory to install PyUSB. Since you’re installing PyUSB system-wide, you do need to run that command with “sudo” to run it as root.
Step 4. Write a short program on your Linux machine to talk to the device. I made a file missile-launcher.py and executable access with “chmod ugo+rx missile-launcher.py” next. Here’s the short program I ended up with:
#!/usr/bin/python import usb.core import usb.util import sys # find our device dev = usb.core.find(idVendor=0x0a81, idProduct=0x0701) # was it found? if dev is None: raise ValueError('Device not found') # Linux kernel sets up a device driver for USB device, which you have # to detach. Otherwise trying to interact with the device gives a # 'Resource Busy' error. try: dev.detach_kernel_driver(0) except Exception, e: pass # already unregistered # set the active configuration. With no arguments, the first # configuration will be the active one dev.set_configuration() print "all done"
Note that my “idVendor=0x0a81, idProduct=0×0701″ parameters use the values I found from lsusb -vv. If you compare against ladyada’s short program you’ll notice one major difference. My code has these lines:
# Linux kernel sets up a device driver for USB device, which you have # to detach. Otherwise trying to interact with the device gives a # 'Resource Busy' error. try: dev.detach_kernel_driver(0) except Exception, e: pass # already unregistered
Ladyada’s PyUSB program for Windows didn’t have anything like that. But when I ran the program under Linux, I got the error message “usb.core.USBError: Resource busy”. It turns out that the Linux kernel tries to use a default kernel driver, and that prevents my program from talking to the device. Detaching the kernel driver lets me talk to the device just fine. I picked up this tip Ken Shirriff’s post about a USB panic button with Linux and Python. In theory you could also unbind the USB device from a command-line, but I prefer to do it right in my PyUSB program directly.
Note that you will need to run the python program as root, e.g. “sudo ./missile-launcher.py” or else you’ll get a warning message like “usb.core.USBError: Access denied (insufficient permissions)”.
At this point, you have a small working program that opens up a connection to the USB rocket launcher. If the USB rocket launcher isn’t plugged in, you’ll get a “Device not found” error, and if the USB device is plugged in, you’ll get an “all done” message and the program exits gracefully.
Step 5: Try to read from the USB device. In ladyada’s guide, she tried sending 0b11000000 = 0xC0 (“Read Vendor data from Device”) to a Kinect. I got no response from that, but I did get a response sending 0b10000000. That corresponds to sending:
- a ’1′ to read from the device
- a message type of ’00′ = standard. Ladyada got a response sending to ’10′ = vendor
- ’000′ (reserved bits, so always 0)
- ’00′ to say the recipient of the message is the device.
Then sending a request of 0 got back a result of two zero bytes:
bRequest 0 array('B', [0, 0])
Interestingly, running the same program again would get a “usb.core.USBError: Unknown error” response. At that point, I would unplug the USB device and then plug it back in to reset it. I didn’t get any other responses from trying to send message types of class or vendor (as opposed to standard), nor did I get any responses from try to send messages to the interface or endpoint (as opposed to the device). See ladyada’s guide for more details about fuzzing the device and what all the various bit fields mean.
Step 6: Set up the Linux computer to use the Total Phase Beagle. The CD worked nicely with HTML documentation on it. First, you copy some udev rules so that the device is writable by anyone when the Beagle is plugged in:
cd /media/Total Phase/drivers/linux/
sudo cp 99-totalphase.rules /etc/udev/rules.d/
sudo chmod 644 /etc/udev/rules.d/99-totalphase.rules
If you’ve already plugged in the Beagle, you’ll need to unplug it and plug it back in for these rules to fire. Next, you’ll need the Data Center software. You can get it off the CD, but I’d recommend downloading the latest software and user’s manual from the website instead. My CD had software version 4.20 for example and the website was up to 5.01. Extract the software zip file (either from online or the CD). Then follow the directions in the online manual (or user manual PDF). The directions according to the manual are
- Install the USB drivers and Data Center software. Copying the udev rules is enough for USB drivers on Linux. Unpacking the zip is all you need for the Data Center software, because the executable is self-contained.
- Plug the Beagle analyzer into the analysis machine. This was my Linux machine.
- Plug the Beagle analyzer into the bus to be analyzed. In this case, this was my XP computer. Don’t plug the USB missile launcher in yet though.
- Start the Data Center software. Run the program “Data Center” in the directory you extracted from the .zip file. Follow the rest of the instruction in the Quick Start section.
What, *another* half-finished blog post about Linux USB drivers? Yup.
Suppose you have a device and want a Linux device driver for it. There are a few steps you’ll need to take. One of the heroes in this area is Greg Kroah-Hartman. Greg wrote USBView, which is a Linux tool to enumerate a list of USB devices. He’s also done a massive amount of documentation as we’ll see below. One of his more eye-catching tricks is to walk a classroom through the process of writing a Linux driver for a USB thermometer live and in real-time. In addition to all the work he does for Linux in general, he recently announced a program to work with manufacturers and provide Linux drivers for new devices for free. That’s right, manufacturers get a free driver. From the original announcement:
All that is needed is some kind of specification that describes how your device works, or the email address of an engineer that is willing to answer questions every once in a while. A few sample devices might be good to have so that debugging doesn’t have to be done by email, but if necessary, that can be done.
In return, you will receive a complete and working Linux driver that is added to the main Linux kernel source tree. The driver will be written by some of the members of the Linux kernel developer community (over 1500 strong and growing).
That is majorly good karma for Linux and Greg. But if you’re not a manufacturer, here are the steps that you’d look into.
1. Get documentation of the USB protocol, or reverse engineer the protocol. It’s far easier if you can get documentation of the protocol. If you do need to reverse engineer the USB protocol, here are some tools that might help:
Windows tools: USB Snoopy let you do actions with your device and log the stream of USB information going downstream/outbound to the device, or upstream/inbound back to your computer. Snoopy Pro is a variant of the same code that evidently has some improvements. It appears that the preferred location for Snoopy Pro is here.
- The previously mentioned USBView will show you devices that are currently plugged in.
- The usbutils package includes a bunch of handy console tools for USB, including lsusb, which shows you the USB devices that are currently plugged in. The output of lsusb looks like this:
Bus 001 Device 006: ID 2222:3061 MacAlly
Bus 001 Device 002: ID 0557:7000 ATEN International Co., Ltd
Bus 001 Device 003: ID 045e:00db Microsoft Corp.
2. Write the driver.
- It seems that the process of writing drivers in Linux is getting easier over time. The Linux Journal has documented this well. Compare this 2001 article by Greg Kroah-Hartman to Greg’s 2004 article on controlling a simple USB lamp device. Then see Greg’s follow-up article on writing a linux driver in user space. It turns out that you can use the libusb library to read/write with USB devices without ever mucking around in the kernel. This is possible because Linux provides a USB filesystem (called USBFS) that automatically mounts USB devices into the Linux directory tree. Note that libusb also works on BSDs, Mac/OSX computers, and that there is a Windows libusb port.
If you really want to delve into this deeply, there’s an O’Reilly book on Linux device drivers that you can buy as well.