I found myself in a, ahem, lively discussion with someone recently. It started when I said “there was always something wrong about the daily deals businesses (i.e. Groupon), but I’m sure they’ll teach us what’s really needed.” Turns out this person ran a local daily-deals site. Oops.

My feeling is that anytime something doesn’t take root and grow into a lasting business, there’s a lesson to learn. Early social-networking sites weren’t quite a match with needs. More recent ones have gotten something right that the early ones missed. More than one thing needs to be gotten right. In particular, to achieve good user adoption, the value you deliver for the user, what you ask in return, and what the user perceives of both the value and the returned favor are all very delicate balances.

I always felt that these were out of balance in daily-deals businesses because they framed things wrong for the customer. If I get a daily-deals coupon to go try something out, I might be motivated to try it, but my core belief is that if I wasn’t already a customer, I’m not going to become one after my cheapo trial. I think one of the main reasons for that is that the daily-deal coupon sends the signal that the product isn’t worth the usual price. I firmly believe that most people live by “you get what you pay for,” and that this is a two-way street. If you don’t pay a lot, then you don’t think it’s worth a lot.

When I joined Percona, their consulting rate was $200 per hour. Complaints about this high rate were widespread, both externally and internally (consultants thought it was too high also). But something funny was going on: customers wouldn’t keep their appointments with consultants, and they didn’t seem to care. I made it a policy that missed appointments would be billed anyway, and that didn’t change anything. So I experimented with the rates. After trying out various rates for about a month, $300 per hour seemed to be a sweet spot. Serious customers were still willing to pay, we weeded out many bad customers who only caused trouble, and most importantly, a lot fewer appointments were missed.

I experienced something similar in my individual consultancy. I worked on a pro bono basis for a local immigrant health clinic. I had a bad feeling that I was taken for granted, and one day when I was at the clinic early to finish up the project, I stood in the early morning cold for half an hour before I realized that nobody was coming to meet me and let me into the building. After that I stopped doing anything pro bono, and I always had a good feeling that my services were appreciated.

So while arguing about the daily deals sites, I recalled the feeling I always had: that there was something wrong with the business model — that it was undermining what it was supposed to promote. And that vague notion came back to me: the decline of daily deals should teach us what ought to be done instead.

Later, it came to me: A daily-deals offer is like a one-night stand, with no expectation of a long-term relationship. Without this agreement at the foundation of the relationship, the match between the ask and the offer is lopsided, and the “customer” becomes an exploiter, instead of a customer.

And with that, I realized that I had it wrong. Far from being the leading indicator, the daily deals sites were actually behind the times. There have been businesses for many years doing what they should have done. I’ve supported some of them myself. I’m referring to BMG’s music subscription, Omaha Steaks, book-of-the-month clubs, gourmet coffee or wine subscriptions, and so on.

You begin these relationships with several subtly intertwined things, including a good introductory offer, an upfront agreement on the real value of what you’re getting in the trial, and an agreement to be a long-term customer. Oh, sure, you can change your mind and cancel after the introductory offer is up. But agreeing and then canceling is different from never agreeing. There’s probably a lot of psychological research related to what I’m claiming, and I’m sure that some of it contradicts me, but I believe that “buy 10 CDs at $1 each, then we’ll send one per month at full price thereafter, and if you buy X more within a year you get another batch at $1 each” is very different from “you can rip this merchant off by walking out of his store with $1 CDs and no actual or implied obligation.” With BMG’s music subscription service, you opt in and then you have to opt out again (either once-and-for-all, or once every month). And maybe your conscious brain says “I’m going to just cancel after I get my cheap CDs,” but I’m an astute enough observer of myself to know that my subconscious feels differently about the matter. In the back of my mind, I feel like I’m cheating BMG if I cancel before I buy enough CDs at full price to be a profitable customer for them. The conscience gets involved. It never gets involved with a daily deal — there’s no opt-in to opt-out of.

Perhaps what the daily deals sites need to do is bring a platform for this kind of long-term relationship, and managing the logistics, to lots of companies so they don’t have to reinvent it on their own. Perhaps there is a “next time we’ll get this right” for daily deals after all.

On that note, it’s a good thing Gearhart’s Chocolates doesn’t have a BMG-like deal, because you’d have me at “chocolate.” (If you don’t know who they are, it’s a local Charlottesville chocolatier that is easily one of the best in the entire world, and priced to match. I send their variety pack as thank-yous fairly often. Just another perk of living in Charlottesville.)

Last week Tokutek announced that they’re open-sourcing their TokuDB storage engine for MySQL. If you’re not familiar with TokuDB, it’s an ACID-compliant storage engine with a high-performance index technology known as fractal tree indexing. Fractal trees have a number of nice characteristics, but perhaps the most interesting is that they deliver consistently high performance under varying conditions, such as when data grows much larger than memory or is updated frequently. B-tree indexes tend to get fragmented over time, and exhibit a performance cliff when data doesn’t fit in memory anymore.

The MySQL community is excited about having access to TokuDB’s source code, and rightly so. TokuDB is, broadly speaking, aimed at the same category of use cases as Oracle’s InnoDB, which has been MySQL’s leading storage engine for a long time.

MySQL’s market size is large for an opensource product (roughly $500M to $1B USD, depending on who you talk to), and in a big pond, a stone causes wide ripples. I think the most significant implications, though, are for MongoDB. Tokutek has published a series of benchmarks of MongoDB performance with TokuDB as the storage engine instead of MongoDB’s default storage engine. The results are compelling.

I think TokuDB will rapidly become the storage engine of choice for MongoDB, and could catapult MongoDB into the lead in the NoSQL database arena. This would have profound implications for opensource databases of all flavors, not just NoSQL databases.

It’s worth revisiting a bit of ancient history for some context.

Way back in the olden days, MySQL’s main storage engine was MyISAM. MyISAM is non-transactional and has table-level locking, meaning that a write (update, insert, delete, or similar) blocked all concurrent access to the table. This is okay for some uses, and can even be very good in special cases, but in the general case it is a disaster. MyISAM introduced some special workarounds for common cases (such as permitting nonblocking inserts to occur at the end of the table), but in the end, you can’t fix table-level locking. A mixed workload needs storage that’s designed for high read and write concurrency without blocking.

MyISAM had other problems, such as lacking transactions, being prone to data corruption, and long repair times after a crash.

As a result, MySQL as a whole was only interesting to a minority of users. For demanding applications it was little more than a curiosity.

Then came InnoDB. InnoDB introduced ACID transactions, automatic crash recovery, and most importantly, row-based locking and MVCC, which allowed highly concurrent access to rows, so readers and writers don’t block each other. InnoDB was the magic that made MySQL a credible choice for a wide range of use cases.

Most of the interesting chapters in MySQL’s history have involved InnoDB in one way or another. To list some highlights: Oracle bought InnoDB’s creator Innobase Oy, MySQL scrambled to find a replacement (Maria, Falcon, PBXT), Sun’s decision to acquire MySQL was said to be influenced by Falcon, Percona created XtraDB, and Oracle acquired Sun. Things are settling down now, but it’s easy to forget how much of a soap opera the MySQL world has lived through because of InnoDB not being owned by MySQL.

And in the middle of all this came NoSQL databases. In the past half-dozen years, more databases have been invented, popularized, and forgotten than I care to think about. In many cases, though, these databases were criticized as ignoring or reinventing (badly) decades of learning in relational database technology, and even computer science in general. I know I’ve looked at my share of face-palm code.

Databases, by and large, depend on reliable, high-performance storage and retrieval subsystems more than anything else. Many of the NoSQL databases have interesting ideas built on top of bad, bad, bad storage code.

MongoDB is a case in point. MongoDB reinvented some of the worst parts of MySQL all over again. Storage was initially little more than mmap over a file. I think Mark Callaghan put it best in 2009, when he said “Reinventing MyISAM is not a feature.” MongoDB’s storage at that time really was MyISAM-like. It’s improved somewhat since then, but it hasn’t had the wholesale rip-and-replace improvement that I think is needed. Not only that, but MongoDB as a whole is still (predictably) built around the limitations of the underlying storage, with coarse-grained locking.

But MongoDB, like MySQL, has been relevant in spite of these shortcomings. Form your own opinion about why this is, but from my point of view there are two main reasons:

• MongoDB was born in an era when the popular databases were frustratingly slow and clunky to work with, and innovation was stalled due to the political drama surrounding them.
• MongoDB simply feels nice to developers. If you’re not a developer, this is a little hard to explain, but it just feels good, like your favorite pair of jeans. Like a hug from a good friend. Like a hammock and a summer day. The difference between an SQL database and MongoDB for many developers is like the difference between an iPod and a cheap knockoff MP3 player. I could go on and on.

It’s difficult to overstate the importance of this, because it means that MongoDB may well become an enterprise database, despite what bad opinions you may have about it now. Why is this? It’s because developers are king in the modern IT enterprise. Developers determine what technologies get adopted in IT. CTOs like to think the decisions come from the top down, but I’ve seen it work the other way time and time again. Developers start to use something that frustrates them less than the alternatives, and a groundswell begins that’s impossible to stop. Someday the CTO discovers that the question of whether to use technology X was decided by a junior developer long ago and deployed to production, and now it’s too late.

I’ve done it myself. At Crutchfield I hijacked the company-wide policy that migration from legacy VB6 to .NET would proceed along the lines of a transition to VB.NET. I was fighting through awful code day in and day out, and I knew that a more restrictive language would prevent a lot of bad practices. So I wrote several major systems in C# without asking permission. It’s a lot easier to get forgiveness than permission. Then I showed off what I’d done. When I left Crutchfield, the IT department had chosen C#, not VB.NET, as its language of the future (even though there were, and probably still are, major VB.NET applications).

Similarly, at Crutchfield I was provided a 15-inch CRT monitor to work on. This was 2003, you understand. Even at that time, it was awful. How can you expect a developer to work on a flickering, small monitor? I bought my own large-screen LCD and put it in my cubicle. Management ordered me to remove it because it was causing a flood of “hey, how did Baron get a nice monitor?” questions, but the camel already had a nose under the tent. I took my monitor home, but not too long after that we all started to get nicer monitors. I brought my own nice chair to work, too. All told I probably forced Crutchfield to spend thousands of dollars upgrading equipment. You have to be careful about headstrong kids like me — don’t turn your backs on us for a moment.

This story illustrates why MongoDB is likely to become a major database: because developers enjoy working with it. It feels pleasant and elegant. Remember, most technology decisions are based on how people feel, not on facts. We’re not rational beings, so don’t expect the best solution to win. Expect people to choose what makes them happy.

And with the availability of TokuDB, MongoDB is lovable by a lot more people. With reliable storage and transactions, uncool kids can like it too.

It goes further than just the storage engine. The kernel of MongoDB has code that needs to be fixed, such as the coarse-grained locking code. Tokutek basically forked MongoDB in order to insert TokuDB into it. They had to, in order to get all that locking out of the way and allow MongoDB to shine with TokuDB on the backend.

I’m not sure exactly how this will play out — will Tokutek start offering a competitive product? Will there be opensource community-based forks of MongoDB that integrate TokuDB? Will 10gen do the engineering to offer TokuDB as a backend? Will 10gen and Tokutek partner to do the engineering and provide support? Will 10gen acquire Tokutek? Will a large company acquire both? You decide.

But I believe that a few things are inevitable, and don’t require a crystal ball to guess.

Anyone who cares about MongoDB is going to be using TokuDB as their storage backend within a matter of months. It’s happened before — look at what happened to MySQL and InnoDB. Look at Riak; people dropped Bitcask like a hot potato when LevelDB storage arrived (although it hasn’t been a perfect solution).

Just to be clear, I do not think that MongoDB’s parallels with MySQL’s history must inevitably repeat in all aspects of the story. The world of databases today (big data, cloud, mobile) is not in the same situation it was when MySQL was creeping into general awareness (web, gaming, social, general lack of good alternatives to commercial databases), and the reasons people use MongoDB now are different from the reasons people chose MySQL back in the day. Still, there’s a good chance that MySQL’s past can teach us about MongoDB’s future, and for some use cases, MongoDB deployments will soon accelerate rapidly. I expect a larger commercial ecosystem to emerge, too; right now the MongoDB market is worth tens of millions, and I’d guess in a few years we’ll look back and see a sharp inflection point in 2013 and 2014. TokuDB could help propel MongoDB’s market size into hundreds of millions of dollars, which is a position occupied uniquely by MySQL today in the opensource database world.

It’s getting real in the MongoDB world — this is going to be interesting to watch.

This confuses lots of people, including most recently Todd Hoff of HighScalability fame, who wrote in last week’s summary post,

Have to say, this distinction has never made sense to me: Concurrency is not parallelism: concurrency is the composition of independently executing processes, while parallelism is the simultaneous execution of (possibly related) computations. Concurrency is about dealing with lots of things at once. Parallelism is about doing lots of things at once.

I think the problem is that words are hard to understand. The Go blog post is confusing because of that. Pictures are easier. Look, a single-threaded, non-parallel, concurrent process:

Lots of tasks can run on the system, but only one of them makes progress at a time. And here’s one that’s both concurrent and parallel:

Hopefully that clears things up.

Years ago I complained bitterly about MySQL’s backwards development and release model, which made guinea pigs out of the paying customers. I think I’d be remiss if I didn’t say it’s been fixed for years. And it’s really fixed right in my opinion — much better than what I proposed.

Congratulations, and thanks, to the MySQL team for superhuman software engineering, release engineering, documentation, bug triage and analysis, and doing a million things right — in other words, making a damn good database, which is hard. You know I still have gripes occasionally, and so do most people, but in the scheme of things — wow. MySQL is awesome.

We’re ironing out a kink that’s preventing Planet MySQL from aggregating VividCortex’s blog feed, so while that’s in progress, I’ll post a quick note on what we’re up to at VividCortex, for the Planet MySQL readers.

VividCortex is a monitoring and analysis product for MySQL, provided as Software-As-A-Service, with agents that run in your systems and report back to our APIs. The agents are super-efficient and non-obtrusive (you’ve probably noticed my posts about Go recently). They gather high-resolution data about your systems and our web application helps you make sense of it.

VividCortex is shockingly easy to install — if you’re slow at the keyboard, it takes 30 seconds. In less than a minute you can get insight into what your MySQL servers are doing. We are in closed beta right now, with a long waiting list. We’re working with a small set of alpha customers to get the product ready for an open beta.

Our technology and UI focus around four key areas: easy observability of all of your systems (including very large numbers of servers), fluidly browsing and inspecting to swiftly drill into areas of interest, advanced and highly accurate problem detection, and problem diagnosis. In all of these areas we’re truly breaking new ground, unlike anything you can get from legacy monitoring systems, whether commercial or opensource. (Can you think of any existing product that can actually diagnose problems for you?)

One of the enabling technologies is a new kind of fault detection algorithm. We’ve written a series of blog posts on this explaining why it’s needed and how it works.

Feel free to ask me more questions, sign up for our beta, or grab me in the halls at the Percona Live MySQL Conference and Expo. I’d be glad to tell you more.

In 2001 I created a PHP 4 web application framework from scratch as the backbone of a sophisticated application. Back then frameworks weren’t cool. Smarty templates were the hotness.

My framework had URL routing, templates with a capable templating syntax similar to mustache, loosely coupled and tightly cohesive object-oriented design, an elegant way to access the database without dumbing it down, and nicely separated business logic and presentation layers — among many other nice things you find in good frameworks. As the application grew more and more complex, the framework continued to serve well with only occasional enhancements.

I mention this because it illustrates that I’ve been aware of how to design maintainable systems for a long time. But the so-called MVC paradigm (model, view, controller) has never made sense to me.

The design goals make sense. See above. It’s just that I find MVC itself — the abstraction and implementation of those goals — to be unhelpful at best, on a day I’m being charitable. On a day I’m telling it like it is, I’ll just bluntly say MVC is the most confusing pile of acronym bulls**t I can think of.

Tell me, without knowing what MVC is, what would you guess the model represents? Controller? View? It would be bad enough if these were abstract terms like “node” or “resource,” but they’re not abstract. They hint at meaning, and then you discover that MVC doesn’t align with the hinted-at meaning. Instead, the M, V, and C represent concepts that are vague and hard to understand, and are kind of a stretch to mentally fit with the terms. This is worse than abstract, it’s misleading. It’s like bait-and-switch for someone who’s trying to grasp what it means.

I don’t know where MVC came from and I’m sure I won’t change my mind if I find out. I have a hard time believing that I’m the only one who wishes someone had formed a committee[1] to discuss what kind of names to represent the concepts. Especially since there are so many frameworks that abstract things slightly differently, because MVC apparently doesn’t suit their designers either, and thus it’s necessary to explain the relationship between Framework X’s view of the world, and how that loosely maps to the MVC view of the world. It becomes a sloppy comparison to a sloppy analogy.

Even more fun: some folks try to invent an MVC framework, like the original ASP.NET, and “get it wrong,” to disdainful criticism. Then they have to reinvent it, sometimes several times.

It’s kind of like the OSI 7-layer model, which everyone agrees doesn’t actually represent modern networking technology stacks cleanly, and yet everyone talks about the OSI layers. There’s a clean way to think about networking layers, but the OSI model ain’t it. Or NoSQL — the most unhelpful, undescriptive name for an … amorphous category of technologies or worldviews or design patterns or whatever… ever.

The lesson: great concepts that teach great truths, and could result in great advances in engineering due to a common mental model around which other things can be built, can be sidetracked by slapping an ill-fitting monicker or set of analogies on them. And sometimes we never seem to get another chance. NoSQL believers can’t seem to think of anything better than NoSQL as a name (and neither can I, as much as I deplore the NoSQL name). We’re stuck.

I’m sure my strong opinions on this are a) somewhat ignorant and b) not shared by everyone. Discuss! :-)

[1] Yes, I’m aware that awful things come from committees, like SQL. But I have faith that it would have been better than MVC.

I’m scheduled to talk about Go a couple times in the upcoming weeks.

The first is May 4th in Arlington at A Day Of Foster.ly. I’ll be a participant in a panel titled Add To Your Programming Toolkit: Languages You Should Know About (Erlang, Clojure, R, Go, etc…). I’m talking about Go, naturally.

The second May 7th is at my local technology guild, the Neon Guild. I’m going to be giving a “gentle introduction” to Go. I’ve been programming in Go for less than a year, so my first impressions are still fresh. If you are in the Charlottesville area and you haven’t been to a Neon Guild meeting, do yourself a favor and show up. It’s free and no RSVP is required. Description:

It seems that every time I tell someone we’ve chosen Go as our primary language at VividCortex, they respond with something like “I’ve heard about Go! Neat! What is it like? Why did you choose it? Are you happy with it?” The answer is that Go is a fascinating language that’s working great for us. After many years of programming in everything from C to Java to C# to Perl to LISP to Shell, I’m pretty much in love with Go. There are particular reasons I chose it for our special use case, but beyond just niche considerations, it’s working great as a general-purpose, high-performance, very productive systems language. I’ll try to contain my enthusiasm and tell a story about evaluating it, learning it, and then learning it more deeply.

I hope to see you at one of these events. If you see me, wave and introduce yourself, and give me a business card so I can stalk connect with you on LinkedIn!

I use LinkedIn a lot. I also use various Google properties a lot. Both of them have a bunch of distracting and annoying UI features that just get in my way. I know Google and LinkedIn have a vested interest in catching my attention and trying to get me to use their products in the ways that will make them the most money, but as a user, I don’t care what they want. I care what I want.

So I’ve used AdBlock to clean up the elements I don’t want. Here’s how.

I’m assuming you use one of the many excellent AdBlock extensions for whatever browser you use. If not, you need to do that — reclaim your browser and view the web the way you want it. Every time I use someone else’s computer and they don’t have an AdBlock extension it blows my mind.

LinkedIn Endorsements

LinkedIn has a new endorsement feature. They show me a big banner at the top of the page every time I view a connection’s profile page.

I don’t care. I don’t know enough about most of my connections to endorse them publicly, and I don’t want to see that nonsense. So I AdBlock it. This is easy to do — just right-click and select the context menu item to block that whole banner.

I’ve also hidden the endorsements on my profile. It’s nice that people want to endorse me (although I think LinkedIn is almost bullying them into it by putting the obnoxious endorsement box in their faces), but most of the endorsements are for things I’m not actually expert in. Some of them I literally know nothing about. It’s interesting to see the difference between what people think I know and what I actually know.

Google+ Notifications

Somewhere I read that Google+ has quietly become a significantly large social network, with a large fraction of the number of users that Facebook has. I know why that is: it’s because Google has tied a Google+ account into nearly all of their services, and most people I know have multiple Google profiles. I have seven, and a couple months ago I had eight. Google can count me multiple times if they wish; that’s their business.

What’s my business is whether I let them annoy me with notifications. You know that irritating notification bar at the top of the page in all of the pages on Google’s domains? This one?

As far as I can tell, Google doesn’t allow you to disable notifications. The only way I know to get rid of that irritating red animated 1 button is to click on it so it turns back into a faint gray 0:

No offense, guys, I do consider you friends, but I don’t care that you added me to your circles :-\ I don’t use Google+, regardless of what Google tries to make you believe. I bet you were probably bullied into adding me, just to make some annoying suggestion box go away.

The Google+ notifications box is a little harder to hide, because it uses CSS classes to generate the numbers in the box. But a little digging into the page source yielded the following custom AdBlock rules:

google.com##DIV[id*="gbgs"]
google.com##SPAN[id*="gbgs"]

And now all of my Google pages are free of one more time-wasting, attention-disrupting, productivity-killing distraction.

I’m really looking forward to Percona Live this year. I will be there, along with Kyle Redinger, my co-founder at VividCortex. I feel that this year the conference has come full circle. 2007 was my first year at the conference, and it was amazing. This year is tremendously exciting for me because it feels like we’re back on the 2007 trajectory.

Those were the golden days. Things were a little sad in 2010 and 2011 as the MySQL community tried to figure out the new landscape and O’Reilly decided to stop running the event, but then in 2012 we all said “we’re back!” with the exception of Oracle, who instead arranged MySQL Connect, as part of Oracle Open World. I don’t think that was a huge success. I don’t know if that’s why Oracle has decided to participate in this year’s Percona MySQL conference, but frankly it doesn’t matter. What matters is that they’re there. Oracle is a vital part of the MySQL community and if we didn’t have them it would be a huge loss.

I think I already said this in a previous blog post, but the schedule is freaking amazing. I have never seen a conference — any conference — with such a high-quality session schedule. To get this many concentrated high-quality sessions you usually have to attend a mega-conference with dozens of sessions at a time, most of them fluff.

I also have to give Percona a huge thank-you for seeing this through. I was involved — but only slightly — in the 2012 conference, and I can tell you that no outsider can possibly understand the amount of time, money, and work involved. I heard some people say things like “oh yeah, I’ve organized a big event, I know what’s required” followed by descriptions of “cost” and “difficulty” that made me laugh. The conference Percona is organizing is unimaginably more expensive and difficult. If you want to say thank-you, I know they will be glad to hear it. The committee will also. And let’s give credit to the sponsors, too. It’s a ton of effort and expense for all of them.

Are you going to be there? If not, what the heck is wrong with you? Seriously. If you are involved with MySQL at all, if you use it, if your company uses it — you need to be there. If you haven’t been in 2007, 2008, or 2012, there’s no way you can understand what you’re missing. You need to take my word and show up. Percona is making it ridiculously cheap — I don’t know if they’re even making money at the price they’re charging. The value of that week in Santa Clara is hard to calculate. Just go.

High Performance MySQL, 3rd Edition has been selling very well. It’s translated into many languages. O’Reilly sends me a hard-copy of the translations, and I have a whole section on my bookshelf dedicated to them. It’s really satisfying to look at it.

Today I’m happy to announce that we’re moving forward with a new batch of translations. Demand has been so strong that we want to make the book accessible to as wide an audience as possible. Plus, I get a fat check every time O’Reilly sells the translation rights.

The new languages will include Australian, l337 (“Leet”), Jive, Ebonics, Elmer Fudd, Blissymbols, and Esperanto. Here’s a sample before-and-after paragraph:

Isolating the Column

We commonly see queries that defeat indexes or prevent MySQL from using the available indexes. MySQL generally can’t use indexes on columns unless the columns are isolated in the query. “Isolating” the column means it should not be part of an expression or be inside a function in the query.

Here’s the same passage, translated to Australian:

˙ʎɹǝnb ǝɥʇ uı uoıʇɔunɟ ɐ ǝpısuı ǝq ɹo uoıssǝɹdxǝ uɐ ɟo ʇɹɐd ǝq ʇou plnoɥs ʇı suɐǝɯ uɯnloɔ ǝɥʇ ”ƃuıʇɐlosI“ ˙ʎɹǝnb ǝɥʇ uı pǝʇɐlosı ǝɹɐ suɯnloɔ ǝɥʇ ssǝlun suɯnloɔ uo sǝxǝpuı ǝsn ʇ’uɐɔ ʎllɐɹǝuǝƃ ˥QSʎW ˙sǝxǝpuı ǝlqɐlıɐʌɐ ǝɥʇ ƃuısn ɯoɹɟ ˥QSʎW ʇuǝʌǝɹd ɹo sǝxǝpuı ʇɐǝɟǝp ʇɐɥʇ sǝıɹǝnb ǝǝs ʎluoɯɯoɔ ǝM

uɯnloƆ ǝɥʇ ƃuıʇɐlosI

And here’s the sample in Jive:

Them Columns Cut a Lemon fo Isolatin’

Ain’t nothin but a thang bout them messin’ up my old lady’s indexes cain’t be runnin’ upside down yo’ head. Slap my fro. MySQL can’t dig it with lay no indexes on dem less’n you gets ‘em say I won say I pray I get the same ol’ same ol’. Yo SQL, MySQL, all them SQL. What it is, Mama, what it is. Knock yoself a pro slick, get ‘em spreshuns ain’t be togetha. Use yo’ gray mattah! True dat, git it out wid de functions. Come on got to be! Sheeeeeeeh.

There may be some rough edges, of course. This is only an early draft.

In addition, we are translating the technical examples and code samples into additional computer languages, including popular ones like LOLCATS, ALGOL (sorry, not the latest release — that will come soon), and even obscure languages like Node.JS and Commodore 64. We’re also extending the book with compatibility plugins — sort of “skins” or “personalities” if you will — that will let you apply all the knowledge in the book to irrelevant, obscure database servers like Oracle, PostgreSQL (a.k.a. “Postgre”), Riak, and FAT32.

Your feedback and suggestions are welcome. Let me know if there’s anything I can do to help make your High Performance MySQL experience more enjoyable. Or, if you prefer: Slide your jib, brother sky, don’t be sayin’ no off-time jive, lay it on, you dig? Mash me a fin.

I’ve been trying out Republic Wireless, a startup that offers very inexpensive wireless service: $19 for unlimited talk, text, and data. In a nutshell: they resell Sprint’s network, and you agree to connect to wifi as much as possible; they use the Internet instead of the cell network when you’re on wifi. I thought for $19/month it wouldn’t hurt to give it a try. After several months, my experience has been that it isn’t worth using at all, no matter how cheap it is.

I don’t want to dwell on the problems at great length, but here are some of the issues I’ve had.

• The phone is junk. It’s a super-low-quality Motorola Defy XT, which is basically hardware and software (Android) from 3 years ago. It has flaky behavior such as turning off the screen when I try to use the keypad during a call, for example, when I try to press the 7 key to delete a message from voicemail. It also does bizarre things like rotating the screen from landscape to portrait repeatedly and unpredictably when I’m in the middle of trying to use it. It’s a really poor experience in every way; my 3-year-old Droid 2 was much better.
• Sprint’s network is awful. Just awful. Voice quality is terrible. I haven’t been able to actually have a call for more than a couple of minutes at a time. It disconnects or goes into some kind of zombie state where the call still seems to be connected but nobody can hear anything — or only one person’s voice gets through. The phone is literally unusable as a telephone.
• Connecting to wifi doesn’t work. I connect to wifi, I have a strong signal, and their VOIP software uses the cell network instead. It’s incredibly buggy. VOIP calls are better than over-the-air calls, but not much.
• Something is wrong with number routing. I’ve gotten calls from people who were dialing completely different numbers (same area code, but different last 7 digits.) When these people apologized and hung up, then tried calling the number they were trying to reach, they’d get me again. This never happened on Verizon.
• Customer service isn’t merely bad, it’s actually nonexistent. They just use a community forum, and they don’t even answer threads on the forum. See, for example, my thread asking them to make voicemail passwords optional, after they removed passwords from everyone’s voicemail (which is a serious and well-demonstrated privacy/security blunder).
• Text messages stopped working several weeks ago, with basically no notification except for a relatively hidden post in some area of the community forums. Last I looked, there’s no update on whether they’re even trying to fix this.

I’m sure there is more, but that should be enough for now. The summary is that the phone is almost completely unusable for voice calls, completely unusable for text, and practically unusable for anything I’d like to do that requires data (e.g. typing an email) because of the hardware’s flakiness. It’s little more than a small, poorly behaved Android tablet that I paid $250 for.

My wife and I switched from our previous price-gouging phone network at the same time, and both of us have had all the same issues. So this isn’t “just me.” There’s a sample size of at least 2.

Last night at the Golang-DC meetup I spoke about building (MySQL) database applications with Go. The meetup was well attended and people were very enthusiastic about Go. I spent a few minutes talking about Go in general, how VividCortex uses Go (we’ve built our agents, API servers, and all backend processes with Go), why we like it, some of the nice things it enables like making it easy to build very resilient programs, and then I gave the presentation, which I’ve embedded below.

Afterwards the discussion ranged to a lot of related topics. This was the best part of the evening for me. There were really great questions on a variety of topics, and insightful answers from everyone.

I really don’t like running my database in “I Love Garbage” mode, so I set the following SQL_MODE:

STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO, NO_AUTO_CREATE_USER,NO_AUTO_VALUE_ON_ZERO, NO_ENGINE_SUBSTITUTION,NO_ZERO_DATE, NO_ZERO_IN_DATE,ONLY_FULL_GROUP_BY

Guess what WordPress does with that? It doesn’t install. If you set the SQL_MODE to empty and install WordPress, then restore the SQL_MODE, WordPress will run, but if you try to create a post you’ll see an error page that says “You are not allowed to edit this post.”

This problem was reported to WordPress at least 7 years ago. Lessons learned:

• There is a huge amount of software that was built to work with MySQL 3.23′s irritating habit of inserting different data than you told it to, with nothing but a warning most people will never see.
• That software will break in unlovely ways when you try to make MySQL behave more correctly.
• Those who gripe about MySQL’s bugs (as I sometimes do) should remember that MySQL is probably better quality than most of the software that is built to use it. This is probably a universal truth — the Linux kernel is probably better quality than most software that runs in Linux, for example.
• MySQL’s bugs often get fixed faster than aforementioned software’s bugs, too.

I am currently finishing some features to make a program highly resilient to occasional crashing bugs. A particular function was found to crash on queries of the form WHERE x IN(NULL), and that crashed the entire program. Now we have a framework for intelligently recovering from arbitrary crashes. I will write more on this in the future, because I think it’s a very interesting thing to share.

In this episode, I want to focus on a related topic: how do you test a program that is supposed to be resilient to bugs you can’t predict? Many new problems are caused by writing clever code that is supposed to detect, avoid, or recover from problems, even known problems. Unknown problems are even riskier.

The approach that has given me a great deal of confidence in the “resilience framework” is to write a “crash injection framework.” (If it’s not a framework, it can’t be any good, right?) Crash injection can be activated at runtime to cause crashes in arbitrary source code locations. A fairly simple analysis of the source code showed me some places I’m particularly interested in crashing to see how the program reacts.

In the first iteration, it turned out that I hadn’t actually solved the problem, even though I was sure I had. (I had already discovered previous mistakes, so this was actually about the 3rd time through, and I was sure I’d found all the bits I’d previously missed.) My crash injection framework allowed me to quickly dispel the notion that my program was resilient to the kinds of bugs we’d already found.

The fix was simple, but the lesson was well learned. I don’t remember doing this before, but I’ve studied and admired other pieces of software that are built to be resilient. For example, CouchDB has a crash-only design. If you’re not familiar with that phrase, it’s well worth reading more about that. Stewart Smith has also written software that helps reveal wrong assumptions, such as libeatmydata, and I think he also wrote something that randomly broke malloc(), but I can’t find that easily. Assumptions like “allocating memory will always succeed” are great places to end up with horrible bugs.

In the MySQL world, it would be great if we had more crash-injection testing of InnoDB as well. If you agree, you might consider subscribing to http://bugs.mysql.com/bug.php to keep up to date with effort on that front.

The technology industry moves incredibly fast, from one bubble to another. Web 2.0. Online auctions. (Remember when the Internet was filled with hundreds of eBay clones?) Social. Mobile. Location-based. Big Data. Whatever.

I don’t think anyone will call me insightful for observing that the general idea of “social” had a peak in its hype cycle some time ago. I’d say three years ago was really the peak. At some point, lots of people were excited about applying social-ness to everything. Social was going to be the way everything happened in the future. Businesses were going to revolutionize the way internal communications happened by making everything social. Government was going to be more transparent by being social. We were all going to be constantly tuned in to all this frenzy of everything happening — events, information, whatnot — and it would all be saturated with socialness. (This sounds like a new level of hell to me, but some people thought it was a good thing.)

There were basically no bounds to what social could do. IT was going to get more agile through things like social monitoring, for example, like Nodeable. It’s since been renamed and pivoted entirely, but what I remember was that people chit-chatting and promoting/liking/recommending and having relevance-based feeds about system problems was going to improve the signal-to-noise ratio of monitoring alerts, and allow IT staff to focus on stuff that really needed to be fixed.

Social became a golden hammer. Got a social hammer? Everything looks like a social nail. Bang, bang, bang. Solve it with social, get rich with social.

Personally, I have no use for most social media. I tried Facebook for a few months. It’s a platform for little info-bites that are distracting, pleasing, fun, endorphin-releasing. But in my opinion, it is not a way to meaningfully engage with people I really care about on a deep level. So I deactivated my account. Google is trying to go the same route with Google+ and I have no use for that either. The idea that a crowd of people babbling is going to produce something relevant and actually valuable for me, is not something I agree with. Both Facebook and Google+ are this way. I would much rather pull my content from carefully selected places than have it pushed to me by algorithms and the so-called “wisdom of crowds.”

In other words, most social media seems to me to be more about popularity than relevance, value, or meaningfulness. I truly do not care what other people think about pictures or events or celebrities or news or whatever. Their opinions have a very small chance of matching mine. Opinion-based social is just another name for popularity-based social. I have observed that the things I value the most are often the least popular. Just look at how most world governments are run and it’s easy to see that crowd-pleasing things and sensible things are pretty much mutually exclusive.

But there is more than one way to do social.

There’s real, enduring value in the concepts of social. Take a look at Github. It isn’t promoted as “the social coding platform where code gets social and social is how the code is socialized,” but their TITLE tag talks about making better code together. Use their service and what do you do? You have conversations in issues. You have comments on commits and pull requests. You can @mention people. You interact with other coders via code, code itself becomes conversational, and coders have meta-conversations on those conversations. It’s totally social. Heck, Git itself, by its very design, is social.

But Github is not overtly social. That’s the difference. They don’t act as if social is first, and code hosting and issue tracking is secondary. They don’t act as if the core value proposition is socialness and code hosting is just the medium through which they do it.

I think that’s the right way to do it. Take a solid business idea — code hosting and revision control and so on — and add the ability for people to have meta-interactions around it, where it makes sense and actually enriches the core of the business.

One of the most rewarding things you can do is help someone get a great job or hire a great person for the position they need to fill. I have traveled a lot, written books, done a bunch of consulting, and spoken widely on MySQL, other databases, open source, and so forth. I’ve gotten to know a lot of people, some I’d call good friends, and many of them are leading large organizations. I think this is both a privilege and a serious responsibility.

It’s a privilege because I can ask some of these people for help or introductions or advice sometimes. It’s a responsibility because I need to be ready to do something for them, too. In many cases it’s a pay-it-forward kind of readiness.

Many, many people contact me looking for people to hire. I keep a list. When someone tells me they are on the job market, I try to match them with openings I’m aware of, if any are appropriate. (Many fewer people tell me they’re looking for jobs than tell me they’re trying to recruit.)

These are my friends, on both sides. I don’t introduce people unless I think everyone is really genuinely bringing something great to the table. A great position — a great talent — the right circumstances and attitudes.

When I do introduce people, I usually do it one-sidedly, depending on the situation. For example, I’ll forward the candidate’s resume and contact information sometimes, with a few words of introduction. At other times I’ll just BCC the candidate. Or I might send the recruiting party’s information to the candidate.

People usually take this very seriously, but occasionally I’ve referred several people to a company only to find out that the company didn’t follow up on the introduction. That’s pretty much the last chance that company will get from me. If I made a special personal effort to send you someone fantastic who’s really qualified for the job, and you didn’t contact that candidate, there’s really no excuse for such rudeness. You’re telling me you don’t think my experience and judgment about who’s good for the job is valid. I won’t offer it again, then.

Recruiters are an interesting lot. They have a bad reputation, and I think that’s because a few aggressive, pushy people poison the well for the rest of the recruiters. Most recruiters are really good people doing a hard and extremely valuable job, with a lot of care and personal attention to their client as well as their candidates. If you’ve ever built a team, you know how hard it can be. Many recruiters are used to being treated very badly because of their profession’s reputation. I try to make up for that by being extra nice to them. Recruiters who contact me by email often get a personal call in return, and always a personal email reply with a thank you. In almost all cases, recruiters will tell me who their client is after I’ve introduced myself. I only refer my friends to recruiters when I know who they’re recruiting for, and I never make an end run around recruiters. They are doing a mostly thankless job, and they should not be cheated out of their commission.

Over the years I’ve been lucky enough to help a half dozen friends find jobs, and there’s pretty much no better feeling. If you’re looking, let me know. I know really good people who are looking for you. I’m never too busy to help make introductions.

I was a guest on the Food Fight Show last week, along with a bevy of really smart people asking and answering tough questions on fault detection. We didn’t talk a lot about MySQL, but given that VividCortex is focusing on MySQL initially, pretty much all of my experience with zero-threshold, zero-configuration fault detection is MySQL-based.

It’s a fun conversation with a lot of insights into the industry, what’s wrong with current monitoring tools, and where monitoring is going. Also, it’s sold out now, but Monitorama is a conference you might be interested in if you’re doing monitoring (and who isn’t?)

Here is a collection of upcoming events that are interesting to me as a MySQL user (in some cases because I’m speaking). I think some of them are must-see events. I am sure I am missing a lot of events, but some of these are only publicized in specific channels, and I wanted to mention them here to help spread the word.

Upcoming Meetups

This coming Thursday, Charlottesville’s Neon Guild tech group will co-host a Meetup with a noted Kanban expert.

I’m joining two Meetups soon to talk about building database (MySQL) applications with the Go programming language:

• Wednesday, March 13, 2013 in Charlottesville Virginia
• Tuesday, March 19, 2013 in Alexandria, Virginia

Upcoming Conferences

• Percona Live MySQL Conference and Expo 2013 in Santa Clara, CA April 22-25, 2013. This is the annual MySQL conference. I’m not speaking this year, but…
• Velocity Web Performance And Operations Conference June 18-20, Santa Clara, CA. This is an amazing event that brings together the best and brightest people who are building the future of the Internet. I am speaking on quantifying system normality.

Upcoming Call For Proposals

• OmniTI Surge, Maryland, Sep 12-13. CfP closes May 1st.
• Velocity New York 2013 Call for Participation (Oct 14-16). Proposals Due Apr 29th.

I’ve been increasingly questioning the current model of university education in the US. Not only the value for the money, but just the entire notion that it’s a good way to learn. I got my Bachelor’s in Computer Science from UVA, which has been going through utter facepalm-worthy madness recently. It may be biasing my point of view.

A friend recently sent me this:

This shop was written up in the WSJ last week: http://devbootcamp.com My daughter is looking at it. What do you know about them, if anything? What do you think?

Here’s how I responded:

I don’t know any of them personally. That looks like a much better use of my time than the 5 years I spent getting my CS degree. The curriculum covers a lot of good material. Someone who goes into that with zero programming background will likely come out of it with enough knowledge and context to start doing something productive, but no awareness of deep underlying principles and fundamentals. That is the exact opposite of UVA’s CS curriculum. People often see someone do silly things due to lack of CS background, and say you should have the fundamentals first and then learn the specifics, but I’m not convinced that isn’t elitist BS. It strikes me that an apprenticeship model might actually be a better way to learn than the ivory-tower-then-someday-you-will-code way.

It’s only 9 weeks. What does she have to lose other than 9 weeks and some cash? She could stand to gain tremendously.

The current model of higher education in the US currently seems completely broken to me, and I’m really interested to see how some of the alternatives shake out. One thing is for sure: higher education, and the university model in particular, is about to get disrupted. Hard. Like being hit by a freight train.

I was discussing how to avoid surprising users and someone pointed out that what seems intuitive and rational to one person is often complete insanity for others. The mental gap between a developer and a user can often be a chasm far too wide to cross. Of all the bug reports I’ve filed against MySQL, here is my all-time favorite:

select * from t where a >= 1.0order by a;

Does not cause an error. I believe it should, because there should be a whitespace before ORDER BY.

Similar syntax errors such as “select 1e0from dual” were also accepted as valid SQL. Much soul-searching later, the official reply from MySQL’s development team:

The server behaves properly here:
- “1″ alone can not be an identifier, because it’s followed by a “.”
- therefore, the lexer parses “1″ as the beginning of a number, and ends up with “1.0″ as one token.
The next token will be “order”, a keyword.

There is nothing special to document here either. What was reported is not a bug in the server, and is not a documentation bug.

What just happened here? It’s fairly simple. Never attribute to malice that which can be adequately explained by tunnel vision. The developer decided that the SQL isn’t a syntax error because the lexer is designed to parse it. If the developer were able to stand in a user’s shoes for a moment, this answer would be obviously absurd. But when you live deep in the dungeons of a large codebase, you lose the ability (despite your best intentions) to see a normal user’s point of view.

For this reason I often doubt my own judgment. I’m ignorant of my ignorance. It’s a very “meta” problem. I have to rely on others to give me a sanity check.

I suggest, as a topic of discussion, that the willingness and ability to accept others’ judgment over one’s own is an essential characteristic of good leadership. I also suggest that knowing when this is appropriate is another attribute a good leader must have. Both of these are very difficult in my experience.