• Shortcuts : 'n' next unread feed - 'p' previous unread feed • Styles : 1 2

» Publishers, Monetize your RSS feeds with FeedShow:  More infos  (Show/Hide Ads)

Date: Tuesday, 26 Aug 2014 20:15
Many people believe the Holy Grail of secure isolation is to use two or more physically separate machines. This belief seems so natural, that we often don't give it much thought. After all, what better isolation could we possible get than physical "airgap"?

I argue with this point of view in this new paper.

I think a good place for in-depth technical discussions around the topics discussed in the paper would be our qubes-devel mailing list.
Author: "Joanna Rutkowska (noreply@blogger.com)"
Send by mail Print  Save  Delicious 
Date: Wednesday, 06 Aug 2014 14:02
Today we're release the second release candidate (rc2) for Qubes OS R2. There are currently no more open tickets for the final R2 release, and we hope that what we release today is stable enough and so will be identical, or nearly identical, to the final R2 ISO, which we plan to release after the summer holidays. Download and installation instructions are here.

After Qubes rc1 release a few months ago we have been hit by a number of problems related to unreliable VM start-ups. The most prevalent problem has been traced down to an upstream bug in systemd, which just happened to be manifesting on Qubes OS due to specific conditions imposed by our startup scripts.

Actually, it has not been the first time when some things related to VM bootup or initialization didn't work quite well on Qubes, a side effect of heavy optimizations and stripping down we do in order to make the VMs as light weight as possible. E.g. we don't start most of the Desktop Environment which otherwise is assumed to be running by various desktop-related applications and services. In most cases these are really NOTOURBUG kind of problems, yet we just happen to be unlucky they manifest on Qubes. We do need more help from the community with testing, debugging and patching such NOTOURBUG problems in the upstream. The more people use Qubes OS, the higher the chances such problems will be addressed much quicker. Ideally, in the future, we could partner with a Linux distro that would include Qubes AppVM as one of the test cases.

Speaking of different Linux distros --  we have also recently built and released an experimental (“beta”) Debian template for Qubes AppVMs, a popular request expressed by our users for quite some time. It can be readily installed with just one command, as described in the wiki. It is supposed to behave as a first class Qubes AppVM with all the Qubes signature VM integration features, such as seamless GUI virtualization, secure clipboard, secure file copy, and other integration, all working out of the box. Special thanks to our community contributors for providing most of the patches required for porting of our agents and other scripts to Debian. This template is currently provided via our templates-community repo, but it nevertheless has been built and signed by ITL, and is also configured to fetch updates (for Qubes tools) from our server, but we look forward for somebody from the community to take over from us the maintenance (building, testing) of the updates for this template.

Also in our "Templates Appstore" you can find now an experimental “minimal” fedora-based template, which might be used by more advanced users to build customized special-purpose VMs and templates.

We have also moved our Wiki server to a bigger EC2 instance so it could better handle the increased traffic and also added a real CA-signed SSL certificate! But I encourage people to read why this is mostly irrelevant from the security standpoint and why they should still be checking signatures on the ISOs.

We also got a new logo (actually we never really had our own logo before). This also means Qubes now got its own distinct set of themes for installer, plymouth and, of course, a bunch of cool wallpapers with Qubes logo nicely engraved on them. However, it turned out that convincing KDE to set our wallpaper as a default one exceeds the collective mental abilities of ITL, and so one needs to right-click on the desktop and choose one of the Qubes-branded wallpapers manually after install or upgrade.

Every once in a while people (re-)discover that monolithic kernel-based desktop operating systems are not the best solution whenever the user even remotely cares about security...

Yes, USB inherent insecurity, as well as widespread GUI insecurity, or networking stack insecurity, trivial physical insecurities, or sick permissions model as used in most desktop systems, have all been known facts for years. The recognition of these problems has been the primary motivator for us to start the work on Qubes OS back in 2009/2010.

And yes, Qubes running on an appropriate hardware (specifically with Intel VT-d) can solve most of these problems. Correction: Qubes OS can allow the user or administrator to solve these problems, as unfortunately this still requires some configuration decisions made by the human operator. So today Qubes R2 is like a sports manual transmission, which requires a bit of skill to get most out of it. In the near future I see no reason why we should not be offering the "automatic 8-speed transmission" edition of Qubes OS. We just need more time to get there. The R3 release (Odyssey-based), whose early code is planned to be released just after the "final" R2, so sometime in September, is all about bringing us closer to that "automatic transmission" version.

With my 10+ years of experience as a system-level security researcher, I believe there is no other way to go. Don't get deluded that safe languages or formally verified microkernels could solve these problems. Security by Isolation, done sensibly, is the only way to go (of course it doesn't preclude making use of some formally verified components, like e.g. microkernel in place of Xen, at least in some editions of Qubes).

Finally one more announcement for today: after writing this blog for 8 years, I've suddenly felt like I might need to try also some new form of expression... And so, for a few days, I now have a twitter account (@rootkovska), which I hope to use for updates on Qubes, as well as more general commentary on various things happening in IT security.

Author: "Joanna Rutkowska (noreply@blogger.com)" Tags: "os security, qubes, usb"
Send by mail Print  Save  Delicious 
Date: Sunday, 20 Apr 2014 21:40
Today we're releasing Qubes OS R2 rc1 (release candidate), which is expected to be the last milestone before the final Qubes OS R2 release. As mentioned previously today's release is bringing mainly UI improvements and polishing and lots of bugfixes, as well as some last new features: 
  • Both Dom0 and VMs have been upgraded to Fedora 20.
  • Support for full templates download via two new repo definitions: templates-itl and templates-community. With a bit of imagination we could call it Qubes “AppStore” for VMs :) Currently we have only published one template there – the new default fc20-based template, but we plan to upload more templates in the coming weeks (such as the community-produced Arch Linux and Debian templates). Even though we have a separate repo for community contributed templates, we still plan on building those templates ourselves, from (contributed) sources.
  • Support for running Windows AppVMs in “full desktop” mode with support for arbitrary window resizing (which automatically adjusts the resolution in the VMs).
  • Support for on-the-fly switching between the “full desktop” and “seamless” modes for Windows AppVMs.
The last two features require, of course, our proprietary Qubes Windows Tools to be installed in the Windows AppVMs to work, which new version we have also published to the new repositories for R2rc1.

We support smooth upgrading for current Qubes R2 Beta 3 users – the procedure is very simple, yet it will take some hours because of the Dom0 distro upgrading.

As can be seen in our ticketing system, there really are only few minor cosmetic tasks left before the final Qubes R2 release. It is expected that upgrade from today's release to the final R2 will be very simple and quick – just standard updates installation.

As usual, the detailed installation and upgrade instructions, as well as the HCL, can be found here. Note however, that the HCL for the today's release will take some days/weeks to compile, as we need to wait for reports from the community, and so for this time the HCL for the previous release (R2 Beta 3) should be used instead. It is reasonable to expect that the new HCL will be a subset of the previous one.

Also, as usual, please keep in mind that we don't control the servers from which the ISO is being served and so please always make sure to verify the digital signature on the downloaded ISO before installing it.

Please direct all the technical questions or comments regarding Qubes OS to our mailing lists.

Author: "Joanna Rutkowska (noreply@blogger.com)"
Send by mail Print  Save  Delicious 
Date: Thursday, 16 Jan 2014 00:32
When I originally described the flexible Qubes Odyssey framework several months ago, I mentioned that we would even consider to use “Windows Native Isolation” mechanisms as a primitive type of isolation provider (“hypervisor”) for some basic edition of Qubes for Windows. The idea has been very attractive indeed, because with minimal effort we could allow people to install and run such Qubes WNI on their normal, consumer Windows laptops.

Sure, the inter-process isolation provided by a monolithic kernel such as Windows or Linux could never be compared to the inter-VM isolation offered even by the most lousy hypervisors. This is simply because the sizes of the interfaces exposed to untrusted entities (processes in case of a monolithic kernel; VMs in case of a hypervisor) are just incomparable. Just think about all those Windows system calls and GDI calls which any process can call and which contains probably thousands of bugs still waiting to be discovered by some kid with IDA. And think about those tens of thousands of drivers, which also expose (often unsecured) IOCTLs, as well as parsing the incoming packets, USB devices infos, filesystem metadata, etc. And then think about various additional services exposed by system processes, which are not part of the kernel, but which are still trusted and privileged. And now think about the typical interface that needs to be exposed to a typical VM: it's “just” the virtualized CPU, some emulated devices (some old-fashined Pentium-era chipset, SVGA graphics adapter, etc) and virtualized memory.

Anyway, knowing all this, I still believed that Qubes WNI would make a whole lot of sense. This is because Qubes WNI would still offer a significant boost over the “Just Windows” default security, which is (still) essentially equivalent to the MS-DOS security model.  And this is a real pity, because Windows OS has long implemented very sophisticated security mechanisms, such as complex ACLs applicable to nearly any object, as well as recent mechanisms such as UIPI/UAC, etc. So, why not use all those sophisticated security to bring some real-world security to Windows desktops!

And, best of all, once people start using Qubes WNI, and they liked it, they could then pretty seamlessly upgrade to Xen-based Qubes OS, or perhaps Hyper-V-based Qubes OS (when we implement it) and their system would look and behave very similarly. Albeit with orders of magnitude stronger security. Finally, if we could get our Odyssey Framework to be flexible enough to support both Qubes WNI, as well as Xen-based Qubes OS, we should then be able to support any hypervisor or other isolation mechanism in the future.

And so we decided to build the Qubes WNI. Lots of work we invested in building Qubes WNI was actually WNI-independent, because it e.g. covered adjusting the core Odyssey framework to be more flexible (after all “WNI” is quite a non-standard hypervisor) as well as some components that were Windows-specific, but not WNI-specific (e.g. could very well be used on Hyper-V based Qubes OS in the future). But we also invested lots of time into evaluating all those Windows security mechanisms in order to achieve our specific goals (e.g. proper GUI isolation, networking isolation, kernel object spaces isolation, etc)...

Sadly this all has turned out to be a story without a happy end, as we have finally came to the conclusion that consumer Windows OS, with all those one-would-think sophisticated security mechanisms, is just not usable for any real-world domain isolation.

And today we publish a technical paper about our findings on Windows security model and mechanisms and why we concluded they are inadequate in practice. The paper has been written by Rafał Wojdyła who joined ITL a few months ago with the main task of implementing Qubes WNI. I think most people will be able to learn a thing or two about Windows security model by reading this paper.

Also, we still do have this little hope that somebody will read the paper and then write to us: “Oh, you're guys so dumb, you could just use this and that mechanism, to solve all your problems with WNI!” :)

The paper can be downloaded from here.
Author: "Joanna Rutkowska (noreply@blogger.com)"
Send by mail Print  Save  Delicious 
Date: Wednesday, 11 Dec 2013 00:14
P { margin-bottom: 0.08in; }A:link { }
Today we're releasing Qubes R2 Beta 3, one of the latest milestones on our roadmap for Qubes R2. Even though it is still called a “beta”, most users should install it, because, we believe, it is the most polished and stable Qubes edition. Looking back, I think it was a mistake to use this alpha/beta/rc nomenclature to mark Qubes releases, and so, starting with Qubes R3 we will be just using version numbers: 3.0, 3.1, etc.

Anyway, back to the R2 Beta 3 – below I discuss some of the highlights of the today's release:
  • The seamless GUI virtualization for Windows 7-based AppVMs, and support for HVM-based templates (e.g. Windows-based templates) is one of the most spectacular feature of this release, I think. It has already been discussed in an earlier blog post, and now instructions have also been added to the wiki for how to install and use such Windows AppVMs.

  • We've also introduced a much more advanced infrastructure for system backups, so it is now possible to make and restore backups to/from untrusted VMs, which allows e.g. to backup easily the whole system to a NAS, or just to an USB device, not worrying that somebody might exploit the NAS client over the network, or that plugging of the USB disk with malformed partition table or filesystem might compromise the system. The whole point here is that the VM that handles the backup storage (and which might be directing it to a NAS, or somewhere) might be compromised, and it still cannot do anything that could compromise (or even DoS) the system, neither can it sniff the data in the backup. I will write more about the challenges we had to solve and how we did it in a separate blog post. I'm very proud to note that majority of the implementation for this has been contributed by the community, specifically Oliver Medoc. Thanks!
  • A very simple feature, trivial almost, yet very important from the security point of view – it is now possible to set 'autostart' property on select VMs. Why is this so important for security? Because I can create e.g. UsbVM, assign all my USB controllers to it, and then once I set it as autostarting, I can have assurance that all my USB controllers will be delegated to such AppVM immediately upon each system boot. Having such a UsbVM is a very good idea, if one is afraid of physical attacks coming though USB devices. And it now could double as a BackupVM with this new backup system mentioned above!
  • To improve hardware compatibility we now ship the installer with multiple kernel versions (3.7, 3.9, and 3.11) allowing to run the installation using any of those, e.g. if it turned out that one kernel doesn't support the graphics card correctly -- a typical problem many users faced in the past. All the kernels are also installed in the final system, allowing the user to easily boot with a select Dom0 kernel later, choosing the one which supports their hardware best.

  • Another popular problem of the past now was the lack of support for dynamically changing resolution/screen layout in the AppVMs when a seccond monitor or a projector was hot-plugged in (which changed only the resolution layout in Dom0). Now this problem has been solved and the new monitor layout is dynamically propagated to the AppVMs, allowing to use all the screen real estate by the apps running there.
  • There has also been a significant amount of cleanups and fixes. This includes the unification of paths and command names (“The Underscore Revolution” as we call it), as well as refactoring of all the source code components (which now closely matches what we have on Qubes Odyssey/R3), and lots of various bugfixes.
We're planning one more release (Qubes R2 RC1) before the final R2, which will bring improvements mostly in the area of more polished UI, such as allowing some of the tasks that currently require commandline to be done from the Qubes Manager. So, this would mostly be a minor cosmetic upgrade, plus bugfixes. And probably we will also upgrade the default Linux template to Fedora 20.

Installation and upgrade instructions can be found here.
Author: "Joanna Rutkowska (noreply@blogger.com)"
Send by mail Print  Save  Delicious 
Date: Wednesday, 27 Nov 2013 02:06
P { margin-bottom: 0.08in; }A:link { }
Finally, after months of hard work, seamless mode for Windows 7 AppVMs is coming to Qubes OS! The new Windows Support Tools will be released together with the Qubes OS R2 Beta 3, which we plan to release in the next 1-2 weeks. Here is an obligatory screenshot showing a few Windows apps running in seamless mode integrated onto Qubes trusted desktop (note the usual Qubes trusted decorations around each of the Win7 windows):

P { margin-bottom: 0.08in; }A:link { }
The seamless mode for Windows AppVMs is not yet as polishedas the one we have for Linux AppVMs, because, unlike what we do for Xorg, the Windows GUI agent is not based on composition buffers extraction. This causes some, ratherminor, cosmetic problems. For example, when we have two overlapping windows from a Win7 AppVM, and move the top window away, its remaining "shadow" will be visible on the underlying window for the duration of the operation. But generally this all works reasonably good, and you should not really feel any slowness or heaviness compared to Linux AppVMs virtualization. It should be noted that we managed to add this seamless support for Windows AppVMs without any changes to our secure GUI virtualization protocol.

Of course, the usual Qubes integration features, such as secure inter-VM clipboard and file copy also work for Windows AppVMs with the tools installed.

The Qubes Windows Support Tools are proprietary, but they are supposed to be installed only in the Windows 7 VMs, which themselves contain millions of lines of proprietary code already. Besides that, the tools do not introduce any other modifications to the system.

As a special bonus we have also added (and releasing also in R2B3) the support for template-based HVMs. So it will now be possible to do something like this:

P { margin-bottom: 0.08in; }A:link { }
qvm-create --hvm work-win7 --template win7-x64 --label green
qvm-create --hvm personal-win7 --template win7-x64 --label purple
qvm-create--hvm testing-win7 --template win7-x64 --label red

P { margin-bottom: 0.08in; }A:link { }... tellingQubes to createthree HVM AppVMs based on the same template.

All such template-based AppVMs use the root filesystem from the Template VM, which is shared in a read-only manner, of course, but Qubes makes it look for the AppVMs as if the root filesystem was writable.Just like in case of Linux AppVMs, the actual writes are stored in COW buffersbacked by files stored in each of the AppVMs directories. Upon AppVM's reboot, thosefilesarediscarded, which reverts the VMs' root filesystems back to that of the template (the “golden image”).

For the above mechanism to make any sense we should configure the OS in the Template VMto usea separate disk for the user's home directory(ies) (e.g. C:\Users in case of Windows). Qubes automatically exposes an additional private disk to each of the AppVMs exactly for this very purpose. Again, just like it has been done for Linux AppVMs for years.

The above feature allows to create lots of Windows AppVMs quickly and with minimal use of disk space, and with an ability to centrally update all the system software in all the AppVMs all at once. Just like for Linux AppVMs.

Users should, however, ensure that their license allows for such instantiating of the OS they use in the template. Note that from the technical point of view the OS is installed, and, in case of Windows, also activated, only once: in the template VM. The installed files are never copied, they are only shared with the running instances of AppVMs. Consult your software licensing lawyer.
Author: "Joanna Rutkowska (noreply@blogger.com)"
Send by mail Print  Save  Delicious 
Date: Monday, 23 Sep 2013 20:35
P { margin-bottom: 0.08in; }
In the first part of this article published a few weeks ago, I have discussed the basics of Intel SGX technology, and also discussed challenges with using SGX for securing desktop systems, specifically focusing on the problem of trusted input and output. In this part we will look at some other aspects of Intel SGX, and we will start with a discussion of how it could be used to create a truly irreversible software.

SGX Blackboxing – Apps and malware that cannot be reverse engineered?

A nice feature of Intel SGX is that the processor automatically encrypts the content of SGX-protected memory pages whenever it leaves the processor caches and is stored in DRAM. In other words the code and data used by SGX enclaves never leave the processor in plaintext.

This feature, no doubt influenced by the DRM industry, might profoundly change our approach as to who controls our computers really. This is because it will now be easy to create an application, or malware for that matter, that just cannot be reversed engineered in any way. No more IDA, no more debuggers, not even kernel debuggers, could reveal the actual intentions of the EXE file we're about to run.

Consider the following scenario, where a user downloads an executable, say blackpill.exe, which in fact logically consists of three parts:

  1. A 1st stage loader (SGX loader) which is unencrypted, and which task is to setup an SGX enclave, copy the rest of the code there, specifically the 2nd stage loader, and then start executing the 2nd stage loader...
  2. The 2nd stage loader, which starts executing within the enclave, performs remote attestation with an external server and, in case the remote attestation completes successfully, obtains a secret key from the remote server. This code is also delivered in plaintext too.
  3. Finally the encrypted blob which can only be decrypted using the key obtained by the 2nd stage loader from the remote server, and which contains the actual logic of the application (or malware).

We can easily see that there is no way for the user to figure out what the code from the encrypted blob is going to do on her computer. This is because the key will be released by the remote server only if the 2ndstage loader can prove via remote attestation that it indeed executes within a protect SGX enclave and that it is the original unmodified loader code that the application's author created. Should one bit of this loader be modified, or should it be attempted to run outside of an SGX enclave, or within a somehow misconfigured SGX enclave, then the remote attestation wouldfail and the key will not be obtained.

And once the key is obtained, it is available only within the SGX enclave. It cannot be found in DRAM or on the memory bus, even if the user had access to expensive DRAM emulators or bus sniffers. And the key cannot also be mishandled by the code that runs in the SGX enclave, because remote attestation also proved that the loader code has not been modified, and the author wrote the loader specifically not to mishandle the key in any way (e.g. not to write it out somewhere to unprotected memory, or store on the disk). Now, the loader uses the key to decrypt the payload, and this decrypted payload remains within secure enclave, never leaving it, just like the key. It's data never leaves the enclave either...

One little catch is how the key is actually sent to the SGX-protected enclave so that it could not be spoofed in the middle? Of course it must be encrypted, but to which key? Well, we can have our 2ndstage loader generate a new key pair and send the public key to the remote server – the server will then use this public key to send the actual decryption key encrypted with this loader's public key. This is almost good, except for the fact that this scheme is not immune to a classic main in the middle attack. The solution to this is easy, though – if I understand correctly the description of the new Quoting and Sealing operations performed by the Quoting Enclave – we can include the generated public key hash as part of the data that will be signed and put into the Quote message, so the remote sever can be assured also that the public key originates from the actual code running in the SGX enclave and not from Mallory somewhere in the middle.

So, what does the application really do? Does it do exactly what has been advertised by its author? Or does it also “accidentally” sniffs some system memory or even reads out disk sectors and sends the gathered data to a remote server, encrypted, of course? We cannot know this. And that's quite worrying, I think.

One might say that we do accept all the proprietary software blindly anyway – after all who fires up IDA to review MS Office before use? Or MS Windows? Or any other application? Probably very few people indeed. But the point is: this could be done, and actually some brave souls do that. This could be done even if the author used some advanced form of obfuscation. Can be done, even if taking lots of time. Now, with Intel SGX it suddenly cannot be done anymore. That's quite a revolution, complete change of the rules. We're no longer masters of our little universe – the computer system – and now somebody else is.

Unless there was a way for “Certified Antivirus companies” to get around SGX protection.... (see below for more discussion on this).

...And some good applications of SGX

The SGX blackboxing has, however, some good usages too, beyond protecting the Hollywood productions, and making malware un-analyzable...

One particularly attractive possibility is the “trusted cloud” where VMs offered to users could not be eavesdropped or tampered by the cloud provider admins. I wrote about such possibility two years ago, but with Intel SGX this could be done much, much better. This will, of course, require a specially written hypervisor which would be setting up SGX containers for each of the VM, and then the VM could authenticate to the user and prove, via remote attestation, that it is executing inside a protected and properly set SGX enclave. Note how this time we do not require the hypervisor to authenticate to the users – we just don't care, if our code correctly attests that it is in a correct SGX, it's all fine.

Suddenly Google could no longer collect and process your calendar, email, documents, and medial records! Or how about a tor node that could prove to users that it is not backdoored by its own admin and does not keep a log of how connections were routed? Or a safe bitcoin web-based wallet? It's hard to overestimate how good such a technology might be for bringing privacy to the wide society of users...

Assuming, of course, there was no backdoor for the NSA to get around the SGX protection and ruin this all goodness...(see below for more discussion on this).

New OS and VMM architectures

In the paragraph above I mentioned that we will need specially written hypervisors (VMMs) that will be making use of SGX in order to protect the user's VMs against themselves (i.e. against the hypervisor). We could go further and put other components of a VMM into protected SGX enclaves, things that we currently, in Qubes OS, keep in separate Service VMs, such as networking stacks, USB stacks, etc. Remember that Intel SGX provides convenient mechanism to build inter-enclave secure communication channels.

We could also take the “GUI domain” (currently this is just Dom0 in Qubes OS) and move it into a separate SGX enclave. If only Intel came up with solid protected input and output technologies that would work well with SGX, then this would suddenly make whole lots of sense (unlike currently where it is very challenging). What we win this way is that no longer a bug in the hypervisor should be critical, as it would be now a long way for the attacker who compromised the hypervisor to steal any real secret of the user, because there are no secrets in the hypervisor itself.

In this setup the two most critical enclaves are: 1) the GUI enclave, of course, and 2) the admin enclave, although it is thinkable that the latter could be made reasonably deprivileged in that it might only be allowed to create/remove VMs, setup networking and other policies for them, but no longer be able to read and write memory of the VMs (Anti Snowden Protection, ASP?).

And... why use hypervisors? Why not use the same approach to compartmentalize ordinary operating systems? Well, this could be done, of course, but it would require considerable rewrite of the systems, essentially turning them into microkernels (except for the fact that the microkernel would no longer need to be trusted), as well as the applications and drivers, and we know that this will never happen. Again, let me repeat one more time: the whole point of using virtualization for security is that it wraps up all the huge APIs of an ordinary OS, like Win32 or POSIX, or OSX, into a virtual machine that itself requires orders of magnitude simpler interface to/from the outside world (especially true for paravirtualized VMs), and all this without the need to rewrite the applications.

Trusting Intel – Next Generation of Backdooring?

We have seen that SGX offers a number of attractive functionality that could potentially make our digital systems more secure and 3rdparty servers more trusted. But does it really?

The obvious question, especially in the light of recent revelations about NSA backdooring everything and the kitchen sink, is whether Intel will have backdoors allowing “privileged entities” to bypass SGX protections?

Traditional CPU backdooring

Of course they could, no question about it. But one can say that Intel (as well as AMD) might have been having backdoors in their processors for a long time, not necessarily in anything related to SGX, TPM, TXT, AMT, etc. Intel could have built backdoors into simple MOV or ADD instructions, in such a way that they would automatically disable ring/page protections whenever executed with some magic arguments. I wrote more about this many years ago.

The problem with those “traditional” backdoors is that Intel (or a certain agency) could be caught using it, and this might have catastrophic consequences for Intel. Just imagine somebody discovered (during a forensic analysis of an incident) that doing:

MOV eax, $deadbeef
MOV ebx, $babecafe
ADD eax, ebx

...causes ring elevation for the next 1000 cycles. All the processors affected would suddenly became equivalents of the old 8086 and would have to be replaced. Quite a marketing nightmare I think, no?

Next-generation CPU backdooring

But as more and more crypto and security mechanisms got delegated from software to the processor, the more likely it becomes for Intel (or AMD) to insert really “plausibly deniable” backdoors into processors.

Consider e.g. the recent paper on how to plant a backdoor into the Intel's Ivy Bridge's random number generator (usable via the new RDRAND instruction). The backdoor reduces the actual entropy of the generator making it feasible to later brute-force any crypto which uses keys generated via the weakened generator. The paper goes into great lengths describing how this backdoor could be injected by a malicious foundry (e.g. one in China), behind the Intel's back, which is achieved by implementing the backdoor entirely below the HDL level. The paper takes a “classic” view on the threat model with Good Americans (Intel engineers) and the Bad Chinese (foundry operators/employees). Nevertheless, it should be obvious that Intel could have planted such a backdoor without any effort or challenge described in the paper, because they could do so at any level, not necessarily below HDL.

But backdooring an RNG is still something that leaves traces. Even though the backdoored processor can apparently pass all external “randomness” testes, such as the NIST testsuite, they still might be caught. Perhaps because somebody will buy 1000 processors and will run them for a year and will note down all the numbers generated and then conclude that the distribution is quite not right. Or something like that. Or perhaps because somebody will reverse-engineer the processor and specifically the RNG circuitry and notice some gates are shorted to GND. Or perhaps because somebody at this “Bad Chinese” foundry will notice that.

Let's now get back to Intel SGX -- what is the actual Root of Trust for this technology? Of course, the processor, just like for the old ring3/ring0 separation. But for SGX there is additional Root of Trust which is used for remote attestation, and this is the private key(s) used for signing the Quote Messages.

If the signing private key somehow got into the hands of an adversary, the remote attestation breaks down completely. Suddenly the “SGX Blackboxed” apps and malware can readily be decrypted, disassembled and reverse engineered, because the adversary can now emulate their execution step by step under a debugger and still pass the remote attestation. We might say this is good, as we don't want irreversible malware and apps. But then, suddenly, we also loose our attractive “trusted cloud” too – now there is nothing that could stop the adversary, who has the private signing key, to run our trusted VM outside of SGX, yet still reporting to us that it is SGX-protected. And so, while we believe that our trusted VM should be trusted and unsniffable, and while we devote all our deepest secrets to it, the adversary can read them all like on a plate.

And the worst thing is – even if somebody took such a processor, disassembled it into pieces, analyzed transitor-by-transitor, recreated HDL, analyzed it all, then still it all would look good. Because the backdoor is... the leaked private key that is now also in the hands of the adversary, and there is no way to prove it by looking at the processor alone.

As I understand, the whole idea of having a separate TPM chip, was exactly to make such backdoor-by-leaking-keys more difficult, because, while we're all forced to use Intel or AMD processors today, it is possible that e.g. every country can produce their own TPM, as it's million times less complex than a modern processor. So, perhaps Russia could use their own TPMs, which they might be reasonably sure they use private keys which have not be handed over to the NSA.

However, as I mentioned in the first part of this article, sadly, this scheme doesn't work that well. The processor can still cheat the external TPM module. For example, in case of an Intel TXT and TPM – the processor can produce incorrect PCR values in response to certain trigger – in that case it no longer matters that the TPM is trusted and keys not leaked, because the TPM will sign wrong values. On the other hand we go back now to using “traditional” backdoors in the processors, whose main disadvantage is that people might got cought using them (e.g. somebody analyzed an exploit which turns out to be triggering correct Quote message despite incorrect PCRs).

So, perhaps, the idea of separate TPM actually does make some sense after all?

What about just accidental bugs in Intel products?

Conspiracy theories aside, what about accidental bugs? What are the chances of SGX being really foolproof, at least against those unlucky adversaries who didn't get access to the private signing keys? The Intel's processor have become quite a complex beasts these days. And if you also thrown in the Memory Controller Hub, it's unimaginably complex beast.

Let's take a quick tour back discussing some spectacular attacks against Intel “hardware” security mechanisms. I wrote “hardware” in quotation marks, because really most of these technologies is software, like most of the things in electronics these days. Nevertheless the “hardware enforced security” does have a special appeal to lots of people, often creating an impression that these must be some ultimate unbreakable technologies....

I think it all started with our exploit against Intel Q35 chipset (slides 15+) demonstrated back in 2008 which was the first attack allowing to compromise, otherwise hardware-protected, SMM memory on Intel platforms (some other attacks against SMM shown before assumed the SMM was not protected, which was the case on many older platforms).

This was then shortly followed by another paper from us about attacking Intel Trusted Execution Technology (TXT), which found out and exploited a fact that TXT-loaded code was not protected against code running in the SMM mode. We used our previous attack on Q35 against SMM, as well as found a couple of new ones, in order to compromise SMM, plant a backdoor there, and then compromise TXT-loaded code from there. The issue highlighted in the paper has never really been correctly patched. Intel has spent years developing something they called STM, which was supposed to be a thin hypervisor for SMM code sandboxing. I don't know if the Intel STM specification has eventually been made public, and how many bugs it might be introducing on systems using it, or how much inaccurate it might be.

In the following years we presented two more devastating attacks against Intel TXT (none of which depending on compromised SMM): onewhich exploited a subtle bug in the processor SINIT module allowing to misconfigure VT-d protections for TXT-loaded code, and another one exploiting a classic buffer overflow bug also in the processor's SINIT module, allowing this time not only to fully bypass TXT, but also fully bypass Intel Launch Control Policy and hijack SMM (several years after our original papers on attacking SMM the old bugs got patched and so this was also attractive as yet another way to compromise SMM for whatever other reason).

Invisible Things Lab has also presented first, and as far as I'm aware still the only one, attack on Intel BIOS that allowed to reflash the BIOS despite Intel's strong “hardware” protection mechanism to allow only digitally signed code to be flashed. We also found outabout secret processor in the chipset used for execution of Intel AMT code and we found a way to inject our custom code into this special AMT environment and have it executed in parallel with the main system, unconstrained by any other entity.

This is quite a list of Intel significant security failures, which I think gives something to think about. At the very least that just because something is “hardware enforced” or “hardware protected” doesn't mean it is foolproof against software exploits. Because, it should be clearly said, all our exploits mentioned above were pure software attacks.

But, to be fair, we have never been able to break Intel core memory protection (ring separation, page protection) or Intel VT-x. Rafal Wojtczuk has probably came closest with his SYSRET attackin an attempt to break the ring separation, but ultimately the Intel's excuse was that the problem was on the side of the OS developers who didn't notice subtle differences in the behavior of SYSRET between AMD and Intel processors, and didn't make their kernel code defensive enough against Intel processor's odd behavior.

We have also demonstrated rather impressive attacks bypassing Intel VT-d, but, again, to be fair, we should mention that the attacks were possible only on those platforms which Intel didn't equip with so called Interrupt Remapping hardware, and that Intel knew that such hardware was indeed needed and was planning it a few years before our attacks were published.

So, is Intel SGX gonna be as insecure as Intel TXT, or as secure as Intel VT-x....?

The bottom line

Intel SGX promises some incredible functionality – to create protected execution environments (called enclaves) within untrusted (compromised) Operating System. However, for SGX to be of any use on a client OS, it is important that we also have technologies to implement trusted output and input from/to the SGX enclave. Intel currently provides little details about the former and openly admits it doesn't have the later.

Still, even without trusted input and output technologies, SGX might be very useful in bringing trust to the cloud, by allowing users to create trusted VMs inside untrusted provider infrastructure. However, at the same time, it could allow to create applications and malware that could not be reversed engineered. It's quote ironic that those two applications (trusted cloud and irreversible malware) are mutually bound together, so that if one wanted to add a backdoor to allow A/V industry to be able to analyze SGX-protected malware, then this very same backdoor could be used to weaken the guarantees of the trustworthiness of the user VMs in the cloud.

Finally, a problem that is hard to ignore today, in the post-Snowden world, is the ease of backdooring this technology by Intel itself. In fact Intel doesn't need to add anything to their processors – all they need to do is to give away the private signing keys used by SGX for remote attestation. This makes for a perfectly deniable backdoor – nobody could catch Intel on this, even if the processor was analyzed transistor-by-transistor, HDL line-by-line.

As a system architect I would love to have Intel SGX, and I would love to believe it is secure. It would allow to further decompose Qubes OS, specifically get rid of the hypervisor from the TCB, and probably even more.

P { margin-bottom: 0.08in; }
Special thanks to Oded Horowitz for turning my attention towards Intel SGX.
Author: "Joanna Rutkowska (noreply@blogger.com)"
Send by mail Print  Save  Delicious 
Date: Friday, 30 Aug 2013 15:14
P { margin-bottom: 0.08in; }
Intel Software Guard Extensions (SGX) might very well be The Next Big Thing coming to our industry, since the introduction of Intel VT-d, VT-x, and TXT technologies in the previous decade. It apparently seem to promise what so far has never been possible – an ability to create a secure enclavewithin a potentially compromised OS. It sounds just too great, so I decided to take a closer look and share some early thoughts on this technology.

Intel SGX – secure enclaves within untrustedworld!

Intel SGX is an upcoming technology, and there is very little public documents about it at the moment. In fact the only public papers and presentations about SGX can be found in the agenda of one security workshop that took place some two months ago. The three papers from Intel engineers presented there provide a reasonably good technical introduction to those new processor extensions.

You might think about SGX as of a next generation of Intel TXT – a technology that has never really took off, and which has had a long history of security problems disclosed by certain team of researchers ;) Intel TXT has also been perhaps the most misunderstood technology from Intel – in fact many people thought about TXT as if it already could provide security enclaves within untrusted OS – this however was not really true (even ignoring for our multiple attacks) and I have spoke and wrote many times about that in the past years.

It's not clear to me when SGX will make it to the CPUs that we could buy in local shops around the corner. I would be assuming we're talking about 3-5 years from now, because the SGX is not even described in the Intel SDM at this moment.

Intel SGX is essentially a new mode of execution on the CPU, a new memory protection semantic, plus a couple of new instructions to manage this all. So, you create an enclave by filling its protected pages with desired code, then you lock it down, measure the code there, and if everything's fine, you ask the processor to start executing the code inside the enclave. Since now on, no entity, including the kernel (ring 0) or hypervisor (ring “-1”), or SMM (ring “-2”) or AMT (ring “-3”), has no right to read nor write the memory pages belonging to the enclave. Simple as that!

Why have we had to wait so long for such technology? Ok, it's not really that simple, because we need some form of attestation or sealing to make sure that the enclave was really loaded with good code.

The cool thing about an SGX enclave is that it can coexist (and so, co-execute) together with other code, such all the untrusted OS code. There is no need to stop or pause the main OS, and boot into a new stub mini-OS, like it was with the TXT (this is what e.g. Flicker tried to do, and which was very clumsy). Additionally, there can be multiple enclaves, mutually untrusted, all executing at the same time.

No more stinkin' TPMs nor BIOSes to trust!

A nice surprise is that SGX infrastructure no longer depends on the TPM to do measurements, sealing and attestation. Instead Intel has a special enclave that essentially emulates the TPM. This is a smart move, and doesn't decrease security in my opinion. It surely makes us now trust only Intel vs. trusting Intel plus some-asian-TPM-vendor. While it might sound like a good idea to spread the trust between two or more vendors, this only really makes sense if the relation between trusting those vendors is expressed as “AND”, while in this case the relation is, unfortunately of “OR” type – if the private EK key gets leaked from the TPM manufacture, we can bypass any remote attestation, and no longer we need any failure on the Intel's side. Similarly, if Intel was to have a backdoor in their processors, this would be just enough to sabotage all our security, even if the TPM manufacture was decent and played fair.

Because of this, it's generally good that SGX allows us to shrink the number of entities we need to trust down to just one: Intel processor (which, these days include the CPUs as well as the memory controller, and, often, also a GPU). Just to remind – today, even with a sophisticated operating system architecture like those we use in Qubes OS, which is designed with decomposition and minimizing trust in mind, we still need to trust the BIOS and the TPM, in addition to the processor.

And, of course, because SGX enclaves memories are protected against any other processor mode's access, so SMM backdoor no longer can compromise our protected code (in contrast to TXT, where SMM can subvert a TXT-loaded hypervisor), nor any other entity, such as the infamous AMT, or malicious GPU, should be able to do that.

So, this is all very good. However...

Secure Input and Output (for Humans)

For any piece of code to be somehow useful, there must be a secure way to interact with it. In case of servers, this could be implemented by e.g. including the SSL endpoint inside the protected enclave. However for most applications that run on a client system, ability to interact with the user via screen and keyboard is a must. So, one of the most important questions is how does Intel SGX secures output to the screen from an SGX enclave, as well as how does it ensure that the input the enclave gets is indeed the input the user intended?

Interestingly, this subject is not very thoroughly discussed in the Intel papers mentioned above. In fact only one paper briefly mentions Intel Protected Audio Video Path (PVAP) technology that apparently could be used to provide secured output to the screen. The paper then references... a consumer FAQ onBlueRay Disc Playback using Intel HD graphics. There is no further technical details and I was also unable to find any technical document from Intel about this technology. Additionally this same paper admits that, as of now, there is no protected inputtechnology available, even on prototype level, although they promise to work on that in the future.

This might not sound very surprising – after all one doesn't need to be a genius to figure out that the main driving force behind this whole SGX thing is the DRM, and specifically protecting Holywwod media against the pirate industry. This would be nothing wrong in itself, assuming, however, the technology could also have some other usages, that could really improve security of the user (in contrast to the security of the media companies).

We shall remember that all the secrets, keys, tokens, and smart-cards, are ultimately to allow the user to access some information. And how does people access information? By viewing in on a computer screen. I know, I know, this so retro, but until we have direct PC-brain interfaces, I'm afraid that's the only way. Without properly securing the graphics output, all the secrets can be ultimately leaked out.

Also, how people command their computers and applications? Well, again using this retro thing called keyboard and mouse (touchpad). However secure our enclave might be, without secured input, the app would not be able to distinguish intended user input from simulated input crafted by malware. Not to mention about such obvious attacks as sniffing of the user input.

Without protected input and output, SGX might be able to stop the malware from stealing the user's private keys for email encryption or issuing bank transactions, yet the malware will still be able to command this super-secured software to e.g. decrypt all the user emails and later steal the screenshots of all the plaintext messages (with a bit of simple programming, the screenshot's could be turned back into nice ASCII text for saving on bandwidth when leaking them out to a server in Hong Kong), or better yet, perhaps just forward them to an email address that the attacker controls (perhaps still encrypted, but using the attackers key).

But, let's ignore for a moment this “little issue” of lack of protected input, and lack of technical documentation on how secure graphics output is really implemented. Surely it is thinkable that protected input and output could be implemented in a number of ways, and so let's hope Intel will do it, and will do right. We should remember here, that whatever mechanism Intel is going to use to secure the graphics and audio output, it surely will be an attractive target of attacks, as there is probably a huge money incentive for such attacks in the film illegal copying business.

Securing mainstream client OSes and why this is not so simple?

As mentioned above, for SGX enclaves to be truly meaningful on client systems we need protected input and output, to and from the secured enclaves. Anyway, lets assume for now that Intel has come up with robust mechanisms to provide these. Let's now consider further, how SGX could be used to turn our current mainstream desktop systems into reasonably secure bastions.

We start with a simple scenario – a dedicated application for viewing of incoming encrypted files, say PDFs, performing their decryption and signature verification., and displaying of the final outcome to the user (via protected graphics path). The application takes care about all the key management too. All this happens, of coruse, inside an SGX enclave(s).

Now, this sounds all attractive and surely could be implemented using the SGX. But what about if we wanted our secure document viewer to become a bit more than just a viewer? What if we wanted a secure version of MS Word or Excel, with its full ability to open complex documents and edit them?

Well it's obviously not enough to just put the proverbial msword.exe into a SGXenclave. It is not, because the msword.exe makes use of million of other things that are provided by the OS and 3rdlibraries, in order to perform all sorts of tasks it is supposed to do. It is not a straightforward decision to draw a line between those parts that are security sensitive and those that are not. Is font parsing security critical? Is drawing proper labels on GUI buttons and menu lists security critical? Is rendering of various objects that are part of the (decrypted) document, such as pictures, security critical? Is spellchecking security critical? Even if the function of some of a subsystem seem not security critical (i.e. not allows to easily leakthe plaintext document out of the enclave), let's not forget that all this 3rdparty code would be interacting very closely with the enclave-contained code. This means the attack surface exposed to all those untrusted 3rdparty modules will be rather huge. And we already know it is rather not possible to write a rendererfor such complex documents as PDFs, DOCs, XLS, etc, without introducing tons of exploitable bugs.And these attack are not coming now from the potentially malicious documents (against thosewe protect, somehow, by parsing only signed document from trusted peers), but are coming from the compromised OS.

Perhaps it would be possible to take Adobe Reader, MS Word, Powerpoint, Excel etc, and just rewrite every of those apps from scratch in a way that they wereproperly decomposed into sensitive parts that execute within SGC enclave(s), and those that are not-sensitive and make use of all the OS-providedfunctionality, and further define clean and simple interfaces between those parts, ensuring the “dirty” code cannot exploit the sensitive code. Somehow attractive, but somehow I don't see this happening anytime soon.

But, perhaps, it would be easier to do something different – just take the whole msword.exe, all the DLLs it depends on, as well as all the OS subsystems it depends on, such as the GUI subsystem, and put all of this into an enclave. This sounds like a more rational approach, and also more secure.

Only notice one thing – we just created... a Virtual Machine with Windows OS inside and the msword.exe that uses this Windows OS.. Sure, it is not a VT-x-based VM, it is an SGX-based VM now, but it is largely the same animal!

Again, we came to the conclusion why the use of VMs is suddenly perceived as such an increase in security (which some people cannot get, claiming that introducing VM-layer only increases complexity) – the use of VMs is profitable because of one of thing: it suddenly packs all the fat libraries- and OS-exposed APIs and subsystems into one security domain, reducing all the interfaces between this code in the VM and the outside world. Reducing of the interfaces between two security domains is ALWAYS desirable.

But our SGX-isolated VMs have one significant advantage over the other VM technologies we got used to in the last decade or so – namely those VMs can now be impenetrable to any other entity outside of the VM. No kernel or hypervisor can peek into its memory. Neither can the SMM, AMT, or even a determined physical attacker with DRAM emulator, because SGX automatically encrypts any data that leave the processor, so everything that is in the DRAM is encrypted and useless to the physical attacker.

This is a significant achievement. Of course SGX, strictly speaking, is not a (full) virtualization technology, it's not going to replace VT-x.. But remember we don't always need full virtualization, like VT-x, often we can use paravirtualization and all we need in that case is a good isolation technology. For examaple, Xen usesparavirtualizationfor Linux-based PV VMs, and usesgood-old ring3/ring0 separation mechanism to implement this, and the level of isolation of such PV domains on Xen is comparable to the isolation of HVMs, which are virtualized using VT-x.

To Be Continued

In the next part of this article, we will look into some interesting unconventional uses of SGX, such as creating malware that cannot be reversed engineered, or TOR nodes or Bitcoin mixers that should be reasonably trusted, even if we don't trust theiroperators. Then we will discuss how SGX might profoundly change the architecture of the future operating systems, and virtualization systems, in a way that we will no longer need to trust (large portions of) their kernels or hypervisors, or system admins (Anti Snowden Protection?) And, of course, how our Qubes OS might embrace this technology in the future.

Finally, we should discuss the important issue of whether this whole SGX, while providing many great benefits for system architects, should really be blindly trusted? What are the chances of Intel building in backdoors there and exposing those to the NSA? Is there any difference in trusting Intel processors today vs. trusting the SGX as a basis of security model of all software in the future?

Author: "Joanna Rutkowska (noreply@blogger.com)"
Send by mail Print  Save  Delicious 
Date: Saturday, 22 Jun 2013 11:56
P { margin-bottom: 0.08in; }A:link { }
In a previous postI have outlined a new direction we're aiming with the Qubes project, which is a departure from using a “hardcoded” hypervisor with Qubes (as well as “hardcoded” Linux as Dom0, GUI domain, etc).

Today I'm happy to announce that we've already completed initial porting of the current Qubes OS into this Hypervisor-Abstraction-Layer-based framework. The new version of Qubes, that we call “R3 Alpha” for now, builds fine, installs fine, and even (mostly) works(!), as can be admired on the screenshot below :) It still uses Xen, of course, but this time in a non-hardcoded way, which allows to replace it easily with another hypervisor, as I discuss below.

P { margin-bottom: 0.08in; }A:link { }
Our Qubes Odyssey backendneeded to support a specific hypervisor comprises essentially three parts:
  1. A libvirt driver to support a given VMM. In our case we got it (almost) for free, because Xen 4.2 is well supported by libvirt. I wrote “almost” for free, because some patches to libvirt were still needed, mostly to get rid of some unjustified simplifying assumptions, such as that all the backends are always in Dom0, which is not the case for Qubes OS, of course. Some of those patches were accepted into upstream libvirt, some (still) not, so we had to fork libvirt.
  2. A VMM-specific implementation of our vchan – a simple, socket-like, VMM shared memory-based communication stack between the VMs. Again, in case of Xen 4.2 we got that (almost) for free, because Xen 4.2 has now included a libxenvchan component, which is modified (improved and cleaned up) version of our original vchan (written in early Qubes days for older Xen versions) contributed and maintained by Daniel De Graff from the NSA.
  3. Some minor configuration files, e.g. to tell libvirt which hypervisor protocol to use (in our case: xen:///), and VM configuration template files.
Now, if one wanted to switch Xen for some other hypervisor, such as e.g. the KVM, we would need to write a KVM Odyssey backend in a form of providing the above mentioned three elements. Again, libvirt driver we would get for free, configuration files would be trivial to write, and the only task which would require some coding would be the vchan for KVM.

P { margin-bottom: 0.08in; }A:link { }
Ok, one thing that is left out (non-HAL'ified) for now, is the xc_map_foreign_pages() Xen-specific function call within our GUI daemon.

Ideally such call could also be handled by the libvirt API, however it's not clear to us whether true zero-copy page access is really supported (and intended). If it is not, we will try to contribute a patch to libvirt to add such functionality, as it is generally useful for many things that involve high-speed inter-VM communication, of which our GUI virtualization is just one example. So, at this moment, one would need to add an ugly #if (BACKEND_VMM == ...) to the code above and use another VMM's function(s), equivalent to the xc_map_foreign_pages() on Xen.

But besides the above, essentially everything else should Just Work (TM). And that's pretty amazing, I think :) While I personally can't immediately see any security benefit of switching from Xen to KVM, it might appeal to some people for other reasons (Performance? Better hardware support?). The point is: this should be now easy to do.

If one wanted to support some Windows-based hypervisor, on the other hand, such as MS Hyper-V, or Virtual Box on top of Windows, then two more things will need to be taken care of:

  1. Our core management stack (the core-admin repository), the core RPC services (mostly the qrexec daemon, currently part of core-admin-linux repo), and the libvirt code (core-libvirt, a forked original libvirt with some custom patches I mentioned above), all would need to build and run fine on Windows. While this is not a big problem for core-admin (it's all python) and core-libvirt (it is supposed to build and run on Windows fine), the qrexec daemon would need to be rewritten with Windows OS in mind. We're currently working on this step, BTW.
  2. The GUI daemon would also need to be ported to run on Windows, instead of on top of X Server. This is somehow orthogonal to the need to get rid of the hardcoded xc_map_foreign_pages() function as mentioned above. This step might be optional, however, if we wanted to use a Linux-based (and so Xorg-based GUI server) as a GUI domain.

Once the above two pieces are made Windows-ready (note how I wrote Windows-ready, and not specific-VMM-ready), we can then use any Windows-based hypervisor we want (i.e. for which we have libvirt driver, and can write vchan).

This is again pretty amazing, because it means we don't need N*M variations of each component (where N is the number of VMMs, and M the number of host/GUI OSes to support) – but only N+M! This is similar to how modern compilers are designed using a language-specific frontends (C, C++, Pascal, C#, etc), and architecture-specific backends (x86, x64, ARM, etc), and an Intermediate Language for internal “grinding”, again achieving this N+M number of needed variants instead of N*M, which otherwise would be just totally impractical.

One other detail I would like to point out, and which is also visible on the screenshot above, is that we also got rid of using the Xen-specific Xenstore infrastructure (a registry-like tree-based infrastructure for inter-VM configuration and status exchange), and we replaced it with our own, vchan-based Qubes DB (core-qubesdb).

One interesting thing about Qubes DB is that it get rids of the (overly complex and unnecessary) permission system that is used by xenstore, and instead uses the most simple approach: each VM has its separate Qubes DB daemon, and so a totally separate configuration/state namespace. This is inline with the rest of the Qubes philosophy, which basically says that: permissions is dead, long live separation!

So, in Qubes OS we just isolate everything by default, unless a user/configuration specifically allows an exception – e.g. no file copy operation between domains is possible, unless the user expresses an explicit consent for it.

Many old-school security people can't imagine a system without permissions, but if wethink about it more, we might get to a conclusion that: 1) permissions are complex and so often difficult to understand and set correctly, 2) require often complex code to parse and make security decisions, and 3) often are absolutely unneeded.

As a practical example of how permissions schemes might sometime trick even (otherwise somehow smart) developersinto making a mistake consider this bug in Qubes we made a long time ago when setting permissions on some xenstore key, which resulted in some information leak (not much of a security problem in general, but still). And just today, Xen.org has published this advisory, that sounds pretty serious, again caused by bad permissions on some xenstore keys. (Yes, we do have updated Xen packages to fix that, of course).

Back to Qubes R3 Alpha, the first successful Qubes based on Odyssey HAL framework. As previously mentioned, we plan to make most of the framework open sourced, specifically all the non-Windows code. However, we're not publishing this Odyssey/R3 code at this moment, mainly for two reasons: 1) we don't want people to immediately start building other backends, such as to support KVM, right at this stage, because we still might want/need to modify some interfaces slightly, e.g. for our vchan, and we don't want to tide our hands now, and 2) the other reason is that we're still in the middle of “Beta” releases for Qubes R2, and we want people to rather focus on testing that, rather stable release, than jumping onto Qubes R3 alpha.

In other news: everybody seems to be genuinely surprised that unencrypted information can be intercepted and analyzed without user consent... Can it be that people will "discover" cryptography now? How many of you use PGP everyday? And how long will it take then to understand that cryptography without secure client devices is useless?
Author: "Joanna Rutkowska (noreply@blogger.com)"
Send by mail Print  Save  Delicious 
Date: Thursday, 21 Mar 2013 17:47
P { margin-bottom: 0.08in; }A:link { }
Qubes OS is becoming more and more advanced, polished, and user friendly OS.
But Qubes OS, even as advanced as it is now, surely have its limitations. Limitations, that for some users might be difficult to accept, and might discourage them from even trying out the OS. One such limitation is lack of 3D graphics support for applications running in AppVMs. Another one is still-far-from-ideal hardware compatibility – a somehow inherent problem for most (all?) Linux-based systems.
There is also one more “limitation” of Qubes OS, particularly difficult to overcome... Namely that it is a standalone Operating System, not an application that could be installed inside the user's existing OS. While installing a new application that increases system's security is a no-brianer for most people, switching to a new, exotic OS, is quite a different story...
Before I discuss how we plan to address those limitations, let's first make a quick digression about what Qubes reallyis, as many people often get that wrong...
P { margin-bottom: 0.08in; }A:link { }
What Qubes IS, and what Qubes IS NOT?
Qubes surely is not Xen! Qubes only usesXen to create isolatedcontainers – security domains (or zones). Qubes also is not a Linux distribution! Sure, we currently use Fedora 18 as the default template for AppVMs, but at the same time we also support Windows VMs. And while we also use Linux as GUI and admin domain, we could really use something different – e.g. Windows as GUI domain.
So, what is Qubes then? Qubes (note how I've suddenly dropped the OS suffix) is several things:
  • The way how to configure, harden, and use the VMM (e.g. Xen) to create isolated security domains, and to minimize overall system TCB.
  • Secure GUI virtualization that provides strong gui isolation, while at the same time, provides also seamless integration of all applications running in different VMs onto one common desktop. Plus a customized GUI environment, including trusted Window Manager that provides unspoofable decorations for the applications' windows.
  • Secure inter-domain communication and services infrastructure with centrally enforced policy engine. Plus some “core” services built on top of this, such as secure file exchange between domains.
  • Various additional services, or “addons”, built on top of Qubes infrastructure, such as Disposable VMs, Split GPG, TorVM, Trusted PDF converter, etc. These are just few examples, as basically the sky is the limit here.
  • Various additional customizations to all the guest OSes that run in various domains: GUI, Admin, ServiceVMs, and AppVMs.
P { margin-bottom: 0.08in; }A:link { }
Introducing Qubes HAL: Hypervisor Abstraction Layer
Because Qubes is a bunch of technologies and approaches that are mostly independent from the underlying hypervisor, as discussed above, it's quite natural to consider if we could easily build an abstraction layer to allow the use of different VMMs with Qubes, instead of just Xen? Turns out this is not as difficult as we originally thought, and this is exactly the direction we're taking right now with Qubes Odyssey! To make this possible we're going to use the libvirt project.
So, we might imagine Qubes that is based on Hyper-V or even Virtual Box or VMWare Workstation. In the case of the last two Qubes would no longer be a standalone OS, but rather an “application” that one installs on top of an existing OS, such as Windows. The obvious advantage we're gaining here is improved hardware compatibility, and ease of deployment.
And we can go even further and ask: why not use Windows Native Isolation, i.e. mechanisms such as user account separation, process isolation, and ACLs, to implement domain isolation? In other words why not use Windows OS as a kind of “VMM”? This would further dramatically improve then lightness of the system...
Of course the price we pay for all this is progressively degraded security, as e.g. Virtual Box cannot be a match to Xen in terms of security, both architecturally and implementation-wise, and not to mention the quality of isolation provided by the Windows kernel, which is even less.

  P { margin-bottom: 0.08in; }A:link { }
But on the other hand, it's still better than using “just Windows” which offers essentially only one “zone”, so no domain isolation at all! And if we can get, with minimal effort, most of our Qubes code to work with all those various isolation providers then... why not?
Being able to seamlessly switch between various hypervisors is only part of the story, of course. The remaining part is the support for different OSes used for various Qubes domains. Currently we use Linux, specifically Fedora 18, in our GUI & Admin domain, but there is no fundamental reason why we couldn't use Windows there instead. We discuss this more in-depth in one of the paragraphs below.
The diagram below tries to illustrate the trade-offs between hardware compatibility and ease of deployment vs. security when using different isolation backends with Qubes. Some variants might also offer additional benefits, such as “super-lightness” in terms of CPU and memory resources required, as is the case with Windows Native Isolation.

  P { margin-bottom: 0.08in; }A:link { }
Some example configurations
Let's now discuss two extreme variants of Qubes – one based on the baremetal Xen hypervisor and the other one based on Windows Native Isolation, so a variant from the opposite endof the spectrum (as shown on the illustration above).
The diagram below shows a configuration that uses a decent baremetal hypervisor, such as Xen, with abilities to securely assign devices to untrusted service domains(NetVM, UsbVM). So, this is very similar to the current Qubes OS.

  P { margin-bottom: 0.08in; }A:link { }
Additionally weseeseparate GUI and Admin domains:the GUI domain might perhaps be based on Windows, to provide users with a familiar UI, while the Admin domain, tasked with domain management and policy enforcement,might be based on some minimal Linux distribution.
In the current Qubes OS there is no distinction between a GUI and an Admin domain --both are hosted within one domain called “dom0”. But in some cases there are benefits of separating the GUI domain from the Admin domain. In a corporate scenario, for example, the Admin domain might be accessible only to the IT department and not to the end user. This way the user wouldhave no way of modifying system-wide policies, and e.g. allowing their “work” domain to suddenly talk to the wild open Internet, or to copy workproject files from “work” to “personal” domains(save for the exotic, low-bandwidthcovert channels, such as through CPU cache).
The ability to deprivilege networking and USB stacks by assigning corresponding devices (NICs, and USB controllers) to untrusted, or semi-trused, domains provides great security benefits. This automatically prevents various attacks against the bugs in WiFi stacks or USB stacks.
What is not seen on the diagram, but what is typical for baremetal hypervisors is that they are usually much smaller than hosted hypervisors, implementing less services, and delegating most tasks, such as the infamous I/O emulation to (often)unprivileged VMs.
Let's now look at the other extreme example of using Qubes – the diagram below shows an architecture of a “Qubized” Windows system that uses either a hosted VMM, such as Virtual Box or VMWare Workstation, or even the previously mentioned Windows Native Isolation mechanisms, as an isolation provider for domains.
P { margin-bottom: 0.08in; }A:link { }
Of course this architecture lacks many benefits discussed above, such as untrusted domains for networking and USB stacks, small hypervisors, etc. But it still can be used to implement multiple security domains, at amuch lower “price”: better hardware compatibility, easier deployment, and in case of Windows Native Isolation – excellent performance.
And it really can be made reasonable, although it might require more effort  than it might seem at first sight. Take Windows Native Isolation – of course just creating different user accounts to represent different domains is not enough, because Windows still doesn't implement true GUI-level isolation. Nor network isolation. So, there is a challenge to do it right, and “right” in this case means to make the isolation as good as the Windows kernel can isolate processes from different users from each other.
Sure, a single kernel exploit destroys this all, but it's still better than “one application can (legally) read all my files” policy that 99% of all desktop OSes out there essentially implement today.
Now, probably the best thing with all this is that once we implement a product based on, say, Qubes for Windows, together with various cool “addons” that will take advantage of the Qubes services infrastructure, and which shall be product-specific, it should then be really easy to upgrade to another VMM, say Hyper-V to boost security. And the users shall not even notice a change in the UI, save for the performance degradation perhaps (well, clearly automatic creation of VMs to handle various users tasks would be more costly on Hyper-V than with Windows Native Isolation, where “VMs” are just... processes).
Qubes building blocks – implementation details
Let's have a look now at the repository layout for the latest Qubes OSsources – every name listed below represents a separate code repository that corresponds to a logical module, or a building block of a Qubes system:

P { margin-bottom: 0.08in; }A:link { }
Because current Qubes R2 still doesn't use HAL layer to support different hypervisors, it can currently be used with only one hypervisor, namely Xen, whose code is provided by the vmm-xenrepository (in an ideal world we would be just using vanilla Xen instead of buildingour own from sources, but in reality we like the ability to build it ourselves, slightly modifying some things).
Once we move towards the Qubes Odyssey architecture (essentially by replacing thehardcoded calls to Xen's management stack, in the core-adminmodule, with libvirt calls), we could then easily switch Xen for other hypervisors, such as Hyper-V or Virtual Box. In case of Hyper-V we would not have access to the sources of the VMM, of course, so would just be using the stock binaries, although we still might want to maintain thevmm-hyperv repository that could contain various hardening scripts and configuration files for this VMM. Or might not. Also, chances are high that we would be just able to use the stock libvirt driversfor Hyper-V or Virtual Box,so no need for creating core-libvirt-hypervor core-libvirt-virtualboxbackends.
What we will need to provide, is ourcustom inter-domaincommunication library for each hypervisor supported.This means we will need to write core-vchan-hypervor core-vchan-virtualbox. Most (all?) VMMs do provide some kind of API for inter-VM communication (or at least VM-host communication), so the main task of such component is to wrap the VMM-custom mechanism with Qubes-standarizedAPI for vchan (and this standardization is one thing we're currently working on). All in all, in most cases this will be asimple task.
If we, on the other hand, wanted to support an “exotic” VMM, such as the previously mentioned Windows Native Isolation, which is not really a true VMM, then we will need to write our own libvirt backend to support is:
... as well as the corresponding vchan module (which should be especially trivial to write in this case):
Additionally, if we're building a system where the Admin domain is not based on Linux, which would likely be the case if we used Hyper-V, or Virtual Box for Windows, or, especially, Windows Native Isolation, then we should also provide core-admin-windowsmodule, that, among other things, should provide Qubes qrexecimplementation, something that is highly OS-dependent.
As can be seen above, we currently only have core-admin-linux, which is understandable as we currently use Linux in Dom0. But the good news is that we only need to write core-admin-XXXonce for each OS that is to be supported as an Admin domain, asthis code should not be depend on the actual VMM used (thanks to our smart HAL).
Similarly, we also need to assure that our gui-daemoncan run on the OS that is to be used as a GUI domain (again, in most cases GUI domain would be the same as Admin domain, but not always). Here the situation is generally much easier because “with just a few #ifdefs” our current GUIdaemon should compile and run on most OSes, from Linux/Xorg to Windows and Macs (which is the reason we only have one gui-daemonrepository, instead of several gui-daemon-XXX).
Finally we should provide some code that will gather all the components neededfor our specific product and package this all into either an installable ISO, if Qubes is to be a standalone OS, like current Qubes,or into an executable, in case Qubes is to be an “application”. The installer, depending on the product, might do some cool things, such as e.g. take current user system and automatically move it into one of Qubes domains.
To summary, these would be the components needed to build “Qubes for Windows” product:

Additionally we will likely need a few qubes-app-*modules that would implement some "addons", such as perhaps automatic links and documents opening in specific VMs, e.g.:
Here, again, the sky's the limit and this is specifically the area where each vendor can go to great lenghts and build killer apps using our Qubes framework.
Now, if we wanted to create "Qubes for Hyper-V" we would need the following components:

Here, as an example, I also left optional core-agent-linux and gui-agent-linux components (the same that are to be used with Xen-based Qubes OS) to allow support for also Linux-based VMs – if we can get those “for free”, then why not!
It should be striking how many of those components are the same in both of those two cases – essentially the only differences are made by the use of different vmm-* components and, of course, the different installer
It should be also clear now how this framework now enables seamless upgrades from one product (say Qubes for Windows) to another (say Qubes for Hyper-V).
Our business model assumes working with vendors, as opposed to end users, and licensing to them various code modules needed to create products based on Qubes.
All the code that comprises the base foundation needed for creation of any Qubes variant (so core-admin, gui-common, gui-daemon, qubes-builderand qubes-manager) will be kept open source, GPL specifically. Additionally all the code needed for building of Xen-based Qubes OS with Linux-based AppVMs and Linux-based GUI and Admin domains, will continue to be available as open source. This is to ensure Qubes OS R3 that will be based on this framework, can remain fully open source (GPL).
Additionally we plan to double-license this core open source code to vendors who would like to use it in proprietary products and who would not like to be forced, by the GPL license, to share the (modified) sources.
All the other modules, especially those developed to support other VMMs (Hyper-V, Virtual Box, Windows Native Isolation), as well as those to support Windows OS (gui-agent-windows, core-agent-windows, core-admin-windows, etc) will most likely be proprietary and will be available only to vendors who decide to work with us and buy a license.
So, if you want to develop an open source product that uses Qubes framework, then you can freely do that as all the required core components for this will be open sourced. But if you would like to make a proprietary product, then you should buy a license from us. I think this is a pretty fair deal.
Current status and roadmap
We're currently working on two fronts: one is rewriting current Qubes code to support Qubes HAL, while the other one is adding a backend for Windows Native Isolation (which also involves doing things such as GUI isolation right on Windows).
We believe that by implementing two such extreme backends: Xen and Windows Native Isolation we can best show the flexibility of the framework (plus our customer is especially interested in this backend;)
We should be able to publish some code, i.e. the framework together with early Qubes OS R3 that will be based on it, sometime in fall or maybe earlier.
We obviously are determined to further develop Xen-based Qubes OS, because we believe this is the most practically secure OS available today, and we believe such OS should be open source.
Qubes R2 will still be based on the Xen-hardcoded code, because it's close to the final release and we don't want to introduce such drastic changes at this stage. The only thing that Qubes R2 will get in common with Qubes Odyssey is this new source code layout as presented above (but still with hardcoded xl calls and xen-vchan).
So, this is all really exciting and a big thing, let's see if we can change the industry with this :)
Oh, and BTW, some readers might be wondering why the framework was codenamed “Odyssey” -- this is obviously because of the “HAL” which plays a central role here, and which, of course, also brings to mind the famous Kubrick's movie.
Author: "Joanna Rutkowska (noreply@blogger.com)"
Send by mail Print  Save  Delicious 
Date: Thursday, 28 Feb 2013 19:26
Qubes R2 Beta 2 with KDE 4.9 environment (click for more screenshots)

We're progressing fast and today I would like to announce the release of Qubes R2 Beta 2 ISO. The installation and upgrade instructions, as well as the ISO itself, can be found via the our wiki page. As usual, please remember to verify the digital signature of the downloaded ISO.

The major changes in this beta release include:
  • Upgraded Dom0 distribution to the latest Fedora 18 (all previous releases used Fedora 13 for Dom0!)
  • Upgraded default VM template also to Fedora 18
  • Upgraded Dom0 kernel to 3.7.6
  • Upgraded KDE environment in Dom0 (KDE 4.9)
  • Introduced Xfce 4.10 environment for Dom0 as an alternative to KDE
  • A few other fixes and improvements, including the recently discussed Disposable VM-based PDF converter

The upgrade of the Dom0 distribution and kernel should significantly improve hardware compatibility – one of the major problems with Qubes adoption so far, as we hear. Now, with the latest GPU drivers and Xorg packages, I hope we will be able to cover a much boarder range of hardware (especially all the newer GPUs).

The upgrade of KDE in Dom0 is mostly an eye-candy type of improvement (but then, who doesn't like eye candies!), as is the introduction of the Xfce4 as its alternative, although, arguably, Xfce4 is considered faster and lighter than KDE. In Qubes the choice of an actual desktop environment that runs in Dom0 is not as important as it is on traditional Linux systems, I think, simply because most of the functionality, typically provided by such environments, such as apps and file management, is simply... disabled, because on Qubes there are no user apps or files in Dom0.

Nevertheless people love and hate particular window managers and environments, and we hope that now, by supporting alternative environments, we could appeal to even more users.

I'm glad that we just completed this difficult phase of upgrading Qubes Dom0 distribution (for the first time since Qubes R1 Beta 1!) -- this forced us to clean up the code and prepare for some even bigger and bolder changes in the near future. But those will come only in Release 3. As far as Release 2 is considered, we do have quite a few more tickets scheduled for R2 Beta 3 milestone, but most of those represent various addons, rather than modifications to Qubes core software.

So what are those brave changes that are to happen in Release 3? That I will write about in a separate blog post... Stay tuned!

So, please enjoy this latest Qubes R2 beta release, and be sure to send all your questions and comments, as well as the HCL info, to the qubes-devel mailing list. I have already upgraded my primary laptop to this release a few days ago and everything seems to be working fine, so fear not!
Author: "Joanna Rutkowska (noreply@blogger.com)"
Send by mail Print  Save  Delicious 
Date: Thursday, 21 Feb 2013 20:07
Author: "Joanna Rutkowska (noreply@blogger.com)"
Send by mail Print  Save  Delicious 
Date: Thursday, 21 Feb 2013 15:48
Today, I would like to showcase some of the cool things that one can do with the Qubes networking infrastructure, specifically with all the new features that have been brought by the just released Qubes Beta 2. This will cover the use of multiple Net VMs for creating isolated networks, the use of a Proxy VM for creating a transparent Tor Proxy VM, as well as demonstration of how to use a Standalone VM with manually assigned devices, to create a “WiFi pen-testing” VM, which surely represents the “for fun” aspect of this post.

p { margin-bottom: 0. Qubes Networking Intro

From the networking point of view there are three types of VMs in Qubes:
p { margin-bottom: 0.08in; } p { margin-bottom: 0.08in; }
  • Net VMs, that have networking devices assigned to them, such as e.g. a WiFi or Ethernet card. Each Net VM contains a Xen network backend that is used to provide networking to all VMs that are connected to this Net VM.
  • Regular VMs (AppVMs) that use the networking provided by Net VMs (so they have Xen network frontends that provide virtual interfaces that are backed by the backend in the corresponding Net VM.
  • Proxy VMs that combine both of the above: to Net VMs they look like regular AppVMs, because they are consumers of the networking they provide, but to other AppVMs they act as if they were Net VMs themselves, allowing other VMs to connect to them. Of course the Proxy VMs do not have directly assigned networking devices – they use the networking provided by the Net VM that they connect to. One can chain many Proxy VMs, as we will see below.
p { margin-bottom: 0.08in; } p { margin-bottom: 0.08in; }
The virtual interfaces in client VMs are called ethX, and are provided by the xen_netfront kernel module, and the corresponding interfaces in the Net/Proxy VM are called vifX.Y and are created by the xen_netback module.

Each Net and Proxy VM implements NAT, specifically masquerading, for all the connected VMs. Additionally to this SNAT, each Net or Proxy VM provides also DNAT redirection for DNS resolutions, so that each VM behind a Proxy or Net VM thinks that it uses a DNS in the Net/Proxy VM, but in fact all the DNS request are DNAT-ed by all the Proxy and Net VMs down the original DNS that is provided to the final Net VM. This smart trick allows us to avoid running a DNS caching server in Proxy/Net VMs.

Also, any VM-to-VM traffic, among the VMs connected to the same Net/Proxy VM is blocked by default.

Additionally, each Proxy VM enforces system-wide firewaling rules, specifically the rules for all the directly connected VMs. Those firewalling rules are centrally managed in Dom0 and exposed to each Proxy VM through Xen store. One useful application of this firewalling mechanism is to limit certain VMs to only specific type of white-listed traffic to minimize likelihood of user mistakes. A good example could be a work VM that might be limited to network connectivity only with the select corporate servers and denied all other traffic. This way, when the user receives an email message with an embedded http link (possibly leading to a malicious website) and accidentally clicks on it, nothing wrong happens.

The current infrastructure doesn't support IPv6 routing, but we will likely add this support in the upcoming Beta 3.

p { margin-bottom: 0.08in; }
The default networking topology in Qubes OS

When you proceed with the default installation of Qubes Beta 2, then your initial networking topology looks like on the diagram below:
The default network configuration in Qubes.
p { margin-bottom: 0.08in; } So, by default there is one Net VM, called 'netvm', that is automatically assigned all the networking devices in the system. There is also one Proxy VM, called 'firewallvm' that is directly connected to the default Net VM, and which provides networking to all other VMs in the system. This Proxy VM is used for firewall rules enforcement. Each such service VM consumes 200MB of RAM by default.

p { margin-bottom: 0.08in; }
Network-isolated VMs
p { margin-bottom: 0.08in; }

For some VMs it might be desirable to completely disconnect them from any kind of networking access. This can be easy done using the following command (issued from Dom0's konsole):

[dom0]$ qvm-prefs -s netvm none

For example I have a 'vault' VM that I use for keeping my master PGP keys, and other secrets, and this machine is not connected to any network.

p { margin-bottom: 0.08in; }
p { margin-bottom: 0.08in; }
Using multiple Net VMs for physically isolated networks 

In some scenarios the machine might be connected to two or more physically separate networks (e.g. safe corporate intranet, reachable via ethernet cable on the user's desk, and the unsafe and evil Internet, reachable via WiFi card).

It is easy to use more than one Net VMs in Qubes, and assign different networking devices to different Net VMs, and also decide which VMs are connected to which Net VMs. The diagram below presents an exemplary such setup:
A simple setup with two isolated networks, and one fully isolated domain ('vault').
  p { margin-bottom: 0.08in; }
We could created such a setup using the following commands (issued in Dom0):

[dom0]$ qvm-create netvm1 --net --label red
[dom0]$ qvm-create netvm2 --net --label yellow

Currently qvm-create when used with the --net option automatically assigns all networking devices to the just created VM, so in the example above you would want to remove extra devices from each Net VM using p { margin-bottom: 0.08 qvm-pci -d, leaving only those you really want, e.g.: 

[dom0]$ qvm-pci -l netvm1 # to get a list of currently assigned devices
p { margin-bottom: 0.08in; } p { margin-bottom: 0.08in; }
[dom0]$ qvm-pci -d netvm1 02:00.0

Now we should create the Firewall VMs:

[dom0]$ qvm-create firewallvm1 --proxy --label green
[dom0]$ qvm-create firewallvm2 --proxy --label green

... and connect them to proper Net VMs:

[dom0]$ qvm-prefs -s firewallvm1 netvm netvm1
[dom0]$ qvm-prefs -s firewallvm2 netvm netvm2

And now, for any other VM, just set the appropriate Net VM (either firewallvm1 or firewallvm2, or 'none), to get it assigned to either of the isolated networks, e.g.:

[dom0]$ qvm-prefs -s banking netvm firewallvm1
[dom0]$ qvm-prefs -s xfiles netvm firewallvm2
[dom0]$ qvm-prefs -s vault netvm none
p { margin-bottom: 0.08in; }

This configuration provides very strong isolation between the VMs belonging to network #1, and the VMs belonging to network #2. Specifically, this becomes significant if we fear about potential remotely exploitable bugs in the client code of the core TCP/IP stack (in this case the Net VM could potentially compromise all the connected VMs -- but the same problem applies to even physically separated machines that use the same network).

p { margin-bottom: 0.08in; }
Setting up Tor Proxy using a Proxy VM

Let's now play a bit with Proxy VMs and see how we can use it to create a simple Tor proxy VM. Such a VM would provide anonymized networking to all its clients, so would allow to easily create VMs for anonymous Internet access. The simple setup we would like to prepare is depicted on the figure below:

The 'torvm' Proxy VM provides anonymized networking to 'anon-web' and 'anon-bitcoin' VMs. All the traffic generated by the VMs behind 'torvm' is either fed into the Tor network, or discarded. Furthermore, any app running in those VMs is not able to read any global system identifiers, such as the external IP, external MAC address, etc.
p { margin-bottom: 0.08in; }
Our Tor proxy would forward only the Tor traffic, so we don't have to fear about some Tor-not-aware applications, or even intentionally malicious ones to compromise the privacy of our connection. This is because such applications have no way to generate traffic to the outside world without going through our Tor proxy (unless they could exploit a hypothetical vulnerability in the Tor process running in the Tor VM). Also, the applications running in any VM behind the Tor proxy are not able to determine any globally identifiable IDs, such as the user's external IP address, the real MAC address used by real NICs, etc.

Interestingly just after writing the above paragraph, I discovered that one of our xenstore keys had wrong permissions and, as a result, any VM could read it and get to know the actual external IP (the key is used by a Net VM to communicate the external IP configuration to the connected Proxy VMs, so they could know when to update the firewall configuration). The fix for this problem is here, and the update (qubes-core-dom0-1.6.32) is now available for Dom0 (just do qvm-dom0-updateto get it installed).

  p { margin-bottom: 0.08in; }
So, this represents a rather strong setup for use with Tor. Let's now have a look at how to practically create such a configuration, step by step.

First, let's create the VM that will become our Tor proxy:

[dom0]$ qvm-create torvm --proxy --label green

This will create a Proxy VM named 'torvm', based on the default template. We will need to now start the template VM and install the Tor client there:

[dom0]$ qvm-run -a fedora-14-x64 gnome-terminal

Alternatively, if we didn't trust the Tor client rpm package to be non-malicious, specifically for its installation scripts to be non malicious, we could have based this on a different template, e.g. one used for less trusted VMs, or we could installed the Tor client in /usr/local, that is backed by the VM's private storage, but this would require compiling Tor from sources.

Now, in the just started template VM, lets install the Tor client and (optionally) the Vidalia graphical frontend:

[fedora-14-x64]$ sudo yum install tor vidalia

And then power off the template VM. Now, every VM based on this template, started after the template shutdown, will also see the Tor binary in its filesystem.

Let's now configure our torvm to properly start Tor proxying at boot:

[dom0]$ qvm-run -a torvm gnome-terminal

Now, we will create the following script for starting up the Tor transparent proxy and setting up traffic redirection using iptables:

[torvm]$ vim /rw/config/start_tor_proxy.sh

...and now paste the following into this file:
killall tor
QUBES_IP=$(xenstore-read qubes_ip)

if [ X$QUBES_IP == X ]; then
echo "Error getting QUBES IP!"
echo "Not starting Tor, but setting the traffic redirection anyway to prevent leaks."
/usr/bin/tor \
--SocksPort 0 \
--TransListenAddress $QUBES_IP --TransPort $TOR_TRANS_PORT \
--DNSListenAddress $QUBES_IP --DNSPort 53 \
--RunAsDaemon 1 --ControlPort 9051 \
|| echo "Error starting Tor!"


echo “0” > /proc/sys/net/ipv4/ip_forward
/sbin/iptables -t nat -F
/sbin/iptables -t nat -A PREROUTING -i vif+ -p udp --dport 53 -j DNAT --to-destination $QUBES_IP:53
/sbin/iptables -t nat -A PREROUTING -i vif+ -p tcp -j DNAT --to-destination $QUBES_IP:$TOR_TRANS_PORT
/sbin/iptables -I INPUT 1 -i vif+ -p udp --dport 53 -j ACCEPT
/sbin/iptables -I INPUT 2 -i vif+ -p tcp --dport 9040 -j ACCEPT
/sbin/iptables -F FORWARD

echo “1” > /proc/sys/net/ipv4/ip_forward

Except for the “QUBES_IP=$(xenstore-read qubes_ip)” line that reads the torvm's IP address, there is nothing Qubes-specific in the above listing. It's just a standard way of setting up transparent Tor proxy.

It is important that this file be located in the /rwdirectory, as this directory is backed by the VM's private storage and will survive VM reboots. The VM's root file-system is read-only and all the changes to it are lost on VM shutdown (VM gets an illusion of the root fs being writeable thanks to Copy-On-Write mechanism, but the actual COW backing device is cleared upon each VM shutdown).

We should also modify the /rw/config/rc.localscript, to ensure that our Tor proxy is automatically started -- just paste the following into this script:

# Uncomment this if you would like to use a custom torrc file:
#rm -f /rw/config/log
#ln -sf /rw/config/torrc /etc/tor/torrc

chkconfig qubes_netwatcher off
chkconfig qubes_firewall off
Finally we should also provide a script that would restart our proxy in case the user dynamically switched the NetVM, which would result in the completely different routing. This could be done by creating a script with predefined name qubes_ip_change_hookwithin /rw/config/directory:
Make sure that all the scripts are executable (chmod +x). And that's all. Now, shutdown the torvm:

[dom0]$ qvm-run --shutdown --wait torvm

From now on, every time you start the torvm (or when Qubes starts it in response to start of some other VM that uses torvm as its Net VM), the Tor transparent proxy should be automatically started.

Let's test this by creating a VM that would be using the just created Tor proxy:

[dom0]$ qvm-create anon-web --label black
[dom0]$ qvm-prefs -s anon-web netvm torvm

Now, every time you start the anon-web VM (e.g. by clicking on the Web browser icon in the anon-web's start menu), Qubes will also ensure that torvm is up and running, and this in turn would configure all the Tor proxying for this VM.

Fo additional control one might want to use Vidalia, the graphical front end for Tor (this should be installed within the template VM that has been used for torvm). We could easily start Vidalia by just typing:

[dom0]$ qvm-run -a torvm vidalia

We should however make sure to disable "Start the Tor software when vidalia starts" option in Settings/General in Vidalia. Otherwise, Vidalia might kill your original Tor (that has transparent proxy open) and start own without transparent proxy enabled.

The web browser runs in the 'anon-web' VM that uses 'torvm' for networking access, and thus all the traffic generated by 'anon-web' is routed through the Tor network, or discarded if it's a different traffic than TCP or DNS.

p { margin-bottom: 0.08in; }
Of course one case easily create more VMs that would be using torvm as their Net VM, as so would have anonymized network access. The beauty of this solution is that in case one of my anonymized VM gets compromised, others do not. Plus, the already mentioned benefit, that no matter whether apps in those VMs are buggy, or even intentionally malicious, they would not be able to leak out the user's external IP address.

Creating a WiFi pen-testing VM

Finally let's have some fun and create a WiFi pen-testing VM. The desired config is depicted below:

p { margin-bottom: 0.08in; }
Because we would like to use all sorts of l33t h4x0r t00lzpen-testing security software in this VM, it would make sense to create it as a Standalone VM, which means that it would get its own copy of the whole file-system (as opposed to just the home directory, /rwand /usr/local, as it is the case with regular Qubes VMs). This would ease the installation of all the extra software we would need there, and also ensure that even if the install/build scripts were malicious, the damages would be contained only to this very VM and nothing else. Also, for some reason the standard Linux WiFi stack and drivers still don't support injection on (all?) most of the WiFi cards out of the box, so we would need to patch the actual kernel drivers -- yet another reason to use a Standalone VM in this case.

So, let's create the VM first, and assign a WiFi card to it:

[dom0]$ qvm-create wififun --standalone --label yellow
[dom0]$ qvm-prefs -s wififun memory 800 # ensure at least this mem at startup
[dom0]$ qvm-prefs -s wififun kernel none # use own copy of kernel and modules
[dom0]$ qvm-pci -a wififun

You can easily find the BDF address of any device using the lspcicommand in Dom0 -- this would be something like e.g. “02:00.0”. You should make sure that this WiFi card is not used by any other VM, specifically by your default Net VM (called 'netvm' in a standard Qubes installation). Ideally you could just use a dedicated Express Card-based WiFi card, leaving the built in WiFi assigned to your default Net VM.

Because it's a Standalone VM, Qubes will make a copy of the whole root filesystem, and thus it would eat about 5GB of your disk (normal VMs would take only as much space as their private fs takes up).

Let's now start the VM...

[dom0]$ qvm-run -a wififun gnome-terminal

... and then install the prerequisite software there, starting with downloading the reasonably new compat-wireless sources, together with the required injection patches, and then building and installing the new kernel modules. All actions below are now executed within the VM. This stuff here is really nothing Qubes- or Xen-specific -- one would do more or less the same on any Linux in order to get injection working (so, treat this as a free bonus WiFi hacking tutorial on Linux).

[wififun]$ wget http://linuxwireless.org/download/compat-wireless-2.6/compat-wireless-2011-07-14.tar.bz2

[wififun]$ wget http://patches.aircrack-ng.org/channel-negative-one-maxim.patch
[wififun]$ wget http://patches.aircrack-ng.org/mac80211-2.6.29-fix-tx-ctl-no-ack-retry-count.patch
[wififun]$ wget http://patches.aircrack-ng.org/mac80211.compat08082009.wl_frag+ack_v1.patch

[wififun]$ sudo yum install kernel-devel patch gcc

[wififun]$ tar xjf compat-wireless-2011-07-14.tar.bz2
[wififun]$ cd compat-wireless-2011-07-14
[wififun]$ patch -p1 < ../channel-negative-one-maxim.patch
[wififun]$ patch -p1 < ../mac80211-2.6.29-fix-tx-ctl-no-ack-retry-count.patch
[wififun]$ patch -p1 < ../mac80211.compat08082009.wl_frag+ack_v1.patch

[wififun]$ make
[wififun]$ sudo make unload
[wififun]$ sudo make install

Now, lets reboot the VM to ensure that all the patched drivers will get properly loaded on each VM boot:

[dom0]$ qvm-run --shutdown --wait wififun
[dom0]$ qvm-run -a wififun gnome-terminal

Let's first see if the WiFi driver got properly loaded and if the interface has been created (look for wlanXinterface):

[wififun]$ ifconfig -a

If yes, then proceed with the steps below (if not, then have a look into dmesg and see what was the problem):

[wififun]$ sudo bash
[wififun]# yum install aircrack-ng dnsmasq
[wififun]# airmon-ng start wlan0
[wififun]# iptables -F INPUT
[wififun]# iptables -F FORWARD
[wififun]# echo “1” > /proc/sys/net/ipv4/ip_forward

Note that you don't need to add any explicit masquerading rules, as they are applied by default on Qubes VMs (you can take a look at the nat table in the VM if you want to see by yourself).

Edit the /etc/dnsmasq.conf, so that it contains at least the following:


and then start the dnsmasq daemon -- we will use it for providing DHCP to our fake AP (the at0 interface will be created by airbase-ng and emulates the “uplink” of a traditional AP):

[wififun]# /etc/init.d/dnsmasq start

And finally the fake AP:

[wififun]# airbase-ng -e free_wifi mon0

and on another console (before any client connects, but after airbase-nggot started), configure the at0interface (make sure it matches what you wrote into dnsmasq.conf):

[wififun]# ifconfig at0 up

(you can also add an udev rule to that automatically).

and just to verify it really is working:

[wififun]# tcpdump -i at0

... and now, just wait for a client to connect to your AP. What you do next is only limited by your imagination... But hey, this article is about Qubes networking and not about 0wning client systems ;)

Here's an innocent example using Moxie's sslstrip (amazing this attack still works so well at the end of 2011...):

My 'wififun' VM in action using a simple sslstrip attack, that surprisingly still works pretty nice...
p { margin-bottom: 0.08in; }
Please note that as your wififun VM is a regular Qubes VM, it is automatically connected to the default Net VM, which in turn provides networking to it. That's why it is so easy to create a fully functioning fake AP.

When using custom driver domains, there are currently some catches you should be aware:

Catch #1: When you start a driver domain lateafter system boot, so after some days of uptime and extensive use of VMs, Xen might not be able to allocate enough continues (in terms of MFNs) memory for a driver domain. And PV driver domains, unlike normal domains or HVM driver domains, do require MFN-continuous memory for their DMA buffers (HVM domains do not need that, because IOMMU can create an illusion of this; even though IOMMU is also used for PV driver domains, for protection, it doesn't actively translate bus addresses into GMFNs).

This is usually not a big problem in practice, because in most cases all the driver domains are started early at system boot, when there is still plenty of non-fragmented memory available. However it might become a problem when one wishes to start e.g. the WiFi pen-testing at some later time. The work around is to close as many VMs as possible before starting such driver domain, and then also reducing, for a moment, the amount of memory assigned to Dom0:

[dom0]$ xm mem-set 0 1600m

and then starting the driver domain should be fine. Now we can start all other domains, and that should no longer be problematic for the already running driver domain.

Catch #2: Some network cards, notably Express Cards, might not work well with the 3.0.4 pvops kernel that we use in all VMs by default. In that case you might want to try to use the xenlinux kernel in your WiFi fun VM -- to do that, follow these steps:

[dom0]$ sudo qvm-dom0-update kernel-qubes-vm-
[dom0]$ cp /var/lib/qubes/vm-kernels/* /var/lib/qubes/appvms/wififun/kernels/
[dom0]$ qvm-prefs wififun -s kernelopts "swiotlb=force"

And then, in the VM:

[wififun]$ sudo yum install kernel-devel-

And rebuild the compat-wireless, unload, install modules, and then load drivers again.


As you can see, Qubes Beta 2 now offers a very advanced networking infrastructure that allows more advanced users to create very sophisticated configurations, allowing for pretty good isolation between various domains and networks. Qubes leaves it up to the user (or admin) to figure out what would be the best configuration -- most users would be happy with the default simple setup with just one Net VM and one Firewall VM, while others would go for much more advanced setups.

A bit more advanced networking setup. The usbvm has a 3G modem assigned, and it is possible to dynamically switch between the Net VMs without restarting any other VMs.

Author: "Joanna Rutkowska (noreply@blogger.com)" Tags: "qubes"
Send by mail Print  Save  Delicious 
Date: Thursday, 21 Feb 2013 15:44
The diagram below illustrates how I have decomposed my digital life into security domains. This is a quite sophisticated scheme and most users would probably want something simpler, but I think it's still interesting to discuss it. The domains are implemented as lightweight AppVMs on Qubes OS. The diagram also shows what type of networking connectivity each domain is allowed.

p { margin-bottom: 0.08in; }
Let's discuss this diagram bit by bit. The three basic domains are work (the green label), personal (the yellow label), and red (for doing all the untrusted, insensitive things) – these are marked on the diagram with bold frames.

p { margin-bottom: 0 A quick digression on domain labels (colors) – in Qubes OS each domain, apart form having a distinct name, is also assigned a label, which basically is one of the several per-defined colors. These colors, which are used for drawing window decorations by the trusted Window Manager (color frames), are supposed to be user friendly, easy noticeable, indicators of how trusted a given window is. It's totally up to the user how he or she interprets these colors. For me, it has been somehow obvious to associate the red color with something that is untrusted and dangerous (the “red light” -- stop! danger!), green with something that is safe and trusted, while yellow and orange with something in the middle. I have also extended this scheme, to also include blue, and black, which I interpret as indicating progressively more trusted domains than green, with black being something ultimately trusted.

Back to my domains: the work domain is where I have access to my work email, where I keep my work PGP keys, where I prepare reports, slides, papers, etc. I also keep various contracts and NDAs here (yes, these are PDFs, but received from trusted parties via encrypted and signed email – otherwise I open them in Disposable VMs). The work domain has only network access to my work email server (SMTP/IMAP4 over SSL), and nothing more.

For other work-related tasks that require some Web access, such as editing Qubes Wiki, or accepting LinkedIn invites, or downloading cool pictures from fotolia.com for my presentations, or specs and manuals from intel.com, for all this I use work-pub domain, which I have assigned the yellow label, meaning I consider it only somehow trusted, and I would certainly never put my PGP keys there, or any work-related confidential information.

The personal domain is where all my non-work related stuff, such as personal email and calendar, holiday photos, videos, etc, are held. It doesn't really have access to the Web, but if I was into social networking I would then probably allow HTTPS to something like Facebook.
Being somehow on the paranoid side, I also have a special very-personal domain, which I use for the communication with my partner when I'm away from home. We use PGP, of course, and I have a separate PGP keys for this purpose. While we don't discuss any secret and sensitive stuff there, we still prefer to keep our intimate conversations private.

I use shopping for accessing all the internet e-commerce sites. Basically what defines this domain is access to my credit card numbers and my personal address (for shipping). Because I don't really have a dedicated “corporate” credit card, I do all the shopping in this domain, from groceries, through sports equipment, on hotel/plane reservations ending. If I had separate business credit cards, then I would probably split my shopping domain into personal-shopping and work-shopping. I also have banking domain, which I use only for managing my bank account (which again combines both my personal and company accounts).

I also have a few specialized work-related domains, that I rarely use. The work-admin domain is used to manage almost all of the ITL servers, such as our webserver, Qubes repo & wiki servers, email server and DNS management, etc. This domain is allowed only SSH traffic to those server, and HTTPS to a few Web-based management servers. The work-blog is used to manage this very blog you're reading now. The reason why it is separate from work-admin or work, is because I'm over paranoid, and because I fear that if somebody compromises the blogger service, and subsequently exploits a bug in my browser that I use for editing my blog, than I don't want this person to be able to also get admin access to all the ITL servers. 

Similarly, if somebody somehow compromised e.g. the Amazon Web Management Console, and then exploited my browser in work-admin, then I would like at least to retain access to my blog. If I used twitter, I would probably also manage it from this work-blog domain, unless it was a personal twitter account, in which case I would run it from personal.

The qubes-dev domain is used for all the Qubes development, merging other developers' branches (after I verify signatures on their tags, of course!), building RPMs/ISOs (yes, Qubes Beta 1 will ship as a DVD ISO!), and finally signing them. Because the signing keys are there, this domain is very sensitive. This domain is allowed only SSH network access to our Qubes git server. Again, even if somebody compromised this Git server, it still would not be a big problem for us, because we sign and verify all the tags in each others repos (unless somebody could also modify the SSH/Git daemons running there so that they subsequently exploit a hypothetical bug in my git client software when I connect to the server).

I also decided to keep all the accounting-related stuff in a separate domain – whenever I get an invoice I copy it to the accounting domain. The rationale for this is that I trust those PDFs much less than I trust the PDFs I keep in my work domain.

Once a year I move the old stuff from my work domain, such as old email spools, old contracts and NDAs, to the work-archives domain. This is to minimize the potential impact of the potential attack on my work domain (my work domain could be attacked e.g. by exploiting a hypothetical bug in Thunderbird's protocol handshake using a MITM attack, or a hypothetical bug in GPG).

The vault domain is an ultimately trusted one where I generate and keep all my passwords (using keepass) and master GPG keys. Of course, this vault domain has no networking access. Most of those passwords, such as the email server access password is also kept in the specific domains which uses them, such as the work domain, and more specifically in the Thunderbird client (there is absolutely no point in not allowing e.g. Thunderbird to remember the password – if it got compromised it would just steal it the next time I manually enter it)

And finally, there is the previously mentioned red domain (I have tried to call it junk or random in the past, but I think red is still a better name after all). The red domain is totally untrusted – if it gets compromised, I don't care – I would just recreate it within seconds. I don't even back it up! Basically I do there everything that doesn't fit into other domains, and which doesn't require me to provide any sensitive information. I don't differentiate between work-related and personal-related surfing even – I don't care about anonymity for all those tasks I do there. If I was concerned about anonymity, I would create a separate anonymous domain, and proxy all the traffic through a tor proxy from there.

Now, this all looks nice and easy ;) but there is one thing that complicates the above picture...
p { margin-bottom: 0.08in; }

Data flows between the domains
p { margin-bottom: 0.08in; }

The diagram below shows the same domains, but additionally with arrows symbolizing typical data flows between them.

p { margin-bottom: 0.08in; }
You can see that most of the usual data flows are from more trusted domains to less trusted domains – e.g. copy and pasting a URL that I receive via email in my work domain, so that I could open it in my untrusted browser in red, or moving invoices from my work domain (where I receive them via email) to the accounting domain.

But there are, unfortunately, also some transfers from less trusted domains to more trusted ones. One example is copy and pasting an interesting URL that I just stumbled upon when surfing in the red domain, and that I would like to share with a college at work, or a friend, and so I need to copy and paste it to my email client in either work (colleague) or personal (friend) domain.

Now, copying data from less trusted domains to more trusted ones presents a significant problem. While one could think that pasting an URL into Thunderbird email editor is a pretty harmless operation, it's still is an untrusted input – and we don't really know what the red domain really pasted into its local clipboard, and so what we will paste into the work domain's Thunderbird email editor (perhaps 64kB of some junk that will overflow some undo buffer in the editor?). And even more scary is the example with copying the cool-looking graphics file from the Web into my work domain so that I could use it in my presentation slides (e.g. an xkcd . Attacks originating through malicious JPEGs or other graphics format, and exploiting bugs in rendering code have been known for more than a decade.

But this problem – how to handle data flows from less trusted systems to more trusted ones – is not easily solvable in practice, unfortunately...
Some people who design and build high-security systems for use by military and government takes a somehow opposite approach – they say they are not concerned about less-trusted-to-more-trusted data transfers as long as they could assure there is no way to perform a transfer in the opposite direction. 

So, if we could build a system that guarantees that a more trusted domain can never transfer data to a less trusted domain (even if both of those domains are compromised!), then they are happy to allow one-way “up transfers”. In practice this means we need to eliminate all the covert channels between two cooperating domains. The word cooperating is a key word here, and which makes this whole idea not practical at all, IMHO.

Elimination of the covert channels between cooperating domains is indeed required in this scheme, because the assumption is that the data transfer from the less trusted domain could have indeed compromised the more trusted domain. But this, at least, should not result in any data leak back to the originating domain, and later to the less-classified network, which this less-trusted domain is presumably connected to. One of the assumptions here is that the user of such a system is connected to more than one, isolated networks. Even in that case, elimination of all the covert channels between domains (or at least minimizing their bandwith to something unusable – what is unusable, really?) is a big challenge, and can probably only could be done when we're ready to significantly sacrifice the system's performance (smart scheduling tricks are needed to minimize temporal covert channels).

I would like to make it clear that we are not interested in eliminating cooperative covert channels between domains in Qubes any time in the near future, and perhaps in the long term as well. I just don't believe into such approach, and I also don't like that this approach does nothing to preserve the integrity of the more-trusted domain – it only focuses on the isolation aspect. So, perhaps the attacker might not be able to leak secrets back to the less trusted domain, but he or she can do everything else in this more trusted domain. What good is isolation, if we don't maintain integrity?

An alternative solution to handling the less-trusted-to-more-trusted data transfers, is to have trusted “converters” or “verifiers” that could handle specific file types, such as JPEGs, and ensure we get a non-malicious file in the destination domain. While this might remind the bad-old A/V technology, it is something different. Here, the trusted converters would likely be some programs written in a safe language, running in another trusted domain, rather than a big ugly A/V with a huge database of signatures of “bad” patterns of what might appear in a JPEG file. The obvious problem with such an approach is that somebody must write those converters, and write them for all file types that we wish to allow to be transferred to more trusted domains. Perhaps doable in the longer-term, and perhaps we will do it in some future version of Qubes...

Right now we are ignoring this problem, and we say that all less-trusted-to-more-trusted transfers are to be done on the user's own risk :) You're welcome to submit trusted converters for your favorite file type(s) in the meantime!
p { margin-bottom: 0.08in; }

Copying files between domains
p { margin-bottom: 0.08in; }

Speaking of copying files between domains, there is another security catch here. If we imagined two physically separated machines that share no common network resources, the only way to move files between those two air-gaped machines would be via something like a USB stick or a CDROM or DVD disc. But inserting a USB drive or CDROM into a machine triggers a whole lot of actions: from parsing device-provided information, loading required drivers (for USB), parsing the driver's partition table, mounting and finally parsing the filesystem. Each of this stage requires the machine's OS to perform a lot of untrusted input processing, and the potential attack space here is quite large. So, even if we could limit ourselves to copy only harmless files between machines/domains (perhaps they were somehow verified by a trusted party in-between, as discussed above), still there is a huge opportunity that the originating domain could compromise the target domain.

In Qubes Alpha we have been using a similar file copy mechanism, using a virtual stick for file copy between domains. In Qubes Beta 1 we will provide a new scheme based on same shared memory channel that we use for GUI virtualization – the technical details of this solution will be available soon in our wiki. The most sensitive element in this new scheme is the un-cpio-like utility that runs in the target domain and unpacks the incoming blob into the pre-defined directory tree (e.g. /home/user/incoming/from-{domainname}/). We believe we can write pretty safe un-cpio-like utility, in contrast to secure all the previously mentioned elements (USB device parsing, partition parsing, fs parsing). The Qubes Beta 1 is planned to be released at the end of March, BTW.

Partitioning enforcement and easy of use

For any security partitioning scheme to make sense in real life, it is necessary to have some enforcement mechanism that would ensure that the user doesn't mistakenly bypass it. Specifically for this purpose we have come up with special, previously-mentioned firewalling support in Qubes Beta 1, that I will cover in a separate article soon.

Anther thing is to make the partitioning easy to use. For instance, I would like to be able to setup a hint in the policy, that when I click on an URL in an email I received in my work domain that it should be automatically opened in the red domain's default Web browser. Currently we don't do that in Qubes, but we're thinking about doing it in the near future.


Partitioning one's digital life into security domains is certainly not an easy process and requires some thinking. This process is also very user-specific. The partitioning scheme that I've come up for myself is quite sophisticated, and most people would probably want something much simpler. In case of corporate deployments, the scheme would be designed by CIO or IT admins, and enforced on users automatically. Much bigger problem are home and small business users, who would need to come up with the partitioning themselves. Perhaps in future versions of Qubes we will provide some ready to use templates for select "typical" groups of users.
Author: "Joanna Rutkowska (noreply@blogger.com)" Tags: "os security, general, qubes"
Send by mail Print  Save  Delicious 
Date: Saturday, 15 Dec 2012 12:46
Author: "Joanna Rutkowska (noreply@blogger.com)"
Send by mail Print  Save  Delicious 
Date: Friday, 02 Nov 2012 18:35
Many people ask how does Qubes OS differ from other approaches to desktop security. Today I'm trying to answer the most popular questions.
  • Why bother with Qubes OS, if any Linux/BSD already allows to setup different user accounts, or some form of light-weight containers or sandboxes, such as chroot, LXC, SELinux?
First, if you use Xorg or similar X-based server as your GUI server, and this is what nearly all Linux, and most of the other non-Windows OSes use, then you don't have any form of GUI-level isolation, which is essential for a desktop system. I wrote more about this surprising problem some time ago. Proper GUI-level isolation was one of the main goals for Qubes.

Second, all mainstream desktop OSes, such as Windows, Linux, BSD, even OSX, are all based on a monolithic kernels, which present a significant security problem. This is because a typical monolithic kernel of a contemporary desktop OS contains tens of millions of lines of code, and to make it worse, most of this code is reachable from (untrusted) applications via all sorts of APIs, making the attack surface on the kernel huge. And it requires just one successful kernel exploit to own the whole system, bypassing any security mechanisms that might have been built on top of it, such as SELinux, LXC, etc.

Additionally, all the various drivers, networking and USB stacks, are also hosted in the kernel, making attacks via buggy networking (e.g. via buggy 802.11 stacksor buggy firmware) or USB stacks a practical possibility. And there is essentially nothing one can do about it, when using an OS based on a monolithic kernel.

In Qubes, on the other hand, we use Xen hypervisor to provide security isolation between domains, and Xen is just a few hundred of thousands lines of code. It also doesn't need to provide all sorts of APIs to applications, because the Xen hypervisor is essentially only interested in CPU scheduling, memory management and power management, and very few things beyond that. Most notably, the Xen hypervisor knows nothing about networking, disk storage, filesystems, USB stacks, etc, as all those tasks are delegated to (often untrusted) service VMs.
  • How is Qubes better than just running a bunch of VMs in VMWare or Virtual Box?

First, products such as VMWare Workstation or Fusion, or Virtual Box, are all examples of type II hypervisors (sometimes called “hosted VMMs”), which means that they run inside a normal OS, such as Windows, as ordinary processes and/or kernel modules. This means that they use the OS-provided services for all sorts of things, from networking, USB stacks, to graphics output and keyboard and mouse input, which in turn implies they can be only as secure as the hosting OS is. If the hosting OS got compromised, perhaps via a bug in its DHCP client, or USB driver, then it is a game over, also for all your VMs.

Second, those popular consumer type II VMM systems have not been designed with security as a primary goal. Instead, their main focus has been on easy of use, performance, and providing seamless integration of the guest OS(es) with the host OS. Especially the latter, which involves lack of good method to identify which domain a given application belongs to (so, lack of trusted Window Manager), support for shared clipboards which every other VM can steal, insecure file sharing methods, and others, all make it not a very desirable solution when strong domain isolation is important. (This is not to imply that Qubes doesn't support clipboard or file sharing between domains, it does – it's just that we do it in a secure way, at least so we believe). On the other hand, there are many usability improvements in Qubes that are specific to multi-domain system, and which you won't find in the above mentioned products, such as trusted Window Manager that, while maintaining great seamless integration of all the applications onto a common desktop, still allows the user to always know which domain owns which window, support for advanced networking setups, per-domain policies, the just mentioned secure mechanisms for clipboard and filesystem sharing, and many other. Qubes also focuses on making the VMs light-weight so that it was possible to run really a lot of them at the same time, and also on mechanism to allow for secure filesystem sharing between domains (templates).

Finally, the commercial hosted VMMs are really bloated pieces of code. They support everything and the kitchen sink (e.g. Open GL exposed to VMs, and various additional interfaces to allow e.g. drag and drop of files to/from the VM), and so, the attack surface on such a VMM system is orders of magnitude bigger than in case of Qubes OS.
  • How does Qubes compare to [your favourite academic microkernel]? 
While the Xen hypervisor can indeed be considered a microkernel if you're not a strict terminology freak, Qubes itself is much more than just the hypervisor. Qubes is everything that is needed to build a reasonably secure desktop OS on top ofa baremetal hypervisor (or microkernel). Theoretically, with just a few cosmetic changes (at least architecture-wise), Qubes could perhaps swap the Xen hypervisor for some other hypervisor or microkernel, such as perhaps Hyper-V, KVM, or some more exotic one. Thus, it makes little sense to compare Qubes with a hypervisor or microkernel project. What makes sense is to compare the Xen hypervisor, as used in Qubes, with some other hypervisor or microkernel.

Ok, so how does Xen compare with other hypervisors or microkernels out there? We think Xen is unique because it combines an elegant architecture (type I, baremetal, hypervisor) with a number of practical features, such as power management, support for Intel VT-d and driver domains, support for both para-virtualizaed, and fully-virtualized VMs, and many more, not found in e.g. academic microkernels/hypervisor projects, that otherwise seem attractive from the architecture point of view.
  • How is Qubes better than Google Chrome OS?
 First, Chrome OS is not a general purpose OS. Second, it's based on Linux with all its security limitation that are a result of using a monolithic kernel described above (e.g. all the networking and USB stacks in the kernel without a possibility to deprivilige them). Not being a traditional general purpose OS, Chrome is able to avoid many of the challenges of desktop computing, such as the need to define security domains, inter-domain file exchange (as there is essentially no filesystem visible to the user), and others, which is good, of course. But then again, Chrome OS is essentially just an environment to run the Chrome Browser, so the comparison to Qubes is a bit of a misunderstanding.

Technical aspects aside, there is always the privacy concern associated with running everything in a browser – why would all my private data be managed and accessible to some 3rd party organizations and their administrators?
  • How is Qubes better than [your favorite commercial military-grade certified secure OS]?
You must have heard about the super secure military-grade, formally verified, 100% certified, and generally “unbreakable” operating systems made by companies such as Green Hills, Lynx Works, and others. How do they compare to Qubes OS?

Really, I have no idea. For a mere mortal like myself (and perhaps not a US citizen), it seems impossible to get any more technical documentation of those systems (anything beyond the marketing pseudo-technical gibberish), not to mention a trial copy to play with...

Thus, from my point of view, those systems are just a vaporware. If you, my dear reader, are privileged enough to have access to such system, then good for you, but don't expect me to treat seriously a security product that is not available for a wider audience to touch and play with... (And the Chineese surely have the copies already to play with ;)
  • How is Qubes different than Bromium's “micro virtualization” solution?
Many people asked recently about the Bromium's upcoming product and how it differs from Qubes OS. Unfortunately there are few public information available on this product – essentially there is one not-very-technical whitepaperand there are Ian Pratt's presentation slides from the recent XenSummit about u-Xen, apparently a hypervisor that is to be ultimately used in their upcoming product.

The whitepaper suggests that Bromium is based on a hosted (type II) hypervisor running within a normal Window OS, and that this hypervisor is used to spawn a new “micro VM” for each new “task”, where apparently the task might be something as granular as opening a new tab in a Web browser, which makes it somehow similar to Google Chrome's approach. Clearly, the Bromium's main goal seem to be to automate the process of creating separation domains, which is in contrast with what we do on Qubes OS, where the user is required to define the domains explicitly.

The Pratt's slides provide also some technical insight into how Bromium intends to secure their hypervisor. As just discussed above, a hosted hypervisor must normally trust the hosting OS, in this case Windows, which, for obvious reasons, is not a good idea from the security standpoint. Pratt, however, clearly states that “host (...) can not interfere with the privacy or integrity of the hypervisor or other guests” (slide #8). This is a strong statement, so let's take a closer look at their approach to this problem.

The Bromium's idea of how to make their hypervisor (and the VMs) protected from a potentially malicious host OS is not really breakthrough: the slides suggest to “deprivilege the host into a VT-container” (I think the verb to bluepillis now an accepted term for such action ;), and to remove the host's access to the hypervisor pages (via EPT), as well as protect DMA access from devices via VT-d, plus to make this all sensible, use DRTM scheme such as Intel TXT, to load such a hypervisor from within a potentially untrusted OS.

So, what's wrong with the idea of a load-on-the-fly-secure-VMM-system? Isn't Ian Pratt correct that one could protect its memory and execution from the interference of the host? Actually that is possible – Intel TXT, VT-x, VT-d, and EPT give us means to achieve that (although there are a number of catches here). But he's missing one important point: it's the untrusted OS that still owns and manages the input devices (e.g. via USB stacks and drivers) and, most importantly, the output (via the GUI subsystem and drivers). Ensuring that the host OS cannot interfere (e.g. sniff the screen of trusted applications) might be very difficult, or even impossible, in practice.

If I ever was to break the security of such a system, I would just follow the simple way:
1) Infect the host e.g. via one of the many USB attacks (remember they cannot have sandboxed USB driver domain, as they have only a type II hosted hypervisor),
2) Hook somewhere into the GUI subsystem and keep recoding all the interesting data from the screen...
... or something like that ;)

There are also many other things that needs to be answered and which the publicly available documents are silent about, such as e.g. how does the system handle installation of new applications? How is clipboard and file exchange between (micro)VMs handled? How large are the interfaces exposed to each (micro)VM? For now, without a solid documentation available, and without any code to play with, it is just another vaporware for me. (Interestingly there seem to be Bromium's Beta program, which however doesn't seem to be working, at least not for me -- I tried to signup twice, but never got any confirmation or response...?)
  • How is Qubes different from Xen Client?
In many aspects, Xen Clientmight be the most similar product to Qubes OS. Like Qubes, it is based on the Xen hypervisor and so it is also a standalone OS, that one must install instead of one's favorite system, and also, like Qubes, it is targeted for desktop systems, and also offers a possibility to run a few VMs at a time.

However, XenClient has been designed with a different goal in mind, namely as a “Virtual Desktops To Go” solution, while Qubes has been designed to provide seamless experience for secure multi-domain desktop system. As a result, lots of focus in Qubes has been put on creating trusted GUI subsystem, support for advanced networking configurations, secure inter-VM clipboard and file sharing, secure method to reuse the same filesystem as a basis for the AppVMs, and also to optimize the AppVMs so they start almost instantly and take little memory, so that one could easily run many of them at the same time. All those things seem to be missing from Xen Client (as well as solid technical documentation about its design).

I surely have missed a few other products or approaches -- feel free to point them out in the comments, and I might write a continuation post one day.
Author: "Joanna Rutkowska (noreply@blogger.com)" Tags: "os security, qubes"
Send by mail Print  Save  Delicious 
Date: Friday, 02 Nov 2012 18:27
Author: "Joanna Rutkowska (noreply@blogger.com)"
Send by mail Print  Save  Delicious 
Date: Friday, 02 Nov 2012 18:27
Author: "Joanna Rutkowska (noreply@blogger.com)"
Send by mail Print  Save  Delicious 
Date: Friday, 02 Nov 2012 18:06
Author: "Joanna Rutkowska (noreply@blogger.com)" Tags: "qubes"
Send by mail Print  Save  Delicious 
Date: Friday, 02 Nov 2012 18:06
Author: "Joanna Rutkowska (noreply@blogger.com)" Tags: "qubes"
Send by mail Print  Save  Delicious 
Next page
» You can also retrieve older items : Read
» © All content and copyrights belong to their respective authors.«
» © FeedShow - Online RSS Feeds Reader