We're pleased to announce PyPy 2.0.1. This is a stable bugfix release over 2.0. You can download it here:

http://pypy.org/download.html

The fixes are mainly about fatal errors or crashes in our stdlib. See below for more details.

• I implemented pickling support on ndarrays and dtypes, it will be compatible with numpy's pickling protocol when the "numpypy" module will be renamed to "numpy".
• I am now working on subarrays.

The following example is on CPython, not PyPy, but moving a third (after Reflex and CINT) backend into place underneath cppyy is straightforward compared to developing the backend in the first place. Take this snippet of C++11 code (cpp11.C):

    constexpr int data_size() { return 5; }

    auto N = data_size();

    template<class L, class R>
    struct MyMath {
       static auto add(L l, R r) -> decltype(l+r) { return l + r; }
    };

    template class MyMath<int, int>;

As a practical matter, most usage of new C++11 features will live in implementations, not in declarations, and are thus never seen by the bindings. The above example is therefore somewhat contrived, but it will serve to show that these new declarations actually work. The new features used here are constexpr, auto, and decltype. Here is how you could use these from CPython, using the PyROOT package, which has more than a passing resemblance to cppyy, as one is based on the other:

    import ROOT as gbl
    gbl.gROOT.LoadMacro('cpp11.C')

    print 'N =', gbl.N
    print '1+1 =', gbl.MyMath(int, int).add(1,1)

    $ python cpp11.py
    N = 5
    1+1 = 2

Work is going to continue first on C++03 on cling with CPython (about 85% of unit tests currently pass), with a bit of work on C++11 support on the side. Once fully in place, it can be brought into a new backend for cppyy, after which the remaining parts of C++11 can be fleshed out for both interpreters.

Cheers,
Wim Lavrijsen

This is the tenth status update about our work on the py3k branch, which we
can work on thanks to all of the people who donated to the py3k proposal.

There's been significant progress since the last update: the linux x86-32
buildbot now passes 289 out of approximately 354 modules (with 39 skips) of
CPython's regression test suite.

That means there's only 26 test module failures left! The list of major items
remaining for 3.2 compatibility are now short enough to list here, with their
related tests:

• Tokenizer support for non-ascii identifiers

• test_importlib
• test_pep263

• memoryview (Manuel Jacob's tackling this on the py3k-memoryview branch)

• test_memoryview

• multiprocessing module currently deadlocks

• test_multiprocessing

• Buggy handling of the new extended unpacking syntax by the compiler:

• test_unpack_ex

• The new Global Interpreter Lock and new thread signal handling

• test_threading
• test_threadsignals
• test_sys

• Upgrade unicodedata to 6.0.0 (requires updates to the actual unicodedata
  generation script)

• test_ucn
• test_unicode
• test_unicodedata

• CPyExt

• test_capi (currently crashes)

• Update int's hash code to match to CPython (float's is already updated on the
  py3k-newhash branch. note that PyPy 2.x doesn't even totally match
  CPython's hashing)

• test_decimal
• test_fractions
• test_numeric_tower

• Miscellaneous:

• test_complex
• test_float
• test_peepholer
• test_range
• test_sqlite (a new cffi based version seems to be coming)
• test_ssl
• test_struct
• test_subprocess
• test_sys_settrace
• test_time

Additionally there are still a number of failures in PyPy's internal test
suite. These tests are usually ran against untranslated versions of PyPy during
development. However we've now began running them against a fully translated
version of PyPy on the buildbot too (thanks to Amaury for setting this
up). This further ensures that our tests and implementation are sane.

We're getting closer to producing an initial alpha release. Before that happens
we'd like to see:

• further test fixes
• the results of test runs on other major platforms (e.g. linux x86-64 and osx
  seem to have some additional failures as of now)
• some basic real world testing

Finally I'd like to thank Manuel Jacob for his various contributions over the
past month, including fixing the array and ctypes modules among other things,
and also Amaury Forgeot d'Arc for his ongoing excellent contributions.

cheers,
Phil

Hello everyone

Last week, Alex Gaynor announced the first public release of Topaz, a Ruby interpreter written in RPython. This is the culmination of a part-time effort over the past 10 months to provide a Ruby interpreter that implements enough interesting constructs in Ruby to show that the RPython toolchain can produce a Ruby implementation fast enough to beat what is out there.

Disclaimer

Obviously the implementation is very incomplete currently in terms of available standard library. We are working on getting it useable. If you want to try it, grab a nightly build.

We have run some benchmarks from the Ruby benchmark suite and the metatracing VMs experiment. The preliminary results are promising, but at this point we are missing so many method implementations that most benchmarks won't run yet. So instead of performance, I'm going to talk about the high-level structure of the implementation.

Architecture

Topaz interprets a custom bytecode set. The basics are similar to Smalltalk VMs, with bytecodes for loading and storing locals and instance variables, sending messages, and stack management. Some syntactical features of Ruby, such as defining classes and modules, literal regular expressions, hashes, ranges, etc also have their own bytecodes. The third kind of bytecodes are for control flow constructs in Ruby, such as loops, exception handling, break, continue, etc.

In trying to get from Ruby source code to bytecode, we found that the easiest way to support all of the Ruby syntax is to write a custom lexer and use an RPython port of PLY (fittingly called RPly) to create the parser from the Ruby yacc grammar.

The Topaz interpreter uses an ObjectSpace (similar to how PyPy does it), to interact with the Ruby world. The object space contains all the logic for wrapping and interacting with Ruby objects from the VM. It's __init__ method sets up the core classes, initial globals, and creates the main thread (the only one right now, as we do not have threading, yet).

Classes are mostly written in Python. We use ClassDef objects to define the Ruby hierarchy and attach RPython methods to Ruby via ClassDef decorators. These two points warrant a little explanation.

Hierarchies

All Ruby classes ultimately inherit from BasicObject. However, most objects are below Object (which is a direct subclass of BasicObject). This includes objects of type Fixnum, Float, Class, and Module, which may not need all of the facilities of full objects most of the time.

Most VMs treat such objects specially, using tagged pointers to represent Fixnums, for example. Other VMs (for example from the SOM Family) don't. In the latter case, the implementation hierarchy matches the language hierarchy, which means that objects like Fixnum share a representation with all other objects (e.g. they have class pointers and some kind of instance variable storage).

In Topaz, implementation hierarchy and language hierarchy are separate. The first is defined through the Python inheritance. The other is defined through the ClassDef for each Python class, where the appropriate Ruby superclass is chosen. The diagram below shows how the implementation class W_FixnumObject inherits directly from W_RootObject. Note that W_RootObject doesn't have any attrs, specifically no storage for instance variables and no map (for determining the class - we'll get to that). These attributes are instead defined on W_Object, which is what most other implementation classes inherit from. However, on the Ruby side, Fixnum correctly inherits (via Numeric and Integer) from Object.

This simple structural optimization gives a huge speed boost, but there are VMs out there that do not have it and suffer performance hits for it.

Decorators

Ruby methods can have symbols in its names that are not allowed as part of Python method names, for example !, ?, or =, so we cannot simply define Python methods and expose them to Ruby by the same name.

For defining the Ruby method name of a function, as well as argument number checking, Ruby type coercion and unwrapping of Ruby objects to their Python equivalents, we use decorators defined on ClassDef. When the ObjectSpace initializes, it builds all Ruby classes from their respective ClassDef objects. For each method in an implementation class that has a ClassDef decorator, a wrapper method is generated and exposed to Ruby. These wrappers define the name of the Ruby method, coerce Ruby arguments, and unwrap them for the Python method.

Here is a simple example:

@classdef.method("*", times="int")
def method_times(self, space, times):
    return self.strategy.mul(space, self.str_storage, times)

This defines the method * on the Ruby String class. When this is called, the first argument is converted into a Ruby Fixnum object using the appropriate coercion method, and then unwrapped into a plain Python int and passed as argument to method_times. The wrapper method also supplies the space argument.

Object Structure

Ruby objects have dynamically defined instance variables and may change their class at any time in the program (a concept called singleton class in Ruby - it allows each object to have unique behaviour). To still efficiently access instance variables, you want to avoid dictionary lookups and let the JIT know about objects of the same class that have the same instance variables. Topaz, like PyPy (which got it from Self), implements instances using maps, which transforms dictionary lookups into array accesses. See the blog post for the details.

This is only a rough overview of the architecture. If you're interested, get in touch on #topaz.freenode.net, follow the Topaz Twitter account or contribute on GitHub.

Hi all,

A short notice to tell you that CFFI 0.5 was released. This contains a number of small improvements from 0.4, but seems to otherwise be quite stable since a couple of months --- no change since January 10, apart from the usual last-minute fixes for Python 3 and for Windows.

Have fun!

Armin

This is the ninth status update about our work on the py3k branch, which
we can work on thanks to all of the people who donated to the py3k
proposal.

Just a very short update on December's work: we're now passing about 223 of
approximately 355 modules of CPython's regression test suite, up from passing
194 last month.

Some brief highlights:

• More encoding related issues were addressed. e.g. now most if not all the
  multibytecodec test modules pass.
• Fixed some path handling issues (test_os, test_ntpath and
  test_posixpath now pass)
• We now pass test_class, test_descr and almost test_builtin (among
  other things): these are notable as they are fairly extensive test suites of
  core aspects of the langauge.
• Amaury Forgeot d'Arc continued making progress on CPyExt (thanks again!)

cheers,
Phil

This is the eight status update about our work on the py3k branch, which
we can work on thanks to all of the people who donated to the py3k
proposal.

Just a short update on November's work: we're now passing about 194 of
approximately 355 modules of CPython's regression test suite, up from passing
160 last month. Many test modules only fail a small number of individual tests
now.

We'd like to thank Amaury Forgeot d'Arc for his contributions, in particular he
has made significant progress on updating CPyExt for Python 3 this month.

Some other highlights:

• test_marshal now passes, and there's been significant progress on
  pickling (thanks Kenny Levinsen and Amaury for implementing
  int.{to,from}_bytes)
• We now have a _posixsubprocess module
• More encoding related fixes, which affects many failing tests
• _sre was updated and now test_re almost passes
• Exception behavior is almost complete per the Python 3 specs, what's mostly
  missing now are the new __context__ and __traceback__ attributes (PEP
  3134)
• Fixed some crashes and deadlocks occurring during the regression tests
• We merged the unicode-strategies branch both to default and to py3k: now we
  have versions of lists, dictionaries and sets specialized for unicode
  elements, as we already had for strings.
• However, for string-specialized containers are still faster in some cases
  because there are shortcuts which have not been implemented for unicode yet
  (e.g., constructing a set of strings from a list of strings). The plan is to
  completely kill the shortcuts and improve the JIT to produce the fast
  version automatically for both the string and unicode versions, to have a
  more maintainable codebase without sacrificing the speed. The autoreds
  branch (already merged) was a first step in this direction.

cheers,
Philip&Antonio

The next PyPy sprint will be in San Francisco, California. It is a
public sprint, suitable for newcomers. It will run on Saturday December 1st and
Sunday December 2nd. The goals for the sprint are continued work towards the
2.0 release as well as code cleanup, we of course welcome any topic which
contributors are interested in working on.

Some other possible topics are:

• running your software on PyPy
• work on PyPy's numpy (status)
• work on STM (status)
• JIT improvements
• any exciting stuff you can think of

If there are newcomers, we'll run the usual introduction to hacking on
PyPy.

Location

The sprint will be held at the Rackspace Office:

620 Folsom St, Ste 100
San Francisco

The doors will open at 10AM both days, and run until 6PM both days.

Thanks to David Reid for helping get everything set up!

This is the seventh status update about our work on the py3k branch, which
we can work on thanks to all of the people who donated to the py3k
proposal.

The biggest news is that this month Philip started to work on py3k in parallel
to Antonio. As such, there was an increased amount of activity.

The py3k buildbots now fully translate the branch every night and run the
Python standard library tests.

We currently pass 160 out of approximately 355 modules of CPython's standard
test suite, fail 144 and skip approximately 51.

Some highlights:

• dictviews (the objects returned by dict.keys/values/items) has been greatly
  improved, and now they full support set operators
• a lot of tests has been fixed wrt complex numbers (and in particular the
  __complex__ method)
• _csv has been fixed and now it correctly handles unicode instead of bytes
• more parser fixes, py3k list comprehension semantics; now you can no longer
  access the list comprehension variable after it finishes
• 2to3'd most of the lib_pypy modules (pypy's custom standard lib
  replacements/additions)
• py3-enabled pyrepl: this means that finally readline works at the command
  prompt, as well as builtins.input(). pdb seems to work, as well as
  fancycompleter to get colorful TAB completions :-)
• py3 round
• further tightening/cleanup of the unicode handling (more usage of
  surrogateescape, surrogatepass among other things)
• as well as keeping up with some big changes happening on the default branch
  and of course various other fixes.

Finally, we would like to thank Amaury Forgeot d'Arc for his significant
contributions.

cheers,
Philip&Antonio