New features coming in Julia 1.7

Julia is an open-source programming language and ecosystem for high-performance scientific computing; its development team has made the first release candidate for version 1.7 available for testing on Linux, BSD, macOS, and Windows. Back in May, we looked at the increased performance that arrived with Julia 1.6, its last major release. In this article we describe some of the changes and new features in the language and its libraries that are coming in 1.7.

Historically, Julia's release candidates have been close to the finished product, and most users who would like to work with the new features can safely download binaries of version 1.7rc1 from Julia headquarters in the "upcoming release" section. Officially, however, the current version is not "production ready"; the developers welcome bug reports to the GitHub issue tracker.

Syntax changes

An exhaustive list of all of the changes can be found in the release notes. We will look at some of those, starting with a small handful of adjustments to the language syntax to add an extra measure of concision and expressiveness. As with all language alterations since version 1.0, nothing in 1.7 should create breakage except in rare cases.

A new form of array concatenation presents one of these rare cases. Previously, the semicolon was used for concatenation along the first dimension, and it retains that meaning. But repeated semicolons were treated as a single semicolon, and now they have a new significance. This should only break programs where a repeated semicolon is present as a typo.

Currently, the semicolon operator works as follows:

    v1 = [1, 2, 3]
    v2 = [4, 5, 6]
    [v1; v2]   # results in [1, 2, 3, 4, 5, 6]

But the operator is extended in version 1.7: n semicolons now concatenates along the n^th dimension. New dimensions are created as needed. For example, the result of [v1;; v2] is to create a new second dimension and join along it, producing the 3×2 matrix:

    1  4
    2  5
    3  6

The new operator syntax allows us to create a third dimension concisely: [v1;;; v2] gets us a 3×1×2 array, with:

    1
    2
    3

as the first plane and:

    4
    5
    6

as the second plane.

Thinking about indexing can help clarify the results of matrix concatenation, but it is important to remember that Julia uses 1-based indexing unlike many other languages. Our initial vectors were 1D, so they're indexed with a single dimension: v2[2] = 5. When joining them together along a new second dimension, the result is 2D, so it has two indices: [v1;; v2][2, 1] = 2. The result of [v1;;; v2] is a 3D array; before the third dimension is added, a second dimension must exist, but the concatenation is along the third dimension, so the result has a shape of 3×1×2. Indexing of this 3D array uses three indices: [v1;;; v2][3, 1, 2] = 6. The second index only goes to 1 because there is one column.

There are also two new destructuring features. The first one is most easily explained with an example. Given a struct:

    struct S
        a
        b
    end

we can instantiate it with newS = S(4, 5). As before, we can access the two properties of newS with newS.a and newS.b, which yield 4 and 5. The new feature allows us to do this:

    (; a, b) = newS

Now a has the value 4 and b has the value 5.

The second destructuring feature eliminates a potential source of bugs when doing a destructuring operation into a mutable container. If we define a vector with a = [1, 2], what would be the result of a[2], a[1] = a? Anyone writing this probably intends to switch the positions of the elements, changing a to [2, 1]. But in previous versions of the language, the result was [1, 1], because a was being mutated as it was iterated during the destructuring.

To see why it worked this way up to now, the following illustrates the steps the language would take when evaluating the operation:

    a[2] = a[1]    # now the vector a == [ 1, 1 ]
    a[1] = a[2]    # not quite what we were after

The new version does what most people probably expect by deferring the mutation during iteration. (Of course, this is another breaking change in the rare cases where people actually depended on the mutating behavior.)

Unicode refinements

Julia continues to refine its embrace of Unicode as part of the syntax. From the start, Julia has allowed juxtaposition to mean multiplication in cases where it wasn't ambiguous, so if y = 7 then 2y was 14. It has also allowed Unicode square- and cube-root symbols to have their familiar mathematical meaning. Now we can combine juxtaposition with these symbols, as this REPL session illustrates:

    julia> x = 64
    64

    julia> √x
    8.0

    julia> 3√x
    24.0

    julia> ∛x
    4.0

    julia> 3∛x
    12.0

Infix operators in Julia are simply functions that can be used with an alternative syntax. For example, the plus operator is really a function: 3   4 is the same as (3, 4).

The programmer can use a wide variety of Unicode symbols as the names of functions, but in general these names cannot be used as infix operators unless they're given special treatment by the language. Julia 1.7 defines two new symbols that can be used this way: ⫪ and ⫫, which can be entered in the REPL using \Top and \Bot followed by TAB. They have the same precedence as comparison operators such as >.

As a little example of what one might do with this, the following REPL session defines a function that tests if the absolute value of its first argument is larger than the absolute value of its second:

    julia> function ⫪(a, b)
               return abs(a) > abs(b)
           end
    ⫪ (generic function with 1 method)

    julia> ⫪(-8, 3)
    true

    julia> ⫪(-9, -12)
    false

    julia> -9 ⫪ -12
    false

The last input shows that this new symbol can be used as an infix operator.

The ability to use Unicode for names and operators helps to make Julia programs look more like math. However, Unicode is notorious for containing distinct characters that appear identical, a circumstance that has led directly to a class of domain-name vulnerabilities. Having different characters that are visually indistinguishable can obviously be a source of serious bugs.

One way Julia tries to prevent this problem is by ensuring that such visually identical characters have identical semantics. Version 1.7 defines three Unicode centered dots, with code points U 00b7, U 0387, and U 22c5 (they all look like this: ·) as functionally identical. They can be used as infix operators. I tested this by defining a function using one of the dots, and found that I could call my function, including in infix form, with either of the others. The LinearAlgebra package uses the centered dot for a vector inner product.

The new version also brings in the Unicode minus sign (U 2212, or \minus in the REPL) to mean the same thing as the ASCII hyphen that we generally use for subtraction.

New REPL features

The REPL has long featured something called "paste mode", where a user could paste in an example session, such as the ones above, directly into the REPL; it would automatically strip out the prompts, recognize user input, and generally do the right thing. This has now been extended to all of the REPL modes (pkg, shell, and help) in addition to the normal mode; it even switches modes automatically based on the prompt string in the pasted text.

The help mode shows documentation mainly by formatting doctrings supplied by the programmer at the function or module level. Now, in case a module is missing a docstring (not uncommon, especially with small packages), help will look around in the package directory for a README file and print the one closest to the module in question. In any case, it will print the list of exported names. A similar feature existed in some pre-1.0 Julia versions, so this is more of a revival than something new.

If the REPL user does something that returns a large array, the REPL will print an abbreviated form, using ellipses to indicate skipped elements. However, until now, the REPL would mercilessly dump any enormous string that it was asked to display. In Julia version 1.7, long strings are elided in a similar manner as long arrays. The REPL will print the first three and a half lines or so, followed by a notation like ⋯ 13554 bytes ⋯, and then the final three and a half lines. The show() function can be used to see the whole string.

If the user attempts to import an uninstalled package, now the REPL will offer some advice:

    julia> using Example
    │ Package Example not found, but a package named Example is available from a
    │ registry. 
    │ Install package?
    │   (@v1.7) pkg> add Example 
    └ (y/n) [y]: 

Answering "y" is all that is needed, whereas before users would need to know to enter package mode and use the add command. This should be particularly helpful for beginners.

Changes in the standard library

The well-established deleteat!(v, i) function mutates the Vector v by deleting elements at the indices in the list i. The new Julia version adds its twin, keepat!(v, i), which is also a mutating function as indicated by the "!" convention. It effectively deletes the elements for all of the indexes not present in i.

Previously, redirecting both standard error and standard output in a Julia session was an exercise in verbosity requiring four levels of nesting. Now, there is a new library function that makes this much easier:

    redirect_stdio(p, stdout="stdout.txt", stderr="stderr.txt")   

This will call the function p() and redirect its print statements to the supplied paths. This new function will be most convenient when used with a do block, which creates an anonymous function and passes it as the first argument to a function call. In this way we can wrap anything we want inside the redirect_stdio() function:

    redirect_stdio(stdout="stdout.txt", stderr="stderr.txt") do
               println("Julia has rational numbers:")
               println(1//5   2//5)
    end

After executing this in the REPL we will have an empty stderr.txt file and a stdout.txt file with contents:

    Julia has rational numbers:
    3//5

Like other languages, Julia has tuples that are immutable vectors; it has a named version of tuples as well:

    tup = (1, 2, 3)
    ntup = (a=1, b=2, c=3)
    # ntup[:a] == ntup.a == 1

As shown in the comment, named tuples can be indexed using a name (e.g. ntup[:a]), which is equivalent to the property-style access (e.g. ntup.a). Subsets of ordinary tuples can be extracted by indexing with a vector of indices:

    tup[ [1, 3]  ]    # yields a new tuple: (1, 3)

New methods added to the getindex() function, which performs indexing behind the scenes, means that we can now index named tuples with a vector of symbols:

    ntup[ [:a, :c] ]   # yields a new named tuple: (a = 1, c = 3)

This new feature makes named tuple indexing consistent with indexing of ordinary tuples.

The existing replace() function can make a single substitution on a string. It has been enhanced to accept any number of replacement patterns, which are applied left to right and "simultaneously". The release notes use the "simultaneously" term, which simply means that replaced substrings are not subject to further replacements. The new function is both convenient and significantly faster than the regular-expression techniques that people usually resort to. A simple example should clarify how it works:

    julia> s = "abc"
    "abc"

    julia> replace(s, "b" => "XX", "c" => "Z")
    "aXXZ"

    julia> replace(s, "c" => "Z", "Z" => "WWW")
    "abZ"

    julia> replace(s, "ab" => "x", "bc" => "y")
    "xc"

The last two examples show how the feature works in the presence of multiple possible matches.

Significant improvements related to pseudo-random number generation are arriving with version 1.7. The default generator has been swapped with one that has better performance in terms of time, memory consumption, and statistical properties. The new generator also makes it easier to perform reproducible parallel computations.

Julia supports several types of parallel and concurrent computation. Most parallel computation in Julia is organized around tasks, which are similar to coroutines in other languages. The new random number generator is "task local", which means that, during a parallel computation, each task gets its own instance of the generator. The same sequence of pseudo-random numbers will be generated on each task, independent of the allocation to threads. This allows for reproducible simulations using random numbers, even with algorithms that dynamically create tasks during run time.

That brings us to one of the most significant advances arriving with the new version. Previously, in a multi-threaded computation, once a task was assigned to a thread, it was stuck on that thread forever. Julia version 1.7 introduces task migration. Now the scheduler can move tasks among all available threads, potentially helping with load balancing and parallel efficiency. In a followup article we'll take a detailed look at this in the context of a brief tutorial on parallel computation in Julia.

Conclusion

The number of new features and improvements in the upcoming version of Julia is impressive, especially coming just six months after version 1.6. Development seems to have entered a mature phase with steady incremental but substantial improvements in performance, consistency, and programmer convenience.

Julia's position as a major platform for science and engineering computation seems secure, with inroads into a wide variety of disciplines. It's also a free-software success story, with every layer—the language, its 5,000 public packages, and the LLVM compiler it's built on—developed in the open, with contributions welcome from everyone. Julia's package manager, a subsystem integrated into the language and the REPL, eases the process of making those contributions, and allows the end user to keep the 5,000 packages straight. In a followup article we will look at how that system works from the points of view of the developer and the user.