Alexander Kyte gsoc

This Summer of Code, the Mono project had many exciting submissions. It’s been great to see what our applicants have been able to accomplish. Some were very familiar with the codebases they worked on, while others had to learn quickly. Let’s summarize how they spent this summer.

CppSharp Defect Removal And General Feature Work

Mohit Mohta and Kimon Topouzidis chose to address a number of bugs and add features to the code of CppSharp. Std::string was added, stacks were fixed, options were added, structure packing was added, and primitive types support was improved. They both seem to have learned a lot about the workflow of methodical debugging of systems code.

Clang Sanitizers

Many software bugs don’t result in immediate errors and crashes. Some corrupt program state in such a way that a cryptic error is seen much later. In the worst case, each such delayed crash may have a different stack trace. Many of these bugs have root causes that can be spotted in a running program the second they go wrong. The tooling to do so has only recently been able to spot race conditions, which can be some of the worst of these bugs. Clang has integrated a number of such sanitizers.

Armin Hasitzka chose to use clang’s runtime sanitizers for race conditions and for memory safety to automatically catch Mono bugs. In his efforts, he ran into false positives and legitimate bugs alike. He fixed a number of bugs, helped silence false positives, and left behind infrastructure to automatically catch regressions as they appear.

CppSharp Qt Bindings And Maintenance

Dimitar Dobrev is familiar to the Mono project. He has done the Google Summer of Code with Mono in 2015, and has helped maintain CppSharp since.

This summer, he sought to commit his time to developing the Qt bindings further. In the development of CppSharp, the problem of mapping C# types onto C++ generics arose. There were many potential solutions, but very few retained the feeling of the underlying API. After some experimentation, the hard problems were solved.

As the summer came to an end, he fixed the minor issues that arose during tests of QtSharp. The burden of maintaining the project and responding to bugs from the community did not stop for Dimiar, resulting in partial completion of milestones yet significant overall contribution. Development of QtSharp proceeds alongside his ongoing maintenance work and contributions.

MonoDevelop C/C++ Extension Feature Enhancements

The CBinding extension for MonoDevelop adds a lot of great functionality for working with C and C++ projects. It is still a work in progress, and Anubhav Singh wanted to add some more functionality. He focused on bringing support for Windows compilers and for CMake. He also chose this moment to update the extension to reflect the newer APIs of MonoDevelop. In the process, he had to begin the process of upstreaming some changes to MonoDevelop.

C# Compiler Caching with CSCache

Something often mentioned around a warm laptop with spinning fans is how nice C developers have it. CCache enables someone to recompile large C projects after minor modifications in a very small amount of time. Going beyond the build system skipping recompilation, the system compiler is wrapped by a program that spits back the old output in a fraction of the time that a compiler takes. This is a trick that managed languages haven’t learned until now.

Daniel Calancea created a tool which wraps mcs and understands the commands sent to it. If it is invoked with the same files and the same options twice, it checks that all of the hashes of all of the files are the same between runs. If so, it returns the output of the C# compiler the first time. Equally important is that this tool will return the same return codes as the first run, and will integrate as seamlessly into any build system as ccache does. It even reports the same warnings that the initial compiler did.

Daniel published this tool for Windows and Linux to Nuget.

Import of System.IO.Pipes.PipeStream from CoreFX

Mono’s implementation of System.IO.Pipes has historically not had some features available to the CLR. After msbuild was made open source, users found that Mono unfortunately could not build in parallel because of the API differences. CoreFX brought with it the promise of a System.IO.Pipes.PipeStream that would enable parallel msbuild. CoreFX’s API surface was not strictly a superset of Mono’s though. Mono implemented a couple of endpoints that CoreFX did not, and we used those endpoints in other places in the BCL.

Georgios Athanasopoulos chose to do the work required to make Mono work with CoreFX’s PipeStream. Modifying both CoreFX and Mono was required. Mono’s build system had to choose to use the new implementation files, rather than looking for them in the BCL directory. His work was a success. Finishing early, he chose to experimentally enable a parallel msbuild and test it. Things seem to be mostly working.

Lamdba Debugger Support

Often when debugging C# code in the middle of a large project, it’s important to invoke code to understand how variables are behaving in a segment of code. Sometimes, the code that one wishes to invoke hasn’t been written yet. The developer is left squinting at variables, invoking existing methods, and manually running code in their head. Much better would be to enable the developer to write a new function and invoke it on the variables in question. Interpreted languages offer support for this without much trouble usually because code doesn’t have as much metadata associated with it, and because they have integrated compilers for the debugged languages.

This summer, Haruka Matsumoto worked on a system that enables developers to use these arbitrary code snippets entered into the debugger. Mono runs the debugger and the debuggee in separate running instances of the runtime. As the running mono runtime for the application being debugged doesn’t have access to a C# compiler, this code has to be compiled by the debugger. The debugger uses Roslyn to compile the code segments, and this assembly is sent to the debugged application’s runtime.

This is made more difficult by the fact that the debugger is trying to run a Lambda that has access to the variables and methods defined in the functions the debugger is currently debugging. Shorter method names need to resolve to what they would if the original function had used them, and variables should be accessible by name. Issues with private types are potentially unsolvable without special casing, as mono prevents arbitrary code from modifying private fields. Haruka handled these and other difficult considerations, and delivered a very strong prototype of Lambda support in the integrated runtime debugger. It should be immediately useful for anybody who spends a lot of time using mono to debug C# code.

Import Synchronization Primitives from CoreRT

It is often the case that small differences in the implementations of core runtime functions can result in perceived bugs introduced by switching runtimes. The differences are due to depending on API behavior that may not be entirely defined by the specification, but works in a certain case on a certain machine. This sensitivity is nowhere more baffling to debug than around threading and synchronization primitives. The .NET Core Project contains an open-source, cross-platform implementation of C# synchronization primitives. We expect this to receive much community development and user testing. We hoped to import them to gain both consistent behavior and quality.

This summer, Alexander Efremov imported EventWaitHandle, AutoResetEvent, ManualResetEvent, Mutex and Semaphore into Mono. He both manually integrated these libraries into Mono and automated the process of building them. System.Private.CoreLib.Native was successfully added to mono. System.Threading was identified as the next API to import, in order to enable importing Thread from CoreFX.

Alex Rønne Petersen profiler, runtime

As part of our ongoing efforts to improve Mono’s profiling infrastructure, in Mono 5.6, we will be shipping an overhaul of Mono’s profiler API. This is the part of Mono’s embedding API that deals with instrumenting managed programs for the purpose of collecting data regarding allocations, CPU usage, code coverage, and other data produced at runtime.

The old API had some limitations that prevented some features and capabilities from being implemented. The upgrade to the API will allow us to:

  • Reconfigure the profiling features at runtime
  • Look at the values of incoming parameters and return values.
  • Ability to instrument the managed allocators, thus allowing these to be profiled.

This is what we did.

Reconfigure Profiling at Runtime

We wanted the ability to reconfigure the profiling option at runtime. This was not possible with the old API because none of the API functions took an argument representing the profiler whose options should be changed.

This means that it was only possible to change options of the most recently installed profiler, and this was not guaranteed to be the one you wanted. Additionally, doing so it was not thread safe.

Why would we want to change profiling options at runtime, you might wonder? Suppose you know that only a particular area of your program has performance issues and you’re only interested in data gathered while your program is executing that code. With this capability, you can turn off profiling features such as allocations and statistical sampling until you get to the point you want to profile, and then turn them on programmatically. This can significantly reduce the noise caused by unneeded data in a profiling session.

Call Context Introspection

Call context introspection allows a profiler to instrument the prologue and/or epilogue of any method and gain access to arguments (including the this reference), local variables, and the return value.

This opens up countless possibilities for instrumenting framework methods to learn how a program is utilizing facilities like the thread pool, networking, reflection and so on. It can also be useful for debugging, especially if dealing with assemblies for which the source code is not available.

Instrumenting Managed Allocators

Another improvement we were able to make thanks to the redesigned API was to use instrumented managed allocators when profiling. In the past, we would disable managed allocators entirely when profiling. This would slow down allocation-heavy programs significantly. Now, we insert a call back to the profiler API at the end of managed allocators if profiling is enabled.

Simpler to Work With

On top of these major features, the new API is also simply more pleasant to use. In particular, you no longer have to worry about setting event flags; you simply install a callback and you will get events. Also, you no longer have to use callback installation functions which take multiple callback arguments. Every kind of callback now has exactly one function to install it. This means you will no longer have code such as mono_profiler_install_assembly (NULL, NULL, load_asm, NULL); where it can be unclear which argument corresponds to which callback. Finally, several unused, deprecated, or superseded features and callbacks have been removed.

Breaking Change

The new API completely replaces the old one, so this is a breaking change. We try very hard to not break API/ABI compatibility in Mono’s embedding API, but after much consideration and evaluation of the alternatives, a breaking change was deemed to be the most sensible way forward. To aid with the transition to the new API, Mono will detect and refuse to load profiler modules that use the old API. Developers who wish to support both the old and new APIs by compiling separate versions of their profiler module may find the new MONO_PROFILER_API_VERSION macro useful.

A presentation with more details is available in PowerPoint and PDF formats.

Alexander Köplinger releases

Mono 5.2 is out in the stable channel !

Check out our release notes for more details about what is new on Mono 5.2.

This release was made up of nearly 1000 commits since Mono 5.0 and is the result of many months of work by the Mono team and contributors!

Miguel de Icaza runtime

We have been experimenting with a couple of approaches to bring Mono to the web using WebAssembly - a technology that can efficiently and safely execute code in web browsers without being limited to Javascript. Running code written in C or C++ inside the browser has been a big motivator, but most major programming languages have plans to target WebAssembly as a platform.

WebAssembly has been out for a few months on desktop computers and Android, and with the introduction of iOS 11 it will become nearly universal.

We have done some exploratory work to identify what needs to be done to run Mono on the browser. The early experiments are promising, let me talk about those.

Mono supports various execution modes, it ranges from the traditional fully just-in-time compiled, to fully statically compiled with a couple of hybrid modes in between (statically compiled with JIT, and statically compiled with an interpreter).

Today we have two prototypes running in WebAssembly.

The first one uses the traditional full static compilation mode of Mono, this compiled both the Mono C runtime and the Mono class libraries along with the user code into WebAssembly code. It produces one large statically compiled application. You can try this fully statically compiled Hello World here. The full static compilation currently lives here.

The second prototype compiles the Mono C runtime into web assembly, and then uses Mono’s IL interpreter to run managed code. This one is a smaller download, but comes at the expense of performance. The mixed mode execution prototype currently lives here was merged to Mono master in November 2017.

You can see C# Hello World, or this F# code in action:

open System
type Shape = | Circle of float | Square of float * float
let main argv =
  let s = Circle 10.
  Console.WriteLine (s)

Currently neither approach has been optimized for size, they are using the full Mono Desktop profiles. The size should come down significantly once we use a custom profile based on the mobile profile and remove many of the unnecessary features.

There are both interesting technical challenges to solve in the WebAssembly space, some hard (stack walks) and some straightforward (single threaded).

The development experience is another component that will be interesting to study. Clearly we want to have our cake and eat it too, so we desire the high performance that comes from doing full linking and statically compiling everything to WebAssembly. But for quick iterative development, clearly something like an interpreter or a JIT compiler in the browser with cached versions of libmono and the core managed libraries is desirable.

Miguel de Icaza releases

After almost 12 years, we are upgrading the guts of HttpWebRequest, the engine that powers the basic HTTP client stack in Mono.

.NET offers two sets of HTTP client APIs, the original HttpWebRequest that offers a comprehensive and configurable API that can communicate with HTTP 1.x servers using the original .NET 1.0 programming model, and a modern, pluggable and more limited API in the form of HttpClient which can also talk to HTTP 2.0 servers.

HttpClient is an interesting API as it has been designed to allow for different providers to be used and was also designed with async programming support in mind.

On each platform, HttpClient tries to use the best available transport:

Host/Runtime Backend
Windows/.NET Framework HttpWebRequest
Windows/Mono HttpWebRequest
Windows/UWP Windows native WinHttpHandler (HTTP 2.0 capable)
Windows/.NET Core Windows native WinHttpHandler (HTTP 2.0 capable)
Android/Xamarin Default to Android’s HTTP transport
Can be configured to be HttpWebRequest.
iOS, tvOS, watchOS/Xamarin Default to NSUrlSession (HTTP 2.0 capable)
Can be configured to use HttpWebRequest.
macOS/Xamarin Default to NSUrlSession (HTTP 2.0 capable)
Can be configured to use HttpWebRequest
macOS/Mono HttpWebRequest
macOS/.NET Core libcurl-based HTTP transport (HTTP 2.0 capable)
Linux/Mono HttpWebRequest
Linux/.NET Core libcurl-based HTTP transport (HTTP 2.0 capable)

As you can see, while HttpWebRequest is not the default across the board and lacks HTTP 2.0 capabilities, it is still available in various configurations in Mono-powered stacks (Mono and Xamarin).

Mono’s original HttpWebRequest stack was written in 2004 and was built using the asynchronous APIs that were available in .NET 1.0, that is the BeginInvoke / EndInvoke patterns, a pattern that was heavily based on queuing work, waiting for a result and resuming execution at a later point.

While the code grew in capabilities, features and reliability over the years, all of this was built on top of these 2004-era programming idioms. The code is difficult to read, to follow and understand. Simple bugs can take a long time to fix, and subtle problems can burn the most passionate developer. It has been a source of frustration for both us, and our users when a rare bug comes up in this stack.

At one point we tried to use the .NET Reference Source implementation, but that version was just too difficult to extract as it relies on many internals that do not exist in Mono, and in turn relies on Windows capabilities that we would have needed to support [1]. But this code also used those old idioms and while it might have fewer bugs and we might increase the compatibility of the stacks, the extraction was too complicated.

Over the years we have embraced expedience in the form of band-aids for this code, over the right fix, given just how complex the code had gotten. We are facing a couple of corner cases that are hard to fix and we wanted to fix the code for good, not apply another band aid.

So we decided to refactor the code from the old idioms from 2004 into 2017 idioms, using everything that is available to us in C# 7.

Chief among those capabilities is the use of async programming that removes both the complexity from the code and makes the code a pleasure to read and understand. It has taken us a few weeks to do the port, and we are currently validating every known scenario and test suite against it, but things are looking very promising.

You can track the state of this effort on Martin’s GitHub pull request.

[1] At one point we did write such a layer, the effort still lives dormant on a branch, for anyone that might be interested in resuming that work.