.NET decimal serialization with protobuf

Sat, Sep 17, 2022 6-minute read

Introduction

When working with .NET and financial data, then its very common that the decimal type is used, since it is precise and does not lose information when you do calculations.

i.e. take the following code:

var x = 3.999999999999999d;
Console.WriteLine(x);
var result = x+x;
Console.WriteLine(result);

You would assume the result would be 7.999999999999998 - but in fact it will print out 4 - which for all intends and purposes is okay for most usages - but if you require precision on all decimal places, double is not good enough because the datatype cannot represent the precision correctly.

So this is why decimal was invented because it retains all the information required to be able to accurately calculate and not lose information.

It is also much slower, since most CPU’s do not have processor instructions to operate on decimals.

Microsofts solution

Back to the story - so when using decimals and you want to send them over the wire to another system serialized with protobuf you run in the issue that protobuf have no way of natively serializing decimals.

People have done crazy stuff like converting to a string and then transferring that - and it works, but its extremely ugly and slow - so microsoft suggested that people use the following solution to serialize decimals:

package CustomTypes;

// Example: 12345.6789 -> { units = 12345, nanos = 678900000 }
message DecimalValue {

    // Whole units part of the amount
    int64 units = 1;

    // Nano units of the amount (10^-9)
    // Must be same sign as units
    sfixed32 nanos = 2;
}

And the corresponding .Net class:

// Generated by protobuf
public partial class DecimalValue 
{
	public long Units;
	public int Nanos;	
}
// part of class with converters
public partial class DecimalValue
{
	private const decimal NanoFactor = 1_000_000_000;
	public DecimalValue(long units, int nanos)
	{
		Units = units;
		Nanos = nanos;
	}

	public static implicit operator decimal(DecimalValue grpcDecimal)
	{
		return grpcDecimal.Units + grpcDecimal.Nanos / NanoFactor;
	}

	public static implicit operator DecimalValue(decimal value)
	{
		var units = decimal.ToInt64(value);
		var nanos = decimal.ToInt32((value - units) * NanoFactor);
		return new DecimalValue(units, nanos);
	}
}

This all works perfectly - your decimals will get converted into a type with two values. One for the value before the decimal point and the other for the data after the decimal point.

The problem I have with this solution is that it takes up CPU cycles to convert - and also it truncates the decimals after the decimal point, i.e. the value 1237546.45678900001 will get truncated to: 1237546.456789 after a roundtrip - because of the “NanoFactor” - so if you need higher precision you would need to increase the “NanoFactor” to a higher number and change the Nanos to a long. But if you have more decimals than what can be represented by a long you can never fully protect yourself from losses no matter how big you make “NanoFactor”.

So if you have a class defined like this:

public class MyData
{
	public DecimalValue Value1 { get; set; }
	public DecimalValue Value2 { get; set; }
	public DecimalValue Value3 { get; set; }
	public DecimalValue Value4 { get; set; }
	public DecimalValue Value5 { get; set; }
}

Then every single time you assign a decimal to these properties - the CPU has to do the calculations in the implicit operators defined above. While this is okay if you do not publish a lot of data, then if you publish many millions of records every minute its a lot of wasted CPU cycles.

My alternative solution

So my alternative approach to this - is to use a feature called unions, which allow structs to overlap data.

First we need a struct that matches the internals of decimal. Internally decimal is storing 3 fields:

        // The lo, mid, hi, and flags fields contain the representation of the
        // Decimal value. The lo, mid, and hi fields contain the 96-bit integer
        // part of the Decimal. Bits 0-15 (the lower word) of the flags field are
        // unused and must be zero; bits 16-23 contain must contain a value between
        // 0 and 28, indicating the power of 10 to divide the 96-bit integer part
        // by to produce the Decimal value; bits 24-30 are unused and must be zero;
        // and finally bit 31 indicates the sign of the Decimal value, 0 meaning
        // positive and 1 meaning negative.
        //
        // NOTE: Do not change the order and types of these fields. The layout has to
        // match Win32 DECIMAL type.
        private readonly int _flags;
        private readonly uint _hi32;
        private readonly ulong _lo64;

So our version would look like:

public struct DecimalEquivalent 
{
	public int Flags;
	public uint Hi32;
	public ulong Low;
}

Then we have our “magic” struct that allows us to reference directly the decimal values without any CPU usage at all:

[StructLayout(LayoutKind.Explicit)]
public struct FastDecimalConverter 
{
	[FieldOffset(0)]
	public DecimalEquivalent Equivalent;
	[FieldOffset(0)]
	public decimal Value;
}

And to make it all work we would add some implicit operators to DecimalEquivalent:

public struct DecimalEquivalent
{
	public int Flags;
	public uint Hi32;
	public ulong Low;
	public static implicit operator decimal(DecimalEquivalent grpcDecimal)
	{
		var conv = new FastDecimal { Equivalent = grpcDecimal };
		return conv.Value;
	}

	public static implicit operator DecimalEquivalent(decimal value)
	{
		var conv = new FastDecimal { Value = value };
		return conv.Equivalent;
	}
}

So when we change our existing class to use the DecimalEquivalent:

public class MyData
{
	public DecimalEquivalent Value1 { get; set; }
	public DecimalEquivalent Value2 { get; set; }
	public DecimalEquivalent Value3 { get; set; }
	public DecimalEquivalent Value4 { get; set; }
	public DecimalEquivalent Value5 { get; set; }
}

We can assign decimals to these properties without having to do multiplications at all.

Conclusion

I have tested the improvement in speed and isolated on my machine its at least 60-90 times as fast to use a union approach than using the multiplication approach.

The only disadvantage is that you will potentially serialize 4 bytes more with the union approach - unless you needed to increase the Nanos part of the microsoft way to get more precision.

Luckily protobuf serializes with variable length integers, so in reality the difference will probably not be 4 bytes.

If you take a look at the following example decimal:

1237546.456789

It will be represented like this inside the DecimalValue:

Units = 1237546 
Nanos = 456789000

And like this inside the DecimalEquivalent:


Flags = 393216 
Hi32 = 0 
Low = 1237546456789

So if you do not want to lose precision when transferring decimals, you need to not use the microsoft approach.

Comments, feel free to leave a comment below or send me an email.