Entity Framework Core 3.1 Bug vs 2.2, Speed and Memory During Streaming

Full source code available here.

A while ago I wrote a blog post about the advantages of streaming results from Entity Framework Core as opposed to materializing them inside a controller and the returning the results. I saw very significant improvements in memory consumption when streaming as shown in the chart below.

For this post I updated the code to use Entity Framework Core 3.1.8 on .NET Core 3.1 and was very surprised to see that streaming data did not work as before. Memory usage was high and the speed of the response was much poorer than I expected.

I tried Entity Framework Core 2.2.6 on .NET Core 3.1, and it behaved as I expected, fast and low memory.

Below is how I carried out my comparison.

The Setup
A simple Web API application using .NET Core 3.1.

A local db seeded with 10,000 rows of data using AutoFixture.

An API controller with a single action method that returns data from the database. Tracking of Entity Framework entities is turned off.

[Route("api/[controller]")]
[ApiController]
public class ProductsController : ControllerBase
{
    private readonly SalesContext _salesContext;

    public ProductsController(SalesContext salesContext)
    {
        _salesContext = salesContext;
    }

    [HttpGet("streaming/{count}")]
    public ActionResult GetStreamingFromService(int count)
    {
        IQueryable<Product> products = _salesContext.Products.OrderBy(o => o.ProductId).Take(count).AsNoTracking();
        return Ok(products);
    }
}

It’s a very simple application.

The Test
I “warm up” the application by making a few requests.

Then I use Fiddler to hit the products endpoint 20 times, each request returns 8,000 products.

I do the same for both Entity Framework Core 3.1.8 and Entity Framework Core 2.2.6.

The Results
Framework Core 2.2.6 out performs Framework Core 3.1.8 in both speed of response and memory consumption.

Here is the comparison of how quickly they both load the full set of results.

Time to First Byte (TTFB) –

  • EF 2.2.6 returns data before we even reach 10ms in all cases and before 2ms in most cases.
  • EF 3.1.8 is never faster than 80ms and six requests take more than 100ms.

Overall Elapsed –

  • EF 2.2.6 returns most requests in the 26ms to 39ms range, with only two taking more than 40ms.
  • EF 3.1.8 returns four requests in less than 100ms. The rest are in the 100ms to 115ms range.

The speed difference between EF Core 2.2.6 and EF Core 3.1.8 is significant, to say the least.

Memory Usage –

Here is the graph of memory usage during the test.

  • EF Core 2.2.6 maintains low memory usage.
  • EF Core 3.1.8 consumes significantly more memory to do the same job.

Remember, they are both using the same application code, it’s only the versions of Entity Framework that differ.

I also performed tests with other versions of Entity Framework Core 3.1.*, and Entity Framework 2.* and same very similar results. It seems something in the 3.1 stack is done very differently than the 2.* stack.

As always, you can try it for yourself with the attached zip.

Full source code available here.

Performance Comparison of Entity Framework Core 2.1 and Dapper 1.5

tl;dr – ignore most (maybe all) of the posts out there comparing Dapper and Entity Framework performance, you need to measure it yourself.
Here’s why –
1. Some are angry opinion pieces from people who don’t like one technology or the other and clearly haven’t run any tests.
2. All are out of date (as this one will be shortly) because the libraries move so quickly.
3. None are running against the database and network you have.
4. Your standard queries are far more important than whatever arbitrary queries they used.

Longer Version
I recently had to decide between a few different ORMs for a project, the specifics of the project are irrelevant, but I was using a Postgres database that had lots of data (my lots and your lots are probably different, that I had hundreds of millions of records should not matter to you).

Most of the posts comparing Dapper and EF that I read came down on the side of Dapper, some by a tiny margin and one by no margin but a lot of bluster.

Rather than trust any of these posts I wrote my own benchmarking application using the BenchmarkDotNet library.

I picked seven of the most representative queries I was making and coded them up in both Dapper and EF. Some of the queries pulled back tens of thousands of records and some brought back scores of records, but the most important thing was that these were queries I was going to run in the finished application.

BenchmarkDotNet takes care of things making sure the code has been jitted, that everything has been “warmed up” and then it runs the same request repeatedly to get a useful average.

On top of this, I ran the whole suite of benchmarking tests multiple time, at different times of the day because my application does not live in a vacuum, I needed to know how it would react when the network or database is under more or less load.

At the end of all this Entity Framework came out on top; for the vast majority of test runs it performed better.

Here are the results of one of the test runs, I didn’t pick the best or the worst, but this one is is indicative of what I saw for the majority of tests.

MethodMeanErrorStdDevMedian 
EF_Query1431.19 ms 13.9615 ms38.1581 ms 445.98 ms
Dapper_Query1689.88 ms 35.9716 ms102.7689 ms668.13 ms
EF_Query245.21 ms 1.5613 ms4.6816 ms46.35 ms
Dapper_Query261.55 ms1.7915 ms5.0850 ms62.19 ms
EF_Query3278.72 ms 51.4681 ms150.8833 ms351.38 ms
Dapper_Query3298.64 ms7.3619 ms20.2561 ms297.15 ms
EF_Query417.97 ms 0.3570 ms0.6310 ms17.93 ms
Dapper_Query426.59 ms 0.6516 ms1.7592 ms25.78 ms
EF_Query570.78 ms 1.4936 ms4.0985 ms71.21 ms
Dapper_Query5116.71 ms2.5632 ms7.4015 ms116.36 ms
EF_Query6189.71 ms50.1289 ms148.7825 ms311.21 ms
Dapper_Query6248.61 ms7.2891 ms21.1751 ms304.16 ms
EF_Query7266.62 ms6.2219 20.3179 ms281.27 ms
Dapper_Query7304.27 ms7.8163 ms15.1561 ms301.21 ms

I was a little surprised at this result, but there you have it. EF Core 2.1 is better for me than Dapper.

Out of interest, I took a small portion of my data and put it into SqlServer, and re-ran the tests. This time, Dapper came out slightly ahead. But the SqlServer is on a different network than the Postgres db, is under different load and had a much small dataset so not a realistic exercise.

Conclusion
The moral of the story is you have to test performance yourself, there is absolutely no way you can extrapolate from a test someone wrote about in a blog (including this one) when judging how an ORM will perform for you.