Design Meeting Notes – October 3, 2013

Entity Framework 6.0.1 performance status

Metrics

The metrics we have been monitoring for EF6 performance include:

  • Model loading
    • Code First
    • EDMX
  • Web scenario throughput
  • Query compilation
  • Query execution
    • Warm
  • CRUD operations

The vast majority of scenarios are showing perf as good or better than EF5.

We were missing some metrics for model loading when working on EF6 RTM, especially around creation of databases using initializers for smaller models. We now have these covered for 6.0.1 and perf is looking better again in this area—see below.

Recent model loading perf work items

Work item Description Status

1272: DbContext initialization

Issue with check for History table

Moved to 6.1.0

1662: EdmModelDiffer

Model diffing regression

Fixed in 6.0.1, verified

1663: MetadataItem.Annotations

Minor regression on GetClrType()

Moved to 6.1.0

1668: ReadOnlyMetadataCollection

Bug fix introduced perf regression

Fixed in 6.0.1, pending verification*

1674: HistoryContext construction

Building instance increased in cost

Fixed in 6.1.0, pending verification

1675: GetStorageMappingItemCollection

Cost increased due to new code

Fixed by Andrew’s fix to 1662

1687: Multiple differ calls Differ call for Seed not needed Fixed in 6.0.1, pending verification
1454: Model built twice Model built once by core, once by Migrations Fixed in 6.0.1

* the fix was tested as a bootleg before it was checked in

Cold startup performance

Model loading using Code First

These tests look at loading a model using the DropCreateDatabaseAlways initializer. This means that they always incur the full cost of database creation. This is not the most important scenario from a production perspective where it is much more common for the database to already exist. However, they are important for apps that create a new database on startup and also as a measure of worst-case startup performance and F5/testing scenarios.

  • Big model (>1000 tables)
    • EF 5: 115 seconds
    • EF 6.0.1 before fix for 1668: 52.8 seconds
    • EF 6.0.1 after fix for 1668: 45.1 seconds
  • Small model
    • EF 5: 1584 milliseconds
    • EF 6.0.1: 2790 milliseconds
    • EF 6.0.1 + ngen: 929 milliseconds

Model loading using EDMX

  • Big model (>1000 tables)
    • EF 5: 26.6 seconds
    • EF 6-rc: 19.5 seconds
    • EF 6.0.1: 19.5 seconds
  • Small model
    • EF 5: 244 milliseconds
    • EF 6.0.1: 1568 milliseconds
    • EF 6.0.1 + ngen: 257 milliseconds

Ngen clearly has a big impact in this scenario. The .NET Framework parts of EF5 were always ngened. We are investigating what can be done to make it easier to ngen EF6.

Web scenario throughput

  • We recommend customers to use one context per request
  • Cost of creating DbContext was not as good as ObjectContext (although DbContext does do more)
  • Fixed in EF 6
  • Added a few tests to keep track of this scenario. One such test yields the following results:
    • EF 5: 1800 requests per second
    • EF 6: 3300 requests per second

All tests: 5 runs, median value, high performance power setting, release build

Other metrics

  • Query compilation
  • Query execution
    • We are only testing warm queries and they are behaving well.
    • Only exception: Spatial data types. For example, materialization of spatial types:
      • Before regression: 8480 samples
      • After regression:11681 samples
      • After regression + AsStreaming: 8654 samples
  • CRUD operations
    • These are not the most consistent tests but their behavior has not changed since EF5; perf is roughly the same in the profiles
    • There is a possible small regression in some SaveChanges scenarios which is being investigated

Last edited Oct 18, 2013 at 8:32 PM by ajcvickers, version 3