Background

Applications connecting to a database server have always been vulnerable to connection breaks due to back-end failures and network instability. However, in a LAN based environment working against dedicated database servers these errors are rare enough that extra logic to handle those failures is not often required. With the rise of cloud based database servers such as Windows Azure SQL Database and connections over less reliable networks it is now more common for connection breaks to occur. This could be due to defensive techniques that cloud databases use to ensure fairness of service, such as connection throttling, or to instability in the network causing intermittent timeouts and other transient errors.

Goals

  • Pri0: Provide the ability to retry actions on a variety of failures automatically, minimizing the amount of defensive code and retry logic that is necessary to handle transient failures in common cloud data access scenarios
  • Pri1: A user should be able to implement their own retry strategies and decide which exceptions should cause a retry and which should not.

Non Goals

  • Fully supporting moving unchanged EF-based applications to scenarios with unreliable connections
    • Resiliency is off by default, although we will try to help people know about it by proving information in transient failure exceptions

Design Meeting Notes

http://entityframework.codeplex.com/wikipage?title=Design%20Meeting%20Notes%20-%20November%208%2c%202012
http://entityframework.codeplex.com/wikipage?title=Design%20Meeting%20Notes%20-%20December%206%2c%202012
http://entityframework.codeplex.com/wikipage?title=Design%20Meeting%20Notes%20-%20January%2024%2c%202013

Experience

When using the connection resiliency feature, EF will take care of retrying a command whenever transient errors occur. As an application writer using connection resiliency you still need to handle scenarios where the error is not transient, or is not resolved before the maximum retries have occurred. In the case of an exception that is not retried, for whatever reason, EF will bubble up the exception in the same way it would have if connection resiliency was not enabled.
Connection resiliency should be able to be enabled with a small change to configuration, with the possibility of writing code to implement your own retry strategies if you need to.
What database operations are automatically retried:
  • Executing an EF query:
    • On the DbContext API: DbSet, DbQuery,
    • On the ObjectContext API: ObjectQuery (both for EntitySQL and LINQ to Entities) and CompiledQuery
  • Executing raw Sql:
    • On the DbContext API: SqlQuery and ExecuteSqlCommand
    • On the ObjectContext API: ExecuteStoreQuery and ExecuteStoreCommand
  • Invoking a stored procedure through ExecuteFunction
  • Calling SaveChanges on a DbContext or ObjectContext
  • Executing Migrations
  • Using Load or Reload on DbContext
  • Using Refresh on an ObjectContext

What database operations are not automatically retried:
  • Executing queries in EntityClient, e.g. using EntityCommand

Implementation

Connection retry is taken care of by an implementation of the IDbExecutionStrategy interface. Implementations of the IDbExecutionStrategy will be responsible for accepting an operation and, if an exception occurs, determining if a retry is appropriate and retrying if it is. EF will use an IDbExecutionStrategy each time it executes an operation against the database. The IDbExecutionStrategy interface looks like the following:

public interface IDbExecutionStrategy
{
    bool RetriesOnFailure { get; }

    void Execute(Action operation);
    TResult Execute<TResult>(Func<TResult> operation);

    Task ExecuteAsync(Func<Task> operation, CancellationToken cancellationToken);
    Task<TResult> ExecuteAsync<TResult>(Func<Task<TResult>> operation, CancellationToken cancellationToken);
}

Default Implementations

EF will ship with four execution strategies:
  1. DefaultExecutionStrategy: this execution strategy does not retry any operations, it is the default for databases other than sql server.
  2. DefaultSqlExecutionStrategy: this execution strategy does not retry at all, however, it will wrap any exceptions that could be transient to inform users that they might want to enable connection resiliency.
  3. DbExecutionStrategy: this class is suitable as a base class for other execution strategies. It implements an exponential retry policy, where the initial retry happens with zero delay and the delay increases exponentially until the maximum retry count is hit. This class has an abstract ShouldRetryOn method that can be implemented in derived execution strategies to control which exceptions should be retried.
  4. SqlAzureExecutionStrategy: this execution strategy inherits from DbExecutionStrategy and will retry on exceptions that are known to be possibly transient when working with SqlAzure.
Note: Execution strategies 2 and 4 are included in the Sql Server provider that ships with EF, which is in the EntityFramework.SqlServer assembly.

Using an Execution Strategy

The main way to use execution strategies is to configure EF so that it will use the given strategy for all actions. In order to do this you need to add a resolver to the DependencyResolver chain that EF will use to find an IDbExecutionStrategy when it requires one. There is a sugar method added to DbConfiguration called ExecutionStrategy. To use the SqlAzureExecutionStrategy you would call the method in your DbConfiguration like this:

public class MyConfiguration : DbConfiguration
{
    public MyConfiguration()
    {
        SetExecutionStrategy("System.Data.SqlClient", () => new SqlAzureExecutionStrategy());
    }
} 

The provider invariant name is used as a key for the factory that will return a new execution strategy. You can also use SqlProviderServices.ProviderInvariantName. We accept a factory rather than an instance of the IDbExecutionStrategy because each retriable operation should use a new instance of DbExecutionStrategy.

EF providers have the opportunity to register an execution strategy with the dependency resolver, the SqlServer provider that ships with EF will register the DefaultSqlExecutioStrategy described in the section above. Other providers will return whichever strategy they think is appropriate for their database. If no execution strategy is found then EF will use the DefaultExecutionStrategy.

The other way that you could use an execution policy is to create an instance of the execution policy and use it to execute an action yourself. This might be useful when you do not need to retry for most things, but want to use one for a small number of queries. An example of using the execution strategy this way would be something like this:

var executionStrategy = new SqlAzureExecutionStrategy();
var blogs = executionStrategy.Execute<IEnumerable<Blog>>(GetBlogs);

NOTE: The DbExecutionStrategy base class that SqlAzureExecutionStrategy inherits from is not designed to be thread safe.

Configuring SqlAzureExecutionStrategy

The constructor of SqlAzureExecutionStrategy can accept two parameters, MaxRetryCount and MaxDelay. MaxRetry count is the maximum number of times that the strategy will retry before throwing a RetryLimitExceededException. The MaxDelay is a TimeSpan representing the maximum delay between retries that the execution strategy will use.

public class MyConfiguration : DbConfiguration
{
    public MyConfiguration()
    {
        SetExecutionStrategy("System.Data.SqlClient", () => new SqlAzureExecutionStrategy(1, TimeSpan.FromSeconds(30)));
    }

}

Another way to modify the SqlAzureExecutionStrategy is to add an exception that you want to retry on. The easiest way to do this is to create a new ExecutionStrategy that derives from SqlAzureExecutionStrategy and adds your special case, for example:

public class MySqlAzureExecutionStrategy1 : SqlAzureExecutionStrategy
{
    private List<int> _errorCodesToRetry = new List<int>
        {
            //List custom error codes here.
        };

    protected override bool ShouldRetryOn(Exception exception)
    {
        var sqlException = exception as SqlException;
        if (sqlException != null)
        {
            foreach (SqlError err in sqlException.Errors)
            {
                // Enumerate through all errors found in the exception.
                if (_errorCodesToRetry.Contains(err.Number))
                {
                    return true;
                }
            }
        }
        return base.ShouldRetryOn(exception);
    }
}

User Transaction

One requirement of using connection resiliency with EF is that EF is responsible for the creation and management of transactions. This is because EF will need to rollback and retry the current transaction in order to retry, which we cannot do with a transaction that was created and passed to EF. You can work around this by wrapping your entire logic, including the creation of your transaction inside an explicit usage of an execution strategy. More details on this, and examples of how to handle user created transactions, can be found here: http://msdn.microsoft.com/en-US/data/dn307226

Buffering of Query Results

Buffering is required when using connection resiliency in order to keep the state manager consistent when there is a retry. However, there are also other benefits to buffering the main ones being:
  1. The connection is potentially open and in-use for a shorter period of time
  2. Multiple Active Result Sets (MARS) would not be needed for nested queries, such as those that come with lazy loading.
Because of this buffering is now the default. We will provide an AsStreaming method on IQueryable to allow the old behavior, similar to the AsNoTracking method for non-tracking queries.

Implementation

When in streaming mode EF uses a data reader that is wrapped in an IEnumerable that reads from the data reader as rows are required for materialization.
With the buffering enabled the process is the same, except that the underlying provider DataReader will be read into memory, and close the connection, before it begins to yield rows from in-memory collection of data.

AsStreaming

An AsStreaming extension method will be added to IQueryable that will return a query that will use the streaming behavior described above.

Last edited Mar 6, 2014 at 11:07 PM by AndriySvyryd, version 18