2

Closed

NullReferenceException in new tooling when populating tree

description

There is a bug in runtime where we started throwing a NullReferenceException if the MSL is invalid instead of returning EdmErrors/throwing MappingException. This affects EF Tooling - when loading an edmx containing invalid mapping instead of showing errors as we used to a message box reading "The operation could not be completed: Invalid Pointer" is shown. It is also possible to make VS crash due to this error.

This particular bug can only occur if the model has Independent Associations but we should check if there are not similar regressions introduced in EF6.

Repro steps:
  • In the designer create a model with relationships from the database (e.g. Customer-Orders from Northwind) - make sure the Include Foreign Key checkbox is unchecked
  • Open the edmx in the Xml editor in VS
  • Change the StoreEntitySet attribute value in the AssociationSetMapping element to point to non-existing entity set
  • Double click on the edmx file to open it the designer
Expected behavior: the file can be open, errors displayed in the VS error pane
Actual behavior: the file is not open, "The operation could not be completed: Invalid Pointer" message box is shown

A standalone repro for the runtime would be just to try create a StorageMappingItemCollection using artifacts from the invalid edmx file.

Repro steps to crash the VS
  • In the designer create a model with relationships from the database (e.g. Customer-Orders from Northwind) - make sure the Include Foreign Key checkbox is unchecked
  • Open the edmx in the external editor (e.g. notepad)
  • Change the StoreEntitySet attribute value in the AssociationSetMapping element to point to non-existing entity set and save the file
  • VS will detect that the file changed and will ask the user if they want to reload
    Clicking "Yes" will make VS crash
This issue was originally found in the NuoDb provider where an invalid design time MSL started causing issues after upgrading tooling to EF6.1. Due to a different exception type in previous versions we would ignore the exception and would probe for v1 schema but in EF6.1 we started failing.

Here are the original steps:

Steps to reproduce:
  • Clean VS 2013 Update 1 installation.
  • Uninstall original EFTools.
  • git clone https://git01.codeplex.com/entityframework
  • BuildEFTools /t:Install
  • Add Reference: NuoDb.Data.Client + EntityFramework.NuoDb
  • Install-Package EntityFramework
  • <provider invariantName="NuoDb.Data.Client" type="EntityFramework.NuoDb.NuoDbProviderServices, EntityFramework.NuoDb" />
  • Build
  • Add > New Item
  • EF Designer from database
  • Select connection to NuoDB
  • NRE:
    An exception of type 'System.NullReferenceException' occurred in Microsoft.Data.Entity.Design.VersioningFacade.dll but was not handled in user code
System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Data.Entity.Core.Mapping.MappingItemLoader.ValidateEntitySetFunctionMappingClosure(XPathNavigator nav, EntityContainerMapping entityContainerMapping)
   at System.Data.Entity.Core.Mapping.MappingItemLoader.LoadEntityContainerMappingChildNodes(XPathNavigator nav, EntityContainerMapping entityContainerMapping, EntityContainer storageEntityContainerType)
   at System.Data.Entity.Core.Mapping.MappingItemLoader.LoadEntityContainerMapping(XPathNavigator nav)
   at System.Data.Entity.Core.Mapping.MappingItemLoader.LoadMappingChildNodes(XPathNavigator nav)
   at System.Data.Entity.Core.Mapping.MappingItemLoader.LoadMappingItems(XPathNavigator nav)
   at System.Data.Entity.Core.Mapping.MappingItemLoader.LoadMappingItems(XmlReader innerReader)
   at System.Data.Entity.Core.Mapping.MappingItemLoader..ctor(XmlReader reader, StorageMappingItemCollection storageMappingItemCollection, String fileName, Dictionary`2 scalarMemberMappings)
   at System.Data.Entity.Core.Mapping.StorageMappingItemCollection.LoadItems(IEnumerable`1 xmlReaders, IList`1 mappingSchemaUris, Dictionary`2 userDefinedQueryViewsDict, Dictionary`2 userDefinedQueryViewsOfTypeDict, Double expectedVersion)
   at System.Data.Entity.Core.Mapping.StorageMappingItemCollection.Init(EdmItemCollection edmCollection, StoreItemCollection storeCollection, IEnumerable`1 xmlReaders, IList`1 filePaths, Boolean throwOnError)
   at System.Data.Entity.Core.Mapping.StorageMappingItemCollection..ctor(EdmItemCollection edmItemCollection, StoreItemCollection storeItemCollection, IEnumerable`1 xmlReaders, IList`1 filePaths, IList`1& errors)
   at System.Data.Entity.Core.Mapping.StorageMappingItemCollection.Create(EdmItemCollection edmItemCollection, StoreItemCollection storeItemCollection, IEnumerable`1 xmlReaders, IList`1 filePaths, IList`1& errors)
   at Microsoft.Data.Entity.Design.VersioningFacade.ReverseEngineerDb.StoreSchemaConnectionFactory.LoadMappingItemCollection(DbProviderManifest providerManifest, String mslName, EdmItemCollection edmItemCollection, StoreItemCollection storeItemCollection)
   at Microsoft.Data.Entity.Design.VersioningFacade.ReverseEngineerDb.StoreSchemaConnectionFactory.GetProviderSchemaMetadataWorkspace(IDbDependencyResolver resolver, String providerInvariantName, DbConnection providerConnection, Version targetSchemaVersion)
   at Microsoft.Data.Entity.Design.VersioningFacade.ReverseEngineerDb.StoreSchemaConnectionFactory.Create(IDbDependencyResolver resolver, String providerInvariantName, String connectionString, Version targetSchemaVersion)
   at Microsoft.Data.Entity.Design.VersioningFacade.ReverseEngineerDb.StoreSchemaConnectionFactory.Create(IDbDependencyResolver resolver, String providerInvariantName, String connectionString, Version maxAllowedSchemaVersion, Version& storeSchemaModelVersion)
   at Microsoft.Data.Entity.Design.VisualStudio.ModelWizard.Engine.DatabaseMetadataQueryTool.ExecuteDatabaseMetadataQuery(String esqlQuery, EntityStoreSchemaFilterObjectTypes types, ModelBuilderSettings settings, DoWorkEventArgs args)
   at Microsoft.Data.Entity.Design.VisualStudio.ModelWizard.Engine.DatabaseMetadataQueryTool.GetTablesFilterEntries(ModelBuilderSettings settings, DoWorkEventArgs args)
   at Microsoft.Data.Entity.Design.VisualStudio.ModelWizard.Engine.DatabaseConnectionSsdlAggregator.GetTableFilterEntries(DoWorkEventArgs args)
   at Microsoft.Data.Entity.Design.VisualStudio.ModelWizard.Gui.WizardPageSelectTables.bgWorkerPopulateTree_DoWork(Object sender, DoWorkEventArgs args)
   at System.ComponentModel.BackgroundWorker.OnDoWork(DoWorkEventArgs e)
   at System.ComponentModel.BackgroundWorker.WorkerThreadStart(Object argument)
It's the line setMappingPerTable.Add(fragment.TableSet, setMapping); where the fragment is null.

Mentioned "NuoDB" files attached.

file attachments

Closed Jun 12 at 4:46 PM by RoMiller
Re-closing this issue as it appears the original scenario that this bug related to is fixed.

The triage team discussed and decided the new error found by Maurycy is "Won't Fix" since you have to edit a section of the file explicitly marked as "do not edit".

comments

moozzyk wrote Mar 3 at 8:12 PM

This is a bug in runtime where we would fail with an NRE if the StoreEntitySet does not exist for the mapping. In this particular bug the provider has the following association set mapping defined:
<AssociationSetMapping Name="TableTableConstraints" StoreEntitySet="SConstraints" TypeName="Store.TableOrViewConstraint">
  <EndProperty Name="Parent">
    <ScalarProperty Name="Id" ColumnName="ParentId" />
  </EndProperty>
  <EndProperty Name="Constraint">
    <ScalarProperty Name="Id" ColumnName="Id" />
  </EndProperty>
</AssociationSetMapping>
However there the SConstraints EntitySet does not exist in the StoreItemCollection.

Note that the artifacts are invalid and therefore the only thing we can do on our side is to check the condtion and throw a better exception instead of failing with a NullReferenceException.

cincura_net wrote Mar 4 at 8:30 AM

Ahh, interesting. Surprisingly it works fine with 6.0.2 tooling (or even EF5 tooling). I'll fix my side as well.

moozzyk wrote Mar 4 at 3:39 PM

In EF 6.0.2 we threw the following exception:

Unhandled Exception: System.Data.Entity.Core.MappingException: Schema specified
is not valid. Errors:
(210,10) : error 2007: The Table 'SConstraints' specified as part of this MSL do
es not exist in MetadataWorkspace.
(218,12) : error 2007: The Table 'SConstraints' specified as part of this MSL do
es not exist in MetadataWorkspace.
(224,10) : error 2007: The Table 'SConstraints' specified as part of this MSL do
es not exist in MetadataWorkspace.
(234,10) : error 2007: The Table 'SConstraints' specified as part of this MSL do
es not exist in MetadataWorkspace.
(240,10) : error 2007: The Table 'SConstraints' specified as part of this MSL do
es not exist in MetadataWorkspace.
(294,6) : error 2007: The Table 'SForeignKeys' specified as part of this MSL doe
s not exist in MetadataWorkspace.
(294,6) : error 2063: At least one property must be mapped in the set mapping fo
r 'TableForeignKeys'.
(324,8) : error 2007: The Table 'SConstraints' specified as part of this MSL doe
s not exist in MetadataWorkspace.
(342,8) : error 2007: The Table 'SForeignKeys' specified as part of this MSL doe
s not exist in MetadataWorkspace.
(351,8) : error 2007: The Table 'SForeignKeys' specified as part of this MSL doe
s not exist in MetadataWorkspace.
(360,8) : error 2007: The Table 'SForeignKeys' specified as part of this MSL doe
s not exist in MetadataWorkspace.
at System.Data.Entity.Core.Mapping.StorageMappingItemCollection.Init(EdmItemC
ollection edmCollection, StoreItemCollection storeCollection, IEnumerable1 xmlR
eaders, IList
1 filePaths, Boolean throwOnError)
at System.Data.Entity.Core.Mapping.StorageMappingItemCollection..ctor(EdmItem
Collection edmCollection, StoreItemCollection storeCollection, IEnumerable`1 xml
Readers)
at NuoDbTest.Program.LoadMappingItemCollection(String versionSuffix) in c:\Us
ers\pawelka\Documents\Visual Studio 2013\Projects\NuoDbTest\NuoDbTest\Program.cs
:line 42
at NuoDbTest.Program.Main(String[] args) in c:\Users\pawelka\Documents\Visual
Studio 2013\Projects\NuoDbTest\NuoDbTest\Program.cs:line 30
Press any key to continue . . .

cincura_net wrote Mar 4 at 4:04 PM

Yes, but the wizard continued working. No error message was shown.

moozzyk wrote Mar 4 at 4:49 PM

Attaching repro - if EF 6.0.2 is being used the above exception is thrown. After upgrading to EF6.1.0 Beta a NullReferenceException is thrown.

moozzyk wrote Mar 4 at 4:58 PM

As pointed by Jiri the incorrect schema did not break tooling in EF 6.0.2. This is because for whatever reason the NullReferenceException is on the list of "non-catchable" exceptions (see StoreSchemaConnectionFactory.IsCatchableExceptionType) and is being bubbled up to the surface. When probing for the schema version if there is a "catchable" exception we just try a lower version and since in EF 6.0.2 we got the MappingException for v3 we would try v1 and because v1 artifacts where valid EF designer just used them.

moozzyk wrote Mar 5 at 4:49 PM

Note that this bug can also be hit by the users if they modify their edmx manually and make a mistake. I updated the bug with the repro steps that allow reproducing the issue without any special setup or a bug in the provider.

kendallmiller wrote Mar 21 at 3:59 AM

We're having the same basic issue with VistaDB - just after we shipped our support for EF 6 the tooling was updated and now we get this problem when users attempt DB-first actions. We've attempted debugging this at some length but we just can't work back from the EF source code a possible code path for it to work with a third party provider and not get this problem unless the provider implements all of the latest version of the metadata schema. In point of fact we'd happily implement that but I can't find a spec for it anywhere, beyond attempting to reverse engineer it from the SQL Server implementation.

Are there tests in the EF tooling for ensuring it's possible to create third party providers? Since there are many code paths that have hard coded paths for SQL Server's provider (for example it skips the whole provider probing process that requires the application to be build first with everything in place before it accepts a provider to put it in the list for EF 6 connections) it seems there'd have to be a reference provider or something to ensure it's possible to extend the EF tooling with a provider that isn't SQL Server.

moozzyk wrote Mar 21 at 5:15 PM

@kendallmiller - have you tried adapting the NuoDB repro and use it agains your provider? From the the pic I saw in the tweet it looked like the design mapping schema was incorrect and if this is the case you should be able to repro this without the designer - the NuoDb should point you in the right direction.

kendallmiller wrote Mar 21 at 6:04 PM

You're right, our case is a bit different, although very similar in the repro steps. We're getting an invalid "ProviderIncompatibleException" when it tries to use the designer and as we dig in its because while it correctly detects that we're an EF 6 designer at some point later in the process it reverts to treating us like an EF 4 provider. We'll keep trying to figure out what changed.

That all said, is there a specification for the provider anywhere that lists exactly what metadata queries are required, the rules for them, etc.? We basically are looking at the SQL Source code and trying to adapt it to our situation and hoping we can reverse engineer the rules.

moozzyk wrote Mar 21 at 7:10 PM

@kendalmiller - the way it works is that EF specifies a CSDL and the provider needs to provide an SSDL and an MSL that describes how the SSDL from the provider maps to the CSDL from EF. Currently there are two CSDL files that EF uses ConceptualSchemaDefinition and ConceptualSchemaDefinitionVersion3. If a provider does not support reverse engineering v3 features ("v3 features" AFAIR is actually just about being able to reverse engineer Table Valued Functions) it just needs to provide the StoreSchemaDefinition.ssdl and StoreSchemaMapping.msl files (exposed via DbProviderManifest.GetInformation() which calls the DbProviderManifest.GetDbInformation() method which has to be implemented in the provider's DbProviderManifest derived class). EF designer probes for supported version by trying to load v3 version first and, if this fails, it tries to load version v1 (note while the provider can only support the v3 schemas there is a bug where things will fail if the user creates a non-v3 model. We decided not to fix it because currently all the known providers support v1 schemas and new tooling always create v3 models - even on .NET Framework 4).

If the issue you are hitting is this: https://twitter.com/hb9tws/status/443437280062365696/photo/1 then the queries EF uses don't really matter. EF was not even able to create a valid MetadataWorkspace because apparently the mapping (MSL) is incorrect. EF requires a valid MetadataWorkspace to run any query. Also, if the MetadataWorkspace is correct any valid query should work. This is the reason why queries are private and there is no contract or guarantee what queries will be run to discover the schema - any query should just work as long as the artifacts/metadata are correct. If you really want to know what we run at the moment you can take a look at the EntityStoreSchemaGeneratorDatabaseSchemaLoader which contains some base queries which are composed on in the EntityStoreSchemaQueryGenerator. There are no guarantees however that the queries won't change in the future. Also be aware that the queries are written in ESQL.

kendallmiller wrote Mar 21 at 7:56 PM

That image is from one of our customers so that was the original problem, although it now shows up slightly differently with the latest (6.1) tooling. The thing is, we worked perfectly right up until some tooling update after EF 6.0 - on EF 4, and EF 6.0 - so I'm wondering what changed. We'll review again what we're mapping and see if we can work out from what you've given us where things are going sideways. Thanks for the detailed response.

moozzyk wrote Mar 21 at 10:09 PM

As I wrote before - when EF Designer probes to find the supported version it catches exceptions. However it has a list of exception it does not catch. I suspect that v3 was always broken but because loading the workspace threw MappingException the exception was caught and the designer tried v1 which worked. Now because of the bug in the runtime the exception that is being thrown is not MappingException but a NullReferenceException. NRE is on the list of exception that are not caught so things ultimately break. The NuoDb repro attached tries creating MetadataWorkspace by loading both v3 and v1 schemas allowing you to confirm (or disprove) my hypothesis.

kendallmiller wrote Mar 24 at 12:01 AM

OK, thanks for walking me through the details. I think our V3 support was incorrect in our provider but that problem was previously masked. We've changed our provider to correctly only support V1 and with the current tooling we look good, so we'll promote our fix to NuGet and be on our way!

moozzyk wrote Apr 1 at 5:28 PM

Fixed with changeset 42d1663

HalfNullHalfEmpty - AssociationTypeMapping returned a collection with a null value for mapping fragments if mapping fragment was not initialized. This later caused an NRE during MSL validation and would eventually break the tooling if the provider schemas were invalid (turned out that it was the case for NuoDb and VistaDb). The fix is to return an empty collection if the mapping fragment has not been set which is what we did before we refactored mapping API.

jemartti wrote Apr 10 at 2:30 AM

Verified that Pawel's fix works. Tried hunting around for other areas where we'd throw a NRE (especially around areas where similar regressions would have occurred) but nothing stood out.

** Closed by jemartti 04/09/2014 7:30PM

maumar wrote Jun 11 at 12:55 AM

Still repros, open edmx and add the following line to the DesignerInfoPropertySet:

<DesignerProperty NAME="FDFDS" Value="ffggf" /> />

moozzyk wrote Jun 11 at 4:18 AM

@maumar: I believe this is a completely different issue. The original issue was about EF runtime failing with an NRE when validating MSL and was a regression. As in its core it was a runtime issue it could be reproduced without the designer. This bug is about the DesignerProperty elements which are part of EDMX and as such are not validated by the runtime. As a result the exception must be coming from the designer itself and not from the runtime. The DesignerProperty elements are for private/internal use only and therefore we don't check whether they are valid or not. We actually put the following comment in the edmx created by the designer:
<!-- EF Designer content (DO NOT EDIT MANUALLY BELOW HERE) -->