Feature request: Add support for CosmosDB

[wbuck]

It would be nice to have support for working directly with the Cosmos DB LINQ provider as opposed to working with the EF provider as EF does not generate valid SQL in many situations.

[wbuck]

The one thing I'm unsure about is how to handle the $expand option here. What makes sense from a users perspective.

1. Ignore the expand option
2. Generate a SelectMany call when $expand is specified (I don't think the user would expect this)
3. Throw an InvalidOperationException
4. For each query generate an implicit Select call which includes all properties except for collection properties on the runtime generated type. If $expand is specified then "include" the specified collection property in the runtime generated type.

[BlaiseD]

It depends on whether the concept of navigation properties (both entities and collections) exists in Cosmos DB. If it does not then I would throw the exception. If it does exist then how is it represented as a LINQ query i.e. do what you would do manually.

Maybe it would help to observe the existing code with regard to select and expand, and what happens when none, all or a partial set have been specified.

[wbuck]

In this case there is no concept of a navigation property in Cosmos DB. You can have collections (primitives or objects) but they're not navigation properties. Nothing officially connects one document to another.

You can of course embed an ID of one document into another:

{
    "id": "some user id",
    "name": "bill",
    "posts": [ 
        "id": "some post id",
        "userId": "some user id"
    ] 
}

But you cannot "join" those documents together (either with the C# SDK or Cosmos DB SQL API).

E.g., Users.Include(u => u.Posts)
Is not valid. Two separate queries would have to be made in this case, one to get the users and one to get the posts of a particular user.

First: Users.Where(u => u.Name.Contains("bill");
Followed by: Posts.Where(p => p.UserId == user.Id);

[wbuck]

@BlaiseD Do you know off the top of your head why GetAsync doesn't call GetQueryAsync and then just call ToListAsync directly on the returned IQueryable<T>?

[BlaiseD]

Get, GetAsync were the initial implementation, used joins (includes), call Map and return a collection. GetQuery, GetQueryAsync came later, use projection and return an IQueryable.

[wbuck]

Just doing manual testing at the moment, everything looks good.
One question I do have is what Odata query options are currently supported?

I know that the following are supported:

• $select
• $filter
• $orderby
• $count
• $top
• $skip
• $expand

I don't believe transformations are currently supported, e.g., $apply correct?

Maybe after this PR I'll look into adding transformations, but it'll probably require creating a dynamic runtime type (which I believe I could use this for). Also, returning an IQueryable<TModel> wouldn't be possible. But that's a problem for another day.

[BlaiseD]

I believe that's all for options. FYI - there's a discussion here about configuring the EnableQueryAttribute to ignore the supported options and handle the apply piece - as a workaround.

[BlaiseD]

Some dedicated classes may be worth considering for transforms. There are scenarios e.g. iOS Xamarin/Maui where anonymous types won't work.

[wbuck]

It looks as if I've found a bit of a bug in how we handle expansions. Currently in order to expand a complex type the following syntax is used $expand=MyComplexType\SomeNavigationProperty which we don't seem to currently handle correctly. I need to take a closer look at the LinqExtensions to get a better idea of how we're handling the segments.

I know we're assuming the following expansion syntax: $expand=MyType($expand=SomeNavigationProperty) but this of course implies that MyType is an entity and not a complex type.

This may be a specific CosmosDB concern as I don't believe a collection of complex types makes sense for a relational DB.

[wbuck]

Ok, so it looks like everything is working as expected (at least when tested manually).
Now comes the harder portion, testing...

For other projects I've used Testcontainers to spin up a docker image running the Cosmos DB emulator.

I was thinking of doing the same here for integration testing. Thoughts?

[BlaiseD]

Don't think I've used one. The only restriction is that anyone should be able to clone the repository build and test. The tests e.g. AutoMapper.OData.EF6.Tests should also work in the CI and release builds - not the web tests we switch those off.

BTW adding this Cosmos DB piece to your repo for your own NuGet library is also an option - adding it here is great too.

[wbuck]

Using Docker containers (using the Testcontainers library) for testing I believe requires Docker to be installed. Unfortunately there is no in-memory Cosmos database, only the Cosmos DB emulator. My biggest concern was the CI.

Originally I was planning on adding the feature here but if you'd feel more comfortable with me creating a separate Nuget package I can do that as well.

[BlaiseD]

I'm not familiar with the technology so whatever works with the CI. The expression builder tests check a string representation of the expression (in this case IQueryable.Expression) so that's also an option - ultimately that's all the libraries are doing.

Cosmos DB looks like a good fit with the other two libraries. I brought up your own NuGet because of the whole dynamic: my lack of knowledge of Cosmos DB, your work etc.. in case you hadn't thought of it.