Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Internal] Query: Adds new header SupportedSerializationFormats #3911

Merged
51 commits merged into from
Jun 22, 2023
Merged
Show file tree
Hide file tree
Changes from 45 commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
8e4977f
Binary Serilaization Response test
hdetroja Jun 10, 2022
7467711
Merge branch 'master' of github.com:Azure/azure-cosmos-dotnet-v3
hdetroja Aug 15, 2022
0476942
Merge branch 'master' of github.com:Azure/azure-cosmos-dotnet-v3
hdetroja Oct 27, 2022
d9de84b
Merge branch 'master' of github.com:Azure/azure-cosmos-dotnet-v3
hdetroja Nov 30, 2022
6f4c554
Merge branch 'master' of github.com:Azure/azure-cosmos-dotnet-v3
hdetroja Dec 5, 2022
213e396
Merge branch 'master' of github.com:Azure/azure-cosmos-dotnet-v3
hdetroja Jan 5, 2023
34a5a93
Added new header SupportedSerializationFormats
hdetroja Jan 11, 2023
17eb226
Modified existing use of CosmosSerializationFormatOptions
hdetroja Jan 11, 2023
55b8038
Modified tests and removed unused code
hdetroja Jan 12, 2023
455f20c
Addressed comments
hdetroja Jan 23, 2023
78be9f1
Added more negative cases
hdetroja Jan 23, 2023
f487689
Revert changes
hdetroja Jan 23, 2023
ff2c2e4
Added spaces
hdetroja Jan 23, 2023
a379d0d
Addressed comments
hdetroja Jan 30, 2023
c88af8f
Addressed comments
hdetroja Jan 31, 2023
19b4253
Merge branch 'master' into users/hdetroja/AddsNewHeader
neildsh Jan 31, 2023
70a55bf
Removed SupportedSerializationFormats from Headers file
hdetroja Feb 1, 2023
2a3cd8f
Merge branch 'users/hdetroja/AddsNewHeader' of github.com:Azure/azure…
hdetroja Feb 1, 2023
7ecbb0d
Removed unused JsonSerilazationFormats option
hdetroja Feb 1, 2023
aa4fa0b
Addressed comments
hdetroja Feb 2, 2023
edab02f
Addressed comments
hdetroja Feb 7, 2023
3ff18f7
Merge branch 'master' of github.com:Azure/azure-cosmos-dotnet-v3
hdetroja Feb 7, 2023
5cd6f17
Merge branch 'master' into users/hdetroja/AddsNewHeader
hdetroja Feb 7, 2023
aa907cd
Addressed comments
hdetroja Feb 13, 2023
ffa661c
Merge branch 'master' into users/hdetroja/AddsNewHeader
ealsur Feb 22, 2023
6f04f1d
Addressed comments
hdetroja Feb 27, 2023
01df5e2
Added new enum TransportSerializationFormat
hdetroja Mar 6, 2023
0d06a9c
Added new enum TransportSerializationFormat
hdetroja Mar 6, 2023
4eb2fa2
Addressed comments
hdetroja Mar 7, 2023
4b71297
Removed unused parameter
hdetroja Mar 7, 2023
1e2451b
Addressed comments
hdetroja Mar 14, 2023
447df35
Merge branch 'master' into users/hdetroja/AddsNewHeader
Maya-Painter May 30, 2023
a272704
Merge branch 'master' into users/hdetroja/AddsNewHeader
Maya-Painter Jun 1, 2023
f79c668
updating API
Maya-Painter Jun 1, 2023
1ca0fc8
Merge branch 'master' into users/hdetroja/AddsNewHeader
Maya-Painter Jun 1, 2023
b9c54e7
Merge branch 'master' into users/hdetroja/AddsNewHeader
Maya-Painter Jun 9, 2023
f4874ca
Merge branch 'master' into users/hdetroja/AddsNewHeader
Maya-Painter Jun 13, 2023
9352b1a
remove tests
Maya-Painter Jun 13, 2023
a5b16ca
Text fixes
Maya-Painter Jun 14, 2023
f083f64
fix typo
Maya-Painter Jun 14, 2023
2535791
Merge branch 'master' into users/mayapainter/AddsSerializationHeader
Maya-Painter Jun 14, 2023
6511d32
remove TransportSerializationFormat header
Maya-Painter Jun 15, 2023
af1727f
Merge branch 'master' into users/mayapainter/AddsSerializationHeader
Maya-Painter Jun 15, 2023
a590482
text reverts
Maya-Painter Jun 15, 2023
b90cd4f
revert
Maya-Painter Jun 15, 2023
59cce85
test update
Maya-Painter Jun 15, 2023
7de7e99
PR comments
Maya-Painter Jun 19, 2023
f4878fe
Merge branch 'master' into users/mayapainter/AddsSerializationHeader
Maya-Painter Jun 19, 2023
bbdd88b
Merge branch 'master' into users/mayapainter/AddsSerializationHeader
Maya-Painter Jun 20, 2023
5a3f53c
remove test owner headers HeadersValidationTests.cs
Maya-Painter Jun 20, 2023
9b396f7
PR comments - remove unsupported tests and scope client
Maya-Painter Jun 22, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -33,16 +33,19 @@ public ChangeFeedPaginationOptions(
JsonSerializationFormat? jsonSerializationFormat = null,
Dictionary<string, string> additionalHeaders = null,
ChangeFeedQuerySpec changeFeedQuerySpec = null)
: base(pageSizeHint, jsonSerializationFormat, additionalHeaders)
: base(pageSizeHint, additionalHeaders)
{
this.Mode = mode ?? throw new ArgumentNullException(nameof(mode));
this.ChangeFeedQuerySpec = changeFeedQuerySpec;
this.JsonSerializationFormat = jsonSerializationFormat;
}

public ChangeFeedMode Mode { get; }

public ChangeFeedQuerySpec ChangeFeedQuerySpec { get; }

public JsonSerializationFormat? JsonSerializationFormat { get; }

protected override ImmutableHashSet<string> BannedAdditionalHeaders => BannedHeaders;
}
Maya-Painter marked this conversation as resolved.
Show resolved Hide resolved
}
37 changes: 17 additions & 20 deletions Microsoft.Azure.Cosmos/src/FeedOptions.cs
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ public FeedOptions()
{
}

internal FeedOptions(FeedOptions options)
ealsur marked this conversation as resolved.
Show resolved Hide resolved
public FeedOptions(FeedOptions options)
{
if (options == null)
{
Expand Down Expand Up @@ -65,10 +65,10 @@ internal FeedOptions(FeedOptions options)
this.EndId = options.EndId;
this.StartEpk = options.StartEpk;
this.EndEpk = options.EndEpk;
this.ContentSerializationFormat = options.ContentSerializationFormat;
this.EnableGroupBy = options.EnableGroupBy;
this.MergeStaticId = options.MergeStaticId;
this.Properties = options.Properties;
this.SupportedSerializationFormats = options.SupportedSerializationFormats;
}

/// <summary>
Expand Down Expand Up @@ -282,7 +282,7 @@ internal FeedOptions(FeedOptions options)
/// <value>
/// Option is true if query tracing is enabled; otherwise, false.
/// </value>
internal bool? EmitVerboseTracesInQuery { get; set; }
public bool? EmitVerboseTracesInQuery { get; set; }

/// <summary>
/// Gets or sets the schema rid which could be used to filter the document feed response
Expand All @@ -292,7 +292,7 @@ internal FeedOptions(FeedOptions options)
/// By default, it is <c>null</c> which means no filtering will be applied.
/// Otherwise, it must be a valid resource id of Schema resource.
/// </value>
internal string FilterBySchemaResourceId { get; set; }
public string FilterBySchemaResourceId { get; set; }

/// <summary>
/// Gets or sets the <see cref="PopulateQueryMetrics"/> request option for document query requests in the Azure Cosmos DB service.
Expand Down Expand Up @@ -359,65 +359,62 @@ internal FeedOptions(FeedOptions options)
/// <summary>
/// Gets or sets the flag that tells the backend to forces the query to perform a scan (at a request level).
/// </summary>
internal bool ForceQueryScan { get; set; }
public bool ForceQueryScan { get; set; }

/// <summary>
/// Gets or sets the EnumerationDirection
/// To be used along with Read feed operation
/// </summary>
internal EnumerationDirection? EnumerationDirection { get; set; }
public EnumerationDirection? EnumerationDirection { get; set; }

/// <summary>
/// Gets or sets the ReadFeedKeyType
/// To be used along with Read feed operation
/// </summary>
internal ReadFeedKeyType? ReadFeedKeyType { get; set; }
public ReadFeedKeyType? ReadFeedKeyType { get; set; }

/// <summary>
/// Gets or sets the StartId
/// To be used along with Read feed operation
/// </summary>
internal string StartId { get; set; }
public string StartId { get; set; }

/// <summary>
/// Gets or sets the EndId
/// To be used along with Read feed operation
/// </summary>
internal string EndId { get; set; }
public string EndId { get; set; }

/// <summary>
/// Gets or sets the StartEpk
/// To be used along with Read feed operation
/// </summary>
internal string StartEpk { get; set; }
public string StartEpk { get; set; }

/// <summary>
/// Gets or sets the EndEpk
/// To be used along with Read feed operation
/// </summary>
internal string EndEpk { get; set; }
public string EndEpk { get; set; }

/// <summary>
/// Gets or sets the ContentSerializationFormat for the feed (query/read feed) operation in the Azure Cosmos DB service.
/// Gets or sets the SupportedSerializationFormats for the query response from the service.
/// </summary>
/// <remarks>
/// If the document is stored in a different serialization format then the one requested, then there will be a rewrite over the wire, but the source document will be untouched.
/// </remarks>
internal ContentSerializationFormat? ContentSerializationFormat { get; set; }
public SupportedSerializationFormats? SupportedSerializationFormats { get; set; }
Maya-Painter marked this conversation as resolved.
Show resolved Hide resolved

internal bool EnableGroupBy { get; set; }
public bool EnableGroupBy { get; set; }

/// <summary>
/// Gets or sets the MergeStaticId.
/// To be used along with Read feed operation when Static Column merge is desired.
/// </summary>
internal string MergeStaticId { get; set; }
public string MergeStaticId { get; set; }

/// <summary>
/// Gets or sets the custom serialization options for query
/// </summary>
internal CosmosSerializationFormatOptions CosmosSerializationFormatOptions { get; set; }
public CosmosSerializationFormatOptions CosmosSerializationFormatOptions { get; set; }

internal IDictionary<string, object> Properties { get; set; }
public IDictionary<string, object> Properties { get; set; }
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,12 @@ public virtual string ContentSerializationFormat
set => this.SetProperty(HttpConstants.HttpHeaders.ContentSerializationFormat, value);
}

public virtual string SupportedSerializationFormats
{
get => this.GetValueOrDefault(HttpConstants.HttpHeaders.SupportedSerializationFormats);
set => this.SetProperty(HttpConstants.HttpHeaders.SupportedSerializationFormats, value);
}

public virtual string ReadFeedKeyType
{
get => this.GetValueOrDefault(HttpConstants.HttpHeaders.ReadFeedKeyType);
Expand Down
6 changes: 6 additions & 0 deletions Microsoft.Azure.Cosmos/src/Headers/StoreRequestHeaders.cs
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,12 @@ public override string ContentSerializationFormat
set => this.requestNameValueCollection.ContentSerializationFormat = value;
}

public override string SupportedSerializationFormats
{
get => this.requestNameValueCollection.SupportedSerializationFormats;
set => this.requestNameValueCollection.SupportedSerializationFormats = value;
}

public override string ReadFeedKeyType
{
get => this.requestNameValueCollection.ReadFeedKeyType;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -192,11 +192,6 @@ public async Task<TryCatch<ReadFeedPage>> MonadicReadFeedAsync(
request.Headers.ContinuationToken = ((CosmosString)readFeedContinuationState.ContinuationToken).Value;
}

if (readFeedPaginationOptions.JsonSerializationFormat.HasValue)
{
request.Headers.ContentSerializationFormat = readFeedPaginationOptions.JsonSerializationFormat.Value.ToContentSerializationFormatString();
}

foreach (KeyValuePair<string, string> kvp in readFeedPaginationOptions.AdditionalHeaders)
{
request.Headers[kvp.Key] = kvp.Value;
Expand Down
4 changes: 0 additions & 4 deletions Microsoft.Azure.Cosmos/src/Pagination/PaginationOptions.cs
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,9 @@ internal abstract class PaginationOptions

protected PaginationOptions(
int? pageSizeLimit = null,
JsonSerializationFormat? jsonSerializationFormat = null,
Dictionary<string, string> additionalHeaders = null)
{
this.PageSizeLimit = pageSizeLimit;
this.JsonSerializationFormat = jsonSerializationFormat;
this.AdditionalHeaders = additionalHeaders != null ? additionalHeaders.ToImmutableDictionary<string, string>() : EmptyDictionary;

foreach (string key in this.AdditionalHeaders.Keys)
Expand All @@ -41,8 +39,6 @@ protected PaginationOptions(

public int? PageSizeLimit { get; }

public JsonSerializationFormat? JsonSerializationFormat { get; }

public ImmutableDictionary<string, string> AdditionalHeaders { get; }

protected abstract ImmutableHashSet<string> BannedAdditionalHeaders { get; }
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,8 @@ internal sealed class QueryPaginationOptions : PaginationOptions
public QueryPaginationOptions(
int? pageSizeHint = null,
bool optimisticDirectExecute = false,
JsonSerializationFormat? jsonSerializationFormat = null,
Dictionary<string, string> additionalHeaders = null)
: base(pageSizeHint, jsonSerializationFormat, additionalHeaders)
: base(pageSizeHint, additionalHeaders)
{
this.OptimisticDirectExecute = optimisticDirectExecute;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ namespace Microsoft.Azure.Cosmos.Query
{
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Globalization;
using System.IO;
using System.Linq;
Expand All @@ -31,6 +32,8 @@ namespace Microsoft.Azure.Cosmos.Query

internal abstract class DocumentQueryExecutionContextBase : IDocumentQueryExecutionContext
{
public static readonly string DefaultSupportedSerializationFormats = string.Join(",", SupportedSerializationFormats.JsonText, SupportedSerializationFormats.CosmosBinary);

public readonly struct InitParams
{
public IDocumentQueryClient Client { get; }
Expand Down Expand Up @@ -326,14 +329,9 @@ await this.Client.EnsureValidOverwriteAsync(
requestHeaders.Set(HttpConstants.HttpHeaders.MergeStaticId, this.feedOptions.MergeStaticId);
}

if (this.feedOptions.CosmosSerializationFormatOptions != null)
{
requestHeaders[HttpConstants.HttpHeaders.ContentSerializationFormat] = this.feedOptions.CosmosSerializationFormatOptions.ContentSerializationFormat;
}
else if (this.feedOptions.ContentSerializationFormat.HasValue)
{
requestHeaders[HttpConstants.HttpHeaders.ContentSerializationFormat] = this.feedOptions.ContentSerializationFormat.Value.ToString();
}
requestHeaders[HttpConstants.HttpHeaders.SupportedSerializationFormats] = this.feedOptions.SupportedSerializationFormats != null
? this.feedOptions.SupportedSerializationFormats.Value.ToString()
: DefaultSupportedSerializationFormats;
Maya-Painter marked this conversation as resolved.
Show resolved Hide resolved

return requestHeaders;
}
Expand Down Expand Up @@ -688,23 +686,7 @@ private DocumentFeedResponse<CosmosElement> GetFeedResponse(
{
content = memoryStream.ToArray();
}

IJsonNavigator jsonNavigator = null;

// Use the users custom navigator first. If it returns null back try the
// internal navigator.
if (this.feedOptions.CosmosSerializationFormatOptions != null)
{
jsonNavigator = this.feedOptions.CosmosSerializationFormatOptions.CreateCustomNavigatorCallback(content);
if (jsonNavigator == null)
{
throw new InvalidOperationException("The CosmosSerializationOptions did not return a JSON navigator.");
}
}
else
{
jsonNavigator = JsonNavigator.Create(content);
}
IJsonNavigator jsonNavigator = JsonNavigator.Create(content);

string resourceName = this.GetRootNodeName(documentServiceRequest.ResourceType);

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,6 @@ public override async Task<TryCatch<QueryPage>> ExecuteItemQueryAsync(
cancellationToken: cancellationToken);

return CosmosQueryClientCore.GetCosmosElementResponse(
requestOptions,
resourceType,
message,
trace);
Expand Down Expand Up @@ -297,7 +296,6 @@ public override void ClearSessionTokenCache(string collectionFullName)
}

private static TryCatch<QueryPage> GetCosmosElementResponse(
QueryRequestOptions requestOptions,
ResourceType resourceType,
ResponseMessage cosmosResponseMessage,
ITrace trace)
Expand Down Expand Up @@ -336,8 +334,7 @@ private static TryCatch<QueryPage> GetCosmosElementResponse(
long responseLengthBytes = memoryStream.Length;
CosmosArray documents = CosmosQueryClientCore.ParseElementsFromRestStream(
memoryStream,
resourceType,
requestOptions.CosmosSerializationFormatOptions);
resourceType);

QueryState queryState;
if (cosmosResponseMessage.Headers.ContinuationToken != null)
Expand Down Expand Up @@ -444,12 +441,10 @@ private Task<PartitionKeyRangeCache> GetRoutingMapProviderAsync()
/// </summary>
/// <param name="stream">The memory stream response for the query REST response Azure Cosmos</param>
/// <param name="resourceType">The resource type</param>
/// <param name="cosmosSerializationOptions">The custom serialization options. This allows custom serialization types like BSON, JSON, or other formats</param>
/// <returns>An array of CosmosElements parsed from the response body.</returns>
public static CosmosArray ParseElementsFromRestStream(
Stream stream,
ResourceType resourceType,
CosmosSerializationFormatOptions cosmosSerializationOptions)
ResourceType resourceType)
{
if (!(stream is MemoryStream memoryStream))
{
Expand Down Expand Up @@ -478,20 +473,7 @@ public static CosmosArray ParseElementsFromRestStream(
// You want to create a CosmosElement for each document in "Documents".

ReadOnlyMemory<byte> content = memoryStream.TryGetBuffer(out ArraySegment<byte> buffer) ? buffer : (ReadOnlyMemory<byte>)memoryStream.ToArray();
IJsonNavigator jsonNavigator;
if (cosmosSerializationOptions != null)
{
// Use the users custom navigator
jsonNavigator = cosmosSerializationOptions.CreateCustomNavigatorCallback(content);
if (jsonNavigator == null)
{
throw new InvalidOperationException("The CosmosSerializationOptions did not return a JSON navigator.");
}
}
else
{
jsonNavigator = JsonNavigator.Create(content);
}
IJsonNavigator jsonNavigator = JsonNavigator.Create(content);

string resourceName = resourceType switch
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,8 @@ internal sealed class ReadFeedPaginationOptions : PaginationOptions
public ReadFeedPaginationOptions(
PaginationDirection? paginationDirection = null,
int? pageSizeHint = null,
JsonSerializationFormat? jsonSerializationFormat = null,
Dictionary<string, string> additionalHeaders = null)
: base(pageSizeHint, jsonSerializationFormat, additionalHeaders)
: base(pageSizeHint, additionalHeaders)
{
this.Direction = paginationDirection;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,7 @@ public async ValueTask<bool> MoveNextAsync()

CosmosArray documents = CosmosQueryClientCore.ParseElementsFromRestStream(
innerReadFeedPage.Page.Content,
Documents.ResourceType.Document,
cosmosSerializationOptions: null);
Documents.ResourceType.Document);
ReadFeedPage page = new ReadFeedPage(
documents,
innerReadFeedPage.Page.RequestCharge,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,10 @@
namespace Microsoft.Azure.Cosmos
{
using System;
using System.ComponentModel;
using System.Text;
using Microsoft.Azure.Cosmos.CosmosElements;
using Microsoft.Azure.Cosmos.Query;
using Microsoft.Azure.Cosmos.Query.Core;
using Microsoft.Azure.Cosmos.Query.Core.Pipeline;
using Microsoft.Azure.Documents;
Expand Down Expand Up @@ -188,6 +190,8 @@ public ConsistencyLevel? ConsistencyLevel

internal CosmosSerializationFormatOptions CosmosSerializationFormatOptions { get; set; }

internal SupportedSerializationFormats? SupportedSerializationFormats { get; set; }

internal ExecutionEnvironment? ExecutionEnvironment { get; set; }

internal bool? ReturnResultsInDeterministicOrder { get; set; }
Expand Down Expand Up @@ -240,11 +244,10 @@ internal override void PopulateRequestOptions(RequestMessage request)
{
request.Headers.Add(HttpConstants.HttpHeaders.ResponseContinuationTokenLimitInKB, this.ResponseContinuationTokenLimitInKb.ToString());
}

if (this.CosmosSerializationFormatOptions != null)
{
request.Headers.CosmosMessageHeaders.ContentSerializationFormat = this.CosmosSerializationFormatOptions.ContentSerializationFormat;
}

request.Headers.CosmosMessageHeaders.SupportedSerializationFormats = this.SupportedSerializationFormats != null
? this.SupportedSerializationFormats.ToString()
: DocumentQueryExecutionContextBase.DefaultSupportedSerializationFormats;
Maya-Painter marked this conversation as resolved.
Show resolved Hide resolved

if (this.StartId != null)
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -52,4 +52,4 @@ public CosmosSerializationFormatOptions(
this.CreateCustomWriterCallback = createCustomWriter;
}
}
}
}
Loading