Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Contract Search #99

Merged
120 commits merged into from
Sep 14, 2023
Merged
Show file tree
Hide file tree
Changes from 93 commits
Commits
Show all changes
120 commits
Select commit Hold shift + click to select a range
d5e8afa
Made entities and import flow
Aug 17, 2023
cdf8265
Added test to aggregate which validates events are stored
Aug 18, 2023
bc505e9
Merge branch 'main' into cbw-1248/implementation-smart-contract-aggre…
Aug 18, 2023
fbf296d
Added test for other read cases
Aug 18, 2023
aa922e0
Added smart contract entity
Aug 18, 2023
81b34ab
Added Smart Contract Aggregate to Startup
Aug 18, 2023
58156e1
Added import job
Aug 21, 2023
dcb5256
Finished repository test
Aug 21, 2023
ca0522a
Fixed aggregate test
Aug 21, 2023
b73da60
Added data source
Aug 21, 2023
91b0eaf
Added awaits on jobs on node import
Aug 21, 2023
fc47cb3
Added contract version to contract initialize and update events
Aug 22, 2023
50f6167
Added tests which validates uniqueness contrains
Aug 22, 2023
293329a
Remove todos and added simple resilience in import loop
Aug 22, 2023
1cfb11c
Fixed dependency injection issue
Aug 22, 2023
aa91b17
Fix issues when testing job
Aug 22, 2023
d575289
Small updates
Aug 22, 2023
f63bf17
Updated changelog
Aug 22, 2023
ef4550d
Moved classes to own files
Aug 22, 2023
eae5cb1
Added logic to Smart Contract Repository which can fetch batch
Aug 24, 2023
bb162ca
Made batch import instead of processing single rows
Aug 24, 2023
952e48b
Clean up unused code
Aug 24, 2023
4a40b00
Fix limit on smart contract database job
Aug 24, 2023
e708617
Remove unnecessary batch
Aug 24, 2023
40e474d
Added duration- and height metrics
Aug 24, 2023
d565919
Added health states to application
Aug 25, 2023
b3cc30b
Fixed health output
Aug 25, 2023
5411768
Adding temp CI
Aug 25, 2023
ea18072
Updating change log
Aug 25, 2023
75f2851
Remove some debugging and added nugets
Aug 25, 2023
54ab1c6
Fix test
Aug 25, 2023
b71562d
Merge branch 'main' into cbw-1248/implementation-smart-contract-aggre…
Aug 25, 2023
2c26643
Merge branch 'cbw-1248/implementation-smart-contract-aggregate' into …
Aug 25, 2023
915777e
Refactor
Aug 28, 2023
9857710
Merge branch 'cbw-1248/implementation-smart-contract-aggregate' into …
Aug 28, 2023
ce21ac8
Clean up after merge
Aug 28, 2023
707c401
Added smart contract paging query
Aug 31, 2023
7e5f664
Added metrics to graphql endpoints
Sep 1, 2023
2171a83
Updated changelog
Sep 1, 2023
09d7102
Added contract address to Smart Contract query
Sep 1, 2023
160f17d
Fix increment issue when reading heights
Sep 1, 2023
9edb613
Added simple view which can list smart contracts
Sep 1, 2023
ee4acf7
Updated naming from Smart Contract to Contract
Sep 1, 2023
9ba8c24
Renamed test and tables
Sep 1, 2023
68db31f
Merge branch 'cbw-1248/implementation-smart-contract-aggregate' into …
Sep 1, 2023
bc4440a
Fix naming conflicts
Sep 1, 2023
0faea97
Merge branch 'cbw-1249/add-observability' into cbw-1261/sc-listing-en…
Sep 1, 2023
c595e17
Resolve conflicts after merge
Sep 1, 2023
c823dde
Merge branch 'cbw-1261/sc-listing-endpoints' into cbw-1260/smart-cont…
Sep 1, 2023
e6f4b82
Renamed to Contract
Sep 1, 2023
aa2f083
Updated with add and remove link entities
Sep 1, 2023
f143293
Fixed off by one error
Sep 1, 2023
4ec857d
Merge branch 'cbw-1248/implementation-smart-contract-aggregate' into …
Sep 4, 2023
8c96ed8
Merge branch 'cbw-1249/add-observability' into cbw-1261/sc-listing-en…
Sep 4, 2023
bfd3257
Merge branch 'cbw-1261/sc-listing-endpoints' into cbw-1260/smart-cont…
Sep 4, 2023
f1f8c70
Update Program.cs
Sep 4, 2023
c33b7e6
Merge branch 'cbw-1249/add-observability' into cbw-1261/sc-listing-en…
Sep 4, 2023
8441f6c
Using activity and better exception handling in diagnostic handler
Sep 4, 2023
5197ede
Enable rewind of request body
Sep 4, 2023
ffaa221
Added tracing to logs
Sep 4, 2023
c42ada0
Updated log formats
Sep 4, 2023
0944d3d
remove extra from console
Sep 4, 2023
bff53f4
Corrected naming
Sep 5, 2023
5bf501e
Made repository for contract jobs
Sep 5, 2023
5843bdc
Migrated to options for feature flags
Sep 5, 2023
37b979e
Moved interfaces to same file as classes where only one implementatio…
Sep 5, 2023
e9bc1c1
Changed to date time offset
Sep 5, 2023
152c486
Added warnings not to change job identifier
Sep 5, 2023
3c682d0
Update range function with comments
Sep 5, 2023
6769b70
Follow standards from EF regarding CS8618 compiler warning for C# 10
Sep 5, 2023
3e7126e
Refactored Contract Aggregate Node Import job to be more readable
Sep 5, 2023
8048a49
Merge branch 'cbw-1248/implementation-smart-contract-aggregate' into …
Sep 5, 2023
a30fb7c
Resolve comments after merge
Sep 5, 2023
9f028f2
Merge branch 'main' into cbw-1249/add-observability
Sep 5, 2023
05dfeb9
Resolve issues after merge
Sep 5, 2023
0c43cc3
Merge branch 'cbw-1249/add-observability' into cbw-1261/sc-listing-en…
Sep 5, 2023
77ec045
Resolve issues after merge
Sep 5, 2023
c8020b3
Merge branch 'cbw-1261/sc-listing-endpoints' into cbw-1260/smart-cont…
Sep 5, 2023
e26cb54
Resolve comments
Sep 5, 2023
de8e391
Minor refactoring after final review
Sep 5, 2023
d47a38c
Merge branch 'cbw-1249/add-observability' into cbw-1261/sc-listing-en…
Sep 5, 2023
36467a9
Merge branch 'main' into cbw-1261/sc-listing-endpoints
Sep 5, 2023
f8fa896
Updates after merge
Sep 5, 2023
bfd0d97
Merge branch 'cbw-1261/sc-listing-endpoints' into cbw-1260/smart-cont…
Sep 5, 2023
9be4c15
Change log level to information when contract job has written heights.
Sep 5, 2023
8313365
Fix wrong tracing enrichment
Sep 5, 2023
f332453
Fix compiler warning and sorted contracts when queried for listing
Sep 5, 2023
0b2264f
Fix logging of users cancel queries.
Sep 5, 2023
4ec1261
Made searchable
Sep 6, 2023
946e160
Included contract address as string in Contract to make it searchable
Sep 6, 2023
722f551
Added search functionality to frontend
Sep 6, 2023
fed1171
moved private to bottom
Sep 6, 2023
f483588
Fix exceptions handling in diagnostics
Sep 8, 2023
0b962a5
Fix missing events when money CCD was transferred between account
Sep 8, 2023
282839d
Remove unused exception
Sep 8, 2023
1387363
Fix search result issue when end with comma or space
Sep 8, 2023
3492061
Added block slot time to all events
Sep 8, 2023
4838845
Updated graphql schema
Sep 8, 2023
bfc7795
Rejected events + Using Parallel in import job + Add Sender
Sep 8, 2023
0302c3c
Added to context, fix migration and fix serializer for reject
Sep 8, 2023
c0c8bb6
Removed wrong event_index from reject tables.
Sep 8, 2023
e9e1f09
Merge branch 'main' into cbw-1261/add-reject-events
Sep 11, 2023
b939614
Remove commented
Sep 11, 2023
4afcdd5
Added documentation to transfer event
Sep 11, 2023
b5928ff
Merge branch 'main' into cbw-1260/smart-contract-listing-view
Sep 11, 2023
842f47d
Merge branch 'cbw-1261/add-reject-events' into cbw-1260/smart-contrac…
Sep 11, 2023
8378a25
Merge branch 'cbw-1260/smart-contract-listing-view' into cbw-1258and1…
Sep 11, 2023
542188d
Added age and module to contract list view
Sep 11, 2023
8755cd1
Validated performance and SQL on split query.
Sep 11, 2023
9bed3d6
Merge branch 'cbw-1260/smart-contract-listing-view' into cbw-1258and1…
Sep 11, 2023
b82bb7f
Resolve comments
Sep 11, 2023
e9af685
Fix missing new line
Sep 13, 2023
f6c6b17
Change to Try pattern
Sep 13, 2023
fd64758
Fixed comments
Sep 13, 2023
2297fe6
Merge branch 'cbw-1261/add-reject-events' into cbw-1260/smart-contrac…
Sep 14, 2023
146666b
Merge branch 'main' into cbw-1260/smart-contract-listing-view
Sep 14, 2023
0301bc2
Merge branch 'cbw-1260/smart-contract-listing-view' into cbw-1258and1…
Sep 14, 2023
fda6801
Merge branch 'main' into cbw-1258and1259/contract-search
Sep 14, 2023
a1bbfd9
Fixed merge issues
Sep 14, 2023
846e271
Fix schema
Sep 14, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions backend/.gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -447,3 +447,5 @@ $RECYCLE.BIN/
!.vscode/tasks.json
!.vscode/launch.json
!.vscode/extensions.json

/Logs
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
using System.Threading;
using System.Threading.Tasks;
using Application.Aggregates.Contract.Jobs;
using Application.Observability;
using Application.Configurations;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Options;
Expand Down Expand Up @@ -33,6 +34,8 @@ IOptions<FeatureFlagOptions> featureFlagsOptions

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
using var _ = TraceContext.StartActivity(nameof(ContractJobsBackgroundService));

if (!_featureFlags.ConcordiumNodeImportEnabled)
{
_logger.Information("Import data from Concordium node is disabled. This controller will not run!");
Expand Down Expand Up @@ -73,4 +76,4 @@ private async Task RunJob(IContractJob job, CancellationToken token)
throw;
}
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
using Application.Aggregates.Contract.Jobs;
using Application.Aggregates.Contract.Observability;
using Application.Api.GraphQL.EfCore;
using Application.Observability;
using Application.Configurations;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Hosting;
Expand Down Expand Up @@ -46,6 +47,8 @@ public ContractNodeImportBackgroundService(

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
using var _ = TraceContext.StartActivity(nameof(ContractNodeImportBackgroundService));

if (!_featureFlags.ConcordiumNodeImportEnabled)
{
_logger.Information("Import data from Concordium node is disabled. This controller will not run!");
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,5 +11,5 @@ public class ContractAggregateJobOptions
///
/// Increasing batch size will increase memory consumption on job since more will be loaded into memory.
/// </summary>
public int BatchSize { get; set; } = 10_000;
public int BatchSize { get; set; } = 5_000;
}
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
using Application.Aggregates.Contract.Entities;
using Application.Api.GraphQL.EfCore.Converters.EfCore;
using Microsoft.EntityFrameworkCore;
using Microsoft.EntityFrameworkCore.Metadata.Builders;
Expand All @@ -11,9 +12,6 @@ public void Configure(EntityTypeBuilder<Entities.Contract> builder)
builder.ToTable("graphql_contracts");
builder.HasKey(x => new
{
x.BlockHeight,
x.TransactionIndex,
x.EventIndex,
x.ContractAddressIndex,
x.ContractAddressSubIndex
});
Expand All @@ -29,12 +27,19 @@ public void Configure(EntityTypeBuilder<Entities.Contract> builder)
.HasColumnName("contract_address_index");
builder.Property(x => x.ContractAddressSubIndex)
.HasColumnName("contract_address_sub_index");
builder.Property(x => x.ContractAddress)
.HasColumnName("contract_address");
builder.Property(x => x.Creator)
.HasColumnName("creator")
.HasConversion<AccountAddressConverter>();
builder.Property(x => x.Source)
.HasColumnName("source");
builder.Property(x => x.CreatedAt)
.HasColumnName("created_at");
.HasColumnName("created_at");

builder
.HasMany<ContractEvent>(sm => sm.ContractEvents)
.WithOne()
.HasForeignKey(sme => new { sme.ContractAddressIndex, sme.ContractAddressSubIndex });
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,6 @@ public void Configure(EntityTypeBuilder<ContractEvent> builder)
builder.Property(x => x.Source)
.HasColumnName("source");
builder.Property(x => x.CreatedAt)
.HasColumnName("created_at");
.HasColumnName("created_at");
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@ public void Configure(EntityTypeBuilder<ModuleReferenceContractLinkEvent> builde
x.EventIndex,
x.ModuleReference,
x.ContractAddressIndex,
x.ContractAddressSubIndex
x.ContractAddressSubIndex,
x.LinkAction
});
builder.Property(x => x.BlockHeight)
.HasColumnName("block_height");
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,6 @@ public void Configure(EntityTypeBuilder<ModuleReferenceEvent> builder)
builder.ToTable("graphql_module_reference_events");
builder.HasKey(x => new
{
x.BlockHeight,
x.TransactionIndex,
x.EventIndex,
x.ModuleReference
});
builder.Property(x => x.BlockHeight)
Expand Down
6 changes: 6 additions & 0 deletions backend/Application/Aggregates/Contract/ContractAggregate.cs
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
using Application.Aggregates.Contract.Observability;
using Application.Aggregates.Contract.Types;
using Application.Api.GraphQL.Transactions;
using Application.Observability;
using Concordium.Sdk.Types;
using AccountAddress = Application.Api.GraphQL.Accounts.AccountAddress;
using ContractAddress = Application.Api.GraphQL.ContractAddress;
Expand All @@ -19,6 +20,8 @@ internal sealed class ContractAggregate
private readonly IContractRepositoryFactory _repositoryFactory;
private readonly ContractAggregateOptions _options;
private readonly ILogger _logger;
private const string NodeImportJobActivity = "NodeImportJobActivity";
private const string NodeImportJobLoopActivity = "NodeImportJobLoopActivity";

public ContractAggregate(
IContractRepositoryFactory repositoryFactory,
Expand All @@ -32,6 +35,8 @@ ContractAggregateOptions options

internal async Task NodeImportJob(IContractNodeClient client, CancellationToken token = default)
{
using var _ = TraceContext.StartActivity(NodeImportJobActivity);

var retryCount = 0;
while (!token.IsCancellationRequested)
{
Expand Down Expand Up @@ -294,6 +299,7 @@ private async Task NodeImportRange(IContractNodeClient client, ulong fromBlockHe
{
for (var height = fromBlockHeight; height <= toBlockHeight; height++)
{
using var __ = TraceContext.StartActivity(NodeImportJobLoopActivity);
using var durationMetric = new ContractMetrics.DurationMetric(ImportSource.NodeImport);
try
{
Expand Down
73 changes: 72 additions & 1 deletion backend/Application/Aggregates/Contract/Entities/Contract.cs
Original file line number Diff line number Diff line change
@@ -1,6 +1,12 @@
using Application.Aggregates.Contract.Exceptions;
using Application.Aggregates.Contract.Types;
using Application.Api.GraphQL;
using Application.Api.GraphQL.Accounts;
using Application.Api.GraphQL.EfCore;
using Application.Api.GraphQL.Transactions;
using HotChocolate;
using HotChocolate.Types;
using Microsoft.EntityFrameworkCore;

namespace Application.Aggregates.Contract.Entities;

Expand All @@ -16,9 +22,11 @@ public sealed class Contract
public uint EventIndex { get; init; }
public ulong ContractAddressIndex { get; init; }
public ulong ContractAddressSubIndex { get; init; }
public string ContractAddress { get; init; } = null!;
public AccountAddress Creator { get; init; } = null!;
public ImportSource Source { get; init; }
public DateTimeOffset CreatedAt { get; init; } = DateTime.UtcNow;
public ICollection<ContractEvent> ContractEvents { get; set; } = null!;

/// <summary>
/// Needed for EF Core
Expand All @@ -42,6 +50,69 @@ internal Contract(
Creator = creator;
ContractAddressIndex = contractAddress.Index;
ContractAddressSubIndex = contractAddress.SubIndex;
ContractAddress = contractAddress.AsString;
Source = source;
}
}

[ExtendObjectType(typeof(Query))]
public class ContractQuery
{
[UsePaging]
public IQueryable<Contract> GetContracts(
GraphQlDbContext context)
{
return context.Contract
.AsNoTracking()
.Include(s => s.ContractEvents)
.OrderByDescending(c => c.ContractAddressIndex);
}
}

/// <summary>
/// Adds additional field to the returned GraphQL type <see cref="Contract"/>
/// </summary>
[ExtendObjectType(typeof(Contract))]
public sealed class ContractExtensions
{
/// <summary>
/// Returns aggregated amount from events on contract.
/// </summary>
public double GetAmount([Parent] Contract contract)
{
if (contract.ContractEvents is null)
{
return 0;
}

var amount = 0L;
foreach (var contractEvent in contract.ContractEvents)
{
switch (contractEvent.Event)
{
case ContractInitialized contractInitialized:
amount += (long)contractInitialized.Amount;
break;
case ContractUpdated contractUpdated:
amount += (long)contractUpdated.Amount;
break;
case Transferred transferred:
if (transferred.From is not ContractAddress contractAddress)
{
throw new ContractQueryException(
$"Got transfer with txHash, {contractEvent.TransactionHash}, with event on contract <{contract.ContractAddressIndex},{contract.ContractAddressSubIndex}> with FROM which wasn't a contract address.");
}
if (contractAddress.Index != contract.ContractAddressIndex ||
contractAddress.SubIndex != contract.ContractAddressSubIndex)
{
throw new ContractQueryException(
$"Got transfer with txHash, {contractEvent.TransactionHash}, with event on contract <{contract.ContractAddressIndex},{contract.ContractAddressSubIndex}> with FROM which wasn't same contract address but instead <{contractAddress.Index},{contractAddress.SubIndex}>");
}
amount -= (long)transferred.Amount;
break;
}
}

return amount;
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
namespace Application.Aggregates.Contract.Exceptions;

/// <summary>
/// Thrown when query of data fails.
/// </summary>
public class ContractQueryException : Exception
{
public ContractQueryException(string message) : base(message)
{}
}
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
using Application.Aggregates.Contract.Jobs;
using Application.Aggregates.Contract.Observability;
using Dapper;
using HotChocolate.Execution.Configuration;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Diagnostics.HealthChecks;
Expand Down Expand Up @@ -45,6 +46,14 @@ internal static void AddDapperTypeHandlers()
SqlMapper.AddTypeHandler(new AccountAddressHandler());
}

internal static IRequestExecutorBuilder AddContractGraphQlConfigurations(this IRequestExecutorBuilder builder)
{
builder
.AddType<Entities.Contract.ContractQuery>()
.AddTypeExtension<Entities.Contract.ContractExtensions>();
return builder;
}

/// <summary>
/// Background service which executes all jobs related to Smart Contracts.
///
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
using Application.Aggregates.Contract.Types;
using Application.Api.GraphQL.Accounts;
using Application.Api.GraphQL.Transactions;
using Application.Observability;
using Microsoft.Extensions.Options;

namespace Application.Aggregates.Contract.Jobs;
Expand Down Expand Up @@ -41,6 +42,8 @@ ContractHealthCheck healthCheck

public async Task StartImport(CancellationToken token)
{
using var _ = TraceContext.StartActivity(nameof(ContractDatabaseImportJob));

try
{
_readCount = -1;
Expand Down Expand Up @@ -129,6 +132,8 @@ private async Task RunBatch(long finalHeight, CancellationToken token)
{
while (!token.IsCancellationRequested)
{
using var _ = TraceContext.StartActivity(nameof(RunBatch));

var height = Interlocked.Increment(ref _readCount);
var blockHeightTo = height * _jobOptions.BatchSize;
if (blockHeightTo > finalHeight)
Expand All @@ -139,7 +144,7 @@ private async Task RunBatch(long finalHeight, CancellationToken token)
var affectedRows = await DatabaseBatchImportJob((ulong)blockHeightFrom, (ulong)blockHeightTo, token);

if (affectedRows == 0) continue;
_logger.Debug("Written heights {From} to {To}", blockHeightFrom, blockHeightTo);
_logger.Information("Written heights {From} to {To}", blockHeightFrom, blockHeightTo);
}
}

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
using System.Diagnostics;
using Application.Aggregates.Contract.Types;
using Prometheus;
using static Application.Observability.ApplicationMetrics;

namespace Application.Aggregates.Contract.Observability;

Expand Down Expand Up @@ -70,25 +71,6 @@ public void Dispose()
.WithLabels(_source.ToStringCached(), _exceptionName)
.Observe(elapsedSeconds);
}

private static string PrettyPrintException(Exception ex)
{
var type = ex.GetType();
if (type.GenericTypeArguments.Length == 0)
{
return type.Name;
}

var name = type.Name.AsSpan();
var indexOfGenericCount = name.IndexOf('`');
if (indexOfGenericCount != -1)
{
name = name[..indexOfGenericCount];
}
var typeArguments = string.Join(",", type.GenericTypeArguments.Select(t => t.Name));

return $"{name}<{typeArguments}>";
}
}
}

Loading