Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avro Extractor Not Working #122

Open
SushantVarshney opened this issue Feb 22, 2018 · 1 comment
Open

Avro Extractor Not Working #122

SushantVarshney opened this issue Feb 22, 2018 · 1 comment

Comments

@SushantVarshney
Copy link

After doing all the steps described in your repository for reading the Avro file from Data Lake Store using U-SQL.
Below is the script I am using which is replication of your sample script,also the data used here is the same provided by you.But after submitting the job I get this error:

Use 'IUnstructuredReader.Length' instead. For more information, see documentation about 'IUnstructuredReader.Length' and 'IUnstructuredReader.BaseStream'.

Can anyone tell me how to resolve this out.I don't wanna change the internal code of the DLL's used here.

`/DROP ASSEMBLY IF EXISTS [Avro];
CREATE ASSEMBLY [Avro] FROM @"/assemblies/Avro.dll";
DROP ASSEMBLY IF EXISTS [Microsoft.Analytics.Samples.Formats];
CREATE ASSEMBLY [Microsoft.Analytics.Samples.Formats] FROM @"/assemblies/Microsoft.Analytics.Samples.Formats.dll";
DROP ASSEMBLY IF EXISTS [Newtonsoft.Json];
CREATE ASSEMBLY [Newtonsoft.Json] FROM @"/assemblies/Newtonsoft.Json.dll";
DROP ASSEMBLY IF EXISTS [log4net];
CREATE ASSEMBLY [log4net] FROM @"/assemblies/log4net.dll";
/

REFERENCE ASSEMBLY [Newtonsoft.Json];
REFERENCE ASSEMBLY [log4net];
REFERENCE ASSEMBLY [Avro];
REFERENCE ASSEMBLY [Microsoft.Analytics.Samples.Formats];
REFERENCE ASSEMBLY [Microsoft.Hadoop.Avro];

USING Microsoft.Analytics.Samples.Formats.Avro;
USING Microsoft.Analytics.Samples.Formats.Json;

DECLARE @input_file string = @"/sample-data/twitter.avro";
DECLARE @output_file string = @"/sample-data/abc.csv";

@rs =
EXTRACT
createdat string,
topic string,
sentimentscore long,
eventprocessedutctime string,
partitionid long,
eventenqueuedutctime string
FROM @input_file
USING new AvroExtractor(@"
{
""type"" : ""record"",
""name"" : ""GenericFromIRecord0"",
""namespace"" : ""Microsoft.Streaming.Avro"",
""fields"" : [ {
""name"" : ""createdat"",
""type"" : [ ""null"", ""string"" ]
}, {
""name"" : ""topic"",
""type"" : [ ""null"", ""string"" ]
}, {
""name"" : ""sentimentscore"",
""type"" : [ ""null"", ""long"" ]
}, {
""name"" : ""eventprocessedutctime"",
""type"" : [ ""null"", ""string"" ]
}, {
""name"" : ""partitionid"",
""type"" : [ ""null"", ""long"" ]
}, {
""name"" : ""eventenqueuedutctime"",
""type"" : [ ""null"", ""string"" ]
} ]
}
");

@cnt =
SELECT topic, COUNT(*) AS cnt
FROM @rs
GROUP BY topic;

OUTPUT @cnt TO @output_file USING Outputters.Text();`

@flmader
Copy link

flmader commented Feb 26, 2018

Do you have the full error details? Unfortunately I cannot reproduce this issue. Only guessing: how did you upload the avro file to ADLS?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants