Convert ObjectDocument to json files

[MisinformedDNA]

I'm trying to create a pipeline that downloads a web page, parses it into an object, converts the object into JSON file and then writes the JSON file to disk.

When I run this pipeline, it says documents are being output, but nothing is in the output directory. One problem I see is that how can it know what to name the file?

        .WithInputModules(
            new ReadWebPages(
                new[] { "https://starwars.fandom.com/wiki/Timeline_of_canon_media",
                            "https://starwars.fandom.com/wiki/Timeline_of_Legends_media" }))
        .WithProcessModules(new ParseWookieepediaTimeline (), new GenerateJson(Config.FromDocument<ObjectDocument<Media>>(d => (ObjectDocument<Media>)d)))
        .WithOutputWriteFiles("*.json")

    public class ReadWebPages : ConfigModule<IEnumerable<string>>
    {
        protected override async Task<IEnumerable<IDocument>> ExecuteConfigAsync(IDocument input, IExecutionContext context, IEnumerable<string> value)
        {
            var httpClient = new HttpClient();

            var urls = value.Select(x => new Uri(x));
            var documentTasks = urls.Select(async url =>
            {
                var page = await httpClient.GetStreamAsync(url);
                return context.CreateDocument(page, "text/html");
            });

            return await Task.WhenAll(documentTasks);
        }

    public class ParseWookieepediaTimeline : ParallelModule
    {
        protected override async Task<IEnumerable<IDocument>> ExecuteInputAsync(IDocument input, IExecutionContext context)
        {
            using var stream = input.ContentProvider.GetStream();
            var media = await TimelineParser.Parse(stream);

            return media.Select(x => new ObjectDocument<Media>(x));
        }
    }

What am I doing wrong? I really have no idea what I'm doing, so feel free to rip my code apart. Thanks.

[daveaglick]

As you suspected, the lack of a destination is likely at issue here (at first glance). The easiest way to add one is to use the SetDestination module to extract something from the document:

new SetDestination(Config.FromDocument(doc => $"{doc.GetString("FooBar")}.json"))

And then you can confirm it's working (or in what way it's not working) by adding the -l Debug switch to the CLI command. This will output all sorts of extra information, including every file that got written and where it was written to.