Migration from Wyam

[Vladekk]

Are there migration path from Wyam? I'm looking at Statiq docs, and this seems very hard. I'm just trying to add some resources (node_modules) folder like this:
.AddPipeline("Yarn",new CopyFiles("node_modules/**"))

I get some incomprehensible errors, possibly b/c Statiq tries to process files inside this folder using additional pipelines?
I need to exclude this folder from processing but I just can't find it in docs.

[devlead]

The clean blog theme has some basic instructions in its Readme
https://github.com/statiqdev/CleanBlog#porting-from-wyam

[Vladekk]

Thanks, now I have another issue. (I'm afraid there will be tens more of them coming). I can't get extensionless url to work. About page works, but post in custom location does not.

Update: found the reason, file name was in Cyrillic script. I guess this may be a bug, b/c it was working in wyam

[daveaglick]

Hopefully not too many more bugs! While a lot of engine functionality changed in Statiq, many of the modules are direct ports. And hopefully by now it's been through some pretty good testing having been mostly solid for almost a year. But yeah, there's definitely going to be at least a few more gotchas like this, especially when they relate to environments I don't typically use. Thanks for the PR! I'll try to get that merged and released quickly to unblock you.

[MarkZither]

Glad i saw this as i missed that CleanBlog had been created with migration instructions, I had started to migrate to Statiq a few months ago but didn't quite get there.
With the CleanBlog theme and instructions I had most of my blog working in about an hour, just need to fix my search pipeline and NetlifyCMS links.
For my NetlifyCMS issue it appears Validate doesn't like links formatted like https://unpkg.com/netlify-cms@^2.0.0/dist/cms.css as it flags the ^ as invalid, current workaround is using a pinned version https://unpkg.com/[email protected]/dist/cms.css

[MarkZither]

also need to figure out the correct replacement for this in my _post-footer,cshtml

/* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */
    var disqus_identifier = '@Model.FilePath(Keys.RelativeFilePath).FileNameWithoutExtension.FullPath';
    var disqus_title = '@Model.String(Wyam.Blog.BlogKeys.Title)';
    var disqus_url = '@Context.GetLink(Model)';

as it is failing with

[CRIT] One or more errors occurred. (One or more compilation failures occurred:
_post-footer.cshtml(15,51): error CS0117: 'Keys' does not contain a definition for 'RelativeFilePath'
_post-footer.cshtml(16,39): error CS0103: The name 'Wyam' does not exist in the current context
_post-footer.cshtml(17,24): error CS1973: 'IExecutionContext' has no applicable method named 'GetLink' but appears to have an extension method
by that name. Extension methods cannot be dynamically dispatched. Consider casting the dynamic arguments or calling the extension method without the extension method syntax.)

[MarkZither]

i settled on this to get it working

    /* * * CONFIGURATION VARIABLES: EDIT BEFORE PASTING INTO YOUR WEBPAGE * * */
    var disqus_identifier = '@Document.Destination.FileNameWithoutExtension.FullPath';
    var disqus_title = '@Document.GetString("Title")';
    var disqus_url = '@Document.GetLink(true)';

[daveaglick]

@MarkZither glad you got it working! I'll take a second look tomorrow when I've got a little more time and make sure there's nothing better I can suggest, but looks like you got it covered. Also planning on adding the porting guide to the main site now that you've reported it's working well - not sure why that didn't occur to me in the first place.

[Vladekk]

Okay, I was not able to find any solution to the issue of post-processed node_modules folder, except maybe not using Statiq.Web bootstrapper at all, and adding pipelines manually.
If there is a way to set config value on a directory, it probably would be possible to set RenderPostProcessTemplates setting to false on the input folder for node_modules folder, and it will not be post-processed.
Any help would be really appreciated.

[daveaglick]

Ahhh, I get it now - it's not so much that they're assets or not, it's that even assets have templates that process like Less and Sass. You were definitely on the right track with WebKeys.RenderPostProcessTemplates - I also would have expected that to work...except I only implemented it for content files 😬. Going to add the flag to the other content types as well for the next release. If it were working, setting that in the _directory.yaml file would have been perfect.

Thankfully there's still another work around. Since the templates are selected based on media type, you can change that to ignore templates. And since the Asset content type has a default behavior of copying to output, the two things together should work. Put this in your _directory.yaml:

ContentType: Asset
MediaType: text/plain

[Vladekk]

Seems to work. Thanks!
I now understand a bit more, so hopefully I can start migration. There is a lot of custom logic, though, so expect more questions ;-)

[daveaglick]

No problem at all - these questions are great because they feed directly back into the docs and help everyone!

[MarkZither]

thanks @daveaglick that also solves the issues I was having getting my admin directory to copy as is for my NetlifyCMS files as well as fixing the problem with unpkg links being seen as code blocks.

[daveaglick]

@MarkZither Awesome!

I know you've got some other discussions open, hoping to take a look at those tonight or tomorrow.

[Vladekk]

Question of the day ;-)

This code does not minify anything (but it should?)

    public class LegacyJsMinifyPipeline : Pipeline
    {
        public LegacyJsMinifyPipeline()
        {
            InputModules = new ModuleList()
            {
                new ReadFiles("js/**.js")
            };
            ProcessModules = new ModuleList
            {

                new MinifyJs(),
                
            };
            OutputModules = new ModuleList()
            {
                new WriteFiles()
            };
        }
    }

[Vladekk]

Okay, I understand the issue. It is b/c Statiq.Web processes all folders using predefined pipelines, I guess. However, how do I replace parts of this for my own pipelines? Is there a way?

[Vladekk]

I found one possible solution.
.AddSetting( WebKeys.InputFiles, "" )
and then add pipelines manually to process files you want.

[daveaglick]

That's exactly what I was going to suggest :). You can either exclude all files by clearing the globbing pattern like you have above, or exclude specific folders with a pattern like "{!exclude_me,}**/{!_,}*" and then use ReadFiles("exclude_me/**/*") as the start of your custom pipeline.

I'm not 100% sure on those globbing patterns, but there's now a tester in the CLI that you can use to experiment with patterns and see what they return or not.

[daveaglick]

However, if you want to augment existing pipelines, there are easier ways to do that. For example, if you want to add MinifyJs to the Assets pipeline you can do something like this from your bootstrapper:

.ModifyPipeline(Statiq.Web.Pipelines.Assets, p => p.PostProcessModules.Add(new MinifyJs()))

[Vladekk]

However, if you want to augment existing pipelines, there are easier ways to do that. For example, if you want to add MinifyJs to the Assets pipeline you can do something like this from your bootstrapper:

.ModifyPipeline(Statiq.Web.Pipelines.Assets, p => p.PostProcessModules.Add(new MinifyJs()))

But how it can work, if assets are of different content types? MinifyJs module understands mime type?

[Vladekk]

New question
This pipeline outputs files in the folder of the last ReadFiles() call.
This seems strange, and not intuitive. Moreover, there are no arguments in WriteFiles(). Are there a way to redefine this behavior?

[UsedImplicitly]
  public class LegacyJsMinifyPipeline : Pipeline
  {
      public LegacyJsMinifyPipeline()
      {
          
          InputModules = new ModuleList()
          {
              new ReadFiles("js/**.js"),
              new ReadFiles("Vladekk2/**.js")
              
          };
          ProcessModules = new ModuleList
          {
  
  //            new MinifyJs(),
              
          };
          OutputModules = new ModuleList()
          {
              new WriteFiles()
          };
      }
  }

[Vladekk]

Okay, I found the right way is
new ReadFiles("js/**.js", "Vladekk2/**.js")

[Vladekk]

Okay, I failed to migrate, so I've decided to spend more time understanding stuff. Hopefully, it'll become easier with time.

Some questions

1. if I just take simplest CleanBlog site, how do I add more folders with posts, and posts that are cshtml, not md? Adding folder using comma in settings does not work.

I mean this:
PostSources: posts/*, post/Legacy/*

2. Where does Document come from in index.cshtml? It is passed to posts partial to render posts, but it i weird that index document contains posts.

@Html.Partial("/_posts.cshtml", Document)

At this point, I am not sure web bootstrapper is such a good idea, b/c is complicates understanding of the framework. Maybe, initial command that generates default pipelines that can be studied would be better.

[Vladekk]

Okay, I found globbing pattern like this works for (1):
PostSources: "{posts,post}/**"

However, question 2 still stands.
I guess it has something to do with
https://statiq.dev/web/content-and-data/document-hierarchy ?

So, document for index.html contains all posts in sub-folders?

[daveaglick]

Where does Document come from in index.cshtml? It is passed to posts partial to render posts, but it i weird that index document contains posts.

The key here is knowing that the input index.cshtml file in the CleanBlog theme is an archive (you can tell because it contains Archive... metadata). Archives are special in that one input file becomes multiple output files. Think of them kind of like a file-based .GroupBy(). In this case, the index.cshtml file contains the following front matter (among other values):

ArchiveSources: => GetString("PostSources")

Normally this would include a globbing pattern, but for flexibility since this is a theme that lots of people will be using, it sets the pattern from a setting. You can see the default PostSources value in settings.yml at https://github.com/statiqdev/CleanBlog/blob/main/settings.yml:

PostSources: posts/*

So what this is doing is creating index.cshtml as an archive with files from posts/*. That's why you end up with child documents in the index - because the CleanBlog theme creates multiple, paged index files in output by being an archive. If you create your own index.cshtml file in the root of your immediate input folder, it would override the one from the theme and you could make it not an archive with simple content, though then you'd loose the paged list of posts (unless you also made your version an archive).

Did that help explain what's going on here?

[daveaglick]

At this point, I am not sure web bootstrapper is such a good idea, b/c is complicates understanding of the framework. Maybe, initial command that generates default pipelines that can be studied would be better.

You probably still want the Bootstrapper - it does a lot more than set up the default pipelines. It configures the engine to look at settings files, sets up the dependency injection, etc. The part that does a lot of magic is Statiq Web. One totally legitimate way to use Statiq is not to use Statiq Web at all and just stick to Statiq Framework. Statiq Framework still has a Bootstrapper but doesn't contain any pipelines so you'd have to create all of your pipelines from scratch. That's great if you have different requirements or don't like the pipelines Statiq Web adds. That said, I've tried to give a lot of thought on how to make Statiq Web flexible enough to handle most use cases.

I think in your case, it's more a problem of understanding how those default pipelines in Statiq Web work so you can extend/customize them since you're porting from an existing site. You can take a look at each pipeline at https://github.com/statiqdev/Statiq.Web/tree/main/src/Statiq.Web/Pipelines. In general, the Inputs pipeline runs first and reads all the files in the input folder and does some triage on them like figuring out which ContentType and MediaType they should have. Then a file is processed by either the Assets, Content, or Data pipeline based on it's ContentType (or the Archives or Feeds pipeline if it's metadata indicates it's one of those special types of files). I've been meaning to document these pipelines a little better on the site, but that's essentially what's going on.