Add support for multiple sources

[seancdavis]

Currently, Contentlayer looks for a single exported method that is pulled from one particular source. We've received requests to support more than one source within the same configuration. I'm working to get more information on specific use cases.

Considerations

What are the challenges with this approach?

Design

What would the API for this look like? Perhaps you could define sources separately and then reference the source at the document level? And include the option for a default source.

Napkin sketch:

const ctfl = makeSource({ type: 'contentful', accessToken: /* ... */ })
const local = makeSource({ type: 'local', path: /* ... */ })

const Page = defineDocumentType({ source: ctfl })
const Post = defineDocumentType({})

export default {
  defaultSource: local,
  documentTypes: [Page, Post]
}

[seancdavis]

We're looking for more examples to better understand the use case before introducing this feature.

[timlrx]

I am looking to use it for md/mdx content from possibly different contentDirPath with different mdx plugins. The API looks mostly good to me. Not sure what defaultSource is meant to do though. To make it a little cleaner how about requiring the user to be more explicit on the type in makeSource.

Example types would include: markdown, mdx or contentful with their respective options. If there are plans to support non-local md or mdx, path could also take the form of an s3 URL path.

[684efs3]

When do you plan on releasing this?

In terms of specific use cases, I have mdx files in multiple files, one for a blog, another for news, etc.

[seancdavis]

@684efs3 We're not actively working on this feature at this time, though still considering it and looking for more use cases to determine how to shape the API.

In this case, "multiple sources" means that your collection of data is coming from multiple sources (local, headless CMS, API, etc.).

In your case, it sounds like all your data is local, but spread across multiple files. This is what Contentlayer is designed to solve today. You can create an individual document type for content in each directory. You can even process some content types as MDX and others as plain markdown.

[684efs3]

Ah I see! Apologies! Your comment is very helpful. I'm looking forward to testing it! Thank you very much! :-)

[cjroebuck]

@schickling @seancdavis I'd like to ask if this feature would make it possible to read in content sources from a remote git(hub) repo, a bit like how nextjs does it for their own docs site (https://github.com/vercel/next.js/tree/canary/docs)

Also see (https://github.com/opstrace/next-product-docs).

This means, for example, that we could then keep our monorepo private, but have a public repo that includes just the markdown content, that anyone can request to make change in, such as typos etc.

I guess it might get tricky when using components inline in the MDX that would be from the private repo, but maybe not.

---

I just figured this is possible now using something like git submodules to link the public repo into the private repos content directory

[seancdavis]

@cjroebuck I like the pattern you're suggesting a lot. And yes, your last two points are important ones:

• Submodules are a good solution for bringing in content from another repo and using Contentlayer to load that content into your private site.
• There is an inherent risk in specifying functionality (i.e. using components) in a content project that shouldn't know anything about its front ends.

This issue is specifically about using multiple sources in the same project — e.g. some content comes from markdown files, other from an API like Contentful.

[ahofmeister]

Another use case would be to separate blog entries from projects for a personal portfolio or is there any other solution to implement it?

[phyesix]

Is there any progress on this? My old blog's posts are md and I write my current posts with mdx. Can we write both md and mdx to filePathPattern in contentlayer.config (ex: filePathPattern: [**/*.md, **/*.mdx])?

[ahofmeister]

@phyesix Yes, it is possible to write both. :) What I have done is the following:

Create multiple document types and refer them in the makeSource method like

export default makeSource({
  contentDirPath: 'src/content',
  documentTypes: [Post, Project],
  mdx: {
    remarkPlugins: [remarkGfm],
    rehypePlugins: [
      [
        rehypePrettyCode, options
      ],
    ],
  },

})

Then you can just create the document type like this

export const Project = defineDocumentType(() => ({
  name: 'Project',
  filePathPattern: `**/projects/*.mdx`,
  contentType: 'mdx',
...
...
...

This allows to have different file types. You could just use *.md and *.mdx.

[Blankeos]

Create multiple document types and refer them in the makeSource method like

@ahofmeister this works like a charm for me. I structured my folders like the following

> src
   > content
      > posts
         - *.mdx
      > projects
         - *.mdx