Performance with 10,000 Pages

[jalandis]

I am very excited about the elm-pages project and am experimenting with it to see how it might work for a larger marketing site.

I ran into 2 issues when creating a test spike and am hoping to report + maybe get some advice.

Decoder Mapping

The first issue is in my API's decoders. I have an API which for testing purposes I am the sole creator/consumer. This API has only one endpoint which returns all data for my site. I am using elm-protobuf to generate ELM code. This project produces Json.Decode and not OptimizedDecoder decoders.

The OptimizedDecoder.Decoder sounds amazing when working with a public API but in this case, no extraneous data is ever expected to be returned. It would be convenient to convert between a Json.Decode.Decoder and an OptimizedDecoder.Decoder in this instance.

Is that possible?

Performance Bottleneck

The second issue I have is related to performance. As I said, my DataSource pulls in JSON for generating 10,000 pages. With that much data, every page takes 2 seconds to generate. The full site would take several hours to build if I let it run. For comparison, when I limit to only 100 results in the JSON the build takes about 10ms per page. I experimented with moving the data to Shared.elm but found I still need to parse all the data for routes anyway.

Is this a known issue for larger sites?
Is there any way to avoid parsing the data for every page generated? Without 10,000 separate network requests of course.

Here is the page module I am working on for reference.
npx elm-pages add EmployeePay/PublicID_

module Page.EmployeePay.PublicID_ exposing (Data, Model, Msg, page)

import DataSource exposing (DataSource)
import DataSource.Http
import Head
import Head.Seo as Seo
import Html exposing (..)
import Http
import Json.Decode
import Json.Encode
import Open_data_api
import OptimizedDecoder
import Page exposing (Page, PageWithState, StaticPayload)
import Pages.PageUrl exposing (PageUrl)
import Pages.Secrets as Secrets
import Pages.Url
import Route exposing (Route)
import Shared
import View exposing (View)


type alias Model =
    ()


type alias Msg =
    Never


type alias RouteParams =
    { publicID : String }


page : Page RouteParams Data
page =
    Page.prerender
        { head = head
        , routes = routes
        , data = data
        }
        |> Page.buildNoState { view = view }


apiCall : DataSource.Http.RequestDetails
apiCall =
    { url =
        Open_data_api.getPublicEmployeeEarningsRequestEncoder { year = 2021, quarter = Open_data_api.First }
            |> Json.Encode.encode 0
            |> (++) "https://poc-playground/getPublicEmployeeEarnings?"
    , method = "GET"
    , headers = []
    , body = DataSource.Http.emptyBody
    }


routes : DataSource (List RouteParams)
routes =
    DataSource.Http.request
        (Secrets.succeed apiCall)
        (OptimizedDecoder.field "employees"
            (OptimizedDecoder.list
                (OptimizedDecoder.field "publicID"
                    (OptimizedDecoder.int
                        |> OptimizedDecoder.map (String.fromInt >> RouteParams)
                    )
                )
            )
        )


data : RouteParams -> DataSource Data
data routeParams =
    DataSource.Http.request
        (Secrets.succeed apiCall)
        (OptimizedDecoder.field "employees"
            (OptimizedDecoder.list
                (OptimizedDecoder.map2
                    (\p t -> { publicID = p, title = t })
                    (OptimizedDecoder.field "publicID"
                        (OptimizedDecoder.int
                            |> OptimizedDecoder.map String.fromInt
                        )
                    )
                    (OptimizedDecoder.field "title" OptimizedDecoder.string)
                )
                |> OptimizedDecoder.map
                    (List.filterMap
                        (\i ->
                            if i.publicID == routeParams.publicID then
                                Just i.title

                            else
                                Nothing
                        )
                    )
                |> OptimizedDecoder.andThen
                    (\i ->
                        case List.head i of
                            Just d ->
                                OptimizedDecoder.succeed (Data d)

                            Nothing ->
                                OptimizedDecoder.fail "no matching route found"
                    )
            )
        )


head :
    StaticPayload Data RouteParams
    -> List Head.Tag
head static =
    Seo.summary
        { canonicalUrlOverride = Nothing
        , siteName = "elm-pages"
        , image =
            { url = Pages.Url.external "TODO"
            , alt = "elm-pages logo"
            , dimensions = Nothing
            , mimeType = Nothing
            }
        , description = "TODO"
        , locale = Nothing
        , title = "TODO title" -- metadata.title -- TODO
        }
        |> Seo.website


type alias Data =
    { title : String }


view :
    Maybe PageUrl
    -> Shared.Model
    -> StaticPayload Data RouteParams
    -> View Msg
view maybeUrl sharedModel static =
    { title = "Employee Pay"
    , body = [ div [] [ text static.data.title ] ]
    }

[dillonkearns]

The OptimizedDecoder.Decoder sounds amazing when working with a public API but in this case, no extraneous data is ever expected to be returned. It would be convenient to convert between a Json.Decode.Decoder and an OptimizedDecoder.Decoder in this instance.

I have a similar use case with elm-graphql! There is something called unoptimizedRequest for exactly this kind of use case. Here's the code I use for my elm-graphql requests:

https://github.com/dillonkearns/incremental-elm-web/blob/92ef3178f0082f66b264ff1bfa0a9f9de0d2ecb8/src/Request.elm

As I said, my DataSource pulls in JSON for generating 10,000 pages. With that much data, every page takes 2 seconds to generate. The full site would take several hours to build if I let it run. For comparison, when I limit to only 100 results in the JSON the build takes about 10ms per page. I experimented with moving the data to Shared.elm but found I still need to parse all the data for routes anyway.

I spoke to someone recently who was using elm-pages v2 to generate around 6k pages. They ran into a similar issue with the routes. So I made a couple of performance tweaks. The first was I made DataSource.combine more performant. And the second is I skip the check for whether a page is pre-rendered during the build step, because the build command starts by getting a list of all pre-rendered routes and then going through and generating each of those. So it's a redundant check, and it turns out that was the bottleneck for this project:

https://github.com/kraklin/elm-docset/tree/spike_elm_pages

So that change was the difference between about 30 minutes vs. 30 seconds for a full build! Hopefully you'll see a similar benefit, but given it's a pretty large site you may encounter other bottlenecks. Either way, I'd love to hear how it goes.

I haven't published those performance improvements yet because it is a non-breaking change, but it requires a major version bump, so I wanted to clean some things up before publishing it to prevent unnecessary major version bumps in the future.

You can try out the latest code on the master branch. If you're up for that, I'd love to hear the results and see the numbers before and after.

You can change out the elm-pages package and replace it with:

npm i -g elm-json

elm-json install avh4/[email protected] danyx23/[email protected] dillonkearns/[email protected] elm/[email protected] elm/[email protected] elm/[email protected] elm/[email protected] elm/[email protected] elm/[email protected] elm/[email protected] elm-community/[email protected] elm-community/[email protected] miniBill/[email protected] noahzgordon/[email protected] tripokey/[email protected] zwilias/[email protected] MartinSStewart/elm-serialize Skinney/murmur3 vito/elm-ansi jfmengels/elm-review elm-community/result-extra

(If there are any complaints about other missing dependencies, you can install them with elm-json install <author/package>).

You can add elm-pages as a git submodule like this:

git submodule add https://github.com/dillonkearns/elm-pages.git

Then add ./elm-pages/src to your source-directories.

[dillonkearns]

If there are any other bottlenecks you run into, let me know, I would love to experiment based on your feedback if anything else comes up. And feel free to ping me with questions in the #elm-pages channel in the Elm slack.