Make /developers JS driven via an API

[joevans]

My thinking is that the /developers search page could be completely JS driven with a fetch request to an internal API. That way the filters can be JS driven and not a REST call each time.

This should improve scalability as users, in-platform devs, & filter features increase.

Any thoughts on this?

[joemasilotti]

What's your thinking on why a JavaScript driven fetch would scale better than standard HTTP requests?

[joevans]

With a JS fetch, the returned object can be all (or some) of the developers and their attributes. For example,

{
  devs: {
      [
        name: "The Turbo Native Guy",
        bio: "asdf asdf",
        avatar: "...avatar.jpg",
        featured: true,
        availability: 2022-06-01,
        work_pref: "part_time",
        timezone: "GMT-3",
        role_level: "c-level",
        ....etc
      ],
      ....
  }
}

Then we can iterate through on the client side which will speed things up and we could give some CSS flare on load (not JS specific required for this at all but an opportunity). Additionally, we can use that data for all the filtering and pagination with JS.

Currently, the initial devs grab, pagination, sorting, filters, etc are all server sided rendered and we could prevent each one of those calls to the server.

[joemasilotti]

Then we can iterate through on the client side which will speed things up

Is /developers slow for you right now? According to Scout APM I'm seeing sub 100ms response times for this endpoint.

Currently, the initial devs grab, pagination, sorting, filters, etc are all server sided rendered and we could prevent each one of those calls to the server.

I'm not sure I follow, there's only a single call to the server right now. What are you proposing moving to JavaScript would avoid?

[joevans]

The server isn't slow today, no. But as for scaling the platform then this could result in a high amount of calls and server resources.

I'm not 100% sure I am going to be able to explain this. But I'll try.

Your active record calls are going to increase as your views increase. This is because every time someone visits the /developers page it's a view + activerecord. But as they start to navigate around that page, like apply one filter for example, today that's another hit on both activerecord & view.

What I am proposing is that if we made that page SPA-like, then the additional navigations of filters & page views will result in near zero server sided resources being called as the page views increase. The whole /developers results could be browser cached & filters are handled with just filtering/sorting the dev results object.

I can probably code this in order to show it better than I can explain it... lol

[joemasilotti]

Even if this ends up being much faster down the road I'm hesitant to add SPA-like pages now (or ever) for a few reasons:

1. I'm trying very hard to keep this codebase as "boring Rails" as possible. In the future that might mean Turbo Streams to do some of what you are explaining.
2. There is a ton of additional code to maintain. We would have to keep the Ruby filters in place for folks with JavaScript disabled while also implementing everything in JavaScript.
3. Contributors to the codebase might not be as proficient in JavaScript as they are in Ruby/Rails. I would hate to wall off a section of the codebase.
4. This still feels like a pre-optimization. Under 100ms, even under 200ms, is totally fine by me.

But hey, prove me wrong! Seed your local database with 10,000 developer profiles and run some filters. How much slower is that than with the current count? How quickly does it get out of hand?

[joevans]

Sounds good. I'll do some testing and let you know of any gains I find.

[joemasilotti]

Sounds great! Looking forward to seeing what you can dig up.

P.S. I converted this issue to a discussion so we can reply to individual comments.

[joevans]

[joemasilotti]

Why can't this be done in Rails code and rendered on the server side though? Why does it need JavaScript?

[joevans]

I'm sorry. I meant to put this under #407. May I delete this here and add it there?

[joemasilotti]

Ah, gotcha. That makes more sense! Yeah, go for it.

[joevans]

Thanks. There's nothing more that's needed on this one just yet. haha.