Skip to content

Add ::always_load to resources for eager loading.#847

Open
mrloop wants to merge 7 commits intoJSONAPI-Resources:masterfrom
mrloop:always_include
Open

Add ::always_load to resources for eager loading.#847
mrloop wants to merge 7 commits intoJSONAPI-Resources:masterfrom
mrloop:always_include

Conversation

@mrloop
Copy link
Copy Markdown
Contributor

@mrloop mrloop commented Sep 22, 2016

Enables resources to be defined with always_includes for eager loading. e.g.

class TagResource < JSONAPI::Resource  
  always_include posts: [:section]
  has_many :posts
end

class PostResource < JSONAPI::Resource
  has_one :section
  has_many :tags, acts_as_set: true, inverse_relationship: :posts, eager_load_on_include: false
end

class SectionResource < JSONAPI::Resource
  has_many :posts
end

Normalize and merge the model includes and always includes, iterate over the model includes and merge their always includes. The model and always includes can both be nested.

See #485

@mrloop
Copy link
Copy Markdown
Contributor Author

mrloop commented Sep 23, 2016

Test suite passing for 5.0 and 4.2 but is failing on rails 4.1. 4.1 is no longer officially supported and isn't supported by railslts. No unpatched secuirty vunrebilities reported yet but it's just a matter on time. Will 4.1 support be dropped or do I need to make this PR 4.1 compatible to be considered for merging?

if no resource for specified always_include presume it is an existing
model with no associated resource and use the symbol.
only use related type includes when loading related resource to reduce
the number of unused includes.
The previous recursive strategy for constructing the Active Record
includes argument could result in unused eager loads being performed.

The new strategy normalizes and merges the model includes and always
includes, then iterates over the model includes and normalizes and
merges their always includes. The model and always includes can both
be nested.

For example the recursive strategy gave the following includes for the
app I'm working on:

    [{:raw_show=>[{:events=>[:raw_show, {:run=>[{:show=>[:events,
    :photos]}]}, :master_allocations, {:venue_layout=>[:venue]}]},
    :photos]}, :bookings, :run, :master_allocations, :venue_layout]

While the merge strategy results in:

    [:run, :master_allocations, :venue_layout, :bookings,
    {:raw_show=>[:events, :photos]}]
@lgebhardt
Copy link
Copy Markdown
Contributor

@mrloop This seems to be in violation of the spec. From http://jsonapi.org/format/#fetching-includes:

An endpoint MAY return resources related to the primary data by default.

Implies it's Ok, but it breaks a subsequent section:

If an endpoint supports the include parameter and a client supplies it, the server MUST NOT include unrequested resource objects in the included section of the compound document.

I think we need to handle this as a default_includes. This would only be used if the request didn't specify any includes. I think this would satisfy compliance with the spec, and hopefully your needs.

@mrloop
Copy link
Copy Markdown
Contributor Author

mrloop commented Sep 30, 2016

Maybe the naming is a bit off and should be always_load as in preload or eager_load to better express the intent, that is to MyActiveRecord.includes related records so not to n+1 load db records when calculating attributes which rely on loading the related records and not includes in the jsonapi-resource sense, that is add the records to the payload.

@lgebhardt
Copy link
Copy Markdown
Contributor

@mrloop I see. I was getting hung up on the name an assuming the feature would extend to including the resources, not just the eager loading. I should have looked at the code and I would have seen how it was used - sorry. A name change would definitely be appreciated, especially since I hope to add the default_includes feature in the future. I'd also like to see some documentation in the readme so people know how to use this.

Rename to better express the intent from `always_include` to `always_load` as in `preload` or `eager_load`
@mrloop mrloop changed the title Add ::always_include to resources for eager loading. Add ::always_load to resources for eager loading. Sep 30, 2016
@mrloop
Copy link
Copy Markdown
Contributor Author

mrloop commented Sep 30, 2016

Have renamed always_include to always_load and similar methods. Documentation to follow.

@mrloop
Copy link
Copy Markdown
Contributor Author

mrloop commented Oct 4, 2016

@lgebhardt that's the renaming done and the documentation added.

@NuckChorris
Copy link
Copy Markdown

Wouldn't this make more sense as default_scope { includes(:thing) } in the model? I know "default scopes are bad", but it seems strange to put this in the Resource layer tbh and if you use it to compute virtual attributes it seems to be a reasonable use of a default scope

@nathanpalmer
Copy link
Copy Markdown
Contributor

@NuckChorris Something like this would be useful at the resource layer when you are exposing a property that would otherwise lazy-load a related model. I wouldn't want it to always load this data directly on the model since it's use-case is only for serialization at the resource.

@mrloop
Copy link
Copy Markdown
Contributor Author

mrloop commented Jan 19, 2017

@lgebhardt I'm thinking that Resource is the incorrect location to define eager loading. Instead eager loading should be defined on Serializers. You could have different Serializers for the same Resource, each Serializer having different eager loading requirements. Doing eager loading from the serializer also allows you to take into account fields and reduce what is eager loaded based on fields. I'm going to spike out this approach.
1. Defer loading records until Serializer requires them.
2. Simple eager loading implement in Serializer similar to current PR
3. Have a look at fields and how these can be used.
My main goals would be to get 1 and 2 working. Would a PR along these lines be considered for merging?

Edit, I don't think the above is correct, going to spike something to get a better idea of what's needed.

@Genkilabs
Copy link
Copy Markdown

@mrloop what is the status of this PR? Did it work for you? We are being bitten by N+c queries loading records that need relations in their computed attributes.

@mrloop
Copy link
Copy Markdown
Contributor Author

mrloop commented Jul 17, 2017

@Genkilabs it's stalled, using similar version on code in small production app at the moment, working on another larger app at the moment using it as well, hope to be able to iron out any issues when that's deployed and in use, looks ok in testing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants