SDK Mesh instance with inlined metadata (caching)

[JustusNBB]

Is your feature request related to a problem? Please describe.

I've setup an nx monorepo, and created a lib, which should export a mesh sdk.
For actual endpoints I still need to overwrite the source/metadata, so I am using a (patched) backup metadata file, for two OpenAPI endpoints (until they are fixed) and a backup of a long OData metadata.xml file to shorten the introspection delay.

Now within the nx monorepo, these metadata files need to be treated as assets and copied around with the sdk-lib, because it goes looking for them at runtime.

Describe the solution you'd like

I want to be able to generate an sdk with mesh instance without handling additional non js assets (json/xml/yaml).
For now I am successful configuring the mesh instance programmatically, but to export a built lib we need to put some more effort in.
For OpenAPI I can easily feed the JSON directly, if I adjust the handler a bit - PR incoming!
For OData, it could be possible to do something similar (include the xml string at build time).

Describe alternatives you've considered

I don't know. I am not supposed to skip introspection I think, but it might still be a common usecase to embed API metadata.

Additional context

Functions like "findAndParseConfig" or "utils.readFileOrUrlWithCache" look for files in a relative directory - this breaks when trying to export a library - relative lookup is by process.cwd

[santino]

Hi @JustusNBB, seems like there are a few things in common between what you're describing and what I've been looking for when starting to use Mesh.

Can you share more context on the "why" you want to generate an instance without handling additional assets?
You mentioned that you don't have endpoints yet, I suppose when you will you'll use those as data sources to be converted into a GraphQL schema, as opposed to the static files you're generating ATM.

As for the relative directory issue you're mentioning, you should try using the dir option to set your base directory.
You can set dir in CLI commands, like --dir ./src and also pass it to findAndParseConfig, like findAndParseConfig({ dir: './src' }).

Let me know if that works.

[JustusNBB]

@santino thanks for the info, I just saw there is also a baseDir argument, I'll be trying your suggestions!

Created my PR anyways ;)

[santino]

I personally don't have experience with the SDK, but looking at the API, the behaviour shouldn't be different from standard getMesh and so executable schema.

I have some concerns about the change proposed, so I would like to get a better understanding.

What is the nature of your "blocker"?
Is it just that the static files of reference wouldn't be served from the relative directory?
If so, then the "dir" option should solve the problem.

Regardless of the fact that Mesh currently needs to generate the GraphQL schema at runtime, you're still going to have to deal with static files at build time even if/when that eventually changes.
Let's say that as part of your build you can generate the GraphQL SDL (and so introspection at runtime would no longer be required), this will still need to be saved locally and be available for read at runtime.
Mesh could make simplify this process, but at the end of the day you're still going to have to make sure that some files generated at build time are available at runtime; regardless of what their content is.

With this in mind, I am not sure implementing the change proposed is really going to address your point, at least as I understood it.

[JustusNBB]

The actual issue is that I need to configure the nx build properly to copy the metadata. A way would be to treat them like assets, I've seen that approach to copy them from lib to app dirs. I just need to tell my build to include json and xml like source files and keep them alongside, I guess.

It would also be a solution to inline the metadata in a typescript file as a string or object.

I found the baseDir option to be working as expected (like path.join), this is my code approach of creating a mesh without loading .meshrc.yml:

// code only approach "replacing" .meshrc
import gql from 'graphql-tag';
import { getMesh } from '@graphql-mesh/runtime';
import OpenAPIHandler from '@graphql-mesh/openapi';
// import ODataHandler from '@graphql-mesh/odata';
import StitchingMerger from '@graphql-mesh/merger-stitching';
import { KeyValueCache } from '@graphql-mesh/types';
import { PubSub } from 'graphql-subscriptions';

import { resolvers } from '../mesh/additional-resolvers';

export function getMeshInstance({ cache }: { cache: KeyValueCache }) {
  const meshContext = {
    cache,
    pubsub: new PubSub(),
  };
  return getMesh({
    additionalResolvers: resolvers,
    additionalTypeDefs: [
      gql`
        extend type ProductViewV2 {
          """
          All selling prices, stitched from price service
          """
          prices: [SalesPrice]
        }
      `,
    ],
    sources: [
      {
        name: 'PIMS',
        handler: new OpenAPIHandler({
          name: 'PIMS',
          baseDir: __dirname,
          config: {
            source: '../metadata/pims.metadata.json',
            // source: join(__dirname, '../metadata/pims.metadata.json'),
            // source: require('../metadata/pims.metadata.json'), <-- works with the PR
          },
          ...meshContext,
        }),
      },
      {
        name: 'PriceService',
        handler: new OpenAPIHandler({
          name: 'PriceService',
          config: {
            source: require('../metadata/price-service.metadata.json'),
            //source: join(__dirname, '../metadata/price-service.metadata.json'),
          },
          ...meshContext,
        }),
      },
      // leaving out OData sources here...
    ],
    merger: StitchingMerger,
    ...meshContext,
  });
}

[JustusNBB]

OK I found ts compilerOption resolveJsonModule to be useful to enabling copying of JSON metadata.
But if I want to do the same for OData xml files I need to create a postbuild task...
Or is there another way to ship OData metadata within a built sdk lib?

Is there a more efficient way of caching the schema and or mesh instance configuration generally (skip config resolution & introspection)?

[santino]

Actually, I started to do some work in terms of building initial cache.
Feel free to have a look at the conversation here.

The idea is to introduce a CLI command that you can trigger at build time, and will go through your sources (may they be local files or remote URIs to download original schema from).
Once we collected all sources we can dump their values into a single static JSON file, which represents the initial cache.

In this case, you generate a single JSON asset at build-time, and you copy this one only, so that it is available at runtime.
At runtime, when starting, Mesh will read this file and hydrate its content into the cache; so your sources will be served from the cache and you'll no longer need to have all the files available.

Does this sound like something useful to your use case?

[JustusNBB]

@santino Yes this goes into the right direction, such a cache artifact would be great for a generated SDK lib to boot faster!

[santino]

Cool.
This shouldn't take long to be finalised.

I will let you know once available.
In the meantime, I hope your workarounds are enough to not keep you blocked.

[JustusNBB]

Nice my PR has been merged! I'd still say we leave this ticket open for your work @santino, I'd like to try it as soon as you think it's ready :)

[JustusNBB]

I have a fourth source, another OData endpoint which is public (in contrast to the other ones mentioned opening the ticket), let's explore my issue with it a bit more if I can improve my workaround:
Oh great ms uservoice is down, maybe this link works again when it comes back - https://microsoftgraph.uservoice.com/forums/920506-microsoft-graph-feature-requests/suggestions/37700059-add-manager-to-list-users-graph-api
The issue is that query { me { manager }} is not a User Entity but an empty object from the introspected metadata. But with a little bit of tinkering with the metadata, it does include queryable User attributes.
So on L2146 (in my copy - just noticed this is already outdated now) I can change the metadata so the manager has user attributes queryable:

<NavigationProperty Name="manager" Type="graph.user"/>

How would you achieve the same with an adaption on the schema level?

[santino]

Hi @JustusNBB,
a quick note to let you know that work for the introspection cache, mentioned above, has been completed.
Please refer to the newly introduced documentation page for more info.

As for your other point, to me, it feels like you want to add a new field "manager" to your "User" type; if I'm getting things right.
In this case, you should be able to achieve this by extending your schema.
Take a look at the relevant page in the documentation, to check if this approach could fit your use-case.

[JustusNBB]

I get the idea, but then I have to implement a resolver with a manual join too right?
Or does something alike:

additionalTypeDefs: |
  extend type Query {
    me {
      manager: graph.user
    }
  }

work without an additional resolver?
The beauty of the metadata fix is that the join (second query) is already implemented, just not declared as returning user attributes. Every other approach (like doing this with axios or msal) requires two separate queries while graphql-mesh stitches them automatically if the source metadata is correct - does it fetch profile and manager in parallel, sequentially or within a nested OData query?

[JustusNBB]

Does not look to get fixed any time soon on microsoft's side: https://docs.microsoft.com/en-us/answers/questions/335590/ms-graph-odata-schema-metadata-for-manager-is-a-di.html

It confirms though, manager is not an addition to the schema: manager is by default a DirectoryObject but it can also be of the user or orgContact resource type.
I am looking for the schema transformation allowing me to switch the type of an existing resource without resorting to changing the OData XML metadata source defaults.