RFC: Structured metadata

[bakkot]

I have a proposal for a spec for metadata, laying out goals and a formal spec.

I'm happy to implement this if there's buy-in.

Thoughts?

---

RFC: Structured metadata

Currently when generating images from the CLI (but not the web), metadata for that is stored as a string kind-of corresponding to the prompt. That metadata is enough to reproduce the original image... sometimes.

I'd like to:

• be more precise about the metadata which gets stored
• allow reproducing any output just from the metadata and necessary input files
  • necessary input files meaning the model weights, the image for img2img, and the embeddings if using embeddings
  • metadata should allow you to confirm that you have the right inputs, by storing hashes of all of those files
  • "any" output includes outputs from seed fuzzing and interpolations (which I haven't written yet, in part because I wanted to work out the metadata format first)
• store it in a structured format, namely JSON
• expand the metadata so it works with grids
• expand the metadata so it works with stuff like variations and interpolations

To that end, I'd like to propose the following spec for metadata.

In this doc, "hash" means "the first 8 characters of the hex-encoded sha-256".

Data location

Metadata is a JSON string following the "top-level data" schema, stored in an uncompressed PNG tEXt or iTXt chunk named "sd-metadata". (This corresponds to what PIL does already when adding text data - it will choose tEXt or iTXt depending on whether it contains non-latin-1 characters. I just figure it's worth writing this down.)

Top-level data

The top-level metadata should have the following fields:

• model: "stable diffusion"
• model_id: string identifying the model. must by the model_id field of a Model card. Optional; there is no default value, but consuming applications may infer a value from model_hash if they recognize that value.
• model_url: a string giving a URL where the model can be downloaded (if public) or read about (if not). Optional, does not have a default.
• model_hash: hash of the weights [precise format TBD depending on implementation feasibility]; see the "model information" section below
• app_id: a string identifying the application consuming the model. It is recommended, but not required, that applications hosted on GitHub use the username/repo_name of the repository in this field; for example, the fork we're on would use lstein/stable-diffusion.
• app_version: a string giving the version of the app from app_id. It is recommended, but not required, that projects with numbered versions use a string of the form v1.0, and that projects built from git repos use the short-form git hash of the commit. Optional, defaults to "unknown".
• app_url: a string giving the canonical location of the application on the web. Optional, does not have a default.
• embeddings_hashes: an an array of the hashes of any textual-inversion embeddings in use. Optional, defaults to an empty array.
• arch: "cuda", "MPS", or another helpful value indicating the GPU architecture. Optional, defaults to "unknown".
• grid: a boolean, whether this was a grid. Optional, defaults to false.
• metadata_version: the string "1.0". Optional, defaults to "1.0". Breaking changes to this metadata format should update this field.

and then also one of the following two fields, depending on whether this is a grid:

• image: an object in one of the formats specified below
• images: an array of such objects

Image data

Every image has the following fields:

• type: either "txt2img" or "img2img"
• postprocessing: either null, indicating no postprocessing was done, or an arbitrary object representing the postprocessing performed. Spec for this will depend on individual postprocessors, but I'll write something up for the ones we support. Optional, defaults to null.
• sampler: one of these samplers
• prompt: a nonempty array of { prompt: string, weight: number } pairs. The single-prompt case is [{ prompt: prompt, wieght: 1 }]
• seed: a seed
• variations: an array of { seed: number, weight: number } pairs used to generate variations. Optional, defaults to an empty array.
• steps: the number of steps configured to be taken
• cfg_scale: the unconditional guidance scale
• step_number: the number of steps actually taken. Normally this will be the full number of steps, but for intermediate images it may be less. Optional, defaults to steps (or strength_steps in the case of img2img).
• width: the specified width (as a number of pixels). Optional only when this metadata is embedded in am image whose width is the same as this value would be, in which case it defaults to that image's width.
• height: the specified height (as a number of pixels). Optional only when this metadata is embedded in am image whose height is the same as this value would be, in which case it defaults to that image's height.
• extra: an object containing any necessary additional information to generate this image. Not to be used for other data, like contact information. Optional, defaults to the empty object.

Images of type img2img also have the following fields:

• orig_hash: hash of the input image
• strength_steps: the configured strength for running img2img (as an integer; as discussed here, that's what it actually is).

Height/width are not stored since you can infer those from the file.

Thoughts on storing the model information

I am proposing to store a hash of the loaded model, which is a lot faster than reading the file from disk a second time, but the hash correspond to the file on disk. Better than nothing, though.

Is it worth also storing a hash of the model config? I don't think so, since you're always going to need the original config for a given model weights file.

[psychedelicious]

Perhaps fork could be a commit reference rather than a repo reference - if a repo changes its implementation, you may not be able to recreate the output.

[bakkot]

I'd love to include the current commit, but unfortunately a lot of users get the source by downloading the zip file from github, which does not include the hash anywhere, as far as I can tell (in particular it doesn't include the .git directory).

I guess it could be an optional field to be included if the code generating the metadata can reasonably figure it out, though.

[fat-tire]

Thought: Could this be coordinated with the URI definition standard request I proposed here? So that attribute names match, etc.? And how might they work together? Would a URI point to the image and a second argument produce this structured metadata?

[fat-tire]

Also, fwiw, the single-file, cross-platform PyQt5-based SD GUI I was contributing to already saves its settings in json, but I was realizing that the json settings actually define the image itself in a way.... Dunno if this is something to build on for a proof-of-concept as it's super easy to add to.

[bakkot]

@fat-tire URIs are inherently somewhat unsuited for structured data, and a full specification for a SD image (when you take into account stuff like variations and grids) is inherently structured. So if you want to do a URI, I think it would be best to have only a single key-value pair, where the value is JSON in the format of the spec proposed here. Then you don't need to try to coordinate two different formats for this specification.

[psychedelicious]

I'd love to include the current commit, but unfortunately a lot of users get the source by downloading the zip file from github, which does not include the hash anywhere, as far as I can tell (in particular it doesn't include the .git directory).

I guess it could be an optional field to be included if the code generating the metadata can reasonably figure it out, though.

Ah, right, didn't think about that. Optional field sounds good - and now that I think about it, if we are including a commit reference, we ought to include a branch reference as well (or does a commit imply a specific branch? I don't know).

[fat-tire]

@fat-tire URIs are inherently somewhat unsuited for structured data, and a full specification for a SD image (when you take into account stuff like variations and grids) is inherently structured. So if you want to do a URI, I think it would be best to have only a single key-value pair, where the value is JSON in the format of the spec proposed here. Then you don't need to try to coordinate two different formats for this specification.

This is good, except it makes the URI super long (is the json value in your keypair further encoded/compressed in some way?) and is effectively a wrapper around the json. I'm wondering if there's an abbreviated but human-readable and easily-edited format that could be used to reference an image(s) resource?

Like imagine an image browser for SD-- if the complete json would show up in View Source, what would go in the URL/Address bar? On a reddit post featuring some cool approach to generating images, you wouldn't attach a json file, but what if you could put a short self-contained link that could be copy/pasted or tweaked by hand by any non-technical person-- what would it ideally look like? Or say I wanted a single text file or google doc full of accumulated copy/pasted references images, say, for some art project-- what's the smallest, one-line-per-image (for example) way to do collect them? What might fit in a small QR code?

These are the types of use cases I'm imagining. I feel like a structured json file, even though it was designed to be interpreted by humans, may be too large and unreadable for non-programmers to easily understand and make changes simply and quickly and that a compressed URI with &param=value and sane defaults is more familiar. Maybe I'm talkin' crazy tho, dunno.

[bakkot]

Eh, they don't get that long unless you have a lot of data, and then, well, it is actually long. The data isn't compressed but there's simply not that much of it.

If you're really worried about length we could make some of the fields optional and specify default values - e.g. variations is assumed to be empty if omitted, etc. If we do that you end up with something like

{"ai-type":"stable diffusion","fork":"lstein","weights-hash":123456,"image":{"type":"txt2img","sampler":"k_euler","prompt":"the whole prompt goes here","seed":1234567890,"steps":50,"cfg_scale":0.7}}

which is pretty much as human-readable as a URL would be. More, arguably. There's not really a lot of overhead from the JSON format itself relative to URI-style k=v&k2=v2 style, just a few extra quotation marks (but fewer characters wasted on %20). And this really is the minimum information you need to unambiguously refer to a specific image.

I think copy-pasting things like the above is at least as easy as copy-pasting a URI, and has the benefit of not confusing people; URIs look like you should be able to point your browser at them, and that's not the case here. Plus URI encoding for spaces - which will come up a lot, because every prompt has spaces in it - is a huge pain for humans to deal with.

[fat-tire]

What you've said in principal makes sense to me, especially w/regard to %20 throwing people, but yes, sensible fallback defaults for omitted fields should be part of this standard to make it compact and easy for normal use by non-techies. However, as I think more about this, any defaults would need to be standard to a particular scenario-- ie,, they'd have to be "known" for consistency across apps supporting this metadata structure to agree to all fallback to defaults, and in the same way. But how would everyone know that say with the v4 model, "512x512 is the agreed-upon size default, so always assume that"? Is there some rule, like "the default size for any model is always the shape of the training images"? That seems too rigid. Who decides the defaults, and where would "sensible defaults" be published that anyone writing an application would know to find them?

Along the lines of defaults, I wonder if some fields should be deemed "required" vs "optional" or something-- the required ones would contain the bare minimum needed to generate an image-- people can then build out from there with greater specificity.

Also, you mention that height/width could be determined from the image (I assume, as this json is metadata embedded within an image) or when offered alongside it-- but if the json metadata alone is defined as sufficient to produce an image from scratch, I'd think you need to include width/height in the metadata. ie, don't assume anything should be inferred from the image, as an image may not be included or may have been resized or screenshotted or whatever (I acknowledge, I've expanded the use case beyond metadata in the file as originally conceived, but if this is a format also meant for copy/pasting in forums or via other human-means, it needs to be complete)

Also-- aside from the hash of the weights file, don't you want to have something to indicate about which version of a model is to be (or was) used, which at the very least would be helpful in providing feedback to the end user when they don't match or when a model is missing? Example-- SD is about to release v1-5 right? The model card already contains this metadata in the form of the model_id, such as "CompVis/stable-diffusion-v1-5"

Other random unformed ideas/thoughts:

• "Ai-Type"- maybe "AI-Family"? "type" makes me think of data types or model types
• "fork" -- maybe "variant"? Perhaps always using the model card model_id (if it exists) might be more consistent. (example: "CompVis/stable-diffusion-v1-4" and even "lstein/stable-diffusion-v1.5" if those are meant to be used together. (for that matter, would branch also be useful as an option-- some experimental branch for example might have some special property or function that isn't on main...)
• Not sure if "fork" should always refer to a fork of the code (CompVis vs lstein) or a fork of a model, such as a retrained model which is almost certain to emerge (stable-diffusion-v1-5+-with-extra-nsfw). Perhaps this can be broken into variant and model_id.
• Timestamp? Since this won't be able to be derived from an image when that image doesn't exist, it might be nice to have a time/date for organizing or even versioning different takes on the same image. This also may or may not match the time of the created image.
• contact_info: string or object
• image_name: string (preferred name to be created-- if the referenced image doesn't already exist)
• When grid=True, is it the grid image that has this metadata? Do the individual output images contain the tEXt chunks as well? Does the grid image contain references to the names of the child images in /samples, and do the child images know about which grid image they belong to?
• grid_name : string (if this image is part of a grid- ie, which numbered image is it?)
• Description: string? Keywords for searching through images apart from what's in the prompt?
• Citation: string? The model card for sd contains additional data about sd itself that may (or may not) be appropriate to include in an SD-created image. Maybe it doesn't belong there, but I wanted to at least offer it for consideration
• Licensing info? either a link to the model card or a string with whatever license might be relevant.
• Notes :string ? -- perhaps for the artist/developer who created it? "I tried to emphasize french expressionists to bring out X,Y,Z. Feel free to ask questions on twitter at.."
• NSFW: boolean- again, probably doesn't belong here, but since it's emphasized so strongly in the sd repository and enabled by default, this flag might make sense.

Sorry there's a lot here and apologies if some of these don't quite relate to the lstein fork-- I'm only just getting familiar with the main repo. I don't wanna get to big and unweildy either, but I think some basics like H&W and model name/version have to be there if it's going to be used as a simple text-based generate-from-scratch "trading card", useful for learning, sharing, research, etc.

And I guess once everything is hammered out, if this turns out to be too heavy a way to do this, someone can always "URI-ize" it, especially once the field names are deemed stable and a good standard.

Anyway, this is pretty exciting stuff! Thanks.

ft

[bakkot]

Along the lines of defaults, I wonder if some fields should be deemed "required" vs "optional" or something-- the required ones would contain the bare minimum needed to generate an image-- people can then build out from there with greater specificity.

Yeah, I was imagining only certain field could be omitted. As to how people would know how to interpret missing fields, we'd write it down. So for examples you could omit grid, and that would be defined to mean false.

Also, you mention that height/width could be determined from the image (I assume, as this json is metadata embedded within an image) or when offered alongside it-- but if the json metadata alone is defined as sufficient to produce an image from scratch, I'd think you need to include width/height in the metadata

Yeah that's a good point. I'll update the OP to add those as fields which are optional only when the metadata is embedded in an image file from which it is possible to derive those values.

Also-- aside from the hash of the weights file, don't you want to have something to indicate about which version of a model is to be (or was) used, which at the very least would be helpful in providing feedback to the end user when they don't match or when a model is missing?

The hash of the weights file is sufficient to uniquely determine the model.

For the sake of giving users helpful feedback, rather than just "the weights file you have doesn't match", it would be kind of nice to also have a version to report, but we don't necessarily know that - for example, the installation instructions for this repo suggest putting the weights at models/ldm/stable-diffusion-v1/model.ckpt. There's nothing there to indicate what version of the model is in use, as far as I'm aware.

That said, I do think that having a list of known hashes would be helpful. That wouldn't be part of the metadata spec per se, though it might live along side it, and tools could hardcode that list to give more useful feedback when they see a hash they know.

"fork" -- maybe "variant"?

I'm fine with either name. I don't just want to use the mode card ID because the point of this field is to distinguish this repo from others. Including the commit hash or branch name would be nice, but as discussed in the comments above it's often not possible.

When grid=True, is it the grid image that has this metadata?

Yes.

Do the individual output images contain the tEXt chunks as well?

At least in this repo, there are not any individual output images. There's just the grid.

Does the grid image contain references to the names of the child images in /samples, and do the child images know about which grid image they belong to?

Per above answer, there are no such images. But if there were, the answer to both of these would be no.

Timestamp, contact_info, image_name , Description, Citation, Licensing, NSFW

For all of this stuff, I don't think it belongs in this spec - this is a specification specifically for the metadata for images to tell you the settings used to generate the image. If you want to include other information alongside the image, put it somewhere else. E.g., wrap the data from this spec: so { contact_info: 'whatever', name: 'whatever', metdata: { [this spec] } }.

That way we don't have to keep a registry of additional optional fields, which in my experience never works, and all of the fields in this spec can be automatically derived, which is important.

[fat-tire]

Yeah, I was imagining only certain field could be omitted. As to how people would know how to interpret missing fields, we'd write it down. So for examples you could omit grid, and that would be defined to mean false.

That makes sense, so long as everyone who is implementing this standard can agree on what the fallback defaults are.

Also-- aside from the hash of the weights file, don't you want to have something to indicate about which version of a model is to be (or was) used, which at the very least would be helpful in providing feedback to the end user when they don't match or when a model is missing?

The hash of the weights file is sufficient to uniquely determine the model.

Yeah, but it's only practical for confirming that the model you have is valid, which can be determined via other methods (usually whoever provides the model in the first place will offer a hash). Wouldn't it make more sense to name the model, version, and creator (the model_id from the card)? I mean, that's what it's for, it would make it easy to find the model or even the latest version of the model, and you wouldn't need to have to maintain a giant table of hashes and their names, versions, and sources.

That said, I do think that having a list of known hashes would be helpful. That wouldn't be part of the metadata spec per se, though it might live along side it, and tools could hardcode that list to give more useful feedback when they see a hash they know.

To me, that seems backwards-- the metadata should point to the model used, not the hash of the model- then you don't need any table that has to be maintained and updated (see below re generic model.ckpt). Validating the file's integrity seems outside the scope of what you would expect-- a regular image file doesn't offer the hash of the binary of photoshop that created it-- though it may have a string that indicates the tool and version used..

At the very least I'd expect a model_id should be included in addition to the model hash (for verifying integrity, I guess). I believe that's what the model_id is specifically intended for-- as a unique id meant to identify the model's name, origin, and version.

For the sake of giving users helpful feedback, rather than just "the weights file you have doesn't match", it would be kind of nice to also have a version to report, but we don't necessarily know that - for example, the installation instructions for this repo suggest putting the weights at models/ldm/stable-diffusion-v1/model.ckpt. There's nothing there to indicate what version of the model is in use, as far as I'm aware.

That's why I'm suggesting using model_id to indicate the version of the model. It seems to me to be a responsibility for the sd application implimenting this spec to know what model it's using (by name), not just offer a hash of whatever's there and say "good luck figuring out what this actually is!". Sure, maybe you can use the hash as a checksum to verify you have the right model once you know which model to use, when you're about to recreate the image.

To me, only providing the mdoel hash is a bit like say "to bake this cake, go to the store and buy the one ingredient they have in a red box that weighs 12 oz and costs exactly $13.54". Okay, I guess if I had a list of the store's inventory with associated prices I could find it, but it would have to be an always maintained, up-to-date list. Why not just tell me the exact ingredient I need, so I can ask for it directly? (Yes, as a safety, I can verify the product with the price and weight, but that's not how I want to look for it.) And going back to the error message for the user, it's a lot clearer to say "To bake this cake you need a 12oz bag of Whitman's Quality Flour." vs "Sorry, you are missing a product in a 12oz red box that costs $13.54." and cross-check that with a hopefully current list of all possible ingredients.

"fork" -- maybe "variant"?

I'm fine with either name. I don't just want to use the mode card ID because the point of this field is to distinguish this repo from others. Including the commit hash or branch name would be nice, but as discussed in the comments above it's often not possible.

But wouldn't a variant called lstein/stable-diffusion do exactly that? That is, it distinguishes this repo from fat-tire/stable-diffusion or others-it tells you where to get it, and what it is-- or maybe I'm missing something?

Do the individual output images contain the tEXt chunks as well?

At least in this repo, there are not any individual output images. There's just the grid.

I've not used lstein, but upstream when you create a grid with say, 4 images, you also specify the number of rows (which has a default) and you get 5 images back-- a "grid" image containing the four images arranged in, well, a grid, and the 4 individual images in the /samples folder contained within the outputs folder. So I think of the grid as more of a "preview" image, and if you want any image independently you can grab it from the /samples. It would be nice at one point to have the grid do smaller versions of the originals-- right now they are full-sized.

Does the grid image contain references to the names of the child images in /samples, and do the child images know about which grid image they belong to?

Per above answer, there are no such images. But if there were, the answer to both of these would be no.

Okay- sounds like someone took out the child images in the lstein fork... fwiw, if it's to be compatible with the upstream repository- might want the child images to look like any other generated individual image. Having an --n_iter of 1 and grid of trueI believe will give you BOTH a grid of one image and a child image in /samples, but don't hold me to that.

Timestamp, contact_info, image_name , Description, Citation, Licensing, NSFW

For all of this stuff, I don't think it belongs in this spec - this is a specification specifically for the metadata for images to tell you the settings used to generate the image. If you want to include other information alongside the image, put it somewhere else. E.g., wrap the data from this spec: so { contact_info: 'whatever', name: 'whatever', metdata: { [this spec] } }.

That way we don't have to keep a registry of additional optional fields, which in my experience never works, and all of the fields in this spec can be automatically derived, which is important.

Two more thoughts then, for possible expansion at a later time:

spec_version : int or string -- a way to identify version 1.0 of this spec to a version 10 years from now, for future backward compatibility
future : object -- a place to attach more data that might become important later or to be used for "whatever" someone wants to stick in there.

[bakkot]

Wouldn't it make more sense to name the model, version, and creator (the model_id from the card)?

That information simply isn't available for most users, is the problem. Users just download a weights file and use it. So the hash of the weights file is literally the only information we can use here.

But wouldn't a variant called lstein/stable-diffusion do exactly that?

Yes, like I said I'm fine with calling this field "variant" instead of "fork". If you are asking for some other difference from what I've proposed, I don't know what it is you're asking for.

(We can't use the commit because that information often isn't available; I'd include it if I could, and I think I probably will add an optional field to store that.)

spec_version

Interesting thought. I'm fine with it because I'm not worried about extra space, but if we're trying to minimize fields it's not strictly necessary - we could just say that any new versions will add a new field. (I'd call it metadata_version, though.)

future: object a place to attach more data that might become important later or to be used for "whatever" someone wants to stick in there

I actually just modified the spec to include an extra field! But it's explicitly not supposed to be used for arbitrary other data, only things necessary to reproduce the image; I do feel that data not related to that goal doesn't belong in this metadata and should live somewhere else.