Re-order extraction metadata union for better parsing #865
Conversation
adrianlyjak
force-pushed
the
adrian/fix-lost-dimension
branch
from
0f33e62 to
1dd0eba
Compare
| @@ -203,7 +203,7 @@ class ExtractedFieldMetadata(BaseModel): | |||
|
|
|||
|
|
|||
| ExtractedFieldMetaDataDict = Dict[ | |||
| str, Union[ExtractedFieldMetadata, Dict[str, Any], list[Any]] | |||
| str, Union[Dict[str, Any], ExtractedFieldMetadata, list[Any]] | |||
There was a problem hiding this comment.
this is the fix 🤦
There was a problem hiding this comment.
I am not sure. I think it iterates through the union parsing the first type that matches, and since ExtractedFieldMetadata is all optional, it will match any dict. However this can't be the full explanation, otherwise the ExtractedFieldMetadata values would be parsed to Dict, which isn't happening
There was a problem hiding this comment.
I suppose after this change, ExtractedFieldMetadata will never be hit, no?
There was a problem hiding this comment.
I suppose after this change,
ExtractedFieldMetadatawill never be hit, no?
@zhaotai you make a good point. Looks like parsing from json has the behavior where ExtractedFieldMetadata wouldn't ever parse (whereas whatever the normalization that happens in the ExtractedData constructor kept the classes)
Modified this so that ExtractedFieldMetadata instead only parses if there are no extra fields, which seems more robust.
| data = json.load(f) | ||
| result = ExtractedData.from_extraction_result(ExtractRun.parse_obj(data), Capacitor) | ||
| assert result.field_metadata == { | ||
| "dimensions": { |
There was a problem hiding this comment.
pydantic was converting this to "dimensions": ExtractedFieldMetadata(None, None, None, None)
| @@ -203,7 +203,7 @@ class ExtractedFieldMetadata(BaseModel): | |||
|
|
|||
|
|
|||
| ExtractedFieldMetaDataDict = Dict[ | |||
| str, Union[ExtractedFieldMetadata, Dict[str, Any], list[Any]] | |||
| str, Union[Dict[str, Any], ExtractedFieldMetadata, list[Any]] | |||
adrianlyjak
force-pushed
the
adrian/fix-lost-dimension
branch
from
08b6315 to
d63d610
Compare
adrianlyjak
force-pushed
the
adrian/fix-lost-dimension
branch
from
0912614 to
625cc82
Compare
pydantic was parsing nested dicts to an empty
ExtractedFieldMetadata, because that was the first value in the union