Skip to content

ARROW-252: Add implementation guidelines to the documentation #120

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
julienledem wants to merge 2 commits into from
Closed

ARROW-252: Add implementation guidelines to the documentation #120

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6b68ce1
ARROW-252: Add implementation guidelines to the documentation
julienledem Aug 16, 2016
caf6994
ARROW-252: review feedback
julienledem Aug 20, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions format/Guidelines.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Implementation guidelines

An execution engine (or framework, or UDF executor, or storage engine, etc) can implements only a subset of the Arrow spec and/or extend it given the following constraints:

## Implementing a subset the spec
### If only producing (and not consuming) arrow vectors.
Any subset of the vector spec and the corresponding metadata can be implemented.

### If consuming and producing vectors
There is a minimal subset of vectors to be supported.
Production of a subset of vectors and their corresponding metadata is always fine.
Consumption of vectors should at least convert the unsupported input vectors to the supported subset (for example Timestamp.millis to timestamp.micros or int32 to int64)

## Extensibility
An execution engine implementor can also extend their memory representation with their own vectors internally as long as they are never exposed. Before sending data to another system expecting Arrow data these custom vectors should be converted to a type that exist in the Arrow spec.
Copy link
Contributor

@emkornfield emkornfield Aug 18, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mention whether custom metadata are in/out of scope for V1 (or ever). I would initially vote for out of scope to keep things simple.

Copy link
Member Author

@julienledem julienledem Aug 20, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

An example of this is operating on compressed data.
These custom vectors are not exchanged externaly and there is no support for custom metadata.