-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Closed
Labels
Description
Expected Behavior
Data written using Feast's SnowflakeOnlineStore.online_write_batch
should be persisted in the Snowflake table and be visible in subsequent reads, even across different connections.
Current Behavior
Data written by Feast is not persisted. It is sometimes visible in the same session (temporary reads succeed), but disappears after the connection is closed.
In Snowflake UI the table shows 0B
, and subsequent reads return empty results.
Steps to Reproduce
- Use a StreamFeatureView, which triggers the creation of a Table for that featureview.
- Use Feast (with
SnowflakeOnlineStore
) and Spark-Kafka Processor to write features viaonline_write_batch
. - Immediately read back data with
online_read
→ values are present. - After the connection/session is closed:
- The table shows
0B
in Snowflake UI. - Reads return no data (entity ids missing).
Specifications
- Version: Feast
main
branch / latest (please update with your Feast version) - Platform: Snowflake
- Subsystem: SnowflakeOnlineStore
Possible Solution
The issue is caused by a missing conn.commit()
in online_write_batch
.
In SnowflakeOnlineStore.online_write_batch
, Feast uses:
with GetSnowflakeConnection(config.online_store, autocommit=False) as conn:
write_pandas_binary(...)
execute_snowflake_statement(conn, query) # INSERT OVERWRITE
# Missing: conn.commit()
Since autocommit=False
, the transaction is not committed - and changes are discarded when the connection is closed.
Fix: Add an explicit conn.commit()
:
with GetSnowflakeConnection(config.online_store, autocommit=False) as conn:
write_pandas_binary(...)
execute_snowflake_statement(conn, query)
conn.commit() # required for persistence
Why This Happens
- Tests arent failing as during the same session, uncommitted data is visible (why initial reads worked) => integreation tests with new connection would be helpful.
- Once the connection closes, uncommitted transaction is discarded (why the table is empty and reads fail later)