Fixing the Java helm charts and adding a demo tutorial on how to use them

docs/SUMMARY.md

@@ -46,6 +46,7 @@
   * [Load data into the online store](how-to-guides/feast-snowflake-gcp-aws/load-data-into-the-online-store.md)
   * [Read features from the online store](how-to-guides/feast-snowflake-gcp-aws/read-features-from-the-online-store.md)
 * [Running Feast in production](how-to-guides/running-feast-in-production.md)
+* [Deploying a Java feature server on Kubernetes](how-to-guides/fetching-java-features-k8s.md)
 * [Upgrading from Feast 0.9](https://docs.google.com/document/u/1/d/1AOsr\_baczuARjCpmZgVd8mCqTF4AZ49OEyU4Cn-uTT0/edit)
 * [Adding a custom provider](how-to-guides/creating-a-custom-provider.md)
 * [Adding a new online store](how-to-guides/adding-support-for-a-new-online-store.md)

docs/how-to-guides/fetching-java-features-k8s.md

@@ -0,0 +1,15 @@
+# How to set up a Java feature server
+
+This tutorial guides you on how to:
+
+* Define features and data sources in Feast using the Feast CLI
+* Materialize features to a Redis cluster deployed on Kubernetes.
+* Deploy a Feast Java feature server into a Kubernetes cluster using the Feast helm charts
+* Retrieve features using the gRPC API exposed by the Feast Java server
+
+Try it and let us know what you think!
+
+| ![](../.gitbook/assets/github-mark-32px.png)[ View guide in Github](../../examples/java-demo/README.md) |
+|:--------------------------------------------------------------------------------------------------------|
+
+

examples/java-demo/README.md

@@ -0,0 +1,162 @@
+
+# Running Feast Java Server with Redis & calling with python (with registry in GCP)
+
+For this tutorial, we setup Feast with Redis, using the Feast CLI to register and materialize features, and then retrieving via a Feast Java server deployed in Kubernetes via a gRPC call.
+> :point_right: for tips on how to run and debug this locally without using Kubernetes, see [java/serving/README.md](https://github.com/feast-dev/feast/blob/master/java/serving/README.md)
+
+## First, let's setup a Redis cluster
+1.  Start minikube (`minikube start`)
+2.  Use helm to install a default Redis cluster
+    ```bash
+    helm repo add bitnami https://charts.bitnami.com/bitnami 
+    helm repo update 
+    helm install my-redis bitnami/redis
+    ```
+    ![](redis-screenshot.png)
+3. Port forward Redis so we can materialize features to it
+
+    ```bash
+    kubectl port-forward --namespace default svc/my-redis-master 6379:6379
+    ```
+4. Get your Redis password using the command (pasted below for convenience). We'll need this to tell Feast how to communicate with the cluster.
+
+   ```bash
+    export REDIS_PASSWORD=$(kubectl get secret --namespace default my-redis -o jsonpath="{.data.redis-password}" | base64 --decode)
+    echo $REDIS_PASSWORD
+    ```
+
+## Next, we setup a local Feast repo
+1. Install Feast with Redis dependencies `pip install "feast[redis]"`
+2. Make a bucket in GCS (or S3)
+3. The feature repo is already setup here, so you just need to swap in your GCS bucket and Redis credentials.
+    We need to modify the `feature_store.yaml`, which has two fields for you to replace:
+     ```yaml
+    registry: gs://[YOUR BUCKET]/demo-repo/registry.db
+    project: feast_java_demo
+    provider: gcp
+    online_store:
+      type: redis
+      connection_string: localhost:6379,password=[YOUR PASSWORD]
+    offline_store:
+      type: file
+    flags:
+      alpha_features: true
+      on_demand_transforms: true
+    ```
+4. Run `feast apply` to apply your local features to the remote registry
+5. Materialize features to the online store:
+    ```bash
+    CURRENT_TIME=$(date -u +"%Y-%m-%dT%H:%M:%S")                                    
+    feast materialize-incremental $CURRENT_TIME
+    ``` 
+
+## Now let's setup the Feast Server
+1. Add the gcp-auth addon to mount GCP credentials:
+    ```bash
+   minikube addons enable gcp-auth
+   ```
+3. Add Feast's Java feature server chart repo
+    ```bash
+    helm repo add feast-charts https://feast-helm-charts.storage.googleapis.com
+    helm repo update
+    ```
+4. Modify the application-override.yaml file to have your credentials + bucket location:
+    ```yaml
+    feature-server:
+      application-override.yaml:
+        enabled: true
+        feast:
+          activeStore: online
+          stores:
+            - name: online
+              type: REDIS
+              config:
+                host: my-redis-master
+                port: 6379
+                password: [YOUR PASSWORD]
+    global:
+      registry:
+        path: gs://[YOUR BUCKET]/demo-repo/registry.db
+        cache_ttl_seconds: 60
+      project: feast_java_demo
+    ```
+5. Install the Feast helm chart: `helm install feast-release feast-charts/feast --values application-override.yaml`
+6. (Optional): check logs of the server to make sure it’s working
+   ```bash
+   kubectl logs svc/feast-release-feature-server
+   ```
+7. Port forward to expose the grpc endpoint:
+   ```bash
+   kubectl port-forward svc/feast-release-feature-server 6566:6566
+   ```
+8. Make a gRPC call:
+    - Python example
+      ```bash
+        python test.py
+      ```
+    - gRPC cli:
+
+        ```bash
+        grpc_cli call localhost:6566 GetOnlineFeatures '
+        features {
+          val: "driver_hourly_stats:conv_rate"
+          val: "driver_hourly_stats:acc_rate"
+        }
+        entities {
+          key: "driver_id"
+          value {
+            val {
+              int64_val: 1001
+            }
+            val {
+              int64_val: 1002
+            }
+          }
+        }'
+        ```
+
+        -   Response:
+
+        ```bash
+        connecting to localhost:6566
+        metadata {
+          feature_names {
+            val: "driver_hourly_stats:conv_rate"
+            val: "driver_hourly_stats:acc_rate"
+          }
+        }
+        results {
+          values {
+            float_val: 0.812357187
+          }
+          values {
+            float_val: 0.379484832
+          }
+          statuses: PRESENT
+          statuses: PRESENT
+          event_timestamps {
+            seconds: 1631725200
+          }
+          event_timestamps {
+            seconds: 1631725200
+          }
+        }
+        results {
+          values {
+            float_val: 0.840873241
+          }
+          values {
+            float_val: 0.151376978
+          }
+          statuses: PRESENT
+          statuses: PRESENT
+          event_timestamps {
+            seconds: 1631725200
+          }
+          event_timestamps {
+            seconds: 1631725200
+          }
+        }
+        Rpc succeeded with OK status
+
+        ```

examples/java-demo/feature_repo/application-override.yaml

@@ -0,0 +1,17 @@
+feature-server:
+    application-override.yaml:
+        enabled: true
+        feast:
+            activeStore: online
+            stores:
+            - name: online
+              type: REDIS
+              config:
+                host: my-redis-master
+                port: 6379
+                password: [YOUR PASSWORD]
+global:
+  registry:
+    path: gs://[YOUR BUCKET]/demo-repo/registry.db
+    cache_ttl_seconds: 60
+  project: feast_java_demo

examples/java-demo/feature_repo/driver_repo.py

@@ -0,0 +1,61 @@
+import pandas as pd
+from feast import Entity, Feature, FeatureView, FileSource, ValueType
+from feast.data_source import RequestDataSource
+from feast.on_demand_feature_view import on_demand_feature_view
+from feast.request_feature_view import RequestFeatureView
+from google.protobuf.duration_pb2 import Duration
+
+driver_hourly_stats = FileSource(
+    path="data/driver_stats_with_string.parquet",
+    event_timestamp_column="event_timestamp",
+    created_timestamp_column="created",
+)
+driver = Entity(name="driver_id", value_type=ValueType.INT64, description="driver id",)
+driver_hourly_stats_view = FeatureView(
+    name="driver_hourly_stats",
+    entities=["driver_id"],
+    ttl=Duration(seconds=86400000),
+    features=[
+        Feature(name="conv_rate", dtype=ValueType.FLOAT),
+        Feature(name="acc_rate", dtype=ValueType.FLOAT),
+        Feature(name="avg_daily_trips", dtype=ValueType.INT64),
+        Feature(name="string_feature", dtype=ValueType.STRING),
+    ],
+    online=True,
+    batch_source=driver_hourly_stats,
+    tags={},
+)
+
+# Define a request data source which encodes features / information only
+# available at request time (e.g. part of the user initiated HTTP request)
+input_request = RequestDataSource(
+    name="vals_to_add",
+    schema={"val_to_add": ValueType.INT64, "val_to_add_2": ValueType.INT64},
+)
+
+# Define an on demand feature view which can generate new features based on
+# existing feature views and RequestDataSource features
+@on_demand_feature_view(
+    inputs={
+        "driver_hourly_stats": driver_hourly_stats_view,
+        "vals_to_add": input_request,
+    },
+    features=[
+        Feature(name="conv_rate_plus_val1", dtype=ValueType.DOUBLE),
+        Feature(name="conv_rate_plus_val2", dtype=ValueType.DOUBLE),
+    ],
+)
+def transformed_conv_rate(inputs: pd.DataFrame) -> pd.DataFrame:
+    df = pd.DataFrame()
+    df["conv_rate_plus_val1"] = inputs["conv_rate"] + inputs["val_to_add"]
+    df["conv_rate_plus_val2"] = inputs["conv_rate"] + inputs["val_to_add_2"]
+    return df
+
+
+# Define request feature view
+driver_age_request_fv = RequestFeatureView(
+    name="driver_age",
+    request_data_source=RequestDataSource(
+        name="driver_age", schema={"driver_age": ValueType.INT64,}
+    ),
+)

examples/java-demo/feature_repo/feature_store.yaml

@@ -0,0 +1,11 @@
+registry: gs://[YOUR BUCKET]/demo-repo/registry.db
+project: feast_java_demo
+provider: gcp
+online_store:
+  type: redis
+  connection_string: localhost:6379,password=[YOUR PASSWORD]
+offline_store:
+  type: file
+flags:
+  alpha_features: true
+  on_demand_transforms: true

examples/java-demo/feature_repo/test.py

@@ -0,0 +1,28 @@
+import grpc
+from feast.protos.feast.serving.ServingService_pb2 import (
+    FeatureList,
+    GetOnlineFeaturesRequest,
+)
+from feast.protos.feast.serving.ServingService_pb2_grpc import ServingServiceStub
+from feast.protos.feast.types.Value_pb2 import RepeatedValue, Value
+
+
+# Sample logic to fetch from a local gRPC java server deployed at 6566
+def fetch_java():
+    channel = grpc.insecure_channel("localhost:6566")
+    stub = ServingServiceStub(channel)
+    feature_refs = FeatureList(val=["driver_hourly_stats:conv_rate"])
+    entity_rows = {
+        "driver_id": RepeatedValue(
+            val=[Value(int64_val=driver_id) for driver_id in range(1001, 1003)]
+        )
+    }
+
+    print(
+        stub.GetOnlineFeatures(
+            GetOnlineFeaturesRequest(features=feature_refs, entities=entity_rows,)
+        )
+    )
+
+if __name__ == "__main__":
+    fetch_java()

infra/charts/feast/charts/feature-server/templates/configmap.yaml

@@ -14,26 +14,40 @@ data:
 {{- if index .Values "application-generated.yaml" "enabled" }}
     feast:
       registry: {{ .Values.global.registry.path }}
-      registry-refresh-interval: {{ .Values.global.registry.cache_ttl_seconds }}
+      registryRefreshInterval: {{ .Values.global.registry.cache_ttl_seconds }}
       {{- if .Values.transformationService.host }}
-      transformation-service-endpoint: {{ .Values.transformationService.host}}:{{ .Values.transformationService.port }}
+      transformationServiceEndpoint: {{ .Values.transformationService.host}}:{{ .Values.transformationService.port }}
       {{- else }}
-      transformation-service-endpoint: {{ .Release.Name }}-transformation-service:{{ .Values.transformationService.port }}
+      transformationServiceEndpoint: {{ .Release.Name }}-transformation-service:{{ .Values.transformationService.port }}
       {{- end }}
 
-      active_store: online
+      activeStore: online
       stores:
       - name: online
         type: REDIS 
         config:
           host: {{ .Release.Name }}-redis-master
           port: 6379
-
-    server:
-      port: {{ .Values.service.http.targetPort }}
+    rest:
+      server: 
+        port: {{ .Values.service.http.targetPort }}
+    grpc:
+      server:
+        port: {{ .Values.service.grpc.targetPort }}
 {{- end }}
 
   application-override.yaml: |
 {{- if index .Values "application-override.yaml" "enabled" }}
-{{- toYaml (index .Values "application-override.yaml") | nindent 4 }}    
+  {{- if index .Values "application-override.yaml" "feast" }}
+    feast: {{- toYaml (index .Values "application-override.yaml" "feast") | nindent 6 }}
+      registry: {{ .Values.global.registry.path }}
+      registryRefreshInterval: {{ .Values.global.registry.cache_ttl_seconds }}
+      project: {{ .Values.global.project }}
+  {{- end }}
+  {{- if index .Values "application-override.yaml" "rest" }}
+    rest: {{- toYaml (index .Values "application-override.yaml" "rest") | nindent 6 }}
+  {{- end }}
+  {{- if index .Values "application-override.yaml" "grpc" }}
+    grpc: {{- toYaml (index .Values "application-override.yaml" "grpc") | nindent 6 }}
+  {{- end }}
 {{- end }}

infra/charts/feast/charts/feature-server/templates/secret.yaml

@@ -12,4 +12,12 @@ metadata:
 type: Opaque
 stringData:
   application-secret.yaml: |
-{{- toYaml (index .Values "application-secret.yaml") | nindent 4  }}
+  {{- if index .Values "application-secret.yaml" "feast" }}
+    feast: {{- toYaml (index .Values "application-secret.yaml" "feast") | nindent 6 }}
+  {{- end }}
+  {{- if index .Values "application-secret.yaml" "rest" }}
+    rest: {{- toYaml (index .Values "application-secret.yaml" "rest") | nindent 6 }}
+  {{- end }}
+  {{- if index .Values "application-secret.yaml" "grpc" }}
+    grpc: {{- toYaml (index .Values "application-secret.yaml" "grpc") | nindent 6 }}
+  {{- end }}

infra/charts/feast/charts/feature-server/values.yaml

@@ -25,7 +25,7 @@ application-generated.yaml:
 
 # "application-secret.yaml" -- Configuration to override the default [application.yaml](https://github.com/feast-dev/feast/blob/master/java/serving/src/main/resources/application.yml). Will be created as a Secret. `application-override.yaml` has a higher precedence than `application-secret.yaml`. It is recommended to either set `application-override.yaml` or `application-secret.yaml` only to simplify config management.
 application-secret.yaml:
-  enabled: true
+  enabled: false
 
 # "application-override.yaml" -- Configuration to override the default [application.yaml](https://github.com/feast-dev/feast/blob/master/java/serving/src/main/resources/application.yml). Will be created as a ConfigMap. `application-override.yaml` has a higher precedence than `application-secret.yaml`
 application-override.yaml: