@@ -31,13 +31,18 @@ Add the following configuration to the Spark application:
31
31
spark.hadoop.fs.hopsfs.impl io.hops.hopsfs.client.HopsFileSystem
32
32
spark.hadoop.hops.ipc.server.ssl.enabled true
33
33
spark.hadoop.hops.ssl.hostname.verifier ALLOW_ALL
34
- spark.hadoop.hops.rpc.socket.factory.class.default io.hop.hadoop.shaded.org.apache.hadoop.net.HopsSSLSocketFactory");
34
+ spark.hadoop.hops.rpc.socket.factory.class.default io.hop.hadoop.shaded.org.apache.hadoop.net.HopsSSLSocketFactory"
35
35
spark.hadoop.client.rpc.ssl.enabled.protocol TLSv1.2
36
36
spark.hadoop.hops.ssl.keystores.passwd.name material_passwd
37
37
spark.hadoop.hops.ssl.keystore.name keyStore.jks
38
38
spark.hadoop.hops.ssl.truststore.name trustStore.jks
39
+
40
+ spark.sql.hive.metastore.jars [Path to the Hopsworks Hive Jars]
41
+ spark.hadoop.hive.metastore.uris thrift://[metastore_ip]:[metastore_port]
39
42
```
40
43
44
+ ` spark.sql.hive.metastore.jars ` should point to the path with the Hive Jars which can be found in the * clients.tar.gz* .
45
+
41
46
## PySpark
42
47
43
48
To use PySpark, install the HSFS Python library which can be found on [ PyPi] ( https://pypi.org/project/hsfs/ ) .
@@ -76,16 +81,16 @@ In Hopsworks, click on your *username* in the top-right corner and select *Setti
76
81
77
82
## Connecting to the Feature Store
78
83
79
- You are now ready to connect to the Hopsworks Feature Store from SageMaker :
84
+ You are now ready to connect to the Hopsworks Feature Store from Spark :
80
85
81
86
``` python
82
87
import hsfs
83
88
conn = hsfs.connection(
84
- ' my_instance' , # DNS of your Feature Store instance
85
- 443 , # Port to reach your Hopsworks instance, defaults to 443
86
- ' my_project' , # Name of your Hopsworks Feature Store project
87
- api_key_file = ' featurestore.key ' , # The file containing the API key generated above
88
- hostname_verification = True ) # Disable for self-signed certificates
89
+ host = ' my_instance' , # DNS of your Feature Store instance
90
+ port = 443 , # Port to reach your Hopsworks instance, defaults to 443
91
+ project = ' my_project' , # Name of your Hopsworks Feature Store project
92
+ api_key_value = ' api_key ' , # The API key to authenticate with the feature store
93
+ hostname_verification = True # Disable for self-signed certificates
89
94
)
90
95
fs = conn.get_feature_store() # Get the project's default feature store
91
96
```
0 commit comments