Skip to content

Commit 23ada7c

Browse files
mateuszkuprowskimateuszkuprowski
authored
Fix databricks sdk version (#605)
<!-- CURSOR_SUMMARY --> > [!NOTE] > Pins/updates Databricks SDK and adjusts the Databricks volumes connector, with version bump, changelog, test fixtures, and lockfile updates. > > - **Databricks**: > - Update `unstructured_ingest/processes/connectors/databricks/volumes.py` implementation for the volumes connector. > - **Dependencies**: > - Pin/update Databricks SDK version in `pyproject.toml` and `requirements/connectors/databricks-volumes.txt`. > - Refresh dependency lockfile `uv.lock`. > - **Release**: > - Bump package version in `unstructured_ingest/__version__.py` and update `CHANGELOG.md`. > - **Tests**: > - Update Notion database expected-result fixtures (`test/integration/connectors/expected_results/notion_database/...`). > > <sup>Written by [Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit a4b4c2d. This will update automatically on new commits. Configure [here](https://cursor.com/dashboard?tab=bugbot).</sup> <!-- /CURSOR_SUMMARY -->
1 parent a72b1e4 commit 23ada7c

File tree

8 files changed

+3223
-3216
lines changed

8 files changed

+3223
-3216
lines changed

‎CHANGELOG.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,9 @@
1+
## [1.2.21]
2+
3+
* **fix: Enforce minimum version of databricks-sdk (>=0.62.0) for databricks-volumes connector**
4+
* **fix: Update databricks-volumes connector to use 'content' parameter (changed from 'contents' in SDK 0.70.0+) and wrap file in io.BytesIO for BinaryIO compatibility**
5+
* **fix: Add constraints to prevent platform compatibility issues with tritonclient/perf-analyzer dependencies**
6+
17
## [1.2.20]
28

39
* **fix: Prevent weaviate cloud precheck from passing with invalid config**

‎pyproject.toml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -165,6 +165,8 @@ ci = [
165165
"lancedb<=0.15.0",
166166
# TODO: versions higher than this are missing the macos wheel
167167
"pykx==2.5.3",
168+
# TODO: Constraint due to perf-analyzer platform compatibility issues
169+
"tritonclient<=2.60.0", # Allow 2.60.0 (was working), prevent 2.61.0 (has perf-analyzer issues)
168170
]
169171

170172
[project.scripts]

‎requirements/connectors/databricks-volumes.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
databricks-sdk
1+
databricks-sdk>=0.70.0

‎test/integration/connectors/expected_results/notion_database/downloads/1722c3765a0a8082b382ebc2c62d3f4c.html‎

Lines changed: 51 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -54,17 +54,17 @@
5454
</td>
5555
<td>
5656
<div>
57-
12
57+
4
5858
</div>
5959
</td>
6060
<td>
6161
<div>
62-
None-4
62+
None-1
6363
</div>
6464
</td>
6565
<td>
6666
<div>
67-
test-page4-in-database
67+
test-page1-in-datab
6868
</div>
6969
</td>
7070
<td>
@@ -79,24 +79,21 @@
7979
</td>
8080
<td>
8181
<div>
82-
2025-01-31
82+
2024-12-01
8383
</div>
8484
</td>
8585
<td>
86-
<div>
87-
Not started
86+
<div style='color: blue'>
87+
In progress
8888
</div>
8989
</td>
9090
<td>
9191
<div>
9292
<span style='color: purple'>
9393
V1
9494
</span>
95-
<span style='color: yellow'>
96-
V5
97-
</span>
98-
<span style='color: gray'>
99-
V7
95+
<span style='color: green'>
96+
V3
10097
</span>
10198
</div>
10299
</td>
@@ -107,7 +104,7 @@
107104
</td>
108105
<td>
109106
<div>
110-
6
107+
2
111108
</div>
112109
</td>
113110
</tr>
@@ -128,17 +125,17 @@
128125
</td>
129126
<td>
130127
<div>
131-
90
128+
46
132129
</div>
133130
</td>
134131
<td>
135132
<div>
136-
None-3
133+
None-2
137134
</div>
138135
</td>
139136
<td>
140137
<div>
141-
test-page3-in-database
138+
test-page2-in-database
142139
</div>
143140
</td>
144141
<td>
@@ -147,27 +144,30 @@
147144
</div>
148145
</td>
149146
<td>
150-
<div style='color: pink'>
151-
Medium
147+
<div style='color: blue'>
148+
Low
152149
</div>
153150
</td>
154151
<td>
155152
<div>
156-
2025-01-06
153+
2025-01-04
157154
</div>
158155
</td>
159156
<td>
160-
<div style='color: orange'>
161-
In Review
157+
<div style='color: green'>
158+
Done
162159
</div>
163160
</td>
164161
<td>
165162
<div>
166-
<span style='color: yellow'>
167-
V5
163+
<span style='color: purple'>
164+
V1
168165
</span>
169-
<span>
170-
V6
166+
<span style='color: orange'>
167+
V2
168+
</span>
169+
<span style='color: brown'>
170+
V4
171171
</span>
172172
</div>
173173
</td>
@@ -178,7 +178,7 @@
178178
</td>
179179
<td>
180180
<div>
181-
45
181+
23
182182
</div>
183183
</td>
184184
</tr>
@@ -199,17 +199,17 @@
199199
</td>
200200
<td>
201201
<div>
202-
46
202+
12
203203
</div>
204204
</td>
205205
<td>
206206
<div>
207-
None-2
207+
None-4
208208
</div>
209209
</td>
210210
<td>
211211
<div>
212-
test-page2-in-database
212+
test-page4-in-database
213213
</div>
214214
</td>
215215
<td>
@@ -218,30 +218,30 @@
218218
</div>
219219
</td>
220220
<td>
221-
<div style='color: blue'>
222-
Low
221+
<div style='color: gray'>
222+
High
223223
</div>
224224
</td>
225225
<td>
226226
<div>
227-
2025-01-04
227+
2025-01-31
228228
</div>
229229
</td>
230230
<td>
231-
<div style='color: green'>
232-
Done
231+
<div>
232+
Not started
233233
</div>
234234
</td>
235235
<td>
236236
<div>
237237
<span style='color: purple'>
238238
V1
239239
</span>
240-
<span style='color: orange'>
241-
V2
240+
<span style='color: yellow'>
241+
V5
242242
</span>
243-
<span style='color: brown'>
244-
V4
243+
<span style='color: gray'>
244+
V7
245245
</span>
246246
</div>
247247
</td>
@@ -252,7 +252,7 @@
252252
</td>
253253
<td>
254254
<div>
255-
23
255+
6
256256
</div>
257257
</td>
258258
</tr>
@@ -273,17 +273,17 @@
273273
</td>
274274
<td>
275275
<div>
276-
4
276+
90
277277
</div>
278278
</td>
279279
<td>
280280
<div>
281-
None-1
281+
None-3
282282
</div>
283283
</td>
284284
<td>
285285
<div>
286-
test-page1-in-datab
286+
test-page3-in-database
287287
</div>
288288
</td>
289289
<td>
@@ -292,27 +292,27 @@
292292
</div>
293293
</td>
294294
<td>
295-
<div style='color: gray'>
296-
High
295+
<div style='color: pink'>
296+
Medium
297297
</div>
298298
</td>
299299
<td>
300300
<div>
301-
2024-12-01
301+
2025-01-06
302302
</div>
303303
</td>
304304
<td>
305-
<div style='color: blue'>
306-
In progress
305+
<div style='color: orange'>
306+
In Review
307307
</div>
308308
</td>
309309
<td>
310310
<div>
311-
<span style='color: purple'>
312-
V1
311+
<span style='color: yellow'>
312+
V5
313313
</span>
314-
<span style='color: green'>
315-
V3
314+
<span>
315+
V6
316316
</span>
317317
</div>
318318
</td>
@@ -323,7 +323,7 @@
323323
</td>
324324
<td>
325325
<div>
326-
2
326+
45
327327
</div>
328328
</td>
329329
</tr>

‎test/integration/connectors/expected_results/notion_database/file_data/1722c3765a0a8082b382ebc2c62d3f4c.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@
1414
},
1515
"date_created": "2025-01-05T18:34:00.000Z",
1616
"date_modified": "2025-04-25T13:45:00.000Z",
17-
"date_processed": "1749468783.3651376",
17+
"date_processed": "1761568832.5411103",
1818
"permissions_data": null,
1919
"filesize_bytes": null
2020
},
@@ -34,6 +34,6 @@
3434
"url": "https://www.notion.so/1722c3765a0a8082b382ebc2c62d3f4c"
3535
},
3636
"reprocess": false,
37-
"local_download_path": "/tmp/tmp2eoxbctx/1722c3765a0a8082b382ebc2c62d3f4c.html",
37+
"local_download_path": "/tmp/tmp1_4nwizo/1722c3765a0a8082b382ebc2c62d3f4c.html",
3838
"display_name": "1722c3765a0a8082b382ebc2c62d3f4c.html"
3939
}

‎unstructured_ingest/__version__.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
__version__ = "1.2.20" # pragma: no cover
1+
__version__ = "1.2.21" # pragma: no cover

‎unstructured_ingest/processes/connectors/databricks/volumes.py‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import io
12
import os
23
from abc import ABC
34
from dataclasses import dataclass
@@ -220,9 +221,12 @@ def run(self, path: Path, file_data: FileData, **kwargs: Any) -> None:
220221
output_path = self.get_output_path(file_data=file_data)
221222
with open(path, "rb") as elements_file:
222223
try:
224+
# Read file bytes and wrap in BytesIO to create BinaryIO object
225+
file_bytes = elements_file.read()
226+
binary_data = io.BytesIO(file_bytes)
223227
self.connection_config.get_client().files.upload(
224228
file_path=output_path,
225-
contents=elements_file,
229+
content=binary_data, # Changed from 'contents' to 'content' in SDK 0.70.0+
226230
overwrite=True,
227231
)
228232
except Exception as e:

0 commit comments

Comments
 (0)