-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Use unzip.Parse
over unzip.Extract
#1666
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
bc5b3a8
81d5e48
b956d8a
9dea373
31c555a
a24b9c0
83731e6
1e326de
d3301c9
8d03fb4
e9005f7
76489f4
4256ea9
ac84a9b
614f27a
90894a8
8a1800c
f77cbc9
7fa864a
6cf4fbc
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,7 @@ | ||
import fs from 'fs/promises' | ||
import * as stream from 'stream' | ||
import {createWriteStream} from 'fs' | ||
import * as path from 'path' | ||
import * as github from '@actions/github' | ||
import * as core from '@actions/core' | ||
import * as httpClient from '@actions/http-client' | ||
|
@@ -44,6 +47,11 @@ async function streamExtract(url: string, directory: string): Promise<void> { | |
await streamExtractExternal(url, directory) | ||
return | ||
} catch (error) { | ||
if (error.message.includes('Malformed extraction path')) { | ||
throw new Error( | ||
`Artifact download failed with unretryable error: ${error.message}` | ||
) | ||
} | ||
retryCount++ | ||
core.debug( | ||
`Failed to download artifact after ${retryCount} retries due to ${error.message}. Retrying in 5 seconds...` | ||
|
@@ -78,6 +86,8 @@ export async function streamExtractExternal( | |
} | ||
const timer = setTimeout(timerFn, timeout) | ||
|
||
const createdDirectories = new Set<string>() | ||
createdDirectories.add(directory) | ||
response.message | ||
.on('data', () => { | ||
timer.refresh() | ||
|
@@ -89,8 +99,51 @@ export async function streamExtractExternal( | |
clearTimeout(timer) | ||
reject(error) | ||
}) | ||
.pipe(unzip.Extract({path: directory})) | ||
.on('close', () => { | ||
.pipe(unzip.Parse()) | ||
.pipe( | ||
new stream.Transform({ | ||
objectMode: true, | ||
transform: async (entry, _, callback) => { | ||
const fullPath = path.normalize(path.join(directory, entry.path)) | ||
if (!directory.endsWith(path.sep)) { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It is important for the comparison below. Otherwise it will fail with, for example, "/tmp/dest/file".startsWith("../dest/") There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We decided not to do the path.resolve above since this library could take relative paths! There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That was definitely failing when we switched to resolve temporarily. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ah, OK, |
||
directory += path.sep | ||
} | ||
if (!fullPath.startsWith(directory)) { | ||
reject(new Error(`Malformed extraction path: ${fullPath}`)) | ||
} | ||
|
||
core.debug(`Extracting artifact entry: ${fullPath}`) | ||
if (entry.type === 'Directory') { | ||
if (!createdDirectories.has(fullPath)) { | ||
createdDirectories.add(fullPath) | ||
await resolveOrCreateDirectory(fullPath).then(() => { | ||
entry.autodrain() | ||
callback() | ||
}) | ||
} else { | ||
entry.autodrain() | ||
callback() | ||
} | ||
} else { | ||
if (!createdDirectories.has(path.dirname(fullPath))) { | ||
createdDirectories.add(path.dirname(fullPath)) | ||
await resolveOrCreateDirectory(path.dirname(fullPath)).then( | ||
() => { | ||
entry.autodrain() | ||
callback() | ||
} | ||
) | ||
} | ||
|
||
const writeStream = createWriteStream(fullPath) | ||
writeStream.on('finish', callback) | ||
writeStream.on('error', reject) | ||
entry.pipe(writeStream) | ||
} | ||
} | ||
}) | ||
) | ||
.on('finish', async () => { | ||
clearTimeout(timer) | ||
resolve() | ||
}) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
const fullPath = path.resolve(directory, entry.path)
will
path.resolve()
be an better option, assumeentry.path
should be a relative path?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That makes sense to me! I think
download-artifact
ensures that the path is absolute, but it doesn't look like toolkit does any validation.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, doing that throws off the check below for the directory path 😞
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would you mind if I changed it back? On second thought, it might be better to keep this generic and not assume path for anyone utilizing this package. For download-artifact, we have the absolute path: https://github.com/actions/download-artifact/blob/348754975ef0295bfa2c111cba996120cfdf8a5d/src/download-artifact.ts#L38
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i thought
directory
is absolute path, looks like it can be relative...There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Directory is absolute when it comes from
download-artifact
, but isn't guaranteed absolute from other sources.