Skip to content

Debug info ci failures #3400

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jschaul merged 9 commits into from
Jul 6, 2023
Merged

Debug info ci failures #3400

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
bd0b6aa
add bash logic to find more context of why things go wrong
jschaul Jul 5, 2023
4ab003c
fail to schedule rabbit; and fail to start brig to see if the logs ar…
jschaul Jul 5, 2023
5f7a88c
changelog
jschaul Jul 5, 2023
002d49f
fixup
jschaul Jul 5, 2023
39a28f0
PR feedback
jschaul Jul 5, 2023
8276754
take bash from env
jschaul Jul 5, 2023
a7db990
undo one on-purpose-unschedulable bug
jschaul Jul 6, 2023
62762f2
no need for context before Events:
jschaul Jul 6, 2023
9621754
undo brig misconfiguration
jschaul Jul 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/5-internal/integration-test-debug-logs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
On CI runs, provide additional context when 'helmfile install' fails.
13 changes: 13 additions & 0 deletions hack/bin/integration-setup-federation.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,20 @@ export FEDERATION_DOMAIN_2="federation-test-helper.$FEDERATION_DOMAIN_BASE"

echo "Installing charts..."

set +e
helmfile --environment "$HELMFILE_ENV" --file "${TOP_LEVEL}/hack/helmfile.yaml" sync --skip-deps --concurrency 0
EXIT_CODE=$?

if (( EXIT_CODE > 0)); then
echo "!! Helm install failed. Attempting to get some more information ..."

kubectl -n "$NAMESPACE_1" get events | grep -v "Normal "
kubectl -n "$NAMESPACE_2" get events | grep -v "Normal "
Comment on lines +59 to +60
Copy link
Member

@akshaymankar akshaymankar Jul 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Grepping out Normal might create red herrings because I guess we won't know if things got better. How much noise does this save?

Copy link
Member Author

@jschaul jschaul Jul 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A lot of noise. It's 95% normal events. The events only show if helm install fails - and then these warnings/errors are likely indicative of a problem, even if sometimes unrelated to the failure, I suppose often there's something to it.

Copy link
Member

@akshaymankar akshaymankar Jul 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense :shipit:

"${DIR}/kubectl-get-debug-info.sh" "$NAMESPACE_1"
"${DIR}/kubectl-get-debug-info.sh" "$NAMESPACE_2"
exit $EXIT_CODE
fi
set -e

# wait for fakeSNS to create resources. TODO, cleaner: make initiate-fake-aws-sns a post hook. See cassandra-migrations chart for an example.
resourcesReady() {
Expand Down
29 changes: 29 additions & 0 deletions hack/bin/kubectl-get-debug-info.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
#!/usr/bin/env bash

USAGE="$0 <NAMESPACE>"
NAMESPACE=${1:?$USAGE}

echo "Checking pods in namespace '${NAMESPACE}' that failed to schedule..."

# Get pods that failed to schedule
UNSCHEDULED_PODS=$(kubectl get pods --namespace "$NAMESPACE" -o json | jq -r '.items[] | select(.status.phase=="Pending") | .metadata.name')

for POD in $UNSCHEDULED_PODS; do
echo "Pod $POD failed to schedule for the following reasons:"
# Get events for pod
kubectl describe pod "$POD" --namespace "$NAMESPACE" | grep -A 10 "Events:"
echo ""
done

echo "Checking pods in namespace '${NAMESPACE}' that are crashlooping..."

# Get pods that are crashlooping
CRASHLOOPING_PODS=$(kubectl get pods --namespace "$NAMESPACE" -o json | jq -r '.items[] | select(.status.containerStatuses[]?.state.waiting.reason=="CrashLoopBackOff") | .metadata.name')

for POD in $CRASHLOOPING_PODS; do
echo "Pod $POD is crashlooping for the following reasons:"
# Get logs of previous run for pod
kubectl logs "$POD" --namespace "$NAMESPACE" --previous
echo ""
done