New integration test suite #3171

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

smatting merged 104 commits into develop from pcapriotti/new-integration

May 2, 2023

Contributor

smatting commented Mar 22, 2023 •

edited

Loading

See README.md

This PR introduces a new integration test suite that is meant to test accross multiple services
What is already included:

making requests to API, e.g. baseRequest, submit, bindResponse
working with JSON-like values as the main workhorse, e.g. %., setField
making assertions, e.g. assertBool, shouldMatch
Starting services with modified configuration withModifiedServices
Opening websockets and selecting events from it, e.g. awaitMatch

zebot added the ok-to-test label

smatting force-pushed the pcapriotti/new-integration branch from 5961ab7 to 18cddd5 Compare

March 22, 2023 14:36

smatting force-pushed the pcapriotti/new-integration branch 4 times, most recently from cffc4f4 to 29c450a Compare

April 18, 2023 13:27

smatting force-pushed the pcapriotti/new-integration branch from 97a2e92 to 68c4b8a Compare

April 21, 2023 13:45

smatting marked this pull request as ready for review

April 21, 2023 13:54

smatting requested a review from pcapriotti

April 21, 2023 13:55

fisx reviewed

View reviewed changes

integration/test/TestLib/App.hs Outdated Show resolved Hide resolved

integration/test/TestLib/App.hs Outdated Show resolved Hide resolved

integration/test/API.hs Outdated Show resolved Hide resolved

integration/test/API.hs Outdated Show resolved Hide resolved

pcapriotti and others added 20 commits

April 27, 2023 14:59


          Initial integration package skeleton

08890c9


          Initial http request machinery

e053064


          Add randomUser utility function

0bbf30c


          Show request and response on failure

6711b4e


          Add user creation to test

8f188f4


          Add stub for custom setup

a217b3c


          Create autogen directory in Setup.hs

c1685c2


          Collect all tests

810632f


          Parse haddock comments for tests

880c795


          AddClient api function

99c5899


          Pretty print JSON in requests and responses

5613acc


          Improve assertion failure output

f274fc0


          Use pattern matching instead of haskell-exts

13460ef


          Add some JSON convenience and improve response printing

c190a29


          use tasty

b80c11c


          Add missing newline in error msg

b77c623


          nix: Add "integration" package and docker image

676a0ec


          Fix failing test

2ddc84e


          Run integration test suite on CI

e66d41f


          Only run "integration" test suite on ci (undo this)

e24f579

smatting force-pushed the pcapriotti/new-integration branch from 115de56 to 3c3ca13 Compare

April 27, 2023 13:01


          hlint

022a443

fisx reviewed

View reviewed changes

integration/test/TestLib/App.hs Outdated Show resolved Hide resolved

integration/test/TestLib/App.hs Outdated Show resolved Hide resolved

integration/test/TestLib/App.hs Outdated Show resolved Hide resolved

integration/test/TestLib/App.hs Outdated Show resolved Hide resolved

integration/test/TestLib/App.hs Outdated Show resolved Hide resolved

charts/wire-server/requirements.yaml

@@ @@ -115,3 +115,8 @@ dependencies: @@
                 repository: "file://../sftd"
                 tags:
                   - sftd
+              - name: integration

Contributor

fisx Apr 28, 2023

probably a dumb question, but why is this not loaded in prod?

Contributor Author

smatting Apr 28, 2023

Good question! It actually would! Good catch. Fixed in ddbc197

Contributor Author

smatting Apr 28, 2023

ddbc197 caused

Error from server (NotFound): pods "test-la9l8u3xfu93-wire-server-integration-integration" not found

(see logs)

We need to double-check, but I think

  annotations:
    "helm.sh/hook": test

annotation in the Pod definition wouldn't deploy this pod in prod.

integration/test/Test/Brig.hs Outdated Show resolved Hide resolved

integration/README.md Outdated Show resolved Hide resolved

integration/README.md Outdated Show resolved Hide resolved

integration/test/Testlib/App.hs Outdated Show resolved Hide resolved

fisx and others added 5 commits

April 28, 2023 13:50


          Make adding new team roles nicer (?)

c3385a8


          Fix comment

60451d9


          Don't deploy integration with wire-server chart

ddbc197


          Fix typos

98d2adf


          Adapt test to #3252

3b69163

fisx approved these changes

View reviewed changes

integration/test/Testlib/App.hs

Comment on lines +93 to +112

+              getPrekey :: App Value
+              getPrekey = App $ do
+                pks <- asks (.prekeys)
+                (i, pk) <- atomicModifyIORef pks getPK
+                pure $ object ["id" .= i, "key" .= pk]
+                where
+                  getPK [] = error "Out of prekeys"
+                  getPK (k : ks) = (ks, k)
+              getLastPrekey :: App Value
+              getLastPrekey = App $ do
+                pks <- asks (.lastPrekeys)
+                lpk <- atomicModifyIORef pks getPK
+                pure $ object ["id" .= lastPrekeyId, "key" .= lpk]
+                where
+                  getPK [] = error "Out of prekeys"
+                  getPK (k : ks) = (ks, k)
+                  lastPrekeyId :: Int
+                  lastPrekeyId = 65535

Contributor

fisx Apr 28, 2023

these look a bit out of place here...

Contributor Author

smatting Apr 28, 2023

You mean because they are domain-specific? I think it's okay: They modify state in Env, so it makes sense they live in the module that defines the Env.

integration/test/Testlib/App.hs Outdated Show resolved Hide resolved

integration/test/Testlib/App.hs

Comment on lines +224 to +282

+              somePrekeys :: [String]
+              somePrekeys =
+                [ "pQABAQECoQBYIOjl7hw0D8YRNqkkBQETCxyr7/ywE/2R5RWcUPM+GJACA6EAoQBYILLf1TIwSB62q69Ojs/X1tzJ+dYHNAw4QbW/7TC5vSZqBPY=",
+                  "pQABAQICoQBYIGoXawUQWQ9ZW+MXhvuo9ALOBUjLff8S5VdAokN29C1OA6EAoQBYILLf1TIwSB62q69Ojs/X1tzJ+dYHNAw4QbW/7TC5vSZqBPY=",
+                  "pQABAQMCoQBYIEjdt+YWd3lHmG8pamULLMubAMZw556IO8kW7s1MLFytA6EAoQBYILLf1TIwSB62q69Ojs/X1tzJ+dYHNAw4QbW/7TC5vSZqBPY=",
+                  "pQABAQQCoQBYIPIaOA3Xqfk4Lh2/pU88Owd2eW5eplHpywr+Mx4QGyiMA6EAoQBYILLf1TIwSB62q69Ojs/X1tzJ+dYHNAw4QbW/7TC5vSZqBPY=",
+                  "pQABAQUCoQBYIHnafNR4Gh3ID71lYzToewEVag4EKskDFq+gaeraOlSJA6EAoQBYILLf1TIwSB62q69Ojs/X1tzJ+dYHNAw4QbW/7TC5vSZqBPY=",
+                  "pQABAQYCoQBYIFXUkVftE7kK22waAzhOjOmJVex3EBTU8RHZFx2o1Ed8A6EAoQBYILLf1TIwSB62q69Ojs/X1tzJ+dYHNAw4QbW/7TC5vSZqBPY=",
+                  "pQABAQcCoQBYIDXdN8VlKb5lbgPmoDPLPyqNIEyShG4oT/DlW0peRRZUA6EAoQBYILLf1TIwSB62q69Ojs/X1tzJ+dYHNAw4QbW/7TC5vSZqBPY=",
+                  "pQABAQgCoQBYIJH1ewvIVV3yGqQvdr/QM9HARzMgo5ksOTRyKEuN2aZzA6EAoQBYILLf1TIwSB62q69Ojs/X1tzJ+dYHNAw4QbW/7TC5vSZqBPY=",
+                  "pQABAQkCoQBYIFcAnXdx0M1Q1hoDDfgMK9r+Zchn8YlVHHaQwQYhRk1dA6EAoQBYILLf1TIwSB62q69Ojs/X1tzJ+dYHNAw4QbW/7TC5vSZqBPY=",
+                  "pQABAQoCoQBYIGs3vyxwmzEZ+qKNy4wpFkxc+Bgkb0D76ZEbxeeh/9DVA6EAoQBYILLf1TIwSB62q69Ojs/X1tzJ+dYHNAw4QbW/7TC5vSZqBPY=",
+                  "pQABAQsCoQBYIGUiBeOJALP5dkMduUZ/u6MDhHNrsrBUa3f0YlSSWZbzA6EAoQBYILLf1TIwSB62q69Ojs/X1tzJ+dYHNAw4QbW/7TC5vSZqBPY=",
+                  "pQABAQwCoQBYIMp6QNNTPDZgL3DSSD/QWWnBI7LsTZp2RhY/HLqnIwRZA6EAoQBYILLf1TIwSB62q69Ojs/X1tzJ+dYHNAw4QbW/7TC5vSZqBPY=",
+                  "pQABAQ0CoQBYIJXSSUrE5RCNyB5pg+m6vGwK7RvJ+rs9dsdHitxnfDhuA6EAoQBYILLf1TIwSB62q69Ojs/X1tzJ+dYHNAw4QbW/7TC5vSZqBPY=",
+                  "pQABAQ4CoQBYIHmtOX7jCKBHFDysb4H0z/QWoCSaEyjerZaT/HOP8bgDA6EAoQBYILLf1TIwSB62q69Ojs/X1tzJ+dYHNAw4QbW/7TC5vSZqBPY=",
+                  "pQABAQ8CoQBYIIaMCTcPKj2HuYQ7i9ZaxUw9j5Bz8TPjoAaTZ5eB0w1kA6EAoQBYILLf1TIwSB62q69Ojs/X1tzJ+dYHNAw4QbW/7TC5vSZqBPY=",
+                  "pQABARACoQBYIHWAOacKuWH81moJVveJ0FSfipWocfspOIBhaU6VLWUsA6EAoQBYILLf1TIwSB62q69Ojs/X1tzJ+dYHNAw4QbW/7TC5vSZqBPY=",
+                  "pQABARECoQBYIA8XtUXtnMxQslULnNAeHBIivlLRe/+qdh2j6nTfDAchA6EAoQBYILLf1TIwSB62q69Ojs/X1tzJ+dYHNAw4QbW/7TC5vSZqBPY=",
+                  "pQABARICoQBYIGgzg6SzgTTOgnk48pa6y2Rgjy004DkeBo4CMld3Jlr6A6EAoQBYILLf1TIwSB62q69Ojs/X1tzJ+dYHNAw4QbW/7TC5vSZqBPY=",
+                  "pQABARMCoQBYIEoEFiIpCHgn74CAD+GhIfIgbQtdCqQqkOXHWxRlG6Y6A6EAoQBYILLf1TIwSB62q69Ojs/X1tzJ+dYHNAw4QbW/7TC5vSZqBPY=",
+                  "pQABARQCoQBYINVEwTRxNSe0rxZxon4Rifz2l4rtQZn7mHtKYCiFAK9IA6EAoQBYILLf1TIwSB62q69Ojs/X1tzJ+dYHNAw4QbW/7TC5vSZqBPY=",
+                  "pQABARUCoQBYIN3aeX2Ayi2rPFbiaYb+O2rdHUpFhzRs2j28pCmbGpflA6EAoQBYILLf1TIwSB62q69Ojs/X1tzJ+dYHNAw4QbW/7TC5vSZqBPY=",
+                  "pQABARYCoQBYIJe5OJ17YKQrNmIH3sE++r++4Z5ld36axqAMjjQ3jtQWA6EAoQBYILLf1TIwSB62q69Ojs/X1tzJ+dYHNAw4QbW/7TC5vSZqBPY=",
+                  "pQABARcCoQBYIASE94LjK6Raipk/lN/YewouqO+kcQGpxIqP+iW2hyHiA6EAoQBYILLf1TIwSB62q69Ojs/X1tzJ+dYHNAw4QbW/7TC5vSZqBPY=",
+                  "pQABARgYAqEAWCBZ222LpS6/99Btlw+83PihrA655skwsNevt//8oz5axQOhAKEAWCCy39UyMEgetquvTo7P19bcyfnWBzQMOEG1v+0wub0magT2",
+                  "pQABARgZAqEAWCDGEwo61w4O8T8lyw0HdoOjGWBKQUNqo6+jSfrPR9alrAOhAKEAWCCy39UyMEgetquvTo7P19bcyfnWBzQMOEG1v+0wub0magT2",
+                  "pQABARgaAqEAWCBMSQoQ6B35plB80i1O3AWlJSftCEbCbju97Iykg5+NWQOhAKEAWCCy39UyMEgetquvTo7P19bcyfnWBzQMOEG1v+0wub0magT2"
+                ]
+              someLastPrekeys :: [String]
+              someLastPrekeys =
+                [ "pQABARn//wKhAFggnCcZIK1pbtlJf4wRQ44h4w7/sfSgj5oWXMQaUGYAJ/sDoQChAFgglacihnqg/YQJHkuHNFU7QD6Pb3KN4FnubaCF2EVOgRkE9g==",
+                  "pQABARn//wKhAFggwO2any+CjiGP8XFYrY67zHPvLgp+ysY5k7vci57aaLwDoQChAFgglacihnqg/YQJHkuHNFU7QD6Pb3KN4FnubaCF2EVOgRkE9g==",
+                  "pQABARn//wKhAFggoChErA5oTI5JT769hJV+VINmU8kougGdYqGd2U7hPa8DoQChAFgglacihnqg/YQJHkuHNFU7QD6Pb3KN4FnubaCF2EVOgRkE9g==",
+                  "pQABARn//wKhAFggPLk4BBJ8THVLGm7r0K7EJITRlJnt6bpNzM9GTNRYcCcDoQChAFgglacihnqg/YQJHkuHNFU7QD6Pb3KN4FnubaCF2EVOgRkE9g==",
+                  "pQABARn//wKhAFggqHASsRlZ1i8dESXRXBL2OvR+0yGUtqK9vJfzol1E+osDoQChAFgglacihnqg/YQJHkuHNFU7QD6Pb3KN4FnubaCF2EVOgRkE9g==",
+                  "pQABARn//wKhAFggx/N1YhKXSJYJQxhWgHSA4ASaJKIHDJfmEnojfnp9VQ8DoQChAFgglacihnqg/YQJHkuHNFU7QD6Pb3KN4FnubaCF2EVOgRkE9g==",
+                  "pQABARn//wKhAFggVL6QIpoqmtKxmB8HToiAPxfjSDEzJEUAoFKfhXou06YDoQChAFgglacihnqg/YQJHkuHNFU7QD6Pb3KN4FnubaCF2EVOgRkE9g==",
+                  "pQABARn//wKhAFggRs74/ViOrHN+aS2RbGCwC0sJv1Sp/Q0pmRB15s9DCBMDoQChAFgglacihnqg/YQJHkuHNFU7QD6Pb3KN4FnubaCF2EVOgRkE9g==",
+                  "pQABARn//wKhAFggtNO/hrwzt9M/1X6eK2sG6YFmA7BDqlFMEipbZOsg0vcDoQChAFgglacihnqg/YQJHkuHNFU7QD6Pb3KN4FnubaCF2EVOgRkE9g==",
+                  "pQABARn//wKhAFgg1rZEY6vbAnEz+Ern5kRny/uKiIrXTb/usQxGnceV2HADoQChAFgglacihnqg/YQJHkuHNFU7QD6Pb3KN4FnubaCF2EVOgRkE9g==",
+                  "pQABARn//wKhAFgg2647mOAVeOdhW57Q1zXDigDxRz/hB8ITFSZ7uo+pXH4DoQChAFgglacihnqg/YQJHkuHNFU7QD6Pb3KN4FnubaCF2EVOgRkE9g==",
+                  "pQABARn//wKhAFggjddbHizABYOY0T6rvJeZCvV20dvTT9BYv95ri9bqSb8DoQChAFgglacihnqg/YQJHkuHNFU7QD6Pb3KN4FnubaCF2EVOgRkE9g==",
+                  "pQABARn//wKhAFggCKT/GspZquUY6vKC4TFvaFqTH1QGG1ptauiaulnfqkUDoQChAFgglacihnqg/YQJHkuHNFU7QD6Pb3KN4FnubaCF2EVOgRkE9g==",
+                  "pQABARn//wKhAFggv7bf/kEsTKFDGSgswsywq6AIxBq5AqZbLjDYDHfGjrcDoQChAFgglacihnqg/YQJHkuHNFU7QD6Pb3KN4FnubaCF2EVOgRkE9g==",
+                  "pQABARn//wKhAFggUbjGhhh8EwZEPSz+Y31rYNUu7jsRR8dy1F5FSiJXfXEDoQChAFgglacihnqg/YQJHkuHNFU7QD6Pb3KN4FnubaCF2EVOgRkE9g==",
+                  "pQABARn//wKhAFgg/4nz1uHiPBVGFvYjTMwGQ31bSFNctbU0r2nBtpsK9kcDoQChAFgglacihnqg/YQJHkuHNFU7QD6Pb3KN4FnubaCF2EVOgRkE9g==",
+                  "pQABARn//wKhAFggwbJDyKl7T3+3Ihc0YF06Dz2J11My5qn7JKG+U+ti8lQDoQChAFgglacihnqg/YQJHkuHNFU7QD6Pb3KN4FnubaCF2EVOgRkE9g==",
+                  "pQABARn//wKhAFgglc6nCoZR2/qjLp0tr7vRyuXqb7ugdHHDadjX7zSl4uMDoQChAFgglacihnqg/YQJHkuHNFU7QD6Pb3KN4FnubaCF2EVOgRkE9g==",
+                  "pQABARn//wKhAFgg5ER8h0/bIADXjBXe/XPKdzekgv6nhJ4hp3vJ3jtTSbUDoQChAFgglacihnqg/YQJHkuHNFU7QD6Pb3KN4FnubaCF2EVOgRkE9g==",
+                  "pQABARn//wKhAFggsgV6jq+GuNuvXk+ctHh570cNqEmfPhz34wcYCMCf9xIDoQChAFgglacihnqg/YQJHkuHNFU7QD6Pb3KN4FnubaCF2EVOgRkE9g==",
+                  "pQABARn//wKhAFggdQdlPqkBw6+phKhohp3YaWQL710euZDnyMLFwf2cS0oDoQChAFgglacihnqg/YQJHkuHNFU7QD6Pb3KN4FnubaCF2EVOgRkE9g==",
+                  "pQABARn//wKhAFggKlsI/snuQMoYcZRw/kN+BobPV5gwYeBClp0Wx9btTGUDoQChAFgglacihnqg/YQJHkuHNFU7QD6Pb3KN4FnubaCF2EVOgRkE9g==",
+                  "pQABARn//wKhAFggtruFBClEgdPKvjpHsYLlWMev9L4OmYZwlxbY0NwvzOwDoQChAFgglacihnqg/YQJHkuHNFU7QD6Pb3KN4FnubaCF2EVOgRkE9g==",
+                  "pQABARn//wKhAFggRUdh4cuYtFNL46RLnPy65goYInyreStKwsEcY3pPlLkDoQChAFgglacihnqg/YQJHkuHNFU7QD6Pb3KN4FnubaCF2EVOgRkE9g==",
+                  "pQABARn//wKhAFggQtT7lLZzH171F4jCbHNwxEAt28FwdQ8Kt2tbxFzPgC0DoQChAFgglacihnqg/YQJHkuHNFU7QD6Pb3KN4FnubaCF2EVOgRkE9g==",
+                  "pQABARn//wKhAFggQeUPM119c+6zRsEupA8zshTfrZiLpXx1Ji0UMMumq9IDoQChAFgglacihnqg/YQJHkuHNFU7QD6Pb3KN4FnubaCF2EVOgRkE9g=="
+                ]

Contributor

fisx Apr 28, 2023

also looks out of place.

Contributor Author

smatting Apr 28, 2023

Same argument as for getLastPrekey. Could move into own module to have less visual noise, though

integration/test/Testlib/App.hs Outdated Show resolved Hide resolved

integration/test/Testlib/App.hs Outdated Show resolved Hide resolved

integration/test/Testlib/App.hs Show resolved Hide resolved

integration/test/Testlib/App.hs Outdated

+                assertFailure msg'
+              -- | Useful for debugging
+              printJSON :: ProducesJSON a => a -> App ()

Contributor

fisx Apr 28, 2023

is this still needed given the test output should always do this for you? i'd prefer to remove this.

Contributor Author

smatting Apr 28, 2023

This comes in handy when you want to print json values for debugging, e.g. printing json responses, etc. I think it's useful.

smatting added 17 commits

April 28, 2023 14:44


          Revert "Don't deploy integration with wire-server chart"

a45be60

This reverts commit ddbc197.


          Fix typos and reformulated README from code review

728cbdb


          Remove unused instance

41c3227


          Refactor shouldMatchInt and add comment

a802ca4


          Add docs to (%.) and lookupField

ee1ae94


          Rename ProducesJSON to MakesValue

da4a4ca


          remove --height - errors are all the way up there

838d2fe

wip

2acdab8


          Make stacktraces nice

974f799


          minor thing

ca186d0


          Add viewFederationDomain

7e07695


          Improve Makefile targets


          Add -m to select tests more easily

fd62f54


          linter

7040fc0


          Make TASTY_MATCH work, and also accept comma-separated list

087e6bf


          Inline context into Env

7beea28


          Update "ci" Makefile target

6180de6

smatting merged commit 9013e73 into develop

smatting deleted the pcapriotti/new-integration branch

May 2, 2023 10:05

supersven pushed a commit that referenced this pull request


          New integration test suite (#3171)

9a76dcc

smatting mentioned this pull request

wire-server chart: Disable integration chart by default #3682

Merged

2 tasks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels