Mutiple joex containers hava same app-id in 'job' table 'worker' column.

[ohagene]

Hi, I started mutiple joex containers to increase processing speed. I have changed the app-id in docker-compose file, and the the joex startup log showed that the app-id had been changed, but 'job' table's 'worker' column are all 'joex1', and web-ui processing queue logs are also all 'joex1'. What was wrong?
docker-compose file :

  joex1:
    image: docspell/joex:latest 
    container_name: docspell-joex1
    restart: unless-stopped
    environment:
      - DOCSPELL_JOEX_APP__ID=joex1
      - DOCSPELL_JOEX_SCHEDULER_POOL__SIZE=2
      - DOCSPELL_JOEX_BASE__URL=http://docspell-joex1:7878
      - DOCSPELL_JOEX_BIND_ADDRESS=0.0.0.0
      -  ...

  joex2:
    image: docspell/joex:latest 
    container_name: docspell-joex2
    restart: unless-stopped
    environment:
      - DOCSPELL_JOEX_APP__ID=joex2
      - DOCSPELL_JOEX_SCHEDULER_POOL__SIZE=2
      - DOCSPELL_JOEX_BASE__URL=http://docspell-joex2:7878
      - DOCSPELL_JOEX_BIND_ADDRESS=0.0.0.0
      -  ...

  joex3:
    image: docspell/joex:latest 
    container_name: docspell-joex3
    restart: unless-stopped
    environment:
      - DOCSPELL_JOEX_APP__ID=joex3
      - DOCSPELL_JOEX_SCHEDULER_POOL__SIZE=2
      - DOCSPELL_JOEX_BASE__URL=http://docspell-joex3:7878
      - DOCSPELL_JOEX_BIND_ADDRESS=0.0.0.0
      -  ...

joex6 start up logs:

docspell-joex6  | Starting unoconv listener
docspell-joex6  | May 16, 2022 10:35:14 AM org.jline.utils.Log logr
docspell-joex6  | WARNING: Unable to create a system terminal, creating a dumb terminal (enable debug logging for more information)
docspell-joex6  | 2022.05.16 10:35:14:563 io-compute-29 INFO docspell.config.ConfigFactory.default:44
docspell-joex6  |     Using config from environment variables!
docspell-joex6  | 2022.05.16 10:35:15:0000 [io-comp...] [INFO ] docspell.joex.Main.run:37 -
docspell-joex6  | ***>  ______                          _ _
docspell-joex6  | ***>  |  _  \                        | | |
docspell-joex6  | ***>  | | | |___   ___ ___ _ __   ___| | |
docspell-joex6  | ***>  | | | / _ \ / __/ __| '_ \ / _ \ | |
docspell-joex6  | ***>  | |/ / (_) | (__\__ \ |_) |  __/ | |
docspell-joex6  | ***>  |___/ \___/ \___|___/ .__/ \___|_|_|
docspell-joex6  | ***>                      | |
docspell-joex6  | ***>                      |_| v0.34.0 (#c6da3eb0)
docspell-joex6  | ***>  << JOEX >>
docspell-joex6  | ***>  Id:       joex6
docspell-joex6  | ***>  Base-Url: http://docspell-joex6:7878
docspell-joex6  | ***>  Database: jdbc:postgresql://xxxx:5432/docspell
docspell-joex6  | ***>  Fts:      PostgreSQL(default)
docspell-joex6  | ***>  Config:
docspell-joex6  | ***>  FileRepo: DefaultDatabase(true)

[eikek]

Can you look whats in the table pubsub (e.g. select distinct node_id from pubsub)? Are all nodes listed there? Is it that jobs are in "waiting" state although there are joex containers that could pick it up?

[ohagene]

@eikek Hi. The table pubsub is correct, all nodes are listed there. And lots of jobs are in "waiting" state.
After restarting docspell, I just docker-compose up restserver and joex4. joex4 is picking up jobs and working. But both web-ui processing queue logs and table job's worker column are joex1. Any additional suggestions?

[eikek]

Hi @ohagene, not really new ideas. It is most likely some regression. I have to look into it. When the job table has everywhere joex1, then all others can't get jobs, because they are already owned by another joex. It could be a bug where one joex takes all jobs, perhaps. When joex4 is picking up jobs but everywhere it is marked as joex1, then there is something really fishy.

One other thing came to mind: if a job is submitted then all joex nodes get notified and then race for the next job in the queue. It could be that this notification always reaches only one of the joex nodes…. This would explain that other are idling.

You could try from a "clean" state like this:

1. stop all joex nodes,
2. "clear" the job table's worker column for all waiting jobs (update job set worker = null where state = 'waiting')
3. start one joex: it should take so many jobs as its pool allows - only these should have the worker set
4. if that seems fine, start another joex, this should now also take some jobs etc. but no one should mark all

[ohagene]

Hi @eikek, Sorry, perhaps I didn't make it clear. waiting state jobs' worker is 'null', joex4 picked up jobs successfully, and have done job well, but when the job is in running and success state, the worker is marked as joex1, although in fact joex4 did the job.
I have tried again in an "clean" state.

1. docker-compose up -d restserver joex4.
2. upload a file.
3. joex4 picking up the file process job, after a while, job succeed. But everywhere it is logged as joex1 have done the job :(

[eikek]

Thank you @ohagene - I take a look, this is obviously a bug. Currently I don't have an explanation.

[eikek]

I cannot reproduce it with the tools/start-local/start-local.sh script (used version 0.35.0 and 2 joex instances). That removes docker from the setup.

With docker-compose i could reproduce it! So it is some problem with the picking up env variables - and then I knew... The config file can refer to other settings in its own file. And the app_id will be interpolated as default value in some other places - like the id for the worker and scheduler. But this doesn't work with env variables! For env variables, you need to set all other values explicitly. You can search for "joex1" in the doc page showing all env vars for what needs to be changed as well (most likely). Try setting these:

- DOCSPELL_JOEX_APP__ID=joex2
- DOCSPELL_JOEX_PERIODIC__SCHEDULER_NAME=joex2
- DOCSPELL_JOEX_SCHEDULER_NAME=joex2

Edit: I don't think I want to copy the behavior of the config lib to env variables, so I think an update in the docs would be good….

[ohagene]

@eikek Thanks, problem solved!