java.lang.IllegalArgumentException: clientSecret is null or empty on Azure Container Instance

[sdebruyn]

java.lang.IllegalArgumentException: clientSecret is null or empty

Possibly related to #1512

The issue below happens only on Azure Container Instances, nowhere else.

Driver version

9.2.0.jre11

SQL Server version

current Azure SQL Database

Client Operating System

Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Default locale: en, platform encoding: UTF-8
OS name: "linux", version: "4.15.0-1106-azure", arch: "amd64", family: "unix"

JAVA/JVM version

Java version: 11.0.10, vendor: Oracle Corporation, runtime: /usr/local/openjdk-11

Table schema

not relevant

Problem description

I am using this JDBC driver in combination with msal4j to connect my liquibase to a an Azure SQL Database using a service principal. This is executed in a Docker container on Azure Container Instance.

I have no issues running this locally on my machine. I have no issues running this in the Docker container on my machine either.

However, the exact same Docker container, gives me the mentioned error when I run it in an Azure Container Instance.

What I checked:

• the code works on my machine (macOS 11.2)
• the code works in the Docker container on my machine (FROM maven:3-openjdk-11)
• the issue occurs only on Azure Container Instance, both with and without vnet integration
• the environment variable with my JDBC string contains the same value on ACI and locally, the client secret is always there
• the client ID and client secret, as well as any other value in the connection string, do not contain any special characters
• the order of the connection string components does not make any difference
• the connection is not blocked by any firewalls
• you can see in the trace logs that the secret is correctly parsed from the connection string

JDBC trace logs

log.xml.log

Stacktrace

ful logs.txt

Reproduction code

Using liquibase with a connection string in this form: jdbc:sqlserver://dl-REDACTED-sqlserver.database.windows.net:1433;database=REDACTED;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=300;authentication=ActiveDirectoryServicePrincipal;aadSecurePrincipalId=51fcffe3-REDACTED;aadSecurePrincipalSecret=REDACTED

POM file

<project  xmlns="http://maven.apache.org/POM/4.0.0"  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>eu.lynxcare</groupId>
    <artifactId>REDACTED</artifactId>
    <version>1.0-SNAPSHOT</version>
    <build>
        <pluginManagement>
            <plugins>
                <plugin>
                    <groupId>org.liquibase</groupId>
                    <artifactId>liquibase-maven-plugin</artifactId>
                    <version>4.3.0</version>
                    <configuration>
                        <promptOnNonLocalDatabase>false</promptOnNonLocalDatabase>
                        <driver>com.microsoft.sqlserver.jdbc.SQLServerDriver</driver>
                        <logging>${env.LOGLEVEL}}</logging>
                        <url>${env.JDBC_STRING}</url>
                        <changeLogDirectory>changelogs</changeLogDirectory>
                        <changeLogFile>main.changelog.xml</changeLogFile>
                    </configuration>
                    <dependencies>
                        <dependency>
                            <groupId>com.microsoft.azure</groupId>
                            <artifactId>msal4j</artifactId>
                            <version>1.9.0</version>
                        </dependency>
                        <dependency>
                            <groupId>com.microsoft.sqlserver</groupId>
                            <artifactId>mssql-jdbc</artifactId>
                            <version>9.2.0.jre11</version>
                        </dependency>
                    </dependencies>
                </plugin>
            </plugins>
        </pluginManagement>
    </build>
</project>

Environment variables

KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_SERVICE_PORT=443
MAVEN_CONFIG=/root/.m2
HOSTNAME=wk-caas-b8ed290fc52d4df8bd6e5047b2c06027-7032143a535651dc8335ce
JAVA_HOME=/usr/local/openjdk-11
reportBuilderPassword=REDACTED
AZCOPY_CONCURRENCY_VALUE=AUTO
PWD=/liquibase
HOME=/root
LANG=C.UTF-8
KUBERNETES_PORT_443_TCP=tcp://10.0.0.1:443
dqSchemaName=REDACTED
omopExtensions20201126=REDACTED
adGroupReader=REDACTED
cdmSchemaName=REDACTED
TERM=xterm
MAVEN_HOME=/usr/share/maven
AZCOPY_CRED_TYPE=Anonymous
SHLVL=1
storageAccountKey=REDACTED
KUBERNETES_PORT_443_TCP_PROTO=tcp
JDBC_STRING=jdbc:sqlserver://dl-REDACTED-sqlserver.database.windows.net:1433;database=REDACTED;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=300;aadSecurePrincipalId=51fcffe3-REDACTED;aadSecurePrincipalSecret=REDACTED;authentication=ActiveDirectoryServicePrincipal
KUBERNETES_PORT_443_TCP_ADDR=10.0.0.1
KUBERNETES_SERVICE_HOST=10.0.0.1
KUBERNETES_PORT=tcp://10.0.0.1:443
KUBERNETES_PORT_443_TCP_PORT=443
masterKey=REDACTED
PATH=/usr/local/openjdk-11/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
vocabulariesLocation=REDACTED
customerId=1
TARGET_VERSION=v23
JAVA_VERSION=11.0.10
LOGLEVEL=INFO
cdmv5SchemaName=REDACTED

Docker

FROM maven:3-openjdk-11

# curl
RUN apt -qq update \
 && apt -y --no-install-recommends install curl \
 && apt clean \
 && rm -rf /var/lib/apt/lists/*

# sqlcmd
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - \
 && curl https://packages.microsoft.com/config/ubuntu/18.04/prod.list | tee /etc/apt/sources.list.d/msprod.list \
 && apt -qq update \
 && ACCEPT_EULA=Y apt -y --no-install-recommends install mssql-tools unixodbc-dev \
 && apt clean \
 && rm -rf /var/lib/apt/lists/* \
 && ln -s /opt/mssql-tools/bin/sqlcmd /usr/local/bin/sqlcmd

# AzCopy
RUN curl -sL -o /tmp/azcopy.tar.gz https://aka.ms/downloadazcopy-v10-linux \
 && tar -zxf /tmp/azcopy.tar.gz  -C /usr/local/bin --wildcards --no-anchored --strip-components=1 'azcopy' \
 && chmod +x /usr/local/bin/azcopy \
 && rm -rf /tmp/*
ENV AZCOPY_CONCURRENCY_VALUE=AUTO
ENV AZCOPY_CRED_TYPE=Anonymous

# project
RUN mkdir -p /liquibase
COPY . /liquibase

# Application Insights
ARG APPLICATION_INSIGHTS_RELEASE="3.0.2"
RUN curl -o /liquibase/ai.jar -sL "https://github.com/microsoft/ApplicationInsights-Java/releases/download/${APPLICATION_INSIGHTS_RELEASE}/applicationinsights-agent-${APPLICATION_INSIGHTS_RELEASE}.jar"

# default settings
ENV LOGLEVEL=INFO
ENV storageAccountKey="REDACTED"
ENV vocabulariesLocation="REDACTED"
ENV omopExtensions20201126="REDACTED"
ENV cdmSchemaName=REDACTED
ENV cdmv5SchemaName=REDACTED
ENV dqSchemaName=REDACTED

# update me when you add a new changelog
ENV TARGET_VERSION=v23

WORKDIR /liquibase
CMD /liquibase/scripts/entrypoint.sh

[sdebruyn]

Changing the keys to AADSecurePrincipalId and AADSecurePrincipalSecret does not make any difference

[lilgreenbird]

hi @sdebruyn

Can you please try the same with just a vanilla java app like in the sample code w/o docker?

I've asked an MSAL developer about this the MSAL library should be platform agnostic. I can see that that exception is thrown only when the clientSecret is not provided or empty I'm not sure how that can happen if it was provided as I mentioned in issue 1512. Could you please set a breakpoint at these places to verify that the clientSecret was set for the driver?

[lilgreenbird]

could you also please check if you have the same problem with the previous 8.4.1 release?

[lilgreenbird]

hi @sdebruyn

I can repro the issue can confirm it is a bug

[sdebruyn]

@lilgreenbird Thank you so much for your time and effort!

Since the issue only occurs when used in Azure Container Instances and not on my local machine or even in Docker on my local machine, I had no idea how to use breakpoints inside that container. Also, the 8.4.1 release didn't have the service principal auth feature IIRC.

[lilgreenbird]

Thanks for the understanding and patience. We will be releasing a hotfix for this shortly.

[lilgreenbird]

hi @sdebruyn,

I am testing this fix, can you please give these jars a try to see if this resolve the issue in your environment?

[sdebruyn]

@lilgreenbird Thank you! I'll let you know tomorrow by the end of the day

[sdebruyn]

@lilgreenbird The hotfix fixes the issue, thank you! Out of curiosity: why did this behave differently on Azure Container Instances?