Keys provided as string in caps being converted to camel case from snake case since selenium v4 base

[vm-iiit]

This is a

• Bug report
• Question
• Freature Request

Summary

Ruby appium is converting capabilities from snake case to camel case

Environment

• Appium version (or git revision): Appium v1.22.0
• ruby_lib version: appium_lib (12.0.1)
• Mobile platform/version/device under test: NA

Actual behaviour and steps to reproduce

1. Start a local appium server
2. Use the following script to start session pass capabilities in snake case

require 'rubygems'
require 'appium_lib'

SERVER_URL = "tcp://localhost:4723/wd/hub"

desired_caps = {
  "platformName" => "android",
  "device" => "Samsung Galaxy S21 Ultra",
  "app" => "/Users/viratmishra/Downloads/fast-speed-test-1-0-5.apk", #any local app path
  "os_version" => "11.0",
  "custom_cap" => "custom_value",
  "custom_cap_2" => {
    "custom_nested_key" => "custom_value"
  },
  "custom_cap_3" => {
    "custom_nested_key_2" => {
      "custom_nested_key_3" => "custom_value"
    }
  },
}

appium_driver = Appium::Driver.new({
  'caps' => desired_caps,
  'appium_lib' => {
    :server_url => SERVER_URL
  }}, true)
driver = appium_driver.start_driver

wait = Selenium::WebDriver::Wait.new(timeout: 40)

driver.quit

3. See the caps received by appium server, in my case

[Appium] 
[Appium]   The next major version of Appium (2.x) will **require** the 
[Appium]   'automationName' capability to be set for all sessions on all 
[Appium]   platforms
[Appium] 
[Appium]   In previous versions (Appium <= 1.13.x), the default was 
[Appium]   'automationName=UiAutomator1'
[Appium] 
[Appium]   If you wish to use that automation instead of UiAutomator2, please 
[Appium]   add 'automationName=UiAutomator1' to your desired capabilities
[Appium] 
[Appium]   For more information about drivers, please visit 
[Appium]   http://appium.io/docs/en/about-appium/intro/ and explore the 
[Appium]   'Drivers' menu
[Appium] 
[Appium] ======================================================================
[Appium] 
[Appium] Appium v1.22.0 creating new AndroidUiautomator2Driver (v1.69.0) session
[Appium] Applying relaxed security to 'AndroidUiautomator2Driver' as per server command line argument. All insecure features will be enabled unless explicitly disabled by --deny-insecure
[BaseDriver] Creating session with W3C capabilities: {
[BaseDriver]   "alwaysMatch": {
[BaseDriver]     "platformName": "android",
[BaseDriver]     "appium:device": "Samsung Galaxy S21 Ultra",
[BaseDriver]     "appium:app": "/Users/viratmishra/Downloads/fast-speed-test-1-0-5.apk",
[BaseDriver]     "appium:os_version": "11.0",
[BaseDriver]     "appium:custom_cap": "custom_value",
[BaseDriver]     "appium:custom_cap_2": {
[BaseDriver]       "customNestedKey": "custom_value"
[BaseDriver]     },
[BaseDriver]     "appium:custom_cap_3": {
[BaseDriver]       "customNestedKey_2": {
[BaseDriver]         "customNestedKey_3": "custom_value"
[BaseDriver]       }
[BaseDriver]     }
[BaseDriver]   },
[BaseDriver]   "firstMatch": [
[BaseDriver]     {}
[BaseDriver]   ]
[BaseDriver] }
[BaseDriver] The following capabilities were provided, but are not recognized by Appium:
[BaseDriver]   device
[BaseDriver]   os_version
[BaseDriver]   custom_cap
[BaseDriver]   custom_cap_2
[BaseDriver]   custom_cap_3
[BaseDriver] Session created with session id: cc06c35d-3066-4edc-ae56-5154170364f6
[BaseDriver] Using local app '/Users/viratmishra/Downloads/fast-speed-test-1-0-5.apk'
[UiAutomator2] Checking whether app is actually present
[ADB] Using 'adb' from '/Users/viratmishra/Library/Android/sdk_old/platform-tools/adb'
[ADB] Running '/Users/viratmishra/Library/Android/sdk_old/platform-tools/adb -P 5037 start-server'
[AndroidDriver] Retrieving device list
[ADB] Trying to find a connected android device

The caps custom_nested_key , custom_nested_key_2 and custom_nested_key_3 got converted to camelcase when received by appium server: "customNestedKey" , "customNestedKey_2" , "customNestedKey_3"

Expected behaviour

Caps should have remained as they were passed, i.e. in snake case, tried with python client as well and this didn't happen there.

Link to Appium/Ruby logs

https://gist.github.com/vm-iiit/f07fc52b1f915f52fd8603e2403d941b

Any additional comments

[KazuCocoa]

Selenium client does so when a key does not have a vendor prefix: https://github.com/SeleniumHQ/selenium/blob/604c23b3a5224e9890a80bf5f97e4512d9586ec0/rb/lib/selenium/webdriver/remote/capabilities.rb#L29

I've added more test cases in appium/ruby_lib_core#412 (I already had some of, but did not nested one) and did more investigation. Then, it seems like the converting behavior changed a bit since selenium webdriver v4. Before the v3, it seems like the convert occurred only for symbols, but since selenium webdriver v4, it concerts both string and symbol. It does not happen when the key has a vendor prefix.

[KazuCocoa]

hm, but still https://github.com/SeleniumHQ/selenium/blob/trunk/rb/lib/selenium/webdriver/remote/capabilities.rb#L295-L303
let me see something more

Actually this behavior changed since selenium base version updated. Maybe, so far, it is when capabilities class converts its value to JSON to send them to the server.

Potentially this is not this client's issue, but we can ask selenium team about this.

[KazuCocoa]

As a workaround, you can use selenium v3 based version, or update your capabilities to follow camel case like JSON mostly follows

[vm-iiit]

Thanks for looking into this so promptly @KazuCocoa .
I still think it's ruby client's issue and not Selenium, reason being that I tried the same thing with python client, and did not encounter this scenario.

Version details

❯ pip3 show Appium-Python-Client                                                                                                                                                                                                                                                        ─╯
Name: Appium-Python-Client
Version: 2.7.0
Summary: Python client for Appium
Home-page: http://appium.io/
Author: Isaac Murchie
Author-email: isaac@saucelabs.com
License: Apache 2.0
Location: /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages
Requires: selenium
Required-by:

This is also based on Selenium 4 as per this page

Since v2.0.0, the base selenium client version is v4. The version only works in W3C WebDriver protocol format. If you would like to use the old protocol (MJSONWP), please use v1 Appium Python client.

Test script used

from unittest import TestCase
from appium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
import time
import sys
from selenium.webdriver.common.actions import interaction
from selenium.webdriver.common.actions.action_builder import ActionBuilder
from selenium.webdriver.common.actions.pointer_input import PointerInput
import base64
import random


hub_url = "0.0.0.0:4723/wd/hub"


desired_caps = {
   "app" : "/Users/viratmishra/Downloads/fast-speed-test-1-0-5.apk",
   "os_version": "11.0",
   "platformName": "Android",
   "deviceName": "Samsung Galaxy S21 Plus",
   "custom_cap" : "custom_value",
   "custom_cap_2" : {
      "custom_nested_key" : "custom_value"
   },
   "custom_cap_3" : {
      "custom_nested_key_2" : {
         "custom_nested_key_3" : "custom_value"
      }
   },
}



driver = webdriver.Remote(
   command_executor=hub_url,
   desired_capabilities=desired_caps
)

wait = WebDriverWait(driver, 60)

driver.quit()

Appium server logs

[Appium] 
[Appium]   In previous versions (Appium <= 1.13.x), the default was 
[Appium]   'automationName=UiAutomator1'
[Appium] 
[Appium]   If you wish to use that automation instead of UiAutomator2, please 
[Appium]   add 'automationName=UiAutomator1' to your desired capabilities
[Appium] 
[Appium]   For more information about drivers, please visit 
[Appium]   http://appium.io/docs/en/about-appium/intro/ and explore the 
[Appium]   'Drivers' menu
[Appium] 
[Appium] ======================================================================
[Appium] 
[Appium] Appium v1.22.0 creating new AndroidUiautomator2Driver (v1.69.0) session
[Appium] Applying relaxed security to 'AndroidUiautomator2Driver' as per server command line argument. All insecure features will be enabled unless explicitly disabled by --deny-insecure
[BaseDriver] Creating session with W3C capabilities: {
[BaseDriver]   "alwaysMatch": {
[BaseDriver]     "platformName": "Android",
[BaseDriver]     "appium:app": "/Users/viratmishra/Downloads/fast-speed-test-1-0-5.apk",
[BaseDriver]     "appium:os_version": "11.0",
[BaseDriver]     "appium:deviceName": "Samsung Galaxy S21 Plus",
[BaseDriver]     "appium:custom_cap": "custom_value",
[BaseDriver]     "appium:custom_cap_2": {
[BaseDriver]       "custom_nested_key": "custom_value"
[BaseDriver]     },
[BaseDriver]     "appium:custom_cap_3": {
[BaseDriver]       "custom_nested_key_2": {
[BaseDriver]         "custom_nested_key_3": "custom_value"
[BaseDriver]       }
[BaseDriver]     }
[BaseDriver]   },
[BaseDriver]   "firstMatch": [
[BaseDriver]     {}
[BaseDriver]   ]
[BaseDriver] }
[BaseDriver] The following capabilities were provided, but are not recognized by Appium:
[BaseDriver]   os_version
[BaseDriver]   custom_cap
[BaseDriver]   custom_cap_2
[BaseDriver]   custom_cap_3
[BaseDriver] Session created with session id: 682cad7e-aac3-4d6a-827d-dddb03dd085d
[BaseDriver] Using local app '/Users/viratmishra/Downloads/fast-speed-test-1-0-5.apk'
[UiAutomator2] Checking whether app is actually present
[ADB] Found 1 'build-tools' folders under '/Users/viratmishra/Library/Android/sdk_old' (newest first):
[ADB]     /Users/viratmishra/Library/Android/sdk_old/build-tools/23.0.1
[ADB] Using 'adb' from '/Users/viratmishra/Library/Android/sdk_old/platform-tools/adb'
[ADB] Running '/Users/viratmishra/Library/Android/sdk_old/platform-tools/adb -P 5037 start-server'
[AndroidDriver] Retrieving device list
[ADB] Trying to find a connected android device
[ADB] Getting connected devices
[ADB] No connected devices have been detected
[ADB] Could not find online devices
[ADB] Reconnecting adb (target offline)
[ADB] Running '/Users/viratmishra/Library/Android/sdk_old/platform-tools/adb -P 5037 reconnect offline'
[ADB] Getting connected devices
[ADB] No connected devices have been detected

As for the workaround suggested by you, it also occurred to me but the issue here would be that anyone using a ruby client can pass snake case and it will eventually be treated as valid input since it would be received as camel case whereas Ideally, this should not happen, and in other clients the same input would be invalid.

[vm-iiit]

Also, just to rule out the unlikely possibility of something going wrong during JSON conversion

❯ irb                                                                                                                                                                                                                                                                                   ─╯
2.6.6 :001 > desired_caps = { "custom_cap" => "custom_value", "custom_cap_2" => { "custom_nested_key" => "custom_value" }, "custom_cap_3" => { "custom_nested_key_2" => { "custom_nested_key_3" => "custom_value" } }  }
 => {"custom_cap"=>"custom_value", "custom_cap_2"=>{"custom_nested_key"=>"custom_value"}, "custom_cap_3"=>{"custom_nested_key_2"=>{"custom_nested_key_3"=>"custom_value"}}}
2.6.6 :002 > require 'json'
 => true
2.6.6 :003 > dc_json = desired_caps.to_json
 => "{\"custom_cap\":\"custom_value\",\"custom_cap_2\":{\"custom_nested_key\":\"custom_value\"},\"custom_cap_3\":{\"custom_nested_key_2\":{\"custom_nested_key_3\":\"custom_value\"}}}"
2.6.6 :004 > orig_hash = JSON.parse(dc_json)
 => {"custom_cap"=>"custom_value", "custom_cap_2"=>{"custom_nested_key"=>"custom_value"}, "custom_cap_3"=>{"custom_nested_key_2"=>{"custom_nested_key_3"=>"custom_value"}}}
2.6.6 :005 > desired_caps == orig_hash
 => true
2.6.6 :006 >

[KazuCocoa]

Sorry, I did not say other selenium clients, just Ruby implementation as my previous links for selenium repository.
Added tests in appium/ruby_lib_core#412 worked as you posted when I downgraded the ruby_lib_core implementation with selenium v3, which behavior was changed in v4 based ruby_lib_core version as you reported.

Maybe related to as_json/to_json thing in ::Selenium::WebDriver::Remote::Capabilities. (I need more time to investigate the cause). So, converting camel case in symbol has been behaved the same, but string changed in them.

[KazuCocoa]

this is selenium v3 based one:

[debug] [AndroidUiautomator2Driver@e7b4]     "appium:custom_cap": "custom_value",
[debug] [AndroidUiautomator2Driver@e7b4]     "appium:custom_cap_2": {
[debug] [AndroidUiautomator2Driver@e7b4]       "custom_nested_key": "custom_value"
[debug] [AndroidUiautomator2Driver@e7b4]     },
[debug] [AndroidUiautomator2Driver@e7b4]     "appium:custom_cap_3": {
[debug] [AndroidUiautomator2Driver@e7b4]       "custom_nested_key_2": {
[debug] [AndroidUiautomator2Driver@e7b4]         "custom_nested_key_3": "custom_value"
[debug] [AndroidUiautomator2Driver@e7b4]       }
[debug] [AndroidUiautomator2Driver@e7b4]     },
[debug] [AndroidUiautomator2Driver@e7b4]     "appium:language": "en

[KazuCocoa]

Probably I found the root cause as #945, I was not sure the root motivation about the method I removed in #945, but it converted keys into symbols forcefully. It was legacy code before I started ruby_lib related code. Nowadays it no makes sense in most cases, so I have removed it.

Then, the caps is now:

[debug] [AndroidUiautomator2Driver@f0cf]     "appium:custom_cap": "custom_value",
[debug] [AndroidUiautomator2Driver@f0cf]     "appium:custom_cap_2": {
[debug] [AndroidUiautomator2Driver@f0cf]       "custom_nested_key": "custom_value"
[debug] [AndroidUiautomator2Driver@f0cf]     },
[debug] [AndroidUiautomator2Driver@f0cf]     "appium:custom_cap_3": {
[debug] [AndroidUiautomator2Driver@f0cf]       "custom_nested_key_2": {
[debug] [AndroidUiautomator2Driver@f0cf]         "custom_nested_key_3": "custom_value"
[debug] [AndroidUiautomator2Driver@f0cf]       }
[debug] [AndroidUiautomator2Driver@f0cf]     },
[debug] [AndroidUiautomator2Driver@f0cf]     "appium:language": "en",

Since selenium v4, the converting keys into camel case applied entire nested ones as well, so this issue started. But I think it would be nice to remove legacy code from the ruby_lib_core is more proper than changing selenium side.

[KazuCocoa]

Since the change, keys will stay string if given ones were string. Only symbols will keep getting the camel case conversion

[KazuCocoa]

Please try out appium_lib_core 5.5.0.
As appium_lib, it needs to be reinstalled or update appium_lib_core dependency to 5.5.0

[vm-iiit]

@KazuCocoa apparently the issue isn't fixed yet.

❯ ruby sample.rb                                                                                                                                           ─╯
No matched driver by platformName:  and automationName:
Traceback (most recent call last):
	43: from sample.rb:20:in `<main>'
	42: from /Users/viratmishra/.rvm/gems/ruby-2.7.2/gems/appium_lib-12.0.1/lib/appium_lib/driver.rb:516:in `start_driver'
	41: from /Users/viratmishra/.rvm/gems/ruby-2.7.2/gems/appium_lib_core-5.5.0/lib/appium_lib_core/driver.rb:369:in `start_driver'
	40: from /Users/viratmishra/.rvm/gems/ruby-2.7.2/gems/appium_lib_core-5.5.0/lib/appium_lib_core/driver.rb:369:in `new'
	39: from /Users/viratmishra/.rvm/gems/ruby-2.7.2/gems/appium_lib_core-5.5.0/lib/appium_lib_core/common/base/driver.rb:58:in `initialize'
	38: from /Users/viratmishra/.rvm/gems/ruby-2.7.2/gems/selenium-webdriver-4.3.0/lib/selenium/webdriver/common/driver.rb:74:in `initialize'
	37: from /Users/viratmishra/.rvm/gems/ruby-2.7.2/gems/appium_lib_core-5.5.0/lib/appium_lib_core/common/base/driver.rb:72:in `create_bridge'
	36: from /Users/viratmishra/.rvm/gems/ruby-2.7.2/gems/appium_lib_core-5.5.0/lib/appium_lib_core/common/base/bridge.rb:81:in `create_session'
	35: from /Users/viratmishra/.rvm/gems/ruby-2.7.2/gems/selenium-webdriver-4.3.0/lib/selenium/webdriver/remote/bridge.rb:592:in `execute'
	34: from /Users/viratmishra/.rvm/gems/ruby-2.7.2/gems/appium_lib_core-5.5.0/lib/appium_lib_core/common/base/http_default.rb:97:in `call'
	33: from /Users/viratmishra/.rvm/gems/ruby-2.7.2/gems/selenium-webdriver-4.3.0/lib/selenium/webdriver/remote/http/default.rb:103:in `request'
	32: from /Users/viratmishra/.rvm/gems/ruby-2.7.2/gems/selenium-webdriver-4.3.0/lib/selenium/webdriver/remote/http/common.rb:83:in `create_response'
	31: from /Users/viratmishra/.rvm/gems/ruby-2.7.2/gems/selenium-webdriver-4.3.0/lib/selenium/webdriver/remote/http/common.rb:83:in `new'
	30: from /Users/viratmishra/.rvm/gems/ruby-2.7.2/gems/selenium-webdriver-4.3.0/lib/selenium/webdriver/remote/response.rb:35:in `initialize'
	29: from /Users/viratmishra/.rvm/gems/ruby-2.7.2/gems/selenium-webdriver-4.3.0/lib/selenium/webdriver/remote/response.rb:56:in `assert_ok'
	28: from     at processTicksAndRejections (internal/process/task_queues.js:77:11)
	27: from     at endReadableNT (_stream_readable.js:1168:12)
	26: from     at IncomingMessage.emit (events.js:208:15)
	25: from     at IncomingMessage.onEnd (/Applications/Appium Server GUI.app/Contents/Resources/app/node_modules/appium/node_modules/raw-body/index.js:273:7)
	24: from     at done (/Applications/Appium Server GUI.app/Contents/Resources/app/node_modules/appium/node_modules/raw-body/index.js:213:7)
	23: from     at invokeCallback (/Applications/Appium Server GUI.app/Contents/Resources/app/node_modules/appium/node_modules/raw-body/index.js:224:16)
	22: from     at /Applications/Appium Server GUI.app/Contents/Resources/app/node_modules/appium/node_modules/body-parser/lib/read.js:130:5
	21: from     at next (/Applications/Appium Server GUI.app/Contents/Resources/app/node_modules/appium/node_modules/express/lib/router/index.js:275:10)
	20: from     at Function.process_params (/Applications/Appium Server GUI.app/Contents/Resources/app/node_modules/appium/node_modules/express/lib/router/index.js:335:12)
	19: from     at /Applications/Appium Server GUI.app/Contents/Resources/app/node_modules/appium/node_modules/express/lib/router/index.js:284:7
	18: from     at trim_prefix (/Applications/Appium Server GUI.app/Contents/Resources/app/node_modules/appium/node_modules/express/lib/router/index.js:317:13)
	17: from     at Layer.handle [as handle_request] (/Applications/Appium Server GUI.app/Contents/Resources/app/node_modules/appium/node_modules/express/lib/router/layer.js:95:5)
	16: from     at logger (/Applications/Appium Server GUI.app/Contents/Resources/app/node_modules/appium/node_modules/morgan/index.js:144:5)
	15: from     at next (/Applications/Appium Server GUI.app/Contents/Resources/app/node_modules/appium/node_modules/express/lib/router/index.js:275:10)
	14: from     at Function.process_params (/Applications/Appium Server GUI.app/Contents/Resources/app/node_modules/appium/node_modules/express/lib/router/index.js:335:12)
	13: from     at /Applications/Appium Server GUI.app/Contents/Resources/app/node_modules/appium/node_modules/express/lib/router/index.js:281:22
	12: from     at Layer.handle [as handle_request] (/Applications/Appium Server GUI.app/Contents/Resources/app/node_modules/appium/node_modules/express/lib/router/layer.js:95:5)
	11: from     at Route.dispatch (/Applications/Appium Server GUI.app/Contents/Resources/app/node_modules/appium/node_modules/express/lib/router/route.js:112:3)
	10: from     at next (/Applications/Appium Server GUI.app/Contents/Resources/app/node_modules/appium/node_modules/express/lib/router/route.js:137:13)
	 9: from     at Layer.handle [as handle_request] (/Applications/Appium Server GUI.app/Contents/Resources/app/node_modules/appium/node_modules/express/lib/router/layer.js:95:5)
	 8: from     at /Applications/Appium Server GUI.app/Contents/Resources/app/node_modules/appium/node_modules/appium-base-driver/lib/protocol/protocol.js:423:15
	 7: from     at asyncHandler (/Applications/Appium Server GUI.app/Contents/Resources/app/node_modules/appium/node_modules/appium-base-driver/lib/protocol/protocol.js:297:34)
	 6: from     at AppiumDriver.executeCommand (/Applications/Appium Server GUI.app/Contents/Resources/app/node_modules/appium/lib/appium.js:549:26)
	 5: from     at AppiumDriver.executeCommand (/Applications/Appium Server GUI.app/Contents/Resources/app/node_modules/appium/node_modules/appium-base-driver/lib/basedriver/driver.js:349:15)
	 4: from     at commandExecutor (/Applications/Appium Server GUI.app/Contents/Resources/app/node_modules/appium/node_modules/appium-base-driver/lib/basedriver/driver.js:335:9)
	 3: from     at AppiumDriver.createSession (/Applications/Appium Server GUI.app/Contents/Resources/app/node_modules/appium/lib/appium.js:325:26)
	 2: from     at parseCapsForInnerDriver (/Applications/Appium Server GUI.app/Contents/Resources/app/node_modules/appium/lib/utils.js:116:21)
	 1: from     at processCapabilities (/Applications/Appium Server GUI.app/Contents/Resources/app/node_modules/appium/node_modules/appium-base-driver/lib/basedriver/capabilities.js:220:13)
InvalidArgumentError: 'platformName' can't be blank: 'platformName' can't be blank (Selenium::WebDriver::Error::InvalidArgumentError)

I've updated the ruby client and core library.

❯ gem list | grep appium                                                                                                                                   ─╯
appium_lib (12.0.1)
appium_lib_core (5.5.0)

Seeing the same behaviour for other existing scripts as well which were working fine.

Script used

require 'rubygems'
require 'appium_lib'

SERVER_URL = "tcp://localhost:4723/wd/hub"

desired_caps = {
  "platformName" => "android",
  "platformVersion" => "9.0",
  "deviceName" => "Google Pixel 3",
  "app" => "/Users/viratmishra/Downloads/fast-speed-test-1-0-5.apk",

}


appium_driver = Appium::Driver.new({
  'caps' => desired_caps,
  'appium_lib' => {
    :server_url => SERVER_URL
  }}, true)
driver = appium_driver.start_driver

wait = Selenium::WebDriver::Wait.new(timeout: 40)

driver.quit