Missing (disappearing) HTTP Headers in a published REST service

A Mendix published REST service operation can have multiple parameter types; Query, Path, Body, Header, Form. The parameter type determines where in the REST message the parameter is expected. This parameter can be mapped to a Microflow parameter (e.g. a string) and used in the microflow for further processing. In our implementation we were expecting an ‘api_key’ in the HTTP Header and would check in the microflow if it matched a valid API key. Which worked fine locally, but once it was deployed to the Mendix cloud it showed up as (empty) in the debugger. Even with TRACE enabled on the REST publish log node, we would only get TRACE - REST Publish: The request does not have a 'api_key' header, using default value: empty. Why does it work locally and not in acceptance/production?

Question: Why are certain HTTP headers missing from the request received in the Mendix cloud and not in local development?

Answer: Mendix applications deployed on the Mendix cloud are running on a NGINX web server. They have a configuration that is disabled by default and will cause HTTP headers containing underscores to be dropped. The local development instance does not run on a NGINX web server but (as is my current understanding) runs on Jetty.

If you do not explicitly set `underscores_in_headers on;``, NGINX will silently drop HTTP headers with underscores (which are perfectly valid according to the HTTP standard). This is done in order to prevent ambiguities when mapping headers to CGI variables as both dashes and underscores are mapped to underscores during that process. NGINX docs

Just spent hours debugging why my Mendix app works fine in development but not on acceptance because of this :/

Fix this by renaming the header, replace the underscores _ with hyphens - for example api_key becomes api-key.