Is it possible with nginx to rewrite "&amp" to "&" if it appears in a URL the client sends to nginx? I'm currently kinda stuck, as some parts of my application (where I have plenty influence on) are calling s3 download links from my minio backend with "&amp" and others with "&" argument separation in the URL. I'm not sure why this issue exactly occurs, but my idea is to simply make nginx fix these URL calls internally, as they are practically the same. The problem is that minio (my S3 Server) interprets these wrong and denies access to a given resource as s3 signatures do not match anymore with &amp in the URL. They always have to be "&" instead of "&amp" !

To be a bit more specific, how the URL differs in some cases, please see the following example:

String I pass:

https://localhost/test/sprites.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=2PsuWGctH4UQmGvEQYjTDsZ2HqGM/20220601/minio/s3/aws4_request&X-Amz-Date=20220601T172937Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&X-Amz-Signature=6da7dc0137d25730d09bebbd54b0e1f0132d58cba318b3cfe11bcde5af608e05


Browser calls:

https://localhost/test/sprites.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=2PsuWGctH4UQmGvEQYjTDsZ2HqGM/20220601/minio/s3/aws4_request&X-Amz-Date=20220601T172937Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&X-Amz-Signature=6da7dc0137d25730d09bebbd54b0e1f0132d58cba318b3cfe11bcde5af608e05

The location at my NGINX config looks like this at the moment of writing:

location / {
            auth_jwt_enabled off;
            limit_req zone=s3 burst=100 nodelay;
            proxy_buffer_size 256k;
            proxy_buffers 4 512k;
            proxy_busy_buffers_size 512k;
            proxy_cache_convert_head off;
            proxy_connect_timeout 180;
            proxy_hide_header Set-Cookie;
            proxy_hide_header x-amz-id-2;
            proxy_hide_header x-amz-request-id;
            proxy_http_version 1.1;
            proxy_ignore_headers Set-Cookie;
            proxy_pass http://127.0.0.1:7777;
            proxy_read_timeout 86400;
            proxy_send_timeout 86400;
            proxy_set_header Authorization $http_authorization;
            proxy_set_header Connection "";
            proxy_set_header Host $http_host;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header Connection keep-alive;
            add_header Content-disposition "attachment; filename=$1";
            default_type application/octet-stream;
        }

My guess here is that the browser converts the & to &amp for some reason, as the strings I'm passing does not contain &amp at all ...

CodePudding user response：

I solved the issue now, on application side. So I cannot answer this in a NGINX kinda way. The Problem is, and you might have this issue with other frameworks too (I'm using Django) that the string I'm passing get placed in a JS code snipped. To have a string transformed the right way while injecting it to JavaScript code in a template, you need a JSON string object. Otherwise, this conversion form & to &amp might occur on call.

In Django, you can create a template tag for this:

/myproject/test123/templatetags/js.py

from django.utils.safestring import mark_safe
from django.template import Library

import json


register = Library()

@register.filter(is_safe=True)
def js(obj):
    return mark_safe(json.dumps(obj))

And later in your templates JS code, you simply do this with your URL variable:

<script>
    url: {{ sprites_url | js }},
</script>

CodePudding user response：

Unfortunately nginx does not have an ability of globally replacing some strings. A workaround is possible, based on the rewrite ... last, when being triggered inside a location, starts over the NGX_HTTP_FIND_CONFIG_PHASE to find a new location for the rewritten URI. However there is a limitation of maximum 10 iteration rounds of this process, or nginx will throw an internal HTTP 500 error. Here is the idea:

location / {
    if ($args ~ "^(?<prefix>. )&amp;(?<suffix>. )$") {
        rewrite ^ $uri?$prefix&$suffix? last;
    }
    ...
}

Nevertheless I doubt the problem is on the nginx side, so most likely this won't help.

CodePudding user response：

According to this you need nginx mod LUA:

map $uri $no_slash_uri {
    ~^/(?<no_slash>.*)$ $no_slash;
}

set_by_lua $escaped_uri 'return ngx.escape_uri(ngx.var.uri)';

set_by_lua $no_slash_escaped_uri 
    'return ngx.escape_uri(ngx.var.no_slash_uri)';

location / {
    try_files $uri /index.php?q=$no_slash_escaped_uri&$args;
}