Caching data downloaded using Invoke-WebRequest-CodePudding

Downloading a text file, filex-latest-info.txt from my S3 site to a Windows server running on AWS EC2. The text file is ~ 100 bytes and consists of a JSON description like

{
    "checksum":  "29CE7AB1669E25E5183E80DDC69152C198F5C7CFB243682238C78A16927B32EA",
    "version":  "1.0.2",
    "name":  "filex-v1.0.2.zip"
}

I'm using PowerShell to download the info file, so I can then find the right zip to download. However, it seems to be that the info file is being cached either by AWS or PowerShell, but I'd guess PowerShell.

I'm using a command like:

$response = Invoke-WebRequest -uri https://download.noyoucant.com/beta/installers/filex-latest-info.txt -headers @{"Cache-Control"="no-cache"} -UseBasicParsing

On Saturday morning about 1AM it was caching and I decided to wait until Monday. When I returned Monday morning, the caching problem had somehow resolved itself--but I was still logged in with the same session from days before.

Now, I just went and made an update to the files 30 minutes ago, and now the caching problem has returned. On another machine where I haven't run the query since Friday, it's returning the newly latest version which is correct.

IDK if there is some other way to do it in PowerShell or not? Any other headers to add or options in Invoke-WebRequest.

I see this running as a regular user. Ultimately it will be running under a service context with the NT_Authority account. But, either way, whether as a regular user or as the system account, I see the problem.

I don't need it interactively, but non-interactively as the system account I need the option -UseBasicParsing because IE has never been initted on that account. Adding or removing the argument when running interactively does not change the results.

CodePudding user response：

For our situation, turns out the problem was not PowerShell but caching in AWS CloudFront. We were able to duplicate the problem using curl on WSL. The solution for now is that after modifying the file, we have to invalidate the CloudFront cache. If you ask me, it's pretty poor design that if you change the file, it doesn't invalidate the cache automatically.

First, you need the distribution ID. The following command gives a list of distributions.

aws cloudfront list-distributions

It is a JSON format, and what is important are the ids of the items in the distributionlist. A command like this lists the IDs

(convertfrom-json (out-string -input (aws cloudfront list-distributions))).distributionlist.items.id

We only have one. IDK what would happen if we had more than one. Let's assume my ID is 987654. And the file that was having the problem of still being cached even after I changed it: https://download.noyoucant.com/beta/installers/filex-latest-info.txt

aws cloudfront create-invalidation --distribution-id 987654 -path /beta/installers/filex-latest-info.txt