We're using Packer to construct a custom centos7 image for an Azure scale set. Part of this includes a custom rpm that we have created that builds git from source (can't use community repos so we make our own) and installs it to the /usr/local/bin directory. In normal practice, the package works perfectly. Everything gets installed appropriately to the right places and we can use our new version of git.

When we run things through Packer, we install it via ansible, and then finally Packer does the deprovisioning step, captures the image and puts it in an azure shared image gallery, which we then pick up for use in our azure scale set.

Scale set uses the image to make a few instances, and we're up and running. Problem is, suddenly, the /usr/local/ directory seems to be as if it has been reset to default. There's nothing in /usr/local/bin anymore, and furthermore, some (not all) of the packages that we install as dependencies to build git (like gcc for example), also just disappear. Our git rpm is still listed as installed, but gcc is not.

/usr/bin/ seems fine (aside from the missing gcc, and though we don't need it at this point anyway, it still seems concerning), so we can probably just install it there, but I'd still like to know if something crazy is happening, and should I look out for it in the future seeing as /usr/local/ seemed a logical spot to install it.

TL;DR:

1. packer gets base centos7 image
2. add our custom git package
3. git installs to /usr/local/bin (it works! git is available)
4. deprovision with waagent and generalized
5. packer captures image and uploads it
6. azure scale set uses image to make new instances
7. /usr/local/ is back to original state? (thus git is missing?)
8. ???

packer azure arm docs

waagent deprovisioning tool docs

CodePudding user response：

Figured this out.

Turns out (at least with version 1.7.2) Packer does not necessarily do idempotent operations with the azure arm in relation to Shared Image Gallery versions, even with the --force flag.

We had created the SIG image version before we had gotten our git package fully working and installed properly, so it was created on a base image that did not have /usr/local/bin/ modified.

When we ran the Packer build with the force flag, it deletes and recreates the base image, but it runs a PUT call with the configuration information for the SIG image version, which is to say it will "Create or Update" if it's following convention (you can't see this unless you set some packer logging vars and output the verbose logs to a file or something).

So while the base image was updated to one that had git properly set up, the SIG version thought it was using the same base image as before (the name was the same, no unique identifier), so as far as it was concerned the configuration hadn't changed and nothing needed to happen. After we deleted the old version or made a new version, it properly spun up a VM based on the base image we had made and everything was where it was supposed to be.

I am definitely of the opinion that a --force should be an idempotent operation from start to finish, I'm not sure if this is fixed in future versions (at the time of writing this they're on 1.7.6) but maybe I'll update once I've checked it out.