The layers can be upsampled or downsampled directly using convolution. The stride used for convolution can be increased to cause downsampling as shown here:
Downsampling by convolution is called atrous convolution or dilated convolution or strided convolution. Similarly, it can be reversed to upsample by learning a kernel as shown here:
Upsampling directly using a convolution can be termed as transposed convolution. Some other synonyms are deconvolution or fractionally strided convolution or up-convolution ...
