gaussian grbm initialization

[jquetzalcoatl]

grbm weights and biases initialization set to Gaussian N(0,1/number of nodes)

Hinton guide suggests 0.01 as standard deviation. See https://www.cs.toronto.edu/~hinton/absps/guideTR.pdf

Moreover, having it set to Gaussian with this dependence on the number of nodes makes the energy extensive and initializes the gRBM in a paramagnetic phase similar to that describen in the Random Energy model paper

https://journals.aps.org/prb/abstract/10.1103/PhysRevB.24.2613

See #48

[kevinchern]

@jquetzalcoatl IIRC, Hinton's recommendation pertains to zero-one-valued RBMs (bipartite with hidden units). Would it make sense to translate the $0.01$ to the spin-valued equivalent?

[jquetzalcoatl]

@kevinchern The REM reference is for spin models i.e., {-1,1}. Ultimately, the initialization pertains to whether the model is ergodic. In this sense, the support only set an offset energy.

I believe the main motivation for initializing with 0.01 in Hinton's guide is to start in a paramagnetic phase, which ties nicely with the REM/SK spin glass model

[kevinchern]

Can you add a release note to go with this?

[jquetzalcoatl]

added release note

[kevinchern]

Following motivation from references, should h be initialized to 0?

[kevinchern]

Tests are failing but otherwise LGTM. Thanks for the much-needed PR @jquetzalcoatl !!

@VolodyaCO offered to take a look at the tests

[kevinchern]

Any updates on this?

[VolodyaCO]

The reason for this test failing is very strange. Essentially, it is making sure that both the DVAE forward (which does encode -> latent to discrete -> decode) matches encode -> latent_to_discrete -> decode, i.e., this is a pretty simple unit test:

expected_latents = self.encoders[n_latent_dims](self.data)
expected_discretes = self.dvaes[n_latent_dims].latent_to_discrete(
    expected_latents, n_samples
)
expected_reconstructed_x = self.decoders[n_latent_dims](expected_discretes)

latents, discretes, reconstructed_x = self.dvaes[n_latent_dims].forward(
    x=self.data, n_samples=n_samples
)

assert torch.equal(reconstructed_x, expected_reconstructed_x)
assert torch.equal(discretes, expected_discretes)
assert torch.equal(latents, expected_latents)

Moreover, self.dvaes is built as

self.encoders = {i: Encoder(i) for i in latent_dims_list}
self.decoders = {i: Decoder(latent_features, input_features) for i in latent_dims_list}
self.dvaes = {i: DVAE(self.encoders[i], self.decoders[i]) for i in latent_dims_list}

So even if the encoders/decoders are updated in other tests (because of training), there should be a permanent tracking of the encoders/decoders in the dvaes.

[VolodyaCO]

Found the issue and fixed it in a PR to @jquetzalcoatl 's repo: jquetzalcoatl#1

Please approve javi, this would update the current PR and solve the issue.

Took me a while to get the error!

[VolodyaCO]

I have definitely had to manually change the initialisation of GRBM weights whenever I use the GRBM. Thanks for this PR. I think it looks good to merge.

[kevinchern]

@jquetzalcoatl I added a couple typo fixes, can you accept them?
The remaining questions/comments are for @VolodyaCO and should be good to merge after.

[thisac]

Better add some line breaks here, splitting the full paragraph on several lines.

[thisac]

Hasn't been addressed.

[jquetzalcoatl]

Please check again. Thnx!

[jackraymond]

nu_i = beta* (h_i + sum_j Jij s_j) controls the typical field (bias) of a variable at initialization, I think we want this to be O(1), i.i.d and high entropy. I think that is inline with the motivation for the pull request, but leads to some additional considerations. We also want h to be small compared to J, because we want to initialize outside of the weakly coupled regime ideally. h should be just large enough to break the macroscopic sign-symmetry (IMO).

I think we want a strongly coupled models, so h should be just large enough to break the symmetry and no more. I.e. the contribution from h should be O(1):
beta * sum_i h_i s_i ~ 1 which for random s implies beta h_i ~ O(1/sqrt(N)).

For extensive energy I think we require J to scale as 1/root(mean-degree). 1/sqrt(N) scaling is appropriate for dense models only.

We might want to think about putting in a beta value, that reflects the QPU. E.g. if single qubit freezeout temperature ~ 1/5 we would want to scale down by a factor 5 (I think current default scales the wrong way).

We might want to think about the fact that the J and h-distributions are bounded, so Gaussian is not the maximum-entropy choice. This is probably a technicality because the bounds turn out to be far from the initialization values, but we should certainly discuss the impacts of clipping.

[kevinchern]

Any updates on this? @jquetzalcoatl

[kevinchern]

LGTM, thanks @jquetzalcoatl!!

[kevinchern]

Suggested change

	torch.testing.assert_close(latents, expected_latents)
	torch.testing.assert_close(discretes, expected_discretes)
	torch.testing.assert_close(reconstructed_x, expected_reconstructed_x)
	with self.subTest("Test latent variables match"):
	torch.testing.assert_close(latents, expected_latents)
	with self.subTest("Test discrete variables match"):
	torch.testing.assert_close(discretes, expected_discretes)
	with self.subTest("Test reconstructed outputs match"):
	torch.testing.assert_close(reconstructed_x, expected_reconstructed_x)

[jquetzalcoatl]

@VolodyaCO can you review this ^?

[VolodyaCO]

The changes are just to separate each test within their own scope. It should be fine.

[thisac]

LGTM

[thisac]

Hasn't been addressed.

[VolodyaCO]

The comments by @jackraymond have not been taken into account. As far as I understood, the average absolute value should scale as 1/(mean degree) for quadratic biases and 1/(number of nodes) for linear biases. If sampling from a normal distribution that means that the standard deviation should be of those scales (you need to account for the sqrt(2/π) factor and also for the beta \approx 5 factor).

[jquetzalcoatl]

@VolodyaCO will be take over the PR

[kevinchern]

Minor comments/questions

[kevinchern]

Why redefine this? Just to include dtype?

[kevinchern]

what's going on with the backslashes \ 🧐?

[kevinchern]

I was wondering if hard-coding expected_std + annotating the code with comments would be safer but this is fine as is.

[kevinchern]

Should check ("a", "b")'s weight is preserved (untouched).
e.g.,

1. set seed -> initialize, track weight of ("a", "b")
2. set seed -> initialize with quadratic=...
3. check 1 and 2's weight of ("a", "b") agree.

[kevinchern]

Update your formatter!!!!! Follow Ocean dev guide

[kevinchern]

Why is this test being updated?