ChatGPT built a neural network that runs directly on bits (XNOR + popcount style)
Exploring Neural Network Computation at the Bit Level with XNOR and Popcount Techniques
In recent advancements in neural network research, there’s growing interest in leveraging bitwise operations to optimize inference speed and reduce memory footprint. This article delves into constructing a minimal yet functional binary neural network that operates directly on packed bits, utilizing efficient XNOR and population count (popcount) operations. We’ll demonstrate how to implement such a network, including packing/unpacking utilities, binary dot product computations, and a toy XOR problem solved with this approach.
Why Binary Neural Networks?
Traditional neural networks rely on floating-point computations, which can be resource-intensive. Binary neural networks (BNNs) represent weights and activations using just one bit, drastically reducing storage and enabling faster inference—especially on hardware supporting bitwise operations.
Core Components
- Bit Packing and Unpacking
To operate efficiently on bits, first, data must be packed into machine-friendly formats. Using numpy, we can pack a boolean array (bits) into 64-bit unsigned integers (uint64) and unpack them later as needed.
“`python
def pack_bits(arr):
“””
Pack a 1D array of bits (0/1) into uint64 words.
Returns the packed array and original length.
“””
arr = np.asarray(arr).astype(np.uint8).ravel()
n = arr.size
words = (n + 63) // 64
out = np.zeros(words, dtype=np.uint64)
for i, bit in enumerate(arr):
if bit:
out[i // 64] |= (np.uint64(1) << (i % 64))
return out, n
def unpack_bits(words, n_bits):
“””
Unpack uint64 words into a 0/1 numpy array of specified length.
“””
out = np.zeros(n_bits, dtype=np.uint8)
for i in range(n_bits):
out[i] = (words[i // 64] >> (i % 64)) & 1
return out
“`
- Efficient Bitwise Dot Product
Calculating inner products in binary space is key. Using XNOR logic, matching bits contribute +1, mismatches -1. The approach:
- Perform an XOR between input and weights.
- Take the complement (~) to get matches.
- Count the number of matches (popcount).
- Convert the count
Post Comment