Lazy loading

Because raw EEG recordings can be quite large this package is very aware of memory restrictions and possible bottlenecks due to long loading times from disk.

Therefore most actions that require loading data from disk into memory are executed lazily, meaning:

1. the data is loaded from disk when you access it for the first time
2. the data remains in memory and can be accessed very fast subsequently

To make working with the data more comfortable, the creation of containers and the loading of metadata on the other hand happens instantly.

from neurone_loader import Recording

The following examples use the Recording container, but all of the features shown below also work with Session and Phase!

# fast: only relevant metadata is loaded from disk
%time rec = Recording(test_data_path)

CPU times: user 15.5 ms, sys: 0 ns, total: 15.5 ms
Wall time: 14.7 ms

%%time
# fast: metadata is already in memory
print(f'Sessions: {len(rec.sessions)}')
print(f'Sampling rate: {rec.sampling_rate}Hz')

Sessions: 2
Sampling rate: 5000Hz
CPU times: user 293 µs, sys: 0 ns, total: 293 µs
Wall time: 206 µs

%%time
# this is slow: the session data needs to be retrieved from disk first
print(f'Session 1 shape: {rec.sessions[0].data.shape}')

(Lazy) loading Session.data
(Lazy) loading Phase.data
(Lazy) loading Phase.data
(Lazy) loading Phase.data
(Lazy) loading Phase.data
Session 1 shape: (2504369, 138)
CPU times: user 9.7 s, sys: 2.73 s, total: 12.4 s
Wall time: 4.05 s

%%time
# this will be faster because the data is already in memory
print(f'Session 1 shape again: {rec.sessions[0].data.shape}')

Session 1 shape again: (2504369, 138)
CPU times: user 2.03 ms, sys: 528 µs, total: 2.55 ms
Wall time: 162 µs

As you can see above the container object can be contructed and used very memory and time efficient. Reading the actual session data, which can take a long time and may consume a lot of memory, is only happening when the data is actually needed. On subsequent calls the already loaded data is retrieved from memory which is much faster.

To save memory the data can be cleared from memory using the .clear_data() function.

rec.clear_data()

Preloading

In some cases you may want to load all of the data from disk at once. There are two ways to achieve this.

1. Invoke the loading of all (not yet loaded) data by calling .preload()
2. Load all the data on initialization by setting the argument preload=True

#Reload all the data cleared in [7]
rec.preload()

Preloading property data of <neurone_loader.loader.Recording object at 0x7f905d013898>
(Lazy) loading Recording.data
(Lazy) loading Session.data
(Lazy) loading Phase.data
(Lazy) loading Phase.data
(Lazy) loading Phase.data
(Lazy) loading Phase.data
(Lazy) loading Session.data
(Lazy) loading Phase.data
(Lazy) loading Phase.data
(Lazy) loading Phase.data
(Lazy) loading Phase.data
(Lazy) loading Phase.events
(Lazy) loading Phase.events
(Lazy) loading Phase.events
(Lazy) loading Phase.events
(Lazy) loading Phase.events
(Lazy) loading Phase.events
(Lazy) loading Phase.events
(Lazy) loading Phase.events
(Lazy) loading Phase.n_samples
(Lazy) loading Phase.n_samples
(Lazy) loading Phase.n_samples
(Lazy) loading Phase.n_samples
(Lazy) loading Phase.n_samples
(Lazy) loading Phase.n_samples
(Lazy) loading Phase.n_samples
(Lazy) loading Phase.n_samples
(Lazy) loading Phase.n_channels
(Lazy) loading Phase.n_channels
(Lazy) loading Phase.n_channels
(Lazy) loading Phase.n_channels
(Lazy) loading Phase.n_channels
(Lazy) loading Phase.n_channels
(Lazy) loading Phase.n_channels
(Lazy) loading Phase.n_channels

# Load all of the data on initialization
not_so_lazy_rec = Recording(test_data_path, preload=True)

Preloading property data of <neurone_loader.loader.Recording object at 0x7f9063412ef0>
(Lazy) loading Recording.data
(Lazy) loading Session.data
(Lazy) loading Phase.data
(Lazy) loading Phase.data
(Lazy) loading Phase.data
(Lazy) loading Phase.data
(Lazy) loading Session.data
(Lazy) loading Phase.data
(Lazy) loading Phase.data
(Lazy) loading Phase.data
(Lazy) loading Phase.data
(Lazy) loading Phase.events
(Lazy) loading Phase.events
(Lazy) loading Phase.events
(Lazy) loading Phase.events
(Lazy) loading Phase.events
(Lazy) loading Phase.events
(Lazy) loading Phase.events
(Lazy) loading Phase.events
(Lazy) loading Phase.n_samples
(Lazy) loading Phase.n_samples
(Lazy) loading Phase.n_samples
(Lazy) loading Phase.n_samples
(Lazy) loading Phase.n_samples
(Lazy) loading Phase.n_samples
(Lazy) loading Phase.n_samples
(Lazy) loading Phase.n_samples
(Lazy) loading Phase.n_channels
(Lazy) loading Phase.n_channels
(Lazy) loading Phase.n_channels
(Lazy) loading Phase.n_channels
(Lazy) loading Phase.n_channels
(Lazy) loading Phase.n_channels
(Lazy) loading Phase.n_channels
(Lazy) loading Phase.n_channels