An illegal memory access was encountered ошибка

Relatively new to using CUDA. I keep getting the following error after a seemingly random period of time:
RuntimeError: CUDA error: an illegal memory access was encountered

I have seen people suggest things such as using cuda.set_device() rather than cuda.device(), setting torch.backends.cudnn.benchmark = False

but I can’t seem to get the error to go away. Here are some pieces of my code:
torch.cuda.set_device(torch.device('cuda:0'))
torch.backends.cudnn.benchmark = False

class LSTM(nn.Module):
    def __init__(self, input_dim, hidden_dim, num_layers, output_dim):
        super(LSTM, self).__init__()
        self.hidden_dim = hidden_dim
        self.num_layers = num_layers

        self.lstm = nn.LSTM(input_dim, hidden_dim, num_layers, batch_first=True, dropout=0.2)

        self.fc = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_dim).requires_grad_().cuda()
        c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_dim).requires_grad_().cuda()

        out, (hn, cn) = self.lstm(x, (h0.detach(), c0.detach()))

        out = self.fc(out[:, -1, :]) 
        
        return out

    def pred(self, x):
        return self(x) > 0

def train(model, loss_fn, optimizer, num_epochs, x_train, y_train, x_val, y_val, loss_stop=60):
    cur_best_loss = 999
    loss_recur_count = 0
    best_model = None
    for t in range(num_epochs):
        model.train()

        y_train_pred = model(x_train)

        train_loss = loss_fn(y_train_pred, y_train)

        tr_l = train_loss.item()
        
        optimizer.zero_grad()

        train_loss.backward()
        optimizer.step()

        model.eval()
        with torch.no_grad():  
            y_val_pred = model(x_val)

            val_loss = loss_fn(y_val_pred, y_val)

            va_l = val_loss.item()
            
            if va_l < cur_best_loss:
                cur_best_loss = va_l
                best_model = model
                loss_recur_count = 0
            else:
                loss_recur_count += 1

        if loss_recur_count == loss_stop:
            break
    if best_model is None:
        print("model is None.")
    return best_model

def lstm_test(cols, df, test_percent, test_bal, initial_shares_test, max_price, last_sell_day):
    wdw = 20
    x_train, y_train, x_test, y_test, x_val, y_val = load_data(df, wdw, test_percent, cols)

    x_train = torch.from_numpy(x_train).type(torch.Tensor).cuda()
    x_test = torch.from_numpy(x_test).type(torch.Tensor).cuda()
    x_val = torch.from_numpy(x_val).type(torch.Tensor).cuda()
    y_train = torch.from_numpy(y_train).type(torch.Tensor).cuda()
    y_test = torch.from_numpy(y_test).type(torch.Tensor).cuda()
    y_val = torch.from_numpy(y_val).type(torch.Tensor).cuda()

    input_dim = x_train.shape[-1]
    hidden_dim = 32
    num_layers = 2
    output_dim = 1
    y_preds_dict = {}
    for i in range(11):
        model = LSTM(input_dim=input_dim, hidden_dim=hidden_dim, output_dim=output_dim, num_layers=num_layers).cuda()

        r = (y_train.cpu().shape[0] - np.count_nonzero(y_train.cpu()))/np.count_nonzero(y_train.cpu())/2
        pos_w = torch.tensor([r]).cuda()

        loss_fn = torch.nn.BCEWithLogitsLoss(pos_weight=pos_w).cuda()

        optimizer = torch.optim.AdamW(model.parameters(), lr=0.01)

        best_model = train(model, loss_fn, optimizer, 300, x_train, y_train, x_val, y_val)
        
        y_test_pred = get_predictions(best_model, x_test)
        y_preds_dict[i] = y_test_pred.cpu().detach().numpy().flatten()

and here is the error msg:

<ipython-input-5-c52edc2c0508> in train(model, loss_fn, optimizer, num_epochs, x_train, y_train, x_val, y_val, loss_stop)
     19         model.eval()
     20         with torch.no_grad():
---> 21             y_val_pred = model(x_val)
     22 
     23             val_loss = loss_fn(y_val_pred, y_val)

~anaconda3libsite-packagestorchnnmodulesmodule.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

<ipython-input-4-9da8c811c037> in forward(self, x)
     10 
     11     def forward(self, x):
---> 12         h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_dim).requires_grad_().cuda()
     13         c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_dim).requires_grad_().cuda()
     14 

RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Take and my 5 cents. On GPU I start mining a week ago. For now i have 14 1070ti -+ OC, 2 farms and mining eth with auto restart ethminer if it stops on errors. This two scripts is not best solution, writen from scratch but works fine. Writed only for nvidia but i think it maybe rewriten for ati too ))
All this tested on Ubuntu 16.04

!!! nvidia coolbits must be enabled if you want OC settings to work. Mine is 13 tested on 381 and 387 drivers, emulated monitor for each card neded my nvidia-xconfig conf for 7 GPU, edid.bin find in google, i made mine from AOC 23 mon

nvidia-xconfig: X configuration file generated by nvidia-xconfig

nvidia-xconfig: version 387.34 (buildmeister@swio-display-x64-rhel04-15) Tue Nov 21 03:31:45 PST 2017

Section «ServerLayout»
Identifier «Layout0»
Screen 0 «Screen0»
Screen 1 «Screen1» RightOf «Screen0»
Screen 2 «Screen2» RightOf «Screen1»
Screen 3 «Screen3» RightOf «Screen2»
Screen 4 «Screen4» RightOf «Screen3»
Screen 5 «Screen5» RightOf «Screen4»
Screen 6 «Screen6» RightOf «Screen5»
InputDevice «Keyboard0» «CoreKeyboard»
InputDevice «Mouse0» «CorePointer»
EndSection

Section «Files»
EndSection

Section «InputDevice»
# generated from default
Identifier «Mouse0»
Driver «mouse»
Option «Protocol» «auto»
Option «Device» «/dev/psaux»
Option «Emulate3Buttons» «no»
Option «ZAxisMapping» «4 5»
EndSection

Section «InputDevice»
# generated from default
Identifier «Keyboard0»
Driver «kbd»
EndSection

Section «Monitor»
Identifier «Monitor0»
VendorName «Unknown»
ModelName «Unknown»
HorizSync 28.0 — 33.0
VertRefresh 43.0 — 72.0
Option «DPMS»
EndSection

Section «Monitor»
Identifier «Monitor1»
VendorName «Unknown»
ModelName «Unknown»
HorizSync 28.0 — 33.0
VertRefresh 43.0 — 72.0
Option «DPMS»
EndSection

Section «Monitor»
Identifier «Monitor2»
VendorName «Unknown»
ModelName «Unknown»
HorizSync 28.0 — 33.0
VertRefresh 43.0 — 72.0
Option «DPMS»
EndSection

Section «Monitor»
Identifier «Monitor3»
VendorName «Unknown»
ModelName «Unknown»
HorizSync 28.0 — 33.0
VertRefresh 43.0 — 72.0
Option «DPMS»
EndSection

Section «Monitor»
Identifier «Monitor4»
VendorName «Unknown»
ModelName «Unknown»
HorizSync 28.0 — 33.0
VertRefresh 43.0 — 72.0
Option «DPMS»
EndSection

Section «Monitor»
Identifier «Monitor5»
VendorName «Unknown»
ModelName «Unknown»
HorizSync 28.0 — 33.0
VertRefresh 43.0 — 72.0
Option «DPMS»
EndSection

Section «Monitor»
Identifier «Monitor6»
VendorName «Unknown»
ModelName «Unknown»
HorizSync 28.0 — 33.0
VertRefresh 43.0 — 72.0
Option «DPMS»
EndSection

Section «Device»
Identifier «Device0»
Driver «nvidia»
VendorName «NVIDIA Corporation»
BoardName «GeForce GTX 1070 Ti»
BusID «PCI:1:0:0»
EndSection

Section «Device»
Identifier «Device1»
Driver «nvidia»
VendorName «NVIDIA Corporation»
BoardName «GeForce GTX 1070»
BusID «PCI:2:0:0»
Option «ConnectedMonitor» «DFP-0»
Option «CustomEDID» «DFP-0:/etc/X11/edid.bin»
EndSection

Section «Device»
Identifier «Device2»
Driver «nvidia»
VendorName «NVIDIA Corporation»
BoardName «GeForce GTX 1070»
BusID «PCI:3:0:0»
Option «ConnectedMonitor» «DFP-0»
Option «CustomEDID» «DFP-0:/etc/X11/edid.bin»
EndSection

Section «Device»
Identifier «Device3»
Driver «nvidia»
VendorName «NVIDIA Corporation»
BoardName «GeForce GTX 1070 Ti»
BusID «PCI:5:0:0»
Option «ConnectedMonitor» «DFP-0»
Option «CustomEDID» «DFP-0:/etc/X11/edid.bin»
EndSection

Section «Device»
Identifier «Device4»
Driver «nvidia»
VendorName «NVIDIA Corporation»
BoardName «GeForce GTX 1070 Ti»
BusID «PCI:6:0:0»
Option «ConnectedMonitor» «DFP-0»
Option «CustomEDID» «DFP-0:/etc/X11/edid.bin»
EndSection

Section «Device»
Identifier «Device5»
Driver «nvidia»
VendorName «NVIDIA Corporation»
BoardName «GeForce GTX 1070 Ti»
BusID «PCI:7:0:0»
Option «ConnectedMonitor» «DFP-0»
Option «CustomEDID» «DFP-0:/etc/X11/edid.bin»
EndSection

Section «Device»
Identifier «Device6»
Driver «nvidia»
VendorName «NVIDIA Corporation»
BoardName «GeForce GTX 1070»
BusID «PCI:8:0:0»
Option «ConnectedMonitor» «DFP-0»
Option «CustomEDID» «DFP-0:/etc/X11/edid.bin»
EndSection

Section «Screen»
Identifier «Screen0»
Device «Device0»
Monitor «Monitor0»
DefaultDepth 24
Option «AllowEmptyInitialConfiguration» «True»
Option «Coolbits» «13»
SubSection «Display»
Depth 24
EndSubSection
EndSection

Section «Screen»
Identifier «Screen1»
Device «Device1»
Monitor «Monitor1»
DefaultDepth 24
Option «AllowEmptyInitialConfiguration» «True»
Option «Coolbits» «13»
SubSection «Display»
Depth 24
EndSubSection
EndSection

Section «Screen»
Identifier «Screen2»
Device «Device2»
Monitor «Monitor2»
DefaultDepth 24
Option «AllowEmptyInitialConfiguration» «True»
Option «Coolbits» «13»
SubSection «Display»
Depth 24
EndSubSection
EndSection

Section «Screen»
Identifier «Screen3»
Device «Device3»
Monitor «Monitor3»
DefaultDepth 24
Option «AllowEmptyInitialConfiguration» «True»
Option «Coolbits» «13»
SubSection «Display»
Depth 24
EndSubSection
EndSection

Section «Screen»
Identifier «Screen4»
Device «Device4»
Monitor «Monitor4»
DefaultDepth 24
Option «AllowEmptyInitialConfiguration» «True»
Option «Coolbits» «13»
SubSection «Display»
Depth 24
EndSubSection
EndSection

Section «Screen»
Identifier «Screen5»
Device «Device5»
Monitor «Monitor5»
DefaultDepth 24
Option «AllowEmptyInitialConfiguration» «True»
Option «Coolbits» «13»
SubSection «Display»
Depth 24
EndSubSection
EndSection

Section «Screen»
Identifier «Screen6»
Device «Device6»
Monitor «Monitor6»
DefaultDepth 24
Option «AllowEmptyInitialConfiguration» «True»
Option «Coolbits» «13»
SubSection «Display»
Depth 24
EndSubSection
EndSection

                     Script is for miner loop with OC settings for each GPU. 
                            Settings apply only ones at start if they enabled
                 Just edit it for your needs and run thats all, main part after it

#!/bin/sh

#nvidia-settings -a GPUFanControlState=0
#nvidia-settings -a GPUGraphicsClockOffset[3]=-100
#nvidia-settings -a GPUMemoryTransferRateOffset[3]=1200
#nvidia-smi -pm 1
#nvidia-smi -pl 155

#nvidia-settings -a [gpu:0]/GPUGraphicsClockOffset[3]=-150
#nvidia-settings -a [gpu:0]/GPUMemoryTransferRateOffset[3]=1200
#nvidia-settings -a [gpu:0]/GPUFanControlState=1
#nvidia-settings -a [fan:0]/GPUTargetFanSpeed=80

#nvidia-settings -a [gpu:1]/GPUGraphicsClockOffset[3]=-150
#nvidia-settings -a [gpu:1]/GPUMemoryTransferRateOffset[3]=1450
#nvidia-settings -a [gpu:1]/GPUFanControlState=1
#nvidia-settings -a [fan:1]/GPUTargetFanSpeed=80

#nvidia-settings -a [gpu:2]/GPUGraphicsClockOffset[3]=-150
#nvidia-settings -a [gpu:2]/GPUMemoryTransferRateOffset[3]=1150
#nvidia-settings -a [gpu:2]/GPUFanControlState=1
#nvidia-settings -a [fan:2]/GPUTargetFanSpeed=80

#nvidia-settings -a [gpu:3]/GPUGraphicsClockOffset[3]=-100
#nvidia-settings -a [gpu:3]/GPUMemoryTransferRateOffset[3]=1050
#nvidia-settings -a [gpu:3]/GPUFanControlState=1
#nvidia-settings -a [fan:3]/GPUTargetFanSpeed=80

#nvidia-settings -a [gpu:4]/GPUGraphicsClockOffset[3]=-150
#nvidia-settings -a [gpu:4]/GPUMemoryTransferRateOffset[3]=1050
#nvidia-settings -a [gpu:4]/GPUFanControlState=1
#nvidia-settings -a [fan:4]/GPUTargetFanSpeed=80

#nvidia-settings -a [gpu:5]/GPUGraphicsClockOffset[3]=-100
#nvidia-settings -a [gpu:5]/GPUMemoryTransferRateOffset[3]=800
#nvidia-settings -a [gpu:5]/GPUFanControlState=1
#nvidia-settings -a [fan:5]/GPUTargetFanSpeed=80

#nvidia-settings -a [gpu:6]/GPUGraphicsClockOffset[3]=-100
#nvidia-settings -a [gpu:6]/GPUMemoryTransferRateOffset[3]=900
#nvidia-settings -a [gpu:6]/GPUFanControlState=1
#nvidia-settings -a [fan:6]/GPUTargetFanSpeed=80

while true; # This will loop your miner even if you kill -9 ethminer it will start again after do
# To stop just CTRL+C or what ever you want =)
do
/home/m1/Miner/ethminer -U -S eth-eu2.nanopool.org:9999 -O 0xb4983146f0047d87c63b5fdb3ef9e2bee4557ea3.M1/vhosted@gmail.com
done

                       Thats was not so hard, the main deal is up to go !!! 
                      While our miner script is working  we will run another one 
                                             Script for monitoring

#!/bin/sh

-i 5 number GPU to monit

gpu=nvidia-smi -i 5 --query-gpu=utilization.gpu --format=csv,noheader,nounits

while true; #Loops :=))
do
while [ $gpu -gt 50 ]
do
gpu=nvidia-smi -i 5 --query-gpu=utilization.gpu --format=csv,noheader,nounits
echo «GPU load $gpu»
echo «All good $(date) GPU load $gpu No errors»
sleep 10
done
if [ $gpu -lt 40 ]
then
killall -9 ethminer
echo «Restart Miner GPU load $gpu $(date) error»
echo «Restart Miner $(date) error» >> /home/m1/Miner/ethminer.log
sleep 60
gpu=nvidia-smi -i 5 --query-gpu=utilization.gpu --format=csv,noheader,nounits
fi;
done

Thats it. Finished it esterday. I think it can be smaller. But nothing need to install, compile etc. All night i tested my GPUs with OC and power -+ very fast to test cloks and + tail -f /var/log/kern.log | grep nvrm to see what gpu couesd an error without long farm stop.
If it will help you. I like good coffe )) b4983146f0047d87c63b5fdb3ef9e2bee4557ea3
Hosted