<p>I'm going to hazard a guess that you're using the <a href="https://psl.noaa.gov/repository/entry/show?entryid=ca90a190-f6b1-475f-bbe0-53cb0a308520" rel="nofollow noreferrer">NOAA OISST data</a>, although I don't think that makes a difference for the answer here.</p>
<p>Note that your loops,</p>
<pre><code>for i in range(len(lat)):
    for j in range(len(lon)):
        for m in range(len(lat)):
            for n in range(len(lon)):
</code></pre>
<p>will just generate the indexes,</p>
<pre><code>i = 0, ..., 8
    j = 0, ..., 31
        m = 0, ..., 8
            n = 0, ..., 31
</code></pre>
<p>When you then subset the global SST array using <code>sst5[59:90,i,j]</code> and <code>sst5[59:90,m,n]</code> you're not extracting data from the tropical points in your <code>lat</code> and <code>lon</code> variables, you're just extracting data from the corner of the full, global array, i.e., <code>sst5[:, 0:8, 0:31]</code>. You can see this by printing out <code>sst5.shape</code> in your loop, which will show,</p>
<pre><code>...
(365, 720, 1440)
(365, 720, 1440)
(365, 720, 1440)
(365, 720, 1440)
...
</code></pre>
<p>This corner is in Antarctica, hence no SST data and <code>numpy.corrcoef</code> will be seeing only missing data values and trying to divide by a zero covariance, yielding NaN values.  My Numpy issues the following warning when this happens:</p>
<pre><code>RuntimeWarning: invalid value encountered in true_divide
c /= stddev[:, None]
</code></pre>
<p>You could modify your program to subset the <code>sst5</code> array before the loops:</p>
<pre><code>sst5_subset = sst5[59:90, 210:510:35, ::45]
</code></pre>
<p>and then refer to this array inside the loops:</p>
<pre><code>mar_05.append(np.corrcoef(sst5_subset[:,i,j],sst5_subset[:, m,n])[0,1])
</code></pre>
<p>But, personally, I would get rid of the loops and do this as a whole-array operation:</p>
<pre><code>sst_array = sst5_subset.reshape(sst5_subset.shape[0], -1).T
corr = np.ma.corrcoef(sst_array)
</code></pre>
<p>A quick picture of the output for one point of interest:
<a href="https://i.sstatic.net/bzCFL.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/bzCFL.png" alt="enter image description here" /></a></p>

CC BY-SA 4.0
2023-03-09T10:44:55.550Z
thanks for the convenient reply, and it saved me from the nested loop. Earlier, I calculated the correlation for the globe and did not realize lat = [ ..] and lon = [] did not fulfill the purpose in the nested loop conditions. Thanks for making me realize the mistake I made.
Ruby
CC BY-SA 4.0
2023-03-07T13:00:19.393Z
2023-03-08T16:57:37.630Z
2023-03-08T16:57:37.630Z
Deditos
<p>I have a sea surface temperature time series, and I want to calculate the (Pearson) correlation coefficients among the nodes in the tropics (30.0N to 30S). In the time series, the land information is present as masked. I do not know how to handle masked data in correlation calculation. Please help. The data I used here is in this link <a href="https://drive.google.com/file/d/1SVKQ4uBDEZOuN7_ftd5tqpF9_NGKY3pZ/view?usp=sharing" rel="nofollow noreferrer">https://drive.google.com/file/d/1SVKQ4uBDEZOuN7_ftd5tqpF9_NGKY3pZ/view?usp=sharing</a></p>
<p>I tried the following code, which did not work:</p>
<pre><code>      temp5 = 'sst.day.mean.1983.nc'
      fh5 = Dataset(temp5, mode = 'r')
      sst5 = fh5.variables['sst'][:365]
      time = fh5.variables['time'][:]
      lat = fh5.variables['lat'][210:510][::35] #tropics latitude
      lon = fh5.variables['lon'][::45]

      mar_05=[]
      for i in range(len(lat)):
        for j in range(len(lon)):
           for m in range(len(lat)):
              for n in range(len(lon)): 
               mar_05.append(np.corrcoef(sst5[59:90,i,j],sst5[59:90, m,n][0,1]))
      df = pd.DataFrame(data = mar_05)
</code></pre>

CC BY-SA 4.0
2023-03-05T19:52:17.093Z
I have tried np.ma.corrcoeff instead of np.corrcoef, but still, the correlation list is NAN. Any plausible solution?
Ruby
CC BY-SA 4.0
2023-03-07T09:31:07.130Z
Are any of your data in `sst5` NaN values, including the masked elements?
Deditos
CC BY-SA 4.0
2023-03-07T11:48:16.240Z
@Deditos There are no NAN values in the data; only masked values exist.
Ruby
CC BY-SA 4.0
2023-03-08T13:12:03.293Z
@Deditos thanks for the suggestion, but my problem is still there. I used the NOAA SST, and I defined the latitude of interest (lat = fh5.variables['lat'][210:510][::35]). The latitude range is from [-37.375 -28.625 -19.875 -11.125  -2.375   6.375  15.125  23.875  32.625], the latitude of the tropic. Over the latitude range, the land area is masked, and the ocean data are available. I tried calculating all possible correlation values among the nodes, so the nested loop. I also use ma.corrcoef instead np.corrcoef,  but the correlation list contains NAN values only.
Ruby
CC BY-SA 4.0
2023-03-08T16:26:55.287Z
Your lines `lat=` and `lon=` don't do what you seem to think they do.  They don't set which parts of the axes are active/visible when you reference `sst5`, they just make lists of the lats and lons that you're interested in, but `sst5` knows nothing about their contents.  I'll edit my answer.
Deditos
<netcdf><correlation>
Correlation calculation of masked array:
Ruby
# Correlation calculation of masked array:

(Asked by: username_0 on 2023-03-03)

I have a sea surface temperature time series, and I want to calculate the (Pearson) correlation coefficients among the nodes in the tropics (30.0N to 30S). In the time series, the land information is present as masked. I do not know how to handle masked data in correlation calculation. Please help. The data I used here is in this link https://drive.google.com/file/d/1SVKQ4uBDEZOuN7_ftd5tqpF9_NGKY3pZ/view?usp=sharing
I tried the following code, which did not work:
      temp5 = 'sst.day.mean.1983.nc'
      fh5 = Dataset(temp5, mode = 'r')
      sst5 = fh5.variables['sst'][:365]
      time = fh5.variables['time'][:]
      lat = fh5.variables['lat'][210:510][::35] #tropics latitude
      lon = fh5.variables['lon'][::45]

      mar_05=[]
      for i in range(len(lat)):
        for j in range(len(lon)):
           for m in range(len(lat)):
              for n in range(len(lon)): 
               mar_05.append(np.corrcoef(sst5[59:90,i,j],sst5[59:90, m,n][0,1]))
      df = pd.DataFrame(data = mar_05)

**Tags:** netcdf, correlation

> **Comments:**
> **username_0**: I have tried np.ma.corrcoeff instead of np.corrcoef, but still, the correlation list is NAN. Any plausible solution?  
> **username_1**: Are any of your data in `sst5` NaN values, including the masked elements?  
> **username_0**: @username_1 There are no NAN values in the data; only masked values exist.  
> **username_0**: @username_1 thanks for the suggestion, but my problem is still there. I used the NOAA SST, and I defined the latitude of interest (lat = fh5.variables['lat'][210:510][::35]). The latitude range is from [-37.375 -28.625 -19.875 -11.125  -2.375   6.375  15.125  23.875  32.625], the latitude of the tropic. Over the latitude range, the land area is masked, and the ocean data are available. I tried calculating all possible correlation values among the nodes, so the nested loop. I also use ma.corrcoef instead np.corrcoef,  but the correlation list contains NAN values only.  
> **username_1**: Your lines `lat=` and `lon=` don't do what you seem to think they do.  They don't set which parts of the axes are active/visible when you reference `sst5`, they just make lists of the lats and lons that you're interested in, but `sst5` knows nothing about their contents.  I'll edit my answer.  

---

## 1 answers

### Answer by username_1 on 2023-03-07. Score: 1 | Accepted answer

I'm going to hazard a guess that you're using the NOAA OISST data, although I don't think that makes a difference for the answer here.
Note that your loops,
for i in range(len(lat)):
    for j in range(len(lon)):
        for m in range(len(lat)):
            for n in range(len(lon)):

will just generate the indexes,
i = 0, ..., 8
    j = 0, ..., 31
        m = 0, ..., 8
            n = 0, ..., 31

When you then subset the global SST array using sst5[59:90,i,j] and sst5[59:90,m,n] you're not extracting data from the tropical points in your lat and lon variables, you're just extracting data from the corner of the full, global array, i.e., sst5[:, 0:8, 0:31]. You can see this by printing out sst5.shape in your loop, which will show,
...
(365, 720, 1440)
(365, 720, 1440)
(365, 720, 1440)
(365, 720, 1440)
...

This corner is in Antarctica, hence no SST data and numpy.corrcoef will be seeing only missing data values and trying to divide by a zero covariance, yielding NaN values.  My Numpy issues the following warning when this happens:
RuntimeWarning: invalid value encountered in true_divide
c /= stddev[:, None]

You could modify your program to subset the sst5 array before the loops:
sst5_subset = sst5[59:90, 210:510:35, ::45]

and then refer to this array inside the loops:
mar_05.append(np.corrcoef(sst5_subset[:,i,j],sst5_subset[:, m,n])[0,1])

But, personally, I would get rid of the loops and do this as a whole-array operation:
sst_array = sst5_subset.reshape(sst5_subset.shape[0], -1).T
corr = np.ma.corrcoef(sst_array)

A quick picture of the output for one point of interest:

> **Comments:**
> **username_0**: thanks for the convenient reply, and it saved me from the nested loop. Earlier, I calculated the correlation for the globe and did not realize lat = [ ..] and lon = [] did not fulfill the purpose in the nested loop conditions. Thanks for making me realize the mistake I made.  

---

